Packet Pushers Open Mic: Real SD-WAN Challenges Live Q&A
What is the reality of deploying an SD-WAN? Does it really work? What are the traps, tricks, and stumbles that you need to look for?
Ethan and Greg from the Packet Pushers will be hosting engineers who have deployed Viptela SD-WAN solutions and will be taking questions from a live audience.
Analysts Greg Ferro and Ethan Banks, on their show “Three Real-World SD-WAN Deployments”, have interviewed many network engineers who have executed large SD-WAN projects in retail, financials, healthcare, and energy. These include companies like Kindred Healthcare, Freedom Pay, Gap, Visa, Bloomberg, Seventy Seven Energy, Greenhill, and more. They will share some of their learnings on this Live session and then open a lively discussion of everything SD-WAN.
Greg: Welcome to the FutureWAN Virtual Summit here today. This is Greg Ferro and Ethan Banks from the Packet Pushers. This show is hosted by Viptela, and it’s going to be quite an unusual one in that we’re actually taking live Q&A. So for those of you who have joined us, in the bottom right hand screen, you will actually see a chat window and you can actually put your questions in there, and then Ethan and I are going to take them on live as best we can, right? So you’re really encouraged to throw questions at us. There will be some filtering, of course, but what we’re asking you to do is obviously be polite, respectful of the event, and so we’ll pick out questions, a, that we can answer so that we look semi-intelligent, but we’re willing to take on just about anything. There’s not too much ground that we don’t really want to cover.
We are from the Packet Pushers, and this is part of the FutureWAN Virtual Summit. I want to just point out that – while we wait to warm up this and get your questions in – we are part of an overall bigger event. What you’ll find is that there’s a whole week of this, there’s many other things, and if you go over to Viptela.com/futurewan, you’ll see an entire agenda of sessions being recorded this week, and then they will go out to you. You can actually just register for them, and even if you don’t make it live, you can tune into them in the week ahead and watch them as you wish.
So if you’re watching today, and you do need to drop off for some reason, because the boss wants to yell at you about the firewall rule you got wrong, or whatever, you know, whatever it is, then you can do that and then still catch up on the session later on in the week. And then by all means, Ethan and I are always available to take your questions whenever you want.
Ethan, I…you know, we’ve been pretty big on SD-WAN this year, and I was just thinking: What is it about SD-WAN that makes it like the topic du jour for 2017?
Ethan: Well, there’s a couple of things. One is money. I think that’s the one that comes up as you talk to folks, because they’re looking for ways to save on their WAN budgets. And part of the promise of SD-WAN is that you can take broadband lines and still get the kind of quality you would get as compared to a private MPLS line. And folks are saying, okay, so if that’s plausible, that I can use the internet as a secure transport and still get the quality I was getting out of MPLS, that’s good for my budget. And so, I think that’s a winner there.
But I think coupled with that is simplicity. You can… Since you can bundle a number of different WAN circuits together and treat it as a pool of bandwidth, and not individual circuits you have to do custom routing for, and so on, to make work, there’s an ease of management there that allows you to add and subtract bandwidth and circuits of different types. And, again, that end result, you’re still getting the same quality of transport that you did as a pure private MPLS subscriber.
Greg: Well, I think that’s true, and I think the other thing, too, is that a lot of us are actually not happy with our carriers, right? Your average telco is an offensive toad. Every time you do business with them, you do sort of feel like you’re being screwed over in a major way. And they’ve sort of got you by the short and curlies because you’ve signed a five-year contract or a ten-year contract to buy dedicated WAN services that take like, you know…What’s the worst time – you know, when – back in the days when you were doing engineering work, what was it, 120 days? 200 days? I think the worst one I ever saw was a 210-day deployment for a DS3 45 Meg circuit.
Ethan: Well, we got our first question in, Greg.
Greg: All right.
Ethan: So let me read it out too and we’ll have a discussion here. Is there a [unintelligible 00:03:46] opportunity for MSPs to make money from reselling SD-WAN from Vendor X, whoever Vendor X might be? Aren’t we just replacing hardware costs with software costs?
Greg: Okay. That’s a really good question. So let’s just do some background about what a managed service provider does. MSP is managed service providers, and these are companies who… There’s a few different definitions of this, but as a general rule, a managed service provider takes a wholesale amount of services from a carrier, and then offers them directly to the customer as a managed service. So what they effectively do is they buy some WAN bandwidth from vendor, they buy routers from vendor, and then they go and put them on your site, and then maybe they add a little bit of icing on the top with a network management platform, or more likely they don’t. And then they manage that for the company – managed service provider. They mix up the bandwidth, the routers, and then actually keeping the WAN running.
Up until now, the only way that you’ve been able to build a WAN, really, with some sort of level of reliability, is to buy all of your WAN services from a single supplier. So if you are a managed service provider, you might do a drug deal with Verizon, and only to resell Verizon. Or if you’re in the UK, you might be reselling Virgin Media or BT. And if you’re in Australia, it’d be Telstra or Optus, or whoever it is. And the ways that managed service providers may add value is really just they get a wholesale rate, you know, 10 percent, 20 percent off; they get a bit of a margin, just like IT resellers do; they get a margin on the product down from retail. And then they add the routers, sell you the routers, rent you the routers, and make value.
So, a couple of things there. One is you’re stuck with your vendor routing platform. You’ve probably only got a single supplier in your WAN, even though your managed service provider is running it for you. And you’ve got a network management platform that really is only about seeing if your routers are up or down or your WAN circuits are up or down, and that’s probably about it.
In the case of SD-WAN, what we now have is that managed service providers are no longer locked into one source of supply for the WAN bandwidth. So instead of just buying your WAN bandwidth from a single backbone carrier, because you’ve got to have just one WAN provider, SD-WAN allows you to have a mix of carriers. So it can be Verizon, Comcast, Telstra, BT, whoever it is, your Level 3, whatever your favorite flavor is, and you can mix and match them by putting in SD-WAN over the top. So if you’re a managed service provider, all of a sudden you can now say we’re differentiated from our competitors because we have multiple carriers, and we can bring those multiple carriers to [a Mr. Customer] and you don’t see those differences.
Secondly, you can load balance between those carriers. So instead of… What we do today on a WAN is you buy an [AE] circuit, and this is my active or this is my master, and let’s say it’s 100 Meg Ethernet, and then over here you’ve got a 40 Meg WAN circuit as a backup. Well, today, you can only use one circuit, and the other one sits around doing nothing all day. You’re spending…50 percent of your WAN budget is probably never ever used, it’s just wasted away on bandwidth that you can’t – if you could load balance across two, then all of a sudden you’re going to get a normal…a performance benefit. And a reliability benefit, but mainly a performance benefit. So, managed service providers who move over to these platforms are now able to actually do something that was never possible before.
Ethan: Yeah, I think as far as the opportunity to make money reselling, I think the answer is yes, and there are a number of SD-WAN vendors that have branding opportunities and built their platform with MSPs in mind. That’s their whole thing. That’s part of their value prop. So there’s several that you can talk to that do that. So the money is there, the branding is there.
As far as replacing hardware costs with software costs, yeah, to some degree; you got to look at the cost model. You are going to a far cheaper and more easily replaceable hardware platform in the form of an SD-WAN termination device, it’s just in that little x86 box, as opposed to a more expensive router. And then you – but then you’ve got subscription costs for software, you’re exactly right. So is that a one for one? That’s going to be an analysis you got to look into.
Greg: Well, I think many service providers have pretty much been of pretty limited business, you know. I buy my WAN [unintelligible 00:08:01] from company A, but there’s no company B, I make 10 to 20 percent off the top, and then I sell some routers, maybe, and I manage those for the customers. And I think that’s a pretty – and you may be providing some sort of management software on the top to monitor them and stuff like that, was the best that they could do. Whereas SD-WAN devices have really powerful visibility and analytics on board. It’s absolutely key to the way SD-WANs work, is that you’ve got to be able to look deeply into the flows, right? You have to know every single flow that’s going through the device, because you have to be able to decide where you want that to go.
So if you look at this diagram that’s on the screen, packets come in through the input, they match a flow record, and we match them onto a VLAN, right? We might choose to put it on a VLAN trunk, or overlay it and put it onto a VXLAN overlay, or maybe we put it on an MPLS overlay. Well, SD-WAN does the same kind of thing. And because those flow records are in here, now you can build a whole analytics platform on top of that, right? So you can get deep. So we’ve seen the different SD-WAN providers like Viptela, they actually don’t focus on just routing via packet, they route via application. Or, more correctly, they move the application over a path.
Well, if you’re saying I want my email to go over this WAN link and I want my printing traffic to go over this WAN link, you have to actually have deep recognition of the actual payload. You need to know that this is printing and that this is email. And so all of a sudden, you’ve got access to much more data than you ever had in your old routers. And you present that on an interface, and you can see what’s going on.
Ethan: Another question here, Greg, has come in for us. Are there templates that are available for an SD-WAN network assessment? So I think I know what you’re looking for here. You’re looking for some kind of a questionnaire that is going to take you through the process of determining the suitability of your network for SD-WAN. And I don’t… That’s not something that’s come up in our questions and interviews that we’ve done with a lot of different SD-WAN companies at this point. So I would think about it this way. SD-WAN as a technology is going to sit in your network at the WAN Edge, so you’re looking at something that could potentially replace your WAN router completely. You’re looking at a device that is probably going to be terminating the Ethernet circuits.
If you want to terminate those circuits directly on that WAN device, then they need to be Ethernet. Other than that, you’re going to be leaving your router in the mix. You need to consider your routing infrastructure. How do you want to announce routes from SD-WAN into your network? So if you’re a OSPF or an EIGRP or a BGP shop, do you want to leverage those protocols with your SD-WAN solution? And if you do, that should be part of your evaluation, because some of them do that, some of them do not. And then you need to look at what sort of services you can leverage in your area. Can you… Is it actually viable to bring in broadband circuits into your market? What are your other circuit options in your market? Some of you that live in rural North America, you know, Lord love you, it’s just – your choices may be few, and some of you that live in a metro area may have everything from a direct fiber handoff within your building to several other choices, including WISP, in the area.
And so part of your evaluation process would include that, and what mix of circuits and diversity that you can terminate on the SD-WAN platform makes sense for you, and fit your budget, etcetera. So it’s a long way to say I don’t know of one single template that – maybe some vendors have that, but there are some big things you need to think about as you go through that evaluation process yourself. And, of course, your vendors… You know, every vendor that’s out there is going to have some kind of a sales process that they take you through to try to figure out how to size you into the right solution and so on. So there’s going to be something there, but there’s not like a one-size-fits-all template –
Greg: It’s not that much different –
Ethan: – that I know about.
Greg: – from routing, though, right? So when you go out to build a WAN with your traditional leg – what I call legacy routing these days, which is a bit, you know…whatever. I –
Ethan: That was patronizing.
Greg: Yeah, I know it’s patronizing, but it’s true, though, right? Because the old routers really aren’t…They’re just…They’re really, really good for the 1970s, but not so much for the 2010s, right? No, I’m joking a little bit. Not much, though. [Laughs]
Greg: I think… So a couple of things to think of. One is today you might be buying all your WAN circuits from a single carrier. Now, if you move into an SD-WAN, and let’s say you’re buying two or three different providers, maybe you’re using a public WAN instead of private WAN. What do I mean by public and private WAN? Private WAN is where you buy a dedicated circuit from a carrier who promises that that bandwidth is guaranteed at some level, and…but otherwise it runs over a mixed bandwidth. But the carriers also say that this bandwidth is private. It’s not safe, it’s not secure, but they [unintelligible 00:13:13] that it actually is, right?
But the public WAN is the internet, and it’s certainly possible to run SD-WAN over the internet and never, ever need to have private. Main reason being is that most SD-WAN appliances use an overlay, an encryption overlay, or whether it’s an IPsec or a DMVPN-style IPsec, or a TLS. Different vendors have different approaches, and different approaches vary according to solutions; some vendors implement both according to their technology stack. The thing here is that if you start to replace private WAN with public WAN like with internet connections, and let’s say you decide to use three different internet providers, now you’ve got three bills for that site instead of one. And it can actually be from three different companies, so now you’ve got a purchase ordering problem.
And this is where managed service providers – to go back to our original question – can add some value. That’s a sort of a trip, a trick trap, where you might move down the path of saying, oh, we can get rid of our dedicated WAN and that high priced WAN bandwidth, now you’ve got a purchase ordering problem and you need to have somebody to just reconcile all those invoices. That’s not my idea of a good day, dealing with purchasing.
Ethan: Another question for us, Greg. I am concerned about maturity. Can we share details of any SD-WAN deployments that you are aware of? And The Gap comes to my mind. You want to…want to chat about that a bit?
Greg: Yeah, so, well, we did a podcast – we’ve done two or three podcasts where…With Viptela, in addition to at the Packet Pushers podcast website; at packetpushers.net you’ll find there’s a whole range of podcasts on SD-WAN. Just do a search on our WordPress platform and you’ll find a whole bunch of podcasts there where we’ve spoken to all different vendors. And it’s really useful to listen to all of those, so you understand all of the SD-WAN vendors, but in particular with Viptela we’ve done three – two or three podcasts where we’ve actually spoken to their customers. And one of those was The Gap. They’re talking over 3,000 sites running on Viptela’s WAN, SD-WAN technology, rolled out [unintelligible 00:15:04].
Ethan: [That’s probably] where they end up, yeah. I think they’re into the…well over a thousand deployed right now.
Ethan: Yeah, and that scenario, Greg, as I recall, a lot of retail stores that needed diversity of carriers because they were pushing payment information out of the retail store into headquarters. And so there were a couple of internet service providers that were involved, so they’re terminating broadband ISPs at the stores, and then using SD-WAN for best performance across whichever of those circuits was available. The biggest challenge that I recall as they got into this was not with SD-WAN, but was with getting circuits terminated [and turned up]. The same problem we’ve always had.
Greg: That’s right. That was in that show where we talked to [Snehal Patel].
Greg: And he was talking about the hardest problem wasn’t actually configuring the SD-WAN, it was that providers would promise that a circuit was ready, and they’d get there, plug it in, and it wasn’t.
Ethan: Exactly right, yeah.
Greg: Yeah. And the interesting… I think the other takeaway that I’d got from that conversation with The Gap was they were actually sending their payment traffic over 3G/4G, so they were using the traffic routing capability so that all of the – you know, the credit card swipe machines, that traffic was being diverted at the SD-WAN Edge to run over the 4G network, because that was more reliable than the DSLs and the internet connections. Which kind of blew my mind, I must say. But keep in mind that payment transactions are very small data, but they were finding it better to divert that directly over the 3G, and then other stuff, you know, brochures, data, email, printing and so forth, that all went over internet connections.
Ethan: Just other comments about SD-WAN deployments. There’s a bunch of carriers that have – or SD-WAN providers that have been releasing mentions of customer wins, you know, people that are deploying it. It’s becoming increasingly common in enterprise deployments to send…to use SD-WAN now. In smaller deployments it’s not really that complicated. If you’re dealing with a 20 or 50-site network, something relatively small, but still with some oomph to it and some routing heft, and maybe a bunch of applications you’re dealing with, that’s kind of bread and butter for these guys.
Scale becomes a question mark, so if you’re trying to go to like Gap size where you’re up over 1,000 sites and you’re evaluating 2,000, 3,000 sites, then you’ve got to look at your management, how you’re actually mapping controllers into forwarders, and distributing policy, and how that is all handled. And that’s an interesting point in that not all SD-WAN providers will scale as big as you might need them to go. So that’s a thing that some folks have run into.
Greg: That’s true. So different SD-WAN products have different scaling sizes. I think the largest… I might be talking off the record here, but anyway, let’s go with it. The largest Viptela deployment I’ve heard of is 6,000 sites today, active. If you want to hear more – so we talked before about the FutureWAN being a whole summit, so it’s a whole group of sessions; it’s like a conference track. And what I’ve got on the screen here is we have customers here actually talking in this FutureWAN Summit. There’s First American talking about their SD-WAN. We’re doing a network engineer roundtable with the tech field [unintelligible 00:18:13], that’s on January the 20th, that’s tomorrow at 09h00, the same time as this. And there will be people from Kindred Healthcare, who are also on the podcast talking about their deployments; they’ve been rolling out SD-WAN for their healthcare network. And they’ve got very large sites, and I’ve – we’ve chatted to them several times with several different people, and they’re really, really big about this thing.
So you can get into these sessions. There’s quite a few of these things – Voice of the Customers is part of the FutureWAN, if you want to hear more, here’s one, Kindred Healthcare, again, on the 17th at 11 o’clock. So listen to those people, they’ll be actually telling you about their exact stories as well. And then go and listen to the podcasts that we’ve done with Viptela where we had customers on talking about their practical experiences. I think I’ve been reasonably convinced now, it’s not one or two people that have done these, but we’re talking about tens of companies that we’ve spoken to with SD-WAN deployments who are successful, happy and accelerating their rollouts. I think when we spoke to one company, they said they started off with a pilot of 10, and within six months they’d rolled it out to hundreds, because it worked and there was so much money to be saved.
Ethan: Right, next question here. SD-WAN is – the longer question. So SD-WAN is consistently perceived as a vehicle to network cost reduction through replacement of MPLS lines, which is kind of how I phrased it at the beginning. There’s some caveats there, I think – this is a really good question, actually. However, comparing tier one carrier internet services to MPLS services, the cost savings are not great. And that’s the key. Tier one, like a DIA kind of a circuit. Tier one carrier internet to MPLS services, the cost savings are not that great, so therefore, you [must] leverage the local ISP services to get a significant savings. But in the large deployments, this results in explosion in supplier relationships, [a complexity], because you’re not dealing with just the one MPLS carrier, now you’ve got a bunch of little guys, that’ll sell you the cheap circuits. Okay.
So, here’s the question. Has SD-WAN been [mis-sold] as a cost reduction proposition? And, if not, how do enterprises unlock those savings in practice? I think it is –
Greg: Well –
Ethan: I think it’s overemphasized. I don’t know about mis-sold, I think it’s overemphasized, because I think the… It is easy to grab people’s attention when you communicate that value prop as a cost savings thing, and certainly that messaging has been very consistent in 2015, less so in 2016. Because part of the value… It all depends. Are you having to go to a manager and show a piece of paper with a dollar cost savings? Here’s our ROI, boss, here’s exactly what we did, and we saved these dollars. Or, can you also point out the fact that service is now more reliable, or service is now…it’s easier to operate the WAN because we’ve reduced our operational complexity? I won’t say complexity as a whole, but operational complexity, because now it’s easier to get that WAN up and running, add and subtract circuits, increase and decrease bandwidth and availability, etcetera. So it’s all how you define unlocking savings.
It is something that we have heard from several people, that, yeah, if you’re going to go with a broadband circuit, internet of some kind, then it’s a tier one, a dedicated internet access style circuit, you’re not saving much money, if any. Because really you’re getting the same sort of service level from the provider that you would have been with a private MPLS, only now it happens to be public internet instead of private MPLS. And so, right, the dollar savings aren’t necessarily there. That’s fair, but I’ll point out that when you go to multiple – let’s say a tier two or a local provider, you can unlock savings that way, because if you’ve got two big, fat cable ISP circuits, let’s say, you got a lot of bandwidth, and then you’ve got two choices. Hopefully you’ve got two choices to go through.
Now, if they’re tier two, you want to understand who their tier one upstream [peering] is, to make sure you’ve truly got diversity, so that tier two provider one and tier two provider two aren’t both going through the same tier one [backbone] provider. So when that tier one backbone provider has an issue, you’ve just hammered both your tier two circuits, if you see what I’m saying. This is a design issue, a design question. But you can unlock savings that way, using those style circuits, and still get the performance, because that’s what the SD-WAN monitoring and SLA enforcement policies are doing for you. I don’t know that you have to have a tier one and spend all that money for the big circuits.
Greg: I think… So my addition is the way that our WANs are built today, with active and standby – so I’ll assume that for the purposes of the discussion that there’s an active and a standby, and in the diagram I’m showing on the screen here, it’s sort of like a pretty typical design for Australia where every capital city has dual routers and dual WAN links. And what you’ll… But 50 percent of that is unused. You can’t load balance over those circuits with a router. Effectively you can only have an active and a standby. You could build an overlay network, like build an MPLS on top of this, and then you can start to do some load sharing and stuff like that, but think about how costly that is now for you to run bandwidth.
So moving to SD-WAN can put you in a situation where you’re using all of your bandwidth all the time, and then you run [degraded]. So if something fails, you actually fail to a half functioning state. Now, that’s pretty much how you data centers work today; you lose a server or two, and your site runs off [degraded]. But most WAN networks of today, 50 percent of your Opex and 50 percent of your Capex, so that routers and the circuits that you’ve bought, are actually just wasted. They’re dead money. So there’s one way to look at it from a value-added perspective.
Now, the second thing I want to point out is that what about if you could start to change your architectures to this? What if you could start to build loop architectures like this, which you can’t do practically for WAN networks? Move into different sorts of typologies, make a looped typology, so if any one of these WAN links went down, now you’ve still got a viable path to the destination. Now, this might not work in terms of performance, so why not start to have a mesh of some sort? Now, building mesh networks with OSPF or BGP is effectively a curse that most people would not want to [laughs]… But why can’t you do this, right? Why are we using protocols from the 1970s that only allow a single [and best] path? And then I can’t build these mesh networks where they… And what I can now do is maybe the link here from Melbourne to Brisbane is almost unused, because these are two regional offices.
But what I want to do is have redundancy from Sydney, and the cheapest way is for this link to exist, instead of being forced to run multiple links between these locations, right? Or, maybe you’re going to move a function to Darwin that only talks to Perth. Well, why would you route that traffic via Sydney, like this over here? Why should these offices [too] talk via Sydney? Why don’t you just buy a link that’s from here to here? Now, there might be good reasons why you wouldn’t, because, actually, in Australia there’s nothing between those two locations. That’s 4,000 miles of desert. But the point is that you don’t have that design choice today, and you would if you start moving into SD-WAN because you’re moving into routing by application, or forwarding by application, instead of routing by some mathematical protocol that used to run on a CPU from the 1970s.
Ethan: Okay, so speaking of older equipment, this is another thing that popped in my head about cost savings, is it is possible to get rid of your Edge router, probably a Cisco device of some kind, retire it completely, and no longer have to be running SMARTnet for [fix] support and so on, on that device. You can engineer it out, the SD-WAN device. And this depends on [circuit termination]. If we’re talking about Ethernet circuits you can do this. You can plug directly into your SD-WAN forwarder, and make that Edge hardware go away, which, in the long run, should definitely be a cost savings. So, Greg, let’s move onto another question, because [they’re starting to stack up on us].
Greg: Well, I’ve just got one more thing to talk about here. One of the things [you can do] –
Ethan: Do it quick, buddy.
Greg: One more thing I wanted to point out here is that it’s also possible to build the overlay networks that you build – so the way SD-WAN works is it does it with encapsulated protocols for most vendors. It’s also possible to build application routing where some…You have an overlay network which is a full mesh, but you might have an application network which is a partial mesh or a hub and spoke. So you might decide that all of your internet traffic has to go down to head office, but you want to build a full mesh network for your voice. So in SD-WAN you can do that, using the recognition of routing by application and saying you – for voice, for example, where all clients need to talk to any client directly, so build a full mesh overlay for that, but then build a second overlay which is for all unrecognized or unclassified traffic, and send that to the data center, so that it can pass through the security [tills]. So –
Ethan: So kind of related to this is a question that comes in. We mentioned using OSPF, BGP, maybe EIGRP to advertise routes from SD-WAN into the lab. Okay. This would steer traffic based on IP subnets being advertised. The question, then: How can I steer traffic based on application to the SD-WAN solution instead of by IP subnets? So, in other words you want to… I want this particular application to go into the SD-WAN, and these other applications to be routed traditionally, not through the SD-WAN. I think the answer there is you would pump it all into the SD-WAN forwarding device, and then the policy that you would apply to that application.
You’ve got to allow the SD-WAN [forwarder to] identify the application, and then forward traffic according to a policy. You could have some traffic just be forwarded over whatever circuits you want, including kick it off to a router on my [traditional LAN or legacy WAN], whatever, and then the other traffic with the applications that need to be handled according to whatever your policy is, be forwarded across the SD-WAN in that manner. I don’t know of a way…I don’t…I think it would be over-engineering or overcomplicated to try to do some kind of application based routing, like with PBR or something like that, like ahead of the SD-WAN forwarder. You actually want it to get the SD-WAN forwarder and then make a policy-based decision from there.
Greg: Yeah. Well, at a fundamental level data center networking can’t do that. You don’t… Because you want it to be high speed, low latency, you know… You know, we even talk about switches being cut through instead of store and forward so we can avoid error checking, because error checking takes too long, because 5 nanoseconds is too much delay, you know, blah, blah, blah. So our data center equipment is actually bought…built to a different purpose. [Campus] is the same. Campus is built to a price point, because there’s so much of it. A data center is built on a performance target, and the SD-WAN should be built more towards a feature capability.
So what you’ll need to do is build your SD-WAN gateways just like your WAN Edge is today, and then once it hits the SD-WAN Edge then the application routing kicks in. And that, I think, is the only way that I can imagine, that you’ve got to say this network has this purpose, and it’s optimized or targeted at these needs, and not get them mixed. What we do in the WAN is different from what we do in the data center LAN, is different from the campus, is different from wireless.
Ethan: Mm. Another question here related to security. Do you except either healthcare or banks to implement SD-WAN? Isn’t the security scare much too severe? In other words, it sounds like the logic is private MPLS, it’s private, it’s mine, and therefore it’s much more secure than pushing traffic over the public LAN. Okay, so security’s been addressed in SD-WAN solutions. They will… They can – and all of these guys have pretty much the same security story. Everything is encrypted and – period. No matter what transport it goes over. You can mix private MPLS with public circuits in an SD-WAN scheme. All the traffic is going to be encrypted. Selectively. By default everything’s encrypted.
There are solutions where you can say I don’t want this traffic to be encrypted. Okay, that’s fine. But there is built-in key management, the encryption is done with a ES256, and so as you push that traffic across a public transport you’re not putting the data in flight at any particular risk. And, in fact, you can use SD-WAN solutions to report to you that you are HIPAA compliant, or PCI compliant, assuming that those modules and that…the vendor you’ve chosen supports that. But that’s a pretty common feature set that’s there.
So, case in point, I was doing work, oh, I guess it goes back a couple of years now, with a healthcare provider. And I was looking at a solution like this in the very early days of SD-WAN. I was considering something like this to replace the private MPLS network that we had built, because it gave us so much more design flexibility. Security was not a big concern there. Yet another thought with SD-WAN is you do have traffic steering capability in many of the platforms, where you can say, all right, for this application I need all of this traffic to get routed through this particular firewall, or intrusion detection device, etcetera.
So you’ve got deep packet inspection capabilities that… It’s not that the forwarder does the packet inspection for you, but that it is pushing traffic into a device that can do that. Again, firewall, IDS, etcetera. You can do that traffic steering and get that security that you’re comfortable with. So there’s nothing holding back healthcare or banks from leveraging SD-WAN. Now, I’m trying to think of a customer I’ve heard of that’s actually going down this road, they’re going to use SD-WAN, but… And nobody’s leaping to mind, but I do know that there’s…From my standpoint –
Greg: There are financial institutions using SD-WAN today.
Greg: Yeah. It’s not a no, it’s just a case of getting through the objections and the requirements. I think just to extend on what you’re saying, is that the private WAN circuits that you get is… The private WAN circuits that you get today aren’t actually secure, they’re assumed secure. In the sense that your carrier probably attempts to keep them as isolated and as safe as possible. And in a carrier environment there’s not open access, there’s not open season, but it’s certainly… The thing is that if you’re carrying unencrypted traffic across those WAN links, it is certainly possible that somebody could tap that and capture that data.
Now, whether that’s somebody inside the carrier themselves, or whether that… And up until – well, actually in the last – just in the last year or two, the price of encryption was beyond the price of…that we could afford to buy. You couldn’t put an encryption device in every branch office. You couldn’t afford to build IPsec or SSL VPN meshes, because the cost of CPUs, the compute power, the crypto chips that you needed, were just very expensive, hard to make, or just overpriced, you know, pick your crime.
What we’re seeing now is that your average Intel x86 can easily do 100 megabits per second of IPsec at very low latency. And that changes the game in terms of the price of encryption. So you should now… Your private WAN circuits are not actually secure, they’re just assumed to be secure because there’s no other choice.
Ethan: I completely agree with that. I was working at a bank – this goes back almost ten years – and for that exact reason, Greg, what you just said, we were migrating all of our private MPLS to the…an IPsec tunnel on top of private MPLS for that reason, so… Okay, another question here –
Greg: The hardest part – well, just quickly. The hardest part about then operating an IPsec or a crypto is then key control and configuration management. And there are definitely things in SD-WAN platforms that solve the key management, and the key distribution and key rotation. You need to change your keys on a regular basis, monthly, quarterly, whatever, and you also need to have a system that will do the configuration. Managing 3,000 IPsec tunnels is not my idea of a good day. DMVPN is a fine technology, it has some weaknesses, and it relies on regular key rotation to be able to address those structural weaknesses. And managing the configuration of 30,000 devices is quite a pain in the butt.
And then monitoring IPsec performance, like the performance inside of a crypto tunnel, is on you. You can’t rely on a WAN provider to tell you what your application – you know, and then those of you who have done router configurations for [unintelligible 00:34:09] inside an IPsec on top of an output interface, and just… [Laughs]. So, you know, they’re assumed to be secure is not enough anymore, it’s now becoming accepted that we can encrypt on everything, and you should. Because your carrier is not to be trusted, because they might be working with the government, or maybe the carrier’s been penetrated by a foreign…by your competitors doing industrial espionage, or foreign nationals.
Ethan: I’d love to say that never happens, only it’s been all over the news for years now, so…
Ethan: New question. Are most use cases deploying a full mesh of sites, or engineering this to be more of hub and spoke, or maybe selectively deciding upon branch to branch? And the answer is, yes, it can be any of [unintelligible 00:34:48].
Greg: All of the above. [Laughs]
Ethan: Yeah, a typical SD-WAN is whatever you need, and you can make different SD-WAN typologies for different applications. I want these apps to be hub and spoke, and I want these apps to be site to site, and you can build different policies that allow you to build whatever sort of typologies that you want. And I think there is no one thing. I’ve heard of… You know, why would you build a retail store that doesn’t need to talk to a retail store as a full mesh? You would build that as hub and spoke, right? So it’s going to depend on your business case, and so on.
Greg: Yeah, but the thing then that comes into it, but you want your voice to be full mesh.
Ethan: Exactly, yeah.
Greg: So that’s the classic use case is voice is full mesh but everything else should go back to the head office, because the branches shouldn’t be talking to each other from a security point of view. But, guess what? So if somebody breaches the branch in your…and puts a scanning device on, well, suddenly they can reach every branch. So in this sense it improves your security posture – going back to the previous question. Keep in mind, of course, that as you build these overlays, and the various arbitrary typologies, you increase the available complexity. And so it’s up to your capacity to, a, configure and operate and understand. You don’t want to build too much complexity into these solutions and then suddenly find that your…you’ve created a cross that now you have to bear [the problems of this]. You should aim as far as possible to keep it simple and straightforward, and not get too carried away and excited and say, ooh, and then just start doing it because I can.
Here’s another arbitrary typology for you. Why do you have to backhaul everything back to an internet connection in your data centers? You’ve got two data centers and two internet connections. Why don’t you use them both at the same time? So build… Here we’ve got a typology where these three branches are sharing the internet connection in this region, and then you’ve got another three branches sharing a typology in this region. So now you can start to build regional breakout points for internet access, but still have your inspection technologies and distribute them around a large geographic location. So that’s just the sort of thing that you can, you know… But be aware of the complexity. Don’t make the mistake of doing things just because you can. Make sure you’re doing it for good reasons.
Ethan: We’ve got more questions than we’re going to be able to answer. We’ve got about eight minutes left, and just about as many questions, so I apologize if we don’t get to your question. I’m just trying to pick the ones that I think are… I don’t know, the best. Based on me. So if I didn’t pick your question, then pick on me.
Greg: It’s all Ethan’s fault. It’s all Ethan’s fault.
Ethan: Next question. Of the SD-WAN solutions you’ve seen implemented, how many are internet plus internet versus MPLS plus internet? And do you see MPLS being pushed into obsolescence by internet plus internet with SD-WAN? So what I’m hearing a lot of folks do is they’re starting… It’s a transition process where they’re starting with MPLS plus internet, with the intention of eventually retiring The MPLS completely and going to all internet. And so they’re starting with proof of concept – let’s make sure this thing works. Where is my traffic –
Greg: Play it safe.
Ethan: – actually flowing?
Greg: Remove the risk. Yeah.
Ethan: We are hearing stories from actually several SD-WAN customers that have come up to us and told us, off the record, they can’t because [of the companies] they haven’t been able to say it publicly, but they’ve found out that as the SD-WAN traff – health checking of the circuits does its business and reports, the public internet circuits are doing better than the private MPLS circuits in many cases, and that the majority of their traffic is actually flowing across the public internet circuits. Which is a surprising finding, but we had several people, unprompted, from different markets come up and tell us this. Which was interesting.
Greg: So most people start off with the MPLS and internet, then they quickly realize that dedicated – that internet circuits are of better quality than the MPLS. Now, that’s sometimes – let’s be careful here. A lot of companies are paying for 10 Meg MPLS circuits, and then buying 100 Meg internet circuits, and of course they’re better, because it’s the last mile that defines most of the problems. Not all, just most. And if you had a 100 Meg MPLS with a 100 Meg internet connection, you might not find the same, but quite often a 10 Meg MPLS is the price of a 100 Meg internet. And more bandwidth solves all problems, so sometimes it’s just that, right?
Ethan: That directly goes to another question that has just come in. It was mentioned earlier that assessments aren’t common, necessarily. So have you found SD-WAN connectivity sized bandwidth compared to existing connectivity? And it’s exactly that. You can get broadband, a lot more broadband pipe for the money versus a private MPLS. So, yeah, maybe it’s a 10 Meg private circuit and a 100 Meg public circuit, and – just like you said, Greg – that solves all problems.
Greg: Yeah, and just to extend that, if you’re in like… If you’ve got a substantial regional office, and maybe you’re going to be buying the only thing available in there, you’re not close to an optical loop, and so you can’t get a DWDM terminating a 100 Meg Ethernet or a 1 Gig Ethernet into that office. And they’re probably going to try and bring in a DS3, you know, some sort of legacy technology, because that’s all that’s in the area, but they’ve probably got cable. And you know what? A 100 Meg cable is still just as good as a 100 Meg MPLS for most use cases.
Ethan: Another question here. You had briefly mentioned that some SD-WAN vendors have certain limitations, depending on what features are needed. Can you elaborate on those examples? Yeah, we’ll hit some things real quick here. One is scale. If you have a very large number of offices, some SD-WAN solutions will only scale to hundreds of sites, others will scale to thousands. So the examples of that, Viptela we’ve mentioned – they can scale up into thousands. Um… Who am I thinking of here? [Tulare], I think, doesn’t cater to sites that big. I think their limit, last I knew, was about 500.
Greg: But they’ve got a different – what they’re doing is focusing on different functions in their product. So the… You know, as this market continues to emerge, my sense is that different vendors are focusing on different things. So vendors who are operating… Some vendors have chosen to focus on their application recognition. Some vendors are focusing on scale. Some vendors are focusing on WAN acceleration. So if you look at the Silver Peaks and the Riverbeds, they’ve got these amazing WAN acceleration portfolios, so they’re actually able to offer you not just SD-WAN, multi-path, arbitrary, overlay, typologies, etcetera, etcetera, but also WAN acceleration.
But if you’re WAN accelerating, you’re burning CPU, so then your scale – you’ve got more states, you’ve got more configuration hassles, so then the scale starts to limit. You look at CloudGenix, for example, very focused on very granular application recognition, and that then – but their development plans include scaling up [unintelligible 00:41:20]. So, you know, there are different SD-WAN solutions for different people, it’s not just a single thing. And we should also just [very much] mention public cloud. Nobody’s asked us about the public cloud question – have you got one of those for us?
Ethan: No-one’s asked about public cloud, but that is another good differentiator to bring up, where some SD-WAN providers have partnerships with public cloud. So if you’re trying to accelerate traffic – or optim – well, neither accelerate nor optimize – well, not both –not either of those words. If you’re trying to use SD-WAN to make sure your user experience is the best going up to public cloud, Riverbed, I know, has some partnerships there, VeloCloud has some partnerships there. Some don’t even need partnerships the way they’re doing their optimizations, they just get you closer to that cloud provider. That’s an integration that’s there, it’s worth checking out.
Another is routing, so if you really want to do BGP or OSPF to your core, and [announce] that way, as opposed to having your SD-WAN appliance having to be in line or redirected [to via] PBR or WCCP, that’s something to look into. More and more of them are doing routing now, not everybody was early on, for sure. So it’s… You need… You need to understand exactly what your use cases are, and the specific features you want, and then make sure that’s part of your proof of concept as you start taking to SD-WAN vendors.
Greg: Yeah. And the other thing is you don’t have to replace your entire WAN. So we’ve talked to customers who’ve done six or ten sites out of several hundred, or 100 sites, and just, you know, you – you can put the SD-WAN appliance in with your router, so it’s not an either or. You can actually run it on top of or side by side with your existing WAN, and pilot it out.
Ethan: Okay, oh boy, there’s two more question left, let’s see how we do here. I see a feature such as per packet or packet duplication, where either a single flow can be load balanced across two or more transports, or packets are duplicated across transports to avoid brownouts. Is this just a gimmick, or is there some value to this functionality? It is not just a gimmick. The… What that is intended for is for guaranteed delivery of traffic that cannot be lost, like voice. So you might duplicate traffic, voice traffic across both links, so that just in case there’s a hiccup, some kind of packet loss, temporary event during a voice conversation, the traffic is assumed to have made it across the other link. It gets reassembled on the other end, and you have a single stream delivery to the end point on that end. But it is intended to be very selective. In other words, you don’t –
Greg: It’s not intended to be a universal, everything duplicated, yeah.
Ethan: Exactly right. Exactly right.
Greg: And the important part here is that just think carefully about what the negative impacts of that are. If you’re doing packet duplication, your SD-WAN appliance is now duplicating packets. It’s also managing state, it’s also reassembling, and obviously that has an impact on CPUs, and then your scale must have – and then you’ve also got more complex software. And so, as Ethan said, what we most [unintelligible 00:44:25] recommended designs for that is pick – just like priority cue routing and priority cue [costs] from decades ago, you only do it to 10 percent of your traffic at most. It’s just for a very limited subset of traffic to solve specific needs.
Ethan: Probably our last question today: What are your thoughts on buying SD-WAN from the SD-WAN providers directly, versus buying a managed SD-WAN service through the big telco’s? Do you feel that the benefit of simplicity of SD-WAN is reduced if you need to manage this through a big bureaucratic telecom organization? [Laughs]
Ethan: So two – I’ll give you two quick opinions here. One is look at your pricing. Are you actually going to save money if you’re buying SD-WAN through telco’s? And in talking to – I gave a talk on SD-WAN in Chicago a few months back, and a lot of those folks were starting to shop SD-WAN from their telco’s, and they said, “I don’t save any money that way, it’s actually a value added service that they’re charging me more for. It didn’t help.” So that’s one thing. From a management perspective, you know, it’s very early in the deployment offerings here, and I haven’t seen too many [interfaces] to have an opinion one way or the other. But, as you kind of put it here, a big bureaucratic telecom organization, I can’t – it’s hard for me to imagine that it’s a better experience than buying it yourself. So I think it goes back a lot to relationship with that telco.
Greg: And your own competency.
Greg: So how good are you as an engineer? How good is your team as engineers? Do you have enough people to run this 24/7? Do you need it to run 24/7? It’s not a hard and fast answer. You might have a CIO who is not very competent, and would much rather put responsibility on other people than to be an adult and do it themselves and take responsibility for what they own and manage. It could actually be that there is a financial arrangement with your incumbent telco, and the only way out is to do this, because you’ve got a five-year contract that’s still got four years to run. I would suggest that operating this yourself is definitely cost effective, as the general rule. And in… You’d be crazy not to, because it’s actually quite straightforward. It’s much easier to do than routers. The reason that routers were so hard to manage is because they were built in sort of like the 1970s, and they were designed to operate as standalone units. And –
Ethan: You keep saying the 1970s. I keep imagining routers dressed in like butterfly collars and [leisure suits] and medallions.
Greg: [Laughs]. Well, you know, we do have a CLI, which is roughly the modern equivalent, and a pretty crappy CLI at that, [as the standing thing]. It is… That is very much the flairs of the 1970s, in my mind. I mean, the CLI is a great thing, don’t get me wrong, we needed it and it’ll always be around. But I’m not sure the extent of value, but… You know, to me it seems like I would go with the assumption that deploying this yourself is absolutely – because there’s software defined, there’s applications to configure and manage this, and they have visibility and analytics, you can do all this yourself. You don’t need to give it away to somebody else who’s supposed to know more, because quite often they actually know less than you do.
Ethan: Well, Greg, we are at time. We just got the warning that we’re about to get kicked off in roughly 90 seconds. You want to hit the wrap for us?
Greg: Yeah, so a couple of things. If you take a tweet of what you’re doing here, and then hashtag it with #futurewan, there is a prize going where you can give it to people, that Viptela will see your various things, and they might send you an Amazon Echo as a special prize. Do something a little bit crazy, because that’ll get you noticed. I’m looking at the person that judges them, and they’ve got like this strange red eyes and a weird grin, so amuse them is probably the best way to win the prize. #futurewan. Tell people about it.
And, as always, we’ve been – I’m Greg Ferro from the Packet Pushers, and my cohost here is the fine and upstanding Mr. Ethan Banks. Thanks to Viptela for sponsoring this event. Remember that FutureWAN is a whole week. So you’re watching this on the second day; there’s two more days of these sessions, including the rest of the day today. What you – if you can’t watch them all at once, don’t hesitate to register, and the backend system where you’ve registered will remind you about the sessions and make sure that you find out when they’ve been recorded. And then you can watch them in the leisure in the john later on today.
Ethan: If your question did not get answered – someone asked about local internet access – I’m going to try to blog about that for you at packetpushers.net. Keep an eye out for it.
Greg: All right. I think that’s a wrap today, and if you’ve got anything – thank you so much for joining us, it’s been great, and nice to see you all.