Kindred Healthcare’s SD-WAN ROI and Results (ONUG 2016)
In this session, Kindred Healthcare discusses how they have realized their strategic vision for a Hybrid WAN, end-to-end security segmentation, and provided a foundation for multi-tenancy across their enterprise network. Kindred will share how their deployment is resulting in significant performance gains, higher availability, reduced equipment cost while providing a real return on investment. Kindred will share their long term strategy, how Viptela’s Software Defined WAN solution is helping solve these requirements, as well as integrating their virtualized data center strategy in a production healthcare environment.
James: As you mentioned, we are thrilled to be back at ONUG. We’ve got a long history here. Viptela came out of stealth in May of 2014 at the ONUG here in New York that City hosted and at that ONUG Cigna actually presented their – what they called Cigna Service Provider Agnostic Use Case for C-Span which is really the first SD-WAN use case that was presented at the ONUG forum. And since then we’ve tried to continue that tradition of bringing customers that are driving software defined networking innovation to come and to share at the conference with all of you on what they’re doing, how they’re driving their use cases, how those deployments are going. We had the Gap come and talk a little bit about how they’ve taken this technology, driven a transformation across over 1300 retail stores worldwide across five different brands.
I know I saw [Snahill Patel] walking around here earlier today, so if anybody wants to catch him, I’m sure you can grab him. Actually in the spring, in Mountain View, we had Pasqual from Agilent talk about how Agilent has driven a transformation across their worldwide enterprise network, driving about an 85% cost per megabyte reduction in service and actually improving their application performance both for real-time applications like voice and telepresence video as well as for a number of big data analysis applications that they’ve been driving.
So that being said, I’m really excited to be here with Eric and get to share a little bit more about what Kindred has done. And Nick, as you mentioned, this is not Eric’s first discussion around software defined networking. He actually presented sort of his problem statement a few years ago at ONS, I think it was in 2012, right?
James: And just for those of you who don’t know, Kindred is the largest post acute care diversified healthcare provider in the United States. Over $6 billion in revenue, 75,000 odd employees and I think you guys have facilities in 46 of the 50 United States, right?
Eric: That’s right.
James: So maybe you can start off with a little bit about the current WAN and some of the challenges that you talked about back in 2012 and then dive into where you’ve gone from there.
Eric: Absolutely. So back in 2012, Nick asked me to participate on a panel discussion about challenges that we faced in enterprise networking and how software defined networking might be able to solve some of these challenges and I had just recently heard of software defined networking. Nick kind of clued me in a little bit and hooked me up with some resources and he was really interested in what are some of the big things that we were facing that might be able to be overcome using this new approach.
Here are some of the excerpts from that presentation that are just direct quotes off that presentation in 2012. I said every hop on the network must have a special configuration applied. Manually configured gnats. All kinds of routing re-distribution. Route filtering, ACLs and etcetera. A large number of locations, they all require different types of configuration to address use cases for those different facilities and different things you’re trying to protect or just basically a lot of bailing and duct tape for all the integration work that you’re doing. Then a limited number of support staff.
Kindred is a company that we’ve grown by acquisition by leaps and bounds. I’ve been there since 2004 so a little over 12 years and since I’ve been there we’ve gone from about a 170, maybe 200 locations, smaller field of nursing center and LTACH hospitals. To now we have thousands of locations, as you said, it’s almost a 100,000 employees. We’re now a Fortune 100 company. We do three to four, five acquisitions every single year. Some of them are small and some of them are pretty large and we’ve had a lot of headaches integrating all those networks.
And then – and then in the broadband, we talked about broadband in that presentation about how broadband for site backup or small site connectivity was cheap but it was difficult to support and unreliable.
And then the last thing we talked about was how about was how could we scale the network resources without SDN. So we have a limited support staff. Back to my conversation, points I made a minute ago about the large number of acquisitions that we’ve done. Back then we had five network engineers and now we have seven. So we haven’t grown our support staff in the same rate of scale as we have the integration.
James: Great. And I know that you guys kind of distilled the problem statements down. We started working with you about a year, a little over a year ago. You’d taken a lot of this and said hey, these are the problem statements we had and they’re really the use cases that we want to drive so we want to cover that.
Eric: Yeah. Sure. We’ll touch on this new era. This new era, we think that we’ve been researching and testing software defined networking, watching who the players are in the market, looking at some of the other technologies and I think it’s really nice to see the technology mature to the point where we can actually begin to deploy enterprise ready solutions right now. We’re at that right now. What’s interesting is that you’re used to doing things the old way and you operate with this mindset of having these blinders on and you can’t even imagine what the possibilities are because you’re so used to looking at things in that mindset. I think now we’re kinda positioned for the first time ever to have those blinders be a lot wider and seeing these technologies come on and be market ready is like you see problems in a whole new light because you think of so many different ways you can solve them.
These are some of the use cases that we’ve kinda been defining over the years. The ability to route different types of traffic over different types of connectivity or transport independence basically. It’ll dynamically change the route for that traffic depending on network conditions. Maybe allocate bandwidth limits depending on policy. Maybe you want certain types of traffic not to route over the preferred link but you want them to route over the backup link, but you don’t want them to take the entire backup link. Just to hit highlights, segmentation and isolation is a huge thing for us right now.
We’re really looking at how we can secure our network from end to end, even internally. The micro segmentation conversation. How can we extend end to end segmentation all the way from the data center to the branch? That’s really big for us right now and that kind of leads up to the isolation of different types of application like PSI or medical devices and have them isolated end-to-end across the network. Maintaining consistent policy and configuration, zero-touch provisioning. Break fix. We have seven network engineers. I don’t want my seven guys calling up techs on site worrying about break fix methodology or even new site implementations.
I want them to hand all that off to a lower skilled set group, to just an operations group or an implementations group and have them easily be able to follow along with the documentation and deploy sites reliably. And then just provide resiliency and redundancy and gain that efficiency of being able to use all the connectivity that we have in a location instead of just routing over one path, you’d be able to route over multiple paths and be able to make decisions on that too. Route voice traffic over MPLS for example except for when the network conditions are bad and route maybe file traffic or video traffic over the broadband connection where it needs more bandwidth but it can deal with a little bit different types of connectivity.
James: That actually helped us I know when we got engaged on the initial proof of concepts. You guys had not only done a lot of work defining these use cases but in many cases, you already had broadband or direct internet access deployed at these sites. It was just configured in a way where it was active passive because of some of the challenges you had with the legacy solution. So we were able to kind of fast start the production pilots because you guys had a lot of that connect to video already provisioned.
Eric: Yeah, that’s exactly right. We had broadband in a lot of locations and we’d even tried some different solutions that didn’t quite work exactly the way we wanted them to. Our CIO has been on us for years. Eric, why can’t I use that broadband connection that I have sitting at the site to route YouTube traffic over for training videos? Why does it have to impact my citrics traffic when I do that? We had QOS in place, but in a healthcare industry, Kindred operates on a very, very thin margin in a tough reimbursement market and the healthcare business. So a lot of these sites are running T1s. Maybe one or two T1s. A hospital with a 100 beds in it running behind two T1s and so we’re kind of trying to find a lot of ways to get the most value that we can out of our network infrastructure.
So that kind of brings us to our feature requirements. We wanted to have policy-based application aware network routing over multiple connections. The intelligent routing where we could say if we were suffering 2% packet loss, change the route dynamically from the voice traffic to route it from one connection over to the second connection and to be able to do that on application basis, not just QOS or DSCP values. The network segmentation/isolation is huge. As many of you know, in healthcare and in many enterprises right now, the security requirements are getting tougher and tougher.
We read about it every day somebody is getting some PHR or some data stolen and a lot of times it’s not from the external perimeter but it’s something that’s been sourced from the inside where they have a large footprint to the internal network. To be able to selectively route traffic through security points is selective service training use case is huge. If we wanna take certain types of traffic, let’s say we have a vendor with some clinical equipment on our network and they’re just using us for basic connectivity, we may want to route that across our enterprise fully separated and micro segmented into our data center and then route that through Apollo firewall or a different vender firewall before it goes out to the internet.
Maybe we only allow one little bitty communication or one little interface to the inside to their corporate network. Then the zero touch provisioning and centralized management and to be able to use any type of connectivity. We’re going to have use cases where we will have MPLS, broadband, LTE, direct internet access, care grade internet. Whatever we can think of. A lot of times in these acquisitions we don’t have time to spend up a 90-day circuit so being able to send something out with our gold configuration on it with LTE connectivity as a stop gap until a circuit comes in is huge.
James: Great. So I think you can share a little bit about the topology that you guys have arrived on for the deployment.
Eric: Sure. So these next three slides, I’ll walk through just kind of an example of what I was talking about with overlays. One of the things we liked about the Viptela solution that became apparent to use really quickly when we started looking into it is that the overlay methodology using VRPs or VPNs is very, very easy to configure unlike some other types of methodologies or approaches that we’ve used before. They’re really complex. You can very easily define in just a few lines of configuration in a policy map style that hey, I want this new VRF, a new VPN where I want to have internet of things where that doesn’t touch my internal network at all and I’m just going to ship it off to a centralized data center and out some security clients somewhere.
And then we might have a different line of business where we have … it’s just a different line of business. So the access to the resources that they need, like in the orange, might be a different tenant or an acquisition that we got, that has overlapping IP address space. We want to be able to take that ride it over any transport into our data center and keep it separated until we can apply policy to it and security policy on it.
Then the last slide that builds on this is what I touched on a minute ago. If we have HVAC equipment out there or a third-party partners or any kind of clinical equipment that may not be controlled or secured by our client systems teams, maybe we don’t know the patch state of it, maybe we don’t know what data is on it but this is a required system such as risk pack or imaging or any kind of telehealth equipment. We now have the capability to put that on our network and to keep it completely isolated from our internal corporate network and have that ride through an overlay, encapsulated, secured end-to-end into our data center and again through our firewall.
James: Awesome. Yeah, this is a common theme that we’re seeing with our customers where being able to deploy really any type of transport connectivity in the underlay and think about a true virtualization layer and then build whatever segments are required for a line of business applications or in your case, for patient medical devices in the overlay and I can make changes in the overlay without any disruption in the underlay and changes in the underlay without any disruption in the overlay. That flexibility is just the distilled value that we’re providing.
Eric: Yeah, it was immediately clear that when we started deploying this just how easy it was to add new features such an another overlay or another VRF to meet a specific use case without any disruption. And it was easy enough where any engineer on our team could quickly be shown how to adjust the policy, put it into action and it just worked. Kyle whose here with me today, he’s one of my colleagues and he’s done a lot of the initial roll out and pilot work and he’s done a great job at it. He went on vacation for a couple … he went off and got married and I’m kind of the guy who is the vision behind it and kind of made the problem statements and helped develop the use cases and how we might achieve it. He took off and I went in while he was gone and added two new overlays and a whole bunch of policy for some new use cases we did in just a matter of minutes and had that site up and running. We put a new pilot site in for that use case and it took about 10 minutes.
James: Great. I think you can also show a little bit more about what you were talking about on the service training.
Eric: Yeah, this is just another drawing that it kind of shows a good … it’s a good view to kind of show how you might be able to do some security segmentation, micro segmentation and use different overlays to meet different use cases. So if you have the orange, the red or I don’t know what color is shown up there, purple and green users, you could have the orange user be on one segment of the network or one isolated sub net route end-to-end all the way through and then tie that in.
Maybe you tie that in with your virtualization infrastructure and you do a hand off and map that to a VLAN or a tenant or a VRF inside your data center and that could be one tenant end-to-end. Then the red could be the exactly the same way except for maybe on the red tenant, you don’t quite trust him as much and you want to ride him through some IPS appliance or a firewall and apply some limited policy to that. It’s very, very easy to do to add that to the configuration where the next hop on the network is now this firewall to route that through and then back into your data center.
It may be on the green users it’s just a use case where you have some partners and they just want internet connectivity. We don’t like people coming into our medical facilities and hooking up broadband in hotspots and other things and then attaching to the local network where they could then gain access, have an entry point for some malware or some other thing that might proliferate the local LAN and then it comes back in on the backside. If we can give them secure internet access and keep them off the public network but keep that secured and still not touching our network internally. I think it’s a huge win for us.
James: So we were able to prove the use cases and prove the deployment model pretty quickly but then it came time to actually sell this to get funding. I know you guys did a tremendous amount of analysis on the economics and can share a little bit about what that impact has been for Kindred.
Eric: Yeah. What we did is we took a couple of things where we looked at how much bandwidth we were getting and what we were paying currently using a legacy solution and we just went … we’ve just been going through an RFP process so we really used a conservative approach that we could to an ROR calculation and kind of calculated what our existing MPLS style, kind of the old school methodology of doing networking would cost compared to the Viptela solution.
Then we evaluated all the hardware we would be able to pull out of the facility. We could pull out the router, we could pull out WAN acceleration appliances, we could pull out … we had a project lined up to do this isolation, this directive from the board to where we had to provide secure segments in the branch for PSI and HIPPA compliance and clinical device use cases. So we calculated in what those firewalls would cost because we looked at one approach of how many firewalls, just small little firewalls in addition to the licensing and the maintenance on all these products that I just referred to and we calculated all that in with ROI along with the Viptela.
The Viptela is it’s kind of a newer model like a lot of companies are going to now with more of a maintenance subscription type model and we wanted to offset that operating cost. The capital cost, the hardware investment was just … it’s not even worth discussing, they basically give this stuff away, but there is an operating cost.
So we had to look at the current cost of doing business is the way we were doing things and what we were trying to accomplish. We were able to show a huge total cost of ownership savings. I calculated that just with … for 700 site deployment, we could easily realize $2-$4 million in five years of savings just in the operation cost, total cost of ownership in that solution for five years.
James: And that’s still driving a 700% bandwidth increase on average across the footprint.
Eric: So we averaged out, with RFP, we averaged out that with the network upgrades that we were going to, we were going to assume a five megabyte on average with the old method of doing it or kind of the defacto method of doing it. With the Viptela solution, we averaged out what kind of broadband we were getting because we use a lot of broadband. So we averaged out around 35 mg. On average, we could get 30 mg broadband in some places. Some places were 15. You might get 15 mg and we got other places where we get it for a 100, 150 mg for very cheap.
James: Great. We just have a couple minutes left. I think you wanted to share just a brief kind of look ahead at what you guys are looking at from both a savings perspective and then also sort of next steps.
Eric: Sure. Some of the future savings that we’re looking at is we’ve had these discussions with our boys teams. We’ve been trying to get the centralized SIP model and we really need to protect the voice traffic and a lot of our voice engineers were concerned about doing centralized SIP. Now we’re able to deploy a network solution in place that gives high availability and resiliency and the ability to not only see and protect that voice traffic with QOS but also to fail that over to broadband dynamically so we have actually deployed in SIP locations and proven that it works well.
They have an actual increased voice up time so it got them a lot more comfortable with that. So now we can accelerate getting rid of PRIs and more expensive voice solutions to go to centralized SIP. We think that when we leveraged this technology with an SD data center solution, such an NSX or some other data center solution where you can do a layer to extension and have some of your VM mobility and do some of the cool software defined things and network, we think that we can take this solution and quickly go into companies that we’re acquiring, put a Viptela appliance in there, run their legacy network and the Kindred new network, even with overlapping IP schemes and have that overlay be able to bring them into either old data center or new data center and be able to migrate that workload from their old data center into our data center.
I know this last acquisition we did of Gentiva Healthcare was 600 servers, 50,000 employees and 500 locations and we spent over a year and a half just re-IPing servers so that we could get them from point a to point b. In addition to re-IPing all the facilities and all the disruption that caused. I think I can take this down now from years to months and that’s a huge cost savings.
Eric: Looking ahead next steps, I just kind of touched on it a little bit. I think if we could map Viptela to a software defined data center. So right now you could probably take just that one Q-tagging or do layer three point to point connections in your core from the Viptela appliance and then hand that off to your data center if you’re using NSX, you could use your V tep to translate V LAN to VX LAN and map those tenants over or you could do just a straight layer three point translation between VRFs. I think one of the possible next steps is what would happen if through a partnership or something else maybe VX LAN got incorporated into Viptela.
That’s not on the roadmap. That’s me saying this as a wish but I think when you look at this technology and the fact that it’s really not about the appliance, it’s about the software, they can put that software, you know, at some point, you can see where this is going to evolve where you can put that software on anything and be able to have those dynamic scalable VPNs or overlays created anywhere you want them to. It’s going to be huge.
James: Yep. And we have that deployment model available today or X86 virtualized.