Cellular Routers versus Cellular Out-Of-Band Management
Dan Baxter, Senior Sales Engineer of Opengear outlines how the cellular revolution is the driving force behind the availability and adoption of technology related to cellular-based networking and its management through cellular failover for out-of-band management. Discover what new options are now available to Network Administrators, and what is needed to understand and manage networks within this new dynamic.
“Hello, everyone. My name’s Dan Baxter, I’m a sales engineer with Opengear. Little bit about me – I’ve been in the out-of-band management field for about 15, 16 years. In the last four years, I’ve seen a big change where cellular technologies have started to be embedded in the out-of-band — and routing technologies — as well.
So I wanted to come and talk to you today to give you an introduction into cellular routing and cellular out-of-band management because what we’re finding is although many of our customers can make the Internet work, and keep their organization’s networks going, when we start to talk about going cellular, it becomes very educational. So I just wanted to give you a broad overview today.
Because cellular adoption, obviously, is no mystery – there’s huge adoption going on today. In America, there are more cell phones than there are people, and I think that’s true around the world: 4 billion people, and 3.7 billion cell phones in the world. But what’s really surprising is that in the last few years, the trend’s shifting for people using their cell phone, using cell technology to access the Internet.
So what this is doing is driving your options and your choices for using cell technology so as network admins, network architects, it’s really important to understand your options, your carrier options, your plan options, what the technology is, so you can make the right decisions about implementing the technology and following best practices around them.
Today, I’ll be talking about two of the applications in the cellular arena: cellular routing, which is using the cellular interface for routing traffic between sites and cellular out-of-band which would be using cellular to get access to a remote site. If there was a network outage, you’d get access to your remote out-of-band management equipment, bring the network up and runing. They’re two different but different scenarios.
So both share the idea of a primary and failover interface, both are similar in the security concerns, and both share similar IP plans that you can get from the carrier, but there’s differences in how you deploy them, what kind of security policies you put in place, and also the costing on the plan. So in the cellular routing form factor, you get two categories: you get adding cell to existing routing infrastructure such as a module that might go in the router or an external USB modem, and you get cellular routers that are actually the cell is embedded right inside the router. You can get two kinds of categories in that: the consumer-grade products and the industrial-grade products.
When we talk about cellular routing, within how people are using routing technologies within these. This category, you’ll find there’s HA – that’s high availability. The other one is called “IP Pass Thru” sometimes in the industry, or a third situation where the unit is standalone and it is the only router for the location.
In HA, there’s a protocol handling the failover, and typically there’s two physical devices: one’s primary, one’s a backup. Primary’s the main router of the site, it’s going out the WAN link, and if there’s a failure with that, then the second cellular-enabled device kicks in and the end hosts don’t know the difference. VRRP is a very commonly used routing technology in these scenarios.
There’s another mode – “IP Pass Thru”. This is typically where you’ll have an existing router at its location, and you’re looking to introduce a separate failover link for that router. So that main router will handle all the failover, it’ll handle all the changeover to that link. In that case, you’re protecting yourself from either router failure or a WAN link failure because there’s really different modes depending on what you’re trying to achieve. So the IP Pass Thru, it really depends on the primary router being available. If that primary router fails, and the hosts were using it for their failover, obviously, they’re not going to know about the change, and know how to get over to that cellular router.
Cellular Out-Of-Band management form factors are very similar to the routing. There’s items where you’ll take an existing install and add cellular to it with a USB dongle or external adapter, or you can get a device that has this cellular built directly into it; it’s embedded inside the device.
So external modems are great for adding cellular to an existing installation. There is a drawback, though – it introduces a failure point: dongles and cables are vulnerable. They can be damaged or moved, and you also have third-party driver considerations. All the devices you bring in, does that match up to existing equipment, and is that driver going to be supported?
Embedded, on the other hand, gives you the advantage of removing that single point of failure. Because this technology is built around current cellular technology, this is going to be a nice fit. It’s a nice fit at a hardware level, and there’s a nice fit at the feature set. You can recognize a purpose-built, embedded, cellular, out-of-band device primarily around the form factors. It’s a little higher port densities, it might be RJ-45, and there’s also a related feature set in the firmware for handling failover and also the notification.
Products that are, I’m going to call them an add-on, they typically have a smaller form factor, smaller port density, there’d be a DB9 on them, usually lower port costs, and they’re typically not meant for a data center – maybe a remote site location.
Crucial to any of this is the idea of failover. So the primary link’s up and running, you’ll use that with the device. But then there’ll be some type of trigger, whether a PING trigger, or an interface state change, and the device will recognize that change. At that point, it’ll bring up the cell interface. In some cases, the cell interface will be dormant , meaning it has a different route/metric on it, or it just might be entirely turned off until it’s needed.
So the challenges around that in terms of the out-of-band cellular strategies is that it’s really vendor focused related. What is the vendor’s focus — is it embedding directly into the device, is it an add on? That kind of interplays with your need of “Am I looking for an add-on to an existing install, or do I want to have something that’s directly embedded into the device?” You’ve got to kind of ask those questions going into it. I recommend avoiding dongles because it’s a point of failure. This bit me as a veteran in the out-of-band world — I know what happens to modems and external cell modems have the same vulnerability.
Now we’ll talk a little bit about some of the cellular considerations. One thing that’s necessary is to understand what your options are, the costing, the types of plans that you can get from your carriers — as well as the wavelengths because, really, it comes down to the bandwidths and the wavelengths. There is no one-size-fits-all, and in America, it’s very, very carrier specific.
So Internationally, it’s a little different — it’s more about the wavelengths than the carriers. The modems that you are getting from the devices are matched to the wavelengths of the carrier. For instance, in America, LTE, both Verizon and ATT are going to use the 750MHz range, but they’re going to use different bands. AT&T uses band 17, and Verizon uses band 13. The carriers in America will certify the devices to be used on their network; you cannot take an AT&T product and put it on the AT&T network. You cannot take that SIM and move it between the vendors of the carriers. They hand out an IME number, that’s much like a MAC address serial number, it’s an identifier that registers that number, that device to your account, but it also certifies it to work on the network. Because of that, you’ll order your equipment for AT&T, or you’ll order it for Verizon in America. Again, in different countries around the world, it’s a little bit different.
3G, 4G considerations. comparing them, to me, it really falls down into coverage and speed. I find that in a lot of not only the geographic coverage that 4G is better than 3G. I’m also finding the strength of the signal being a lot better for 4G. Of course, it’s really going to depend on where you are and the geography of the towers and services around you. But the speed’s also better, the carriers are saying 15 megabits download, 100 megabits upload, theoretically, I guess. Compare that to 3G: 1.8 and 3.1. So if you’re doing out-of-band management, and you’re SSH primarily, your speed difference, for instance, may be different than someone who needs to route that traffic across that link when there’s an outage.
There’s a nice table on Wikipedia for both America and international because it becomes a matching game of saying which country am I going to, what carrier is it, what megahertz do they support, what bandwidth. Then you have to go to the hardware and you have to match up a similar hardware on the cell modem and also on the firmware. It really gets complicated internationally sometimes. Different regions in Canada will have different bands supported, so you really need to – these are great references up on Wikipedia.
There’s two types of accounts that typically I see customers going with: there’s a consumer-grade account, and then there’s business-grade accounts. The difference really is that a consumer-grade account is the iPad, tablet – you might walk in a Verizon store and sign up and get an account with Verizon. A business, you’re going to be a business account, everybody gets their smartphones in the company from one carrier, and maybe there’s not a mobility person in sight. There’s typically a sales rep and an SE assigned to each customer, and ATT and Verizon, I know, are both this way. You get quantity discounts on the business side, so that’s important to know.
When we talk about the plans, they fall into those two categories: routing and out of band. For the routing, it’s an LTE network routing plan – 30 gigabyte is about the smallest data rate you can commit to. In the out-of-band arena, it’s different. It’s a machine – a machine plan. They categorize it differently because it’s used differently, and 1 meg is where you can start off. So there’s a big difference there between those two types of plans.
Regardless of the types, there’s three types of IP addresses you can get: you can get a private NAT address, that’s usually like your iPad, your phone falls into that category. Or there’s a public address where there’s a static IP or DHCP, sits out on the Internet – that’s handy for some people – and then there’s a private carrier network cloud (not a lot of people know this option) but the carriers can create a private network cloud for you.
In the private IP plan, the pro’s and con’s are that it’s easy to get going, there’s a low up-front cost… the downside is that it’s a private NAT. So you have to have some kind of way to get to that device, and usually the device is initiating that as an outbound SSH session – it could be an outbound VPN session back to a VPN gateway, and that way you can get over that limitation on the private NAT plan.
Public IP plan is great if you need to have a device sit out on the Internet. It’s going to be static IP, or it’s going to be DHCP. If you do DHCP, you can use DNS services to let you know if the IP address is changed. You may want to think about VPN technologies, and we’ll talk a little bit about security in a little but here, but you can leverage VPN technology on that device since it sits directly on the Internet. It’s easy to access, but it’s got to be secured. There is a one-time setup for this type of account – it’s $500 – if you call up the carrier and say, “I want a private network or a public IP,” it’s going to be $500 one time, and as you add devices in, you just pay a monthly recurring fee on those devices.
So the pro’s and cons of the plan: the pro is it’s on the Internet. The con is it’s on the Internet. So it all comes down to security – you want to firewall the interface, you want to control the services that are active on the interface, and you want to control the amount of time that interface is active. If it’s not going to be used all the time, there’s some sense in making it not turned on until you have a failover situation.
A private carrier network is really popular for customers with a large deployment or someone with very secure data, and they don’t want their traffic going out with everybody else’s smartphone traffic. The carrier provides an isolated network and they can even create hardware-isolated networks for you, and you’ll MPLS into that network or you’ll VPN through gateway on the Internet. So you’re going to route between devices there, they want you to be using BGP, maybe use a GRE tunnel. They have very specific requirements inside that tunnel – what you can and can’t do inside that cloud that they’ve created for you.
So it’s a great option if you need to secure your data. The only drawback is if you’re a road warrior, how are you going to get in? You have to get into corporate to get into that connection. If your corporate site’s down, you can’t get into it. The other part is, not every customer has VPN- or MPLS-savvy people on board or hardware. So it really needs to – it’s just a pro’s and con’s, what one works best for your customer.
So really, our recommendations and best practices is: choose the best plan based on your security needs and pursue the business plans over consumer – buy in bulk; you get a better discount. The more you buy, the more units you put on that plan. And you might have a mobility group in your organization – that’s a great place to start.
So what are the costs of these different plans? If I’m looking at a cellular routing option for a remote location, and I go to Verizon, ATT and I say I want to set up a routing account, it’s going to start at 30 gigabytes, and it’s going to cost me $185 a month. If it’s an out-of-band arrangement, then it’s going to be a M2M plan, and it’s $5 a month, and it’s 1 meg as my entry point. The reason for that difference is out-of-band management over cellular is really about SSH. I’ve got a remote site down, and I need to get in and remediate that situation. So getting in, remediating the situation, getting things back up and running, I could use as much as 100 kilobytes. I could have ten of those instances in one month and still not go over that 1 meg. So when you consider you’re pulling it, that’s primarily the reason why the two plans cost differently.
So the idea here is that this is not your dad’s modem. I don’t even know if modem’s a good word for it, but it’s really not an analog VT-100 modem like you used to get into AOL. It’s an IP interface, and that’s the thing that’s most important to recognize. The plan you get from the carrier is going to be an IP interface, and it’s going to dictate your security policy. So all the standard firewalling rules and IP rules you’d apply to any IP interface applies to the cellular interface. So the specific plan, it’s going to be driven by your security policy. Consider tunneling – even if you have a private network with the carrier, it’s good to tunnel over that network. Consider turning off services – do you need HTTPS on that interface. Limit the services – I mean this is all very basic security policies and it’s all dictated by the fact that this is an IP interface. The other thing is, maybe you need to have some brute force protection on there like to fail-to-band if you’re facing the Internet with a static IP, and also implement alerting. You want to know if that interface goes active and it’s facing on to the internet because of some kind of failover situation, you should be notified that that has happened.
So we’ll talk about going into deployments. Initially, the biggest concern we’ll get from the customers is the signal strength of the data center or the remote site. So it all comes down to RSSI or LTE signal strength. It’s a negative number in decibels, and anything -40 is strong to -98, which is weak. If it’s beyond -98, then the device may not activate.
So there’s some places you could check. You could always ask people, “Hey, go log in with with your cell phone,” or I’ve been in there, I know what my signal strength was. You can also go to a site. There’s many like this, for instance downdetector.com. It’s a great way to track by carrier availability. The only word of warning on this is that smartphones fall into this category, but it’s a good starting point. I think OpenSignal is pretty good as well, opensignal.com. What’s nice there is that they’ve got a graphic overlay, so you can look at the region where your datacenter, your site is, and then you can drill in and see by 2G, 3G, 4G. You can see, is 3G’s going to be a stronger signal for me, is 4G better, what is the reliability in the area, and what is the uptime, what is the ping in download/upload speeds in that area.
My experience is that 4G really has better coverage – both coverage and signal strength. Cell phones, they can be conservative, so be careful if someone says “I didn’t get good signal so cell’s out for us.” Be careful about that because a cell phone is conservative in its reading of that signal strength.
If you look at a site survey and you look at the best, better, and good, the cell phone’s a great place to start. You can get an app on your Android, there’s GNet trackers — one that Verison’s SE’s use and other customers have installed it; It’s a great tool. I don’t recommend it but you can put your iPhone in Field Test Mode and it’ll show you RSSI, it’ll show you the cell signal strength. Even better is to take the actual router or the embedded cellular out-of-band management device and walk it around. They all tell you the RSSI from the user interface. Even better – the best – is a measurement too. So you can get measurement tools that will be carrier agnostic and will tell you the 3G, 4G signal strength at your sites.
Typically when we see customers with out-of-band cellular, we see two different models, whether they’re going into the datacenter or a remote site. If the datacenter, it’s usually one cellular-enabled device acting as a gateway to all the other out-of-band management devices, so if there’s an outage, they just need a single entry point. At a remote site, you’ll see a device installed at each location.
The other thing that’s really important in terms of deployment is that interface handling. You need to consider how the hosts are going to route to that interface to how do they know if there’s been a failover or if there’s a new route. Also, do you want to have that interface dormant? That saves you on costing; you’re only going to use that interface when there’s an outage or emergency. It’s also going to limit the amount of time the interface is available for malicious activity. You could also consider QOS – maybe not everything needs to go over the cellular interface when there is a failover. Maybe you just need to get the most critical data across that interface. And there’s also the ability to watch the interface to make sure it doesn’t go above your plan level. One thing you don’t want to be surprised by is some process running out of control or the cell interface being used in a way you didn’t expect and only knowing about it when the bill comes.
So what are the best practices in general? As a summary, really, know your carrier, know your plan options, and get your routing options in place ahead of time, conduct a site survey, know the signal strengths in your location. Choose a plan ahead of time because if you have to switch plans, the carrier may ding you again on the setup fee. Get to know your carrier reps, they want to talk to you, they want to sell these lines, they want to sell the cellular plans to you. They’re very good, they’re very consultative, and know that if you’re going to go internationally, it’s going to be different; there’ll be different standards internationally than in America. Don’t forget about budgeting – this is one that gets passed up. Traditionally, phone lines as an out-of-band option. might have been billed regionally. Now, if it’s a cell plan, it might be billed centrally. Either way, it’s an item that you should include in the budget as a monthly recurring fee. It may not be high, but you should account for it. And can you roll this into your existing management fabric? It’s a separate IP interface, it’s accessible during an outage, and you can get data, and you can get information from that site even during an outage. Of course, make sure your vendor is an expert – they’re focused – it’s a lot of work to try to match up the right equipment for the right carrier in the right country. Your vendor should be able to help you out with that process.
So coming up on my timeframe here – any questions? Here and here?”
“Hi. Brandon Ross, Network Utility Force, this is actually more of an observation than a question: We deployed some Opengear actual terminal servers with the cellular network for some troubleshooting we were doing, and this is hopefully so no one else has to spend all this time trying to figure this out, we notice that we got what appeared to be a public IP when we first connected to the cellular network while we were kind of expecting that it would be carrier grade NAT’ed. So we were like, “Great, this is public IP, we don’t have to worry about the NAT. Well it turns out, and I am going to name a name T-Mobile uses addresses that aren’t assigned to them on the cellular network that are actually NAT’ed. So I forget whose address block it was but it was very surprising once I looked into it that we thought we had real public IP’s, but they were actually NAT’ed but not using RFC 1918 addresses, so we were scratching our heads for quite some time until we figured that out.
“Interesting, interesting. Thanks.”
“I don’t know if any of the other carriers do that too. Maybe I could turn that into the question – have you experienced other carriers using address space that was not assigned to them?”
“I haven’t. It doesn’t mean – You know, I don’t hear about everything. What’s that? Rogers. Yeah, I haven’t heard of that, but that’s interesting. Thanks for sharing that.”
“Andy Liu, JHU, just wanted to know, with the public addresses, if you go to different states, do they change, or does it stay the same?”
“If the device travels?”
“Typically, with Verizon and AT&T, that IP address is always assigned to that device. It’s got a phone number – you can text the device, say, “Hey, run a script, do some things,” but you can also always address it by the same address. In Canada, it’s a little different. It may be a DHCP address, so as you move from system to system, it might change. Thank you.”
“Joshua Goldbard, Tevnos – My question is about cellular frequencies. So Sprint uses primarily 2.5 gigahertz for their LTE, and ATT and Verizon are using about 90 megahertz. Inside of datacenters where the equipment you’re talking about is primarily deployed, have you seen coverage issues with higher-frequency bands or is that a non-issue?”
“I haven’t, necessarily. We typically will address situations like that by putting a cable that’s longer, 2-, 30-, 40-foot low-loss cable and taking the antennae over to a spot where the signal strength is greater, but bandwidthwise, I haven’t seen anything regarding that. It may be going on, I’m just not aware.”
“Have you had a situation where you had to implement a small cell? Or is that not a thing you had to do?”
“Had to what?”
“Implement a small cell?”
“Net yet. Net yet. Again, the customers may be doing it, a lot of customers are very self-sufficient, but I haven’t heard of that. I haven’t had a customer either say it wasn’t strong enough and we had to switch or we had to implement anything. Usually, they’ll kind of know – it’s either a dead spot in the data center or not. ”
“Got it. Thank you.”
“Wes George, Time Warner Cable. Everybody can go ahead and check their Bingo cards ahead of time, but I’d like to congratulate you for spending all this time talking about IP addressing and different flavors of carrier grade, public versus private and static, whatever, without ever mentioning IP address exhaustion or IPv6”
“Yeah, I have seen on the NANOG groups a lot of discussion around IP version 6, and we’ve had some discussion with customers over that, and they go to the carriers. Initially the sales teams at the carriers are like, ‘I’ll have to check on that.’ The big question is if we’re IP6 here and you’re not, how are we going to deal with that?
Any other questions? Thank you everybody for your attention.”View all Videos