Los entornos de centros de datos continúan evolucionando, y su función y composición han cambiado considerablemente en los últimos veinte años. Desde centros de datos a hiperescala hasta colocaciones, cualquiera que sea el tipo de instalación, las empresas esperan velocidad, rendimiento y eficiencia.
El aumento de tecnologías nuevas como Internet de las cosas (IoT) es un importante motor de crecimiento en el mercado de centros de datos. Se prevé que el mercado de centros de datos de EE. UU. obtenga ingresos por encima de los 60,000 millones de dólares para 2024. Y la alta demanda de instalaciones, hoy más que nunca, exige una solución inteligente para supervisar equipos de centros de datos.
Smart Out-of-Band de Opengear eleva la gestión de la infraestructura a un nuevo nivel. Al dar un paso más que las soluciones tradicionales de gestión fuera de banda, garantiza un acceso eficiente y seguro a la infraestructura del centro de datos.
Gracias a su optimización de la gestión remota de la infraestructura de red, Smart Out-of-Band proporciona una ruta alternativa hacia los dispositivos cuando la red principal se cae. La inteligencia automatizada garantiza el tiempo de actividad de las aplicaciones críticas al permitir la supervisión y el control remotos del entorno. La identificación proactiva les permite a las empresas solucionar rápidamente los problemas para garantizar la continuidad del negocio y reducir los costos de tiempo de inactividad.
Lea nuestro informe técnico para obtener más información sobre las ventajas de las soluciones inteligentes de Opengear.
Smart Out-of-Band les permite a las organizaciones de todo el mundo garantizar una conectividad de reserva resiliente en todos los dispositivos en centros de datos distribuidos. Los servidores de consola y las soluciones de gestión de infraestructura de Opengear garantizan la resiliencia de redes para permitir la continuidad del negocio.
Opengear colaboró con i3D.Net, proveedor global líder de servicios de infraestructura y alojamiento de alto rendimiento, para ayudar a mantener sus operaciones globales. Como necesitaban que la red estuviera constantemente en funcionamiento, implementaron soluciones de Opengear para garantizar la resiliencia de los enrutadores en todo el mundo.
Ensuring network resilience is an important business priority for i3D.net. To maintain its global operation, i3D.net runs a complex low-latency network, with thousands of servers spread over more than 35 points of presence on 6 continents.
It is key that this network is kept up and running at all times. As Rick Sloot, chief operations officer, i3D.net explains: “Ten years ago, this was very difficult to achieve. There were very few ways of handling and managing your servers when the internet went down and those that were available were expensive and did not work well. As technology has advanced over the past decade, customer expectations have grown in line, and a high level of network resilience is now demanded by our users.”
“As we expanded as a business, opening up new locations around the world, it was becoming increasingly difficult to find a vendor capable of delivering a reliable way of keeping us up and running at all times, across all our locations.” continued Sloot.
“We struggled to find local vendors who could support us. Often, we weren’t able to pinpoint the issue or access the faulty equipment. We needed to find a way of managing equipment remotely, identifying and resolving the problem, while keeping the network online. “We started to look for a vendor who could work with cellular technologies such as 4G and who could deliver out-of-band management. Opengear fitted the bill perfectly. Opengear’s tools can support 4G cellular out-of-band connections in all our global locations which have enabled us to save time and money. We were looking for a partner that could deliver triple-A equipment on time all over the globe and knew we could rely on Opengear for this.”
i3D.net decided to work in partnership with Opengear and implement its ACM 7008-2-L Resilience Gateway to provide smart out-of-band management to its entire IT infrastructure, and also to provide a built-in backup LAN and/ or backup Internet connectivity option for all their remote sites.
Today, i3D.net uses the Opengear Resilience Gateway as a fall-back, providing out-of-band management, as and when required, to its routers all over the world. It has achieved a raft of benefits as a result. For example, it has helped to significantly streamline the process of bringing new sites online.
In the past, when i3D.net had to carry out a deployment outside the Netherlands, a technical operations engineer flew to the location and installed the new switches but would typically struggle to configure them. Today, i3D.net simply connects the Opengear Resilience Gateways to these network switches and its network operations engineers back in the Netherlands are then able to configure the whole site remotely via the Opengear device. “This is very powerful and a huge benefit for us in terms of the time and costs saved not only in provisioning but also ongoing maintenance” commented Sloot.
“We were looking for a partner that could deliver triple-A equipment on time all over the globe and knew we could rely on Opengear for this.”
Using Opengear Resilience Gateways also allows i3D.net to stay one step ahead of any pending network failures. According to Sloot: “We don’t like downtime and neither do our customers. Today, if there is an issue, we can directly connect to our routers anywhere in the world without having to wait for someone at the locations to access and fix it. The agility of the Opengear solutions means we can quickly help our customers and maintain our high standards as a performance hosting company.”
Today, i3D.net continues to expand and open global locations dynamically. It is also in the process of switching to a new Juniper Networks router platform. As Sloot explains: “This will give us the opportunity to launch our Global Low-latency Anti-DDoS solution (GLAD). Every region where we are changing the router and do not have an Opengear Resilience Gateway will now receive one.
“Finally, we have an enterprise customer for which we are remotely managing servers, running routers and firewalls in their own environment. We are planning to place the Opengear solution there. It marks the first time we have carried out a deployment of Opengear’s Resilience Gateway outside our own data centres.”
The rapidly expanding use of Opengear at i3D.net bears witness to the broad benefits it brings the hosting provider and also highlights a partnership that goes from strength to strength today and will look to evolve further in the future. i3D.net is on a fast growth path and Opengear continues to support its dynamic expansion worldwide.
Hosting Company
i3D.net is a leading provider of high-performance hosting and global infrastructure services. The company notably excels in-game hosting and infrastructure, serving 100 million gamers daily for game publishers and independent developers on consoles, PC and mobile. The i3D.net network is one of the world’s top-10 most connected hosting provider networks. i3D.net is now a Ubisoft company, having been bought by the French publisher in 2018.
Transcript: Living on the Edge Podcast with TJ Gibson of PayPal
Steve Cummins (00:00):
TJ Gibson is the director of site operations at PayPal. In his 14 years there, TJ has worked in a number of technical roles and currently runs PayPal’s network command center. So TJ, I saw one of your LinkedIn posts that you said a big part of your job is talking. So I guess podcasts is a natural outlet for you. Thanks for talking to us on the Living on the Edge podcast.
TJ Gibson (00:27):
Absolutely. Thank you so much for having me.
Steve Cummins (00:30):
Great. So first thing is, there’s a lot of things I think we can dig into. Site reliability engineering, I think, is something that you’re deep into. And I really want to get into that, but before we dive in, just give me a rundown of how the IT organization is set up at PayPal.
TJ Gibson (00:49):
I mean, as anybody probably imagined PayPal is a very large organization, but my organization, specifically the organization I belong to, I should say, is called site reliability and cloud engineering. And it includes everything you would traditionally see in sort of a legacy infrastructure and operations organization, as well as some platform measurement capabilities. My specific organization there is sort of incident response, network operations, site reliability monitoring and alerting and response. And then we also have a function called embedded SRE. That really is maybe the most closely thing aligned to a pure SRE role within that organization. But it’s an organization of about 600 people today that it really has their fingers in everything. I think my boss likes to refer to it as the engine room of PayPal. It’s where all of the capabilities come from, that our products really leverage to deliver services to our customers.
Steve Cummins (01:43):
Well, that’s great. And a lot of us use PayPal a lot of the time and as with everything we’re never happy when it’s not working. So I guess it’s thanks to you and the other 600 folks in your group that means we’re never sitting there and cursing at the PayPal app. So, let’s talk about this site reliability engineering. And I think, Google are often credited with being the ones that really did a lot of the early work and launched the idea of it, but I’m sure it means different things in different organizations. So what does SRE mean at PayPal and how has that changed over the last few years?
TJ Gibson (02:21):
That’s a big question. I would say that SRE at PayPal really started before SRE was really a term that we were throwing around at the industry. I think in a lot of ways, the early days of it was almost looked down upon by traditional technologists, right? They looked at it like a bug fixing team or like somebody that was really just coming in to clean up some messes somewhere and as Google kind of put their thoughts into words and started to really define this as a niche within the industry is when we said, «Hey, there’s so much overlap with what we’re already doing. Let’s look at this framework, let’s look at this way that they’re defining this role and find out how do we squeeze the maximum amount of value out of it.» And we’ve really seen it grow from that point forward.
TJ Gibson (03:02):
Again, I think the way it’s structured today has really grown beyond what traditional or pure SRE is. But we’ve really seen this conglomeration, I guess, or this aggregation of some of the things that PayPal needed, specifically. Some of the things in the industry that were really coming along in terms of technology and resiliency, and then bringing this framework of site reliability engineering as a practice, and really kind of put those things together to deliver what we’re delivering today. And really when I think about our mission, it’s not just providing these capabilities from an infrastructure or a platform perspective, but it’s really ensuring that all of the products that we deliver and all of the capabilities that we expose to customers have this reliability, this resiliency, this fault tolerance, this usability kind of baked into it from the beginning. Much like, I think, the industry 10 or 15 years ago came around to the idea of information security and really that became a niche practice and still is a niche practice, but it’s much more just a core element of a lot of products and services that people are delivering today.
Steve Cummins (04:07):
So do you think that that’s one of the defining things of SRE? Is this idea that it’s a holistic approach. It’s not looking at one piece of the operation or one location, but really looking at the whole networking ecosystem and make sure it’s covered sort of from A through Z.
TJ Gibson (04:25):
Yeah. I think the way that we tend to talk about it really is from a customer perspective and everything that a customer would expect from our products and services operate the way that they intended to be. I don’t think that it’s a niche in the way that we would think about security or networking or application development. But I do think that it’s niche in the way that it brings the sort of core skillsets of each of those, or at least the core awareness of each of those domains and brings it under the umbrella and provides this lens of resiliency and operability and scalability kind of to the way that these things come together. I don’t know if that was helpful or if that was a little bit too ambiguous.
Steve Cummins (05:03):
No, that makes sense. You mentioned an embedded SRE function and I’m curious, are you talking about that you have folks that are sort of out in the various operating units that are reporting back into your group or is that something else?
TJ Gibson (05:21):
No, I mean, it’s essentially that it is a more of a dotted line from the SRE organization out to these various domains. I think a lot of places when they implement SRE as a specific discipline will look for filling a role within an engineering team. A lot of times when you talk about an agile team, you’ll have your software developers, your product manager and so on. And I think a lot of organizations will bring that site reliability at that level. The way that PayPal has really approached this is more from a centralized aspect, but building these dotted lines between dedicated teams who have the awareness, not just of the domain that they’re embedded but the business that they’re embedded with.
TJ Gibson (05:58):
So within our payments organization or within our identity organization and really understanding how their products and their platforms are put together and bringing that SRE discipline or that SRE mindset to help them holistically figure out how do we make this thing the best it can be? How do we make it a great citizen within the application, so that as those handoffs upstream and downstream are happening, that all of this reliability and operability components are being accounted for.
Steve Cummins (06:24):
Got it. That makes a lot of sense. So in terms of security, which you mentioned is sort of a similar evolution, is security interwoven with SRE or do you still see that as a separate entity?
TJ Gibson (06:37):
I don’t know that I would say it’s woven into SRE. I think it’s become more a part of everything we do. So yes, it’s a part of SRE, but it’s not something that SRE is bringing to the table. It’s something that we hold ourselves accountable to. It’s something that our customers and our business holders hold us accountable for, but really it comes down to the security aspect of it being interwoven to everything we do. if I could kind of project this forward in terms of SRE, I think where I see this progressing in the coming years is that we start to see a lot of these core things today that we would say belong to site reliability engineering. We will see them adopted and integrated with the ways people work when they’re developing products, or when they’re deploying a new network segments or new data centers.
TJ Gibson (07:26):
I think it will become more just a sort of foundational component of how we do IT, the same way that security has over time. I think we’re in a little bit of a transition period here where SRE has kind of come into its own, it’s become a mature role or discipline within the industry. And I think that will stay true as we go forward. But I think some of those things that SRE are bringing today will become part of core platforms and become part of the core workflows as opposed to always being a centralized single accountability kind of function.
Steve Cummins (07:57):
And I guess even the role that you have, although I focused in on this part of your job, which is the SRE, I mean, you really have responsibility for the overall site operations. Right. And SRE is a part of that. So, I guess it emphasizes that idea that SRE is a part of everything that you do rather than being looked at as a standalone piece of the business.
TJ Gibson (08:18):
Sure. And I think in a lot of ways before we actually started recording here, we talked a little bit about DevOps and SRE and what are these things mean and how do they play together? And when I think about my role specifically in that operation center, there’s definitely a push to a more federated accountability model. Right. And so it kind of speaks to DevOps, but I don’t know that either one of these things could really exist independently. I think they’re flavors of the same kind of thing. Right. It’s the way that we work and the way that we provide services and capabilities to our customers and making sure that we have the right mechanisms or the right levers in place to be able to respond when we need to, when something’s broken, but also to be able to plan and design for failure and to be able to plan a design for customer outcomes.
Steve Cummins (09:05):
And I think, particularly in a big organization like yourselves with many moving parts, it makes sense that it covers across it. Do you see any drivers over the next few years that’s going to change the way that SRE is implemented or you think it’s just a continuing evolution?
TJ Gibson (09:25):
I certainly think that there’s always going to be changes. I think, as we see more and more large enterprises really grab a hold of this idea of public cloud and really start to move workloads, I think I saw recently that Capital One essentially was declaring victory in their cloud journey. I think that brings an entirely new perspective on sort of large scale applications and site reliability engineering. But I also think going forward there’re things that I think we haven’t quite yet accounted for. I think machine learning and artificial intelligence is going to bring aspects to our technology stacks that we just don’t have an idea of what that means to be resilient and what that means to be operable. And how it is that we use that sort of technology most effectively, I think SRE has a role to play there as well.
TJ Gibson (10:14):
I think when we look at some of the things that SRE is bringing to the table today, in terms of frameworks and structure and accountability, I think a lot of that we will start to see baked into our applications. I think there’ll become more natural for people as we’re going forward. And so that’s going to, I think, create an opportunity for SRE to, again, maybe step up and up level their viewpoints, similar to maybe how information security practitioners have been able to up-level and make a tighter connection to the policy or the regulatory obligations. I think SREs will be able to continue to look for those opportunities to step up and bring sort of business aspects further down into the stack. And I know that’s a little bit fluffy, so I’m happy to kind of call out some context there if we need to. Okay.
Steve Cummins (11:00):
Hey, I think when you’re talking about trends for the future, if you’re not being fluffy, then you’re fooling yourself. Right. Because who knows what’s coming up, but I think some of the trends you talk about, machine learning and AI, we all know it’s going to have a big impact. And I think it’s just that question mark of how quickly and in what ways, but it’s a fair point, SRE is going to have to adapt to as everything else is along with that. So just shifting gears a little bit. So I’m always curious about career paths and how people get to where they are. So I know you started out in the Air Force, you worked in a couple of networking roles in some other companies, and now you’ve been at PayPal for a number of years, and you’ve talked to a lot of technologists in that time, as you mentioned to me, and you have this idea of how the three phases in a technical career. So maybe you could just sort of talk through that a little bit.
TJ Gibson (11:57):
Yeah. And I wouldn’t say that it’s really just limited to the technology career field, but that’s obviously where my experience is. And the way I try to talk to my folks about it is, and I think early on in your career, the value that the business perceives from you as an individual, from you as a technologist, really boils down to how much you know. Like what do you know? And I think the deeper you know a particular technology or domain area, the more technology or domain areas you know, the more your value is to the organization. I think you hit a point probably somewhere in that five to 10 year point in your career. And it’s obviously very different for everybody, but at some point if you could call it a journeyman, I think you hit a place where it becomes about who you know. It’s not possible to know the depth that you have to, it’s not possible to have the breadth that you need, in order to deliver the outcomes you’re being asked to deliver.
TJ Gibson (12:50):
And so the way that you provide value to the organization becomes much more about how do you bring people together? How do you find the right answers and the right resources within the organization? And I think for the most part, technologists do this pretty well. I think it makes sense. I think it’s a pretty natural evolution. I think where a lot of people tend to get tripped up is in that kind of third step, but it becomes more about who knows you. And yeah, there’s an element of politics there, every place no matter how apolitical they claim to be, there’s always politics, but I don’t think that’s the core of it. I think it’s really about demonstrating value. And so couple of things I try to tell my people are, what is it that keeps your boss up at night?
TJ Gibson (13:28):
Those are the things that you should be looking for solutions for. Those are the things that should also be keeping you up at night. You need to be looking for, how do you bring your talents, your experience, your responsibilities to bear on that problem statement, to help solve it. And I think that’s where a lot of people tend to struggle with wrapping their heads around it. It’s not that it’s incredibly difficult to do, but it’s certainly a mind shift. And so when I say, who knows you it’s about who sees you as the problem solver who sees you as the one that’s going to bring your experiences and skillsets to bear on my problem and understand the context quickly, get to relevance quickly and be able to help me find solutions that technology will solve for my particular business problem. And of course, we’re doing all of these things throughout our career.
TJ Gibson (14:12):
But when I think where I see people really kind of get stuck in a rut, that 10, 12, 15 year mark in their career, is because they’re struggling, maybe I shouldn’t say they’re struggling, where they really start to have challenges is when they start to get uncomfortable. When they start to have problems solving business problems or connecting with their peers that are stakeholders. They tend to fall back on the things that have worked for them in the past. And so they might go out and say, I’ve been in the career for 10 years or 15 years, the technology has moved so fast and I’ve been thinking at this higher level, my problem is I need to go learn the latest and greatest.
TJ Gibson (14:49):
I need to go dive in deeper into whatever technology stack is in front of me. And that is always valuable, but that’s not the only answer. I think what the business is looking for people, technologists specifically, as they reach that higher level in their career is more of a connection back to how technology solves those business problems. Where very early in your career, it’s very much about how to implement a particular solution that’s been given to you or that has been defined.
Steve Cummins (15:16):
Yeah, it’s very true. We tend to focus on the technical skills, right. But I think as you move up in an organization and you sort of broaden your influence, it really does come down to being known, knowing how to get things done. I mean, for me personally, this always flags up. When I move companies, as I’ve done in the past and you’re in a new job, and you realize at that point, how important it was to you in your previous role, because you knew everybody and you knew how to get things done and people knew who you were and if you change companies or organizations, you often have to relearn that. So I think the three phrases you describe make a lot of sense.
TJ Gibson (15:57):
Yeah. And I think the important thing to call out is that it’s not that you’re leaving the technology career path, right? It’s not the technology becomes secondary to the value that you’re bringing to the organization. It’s about being able to layer on that relationship skill set, that business understanding skillset, that translation skillset in a lot of cases to get from business outcomes into technology solutions. And so it’s not about leaving technology behind or ignoring it or not being hands on keyboard anymore. It’s about being able to make stronger connections back to the organization, back to where your customers are and back to where the value that you’re delivering to them is coming from.
Steve Cummins (16:37):
Yeah, for sure. I think that all makes sense. So having just said how important it is to look at the non-technical side of it, I am curious about the technical training side of things. So in terms of that sort of technical… there’s a number of certification path, like I guess CCIE is sort of the most well-known one. And recently Cisco added a DevOps section to it or DevNet section to it. I’m not aware of anything similar that focuses on this SRE or a liability or even site operations part of it. Is there something out there that a certification path that people would follow or is that something that you think needs to be worked on?
TJ Gibson (17:25):
There are some folks that are offering different certification paths for SRE, but I haven’t seen anything really rise to the top as kind of an industry standard. And I think if I look at kind of how the industry is handling the learning and the training around SRE, USENIX has every year a conference they call SREcon. And I think if you look at some of the agendas or the seminar schedules, excuse me, schedules for these events, these conferences, you’ll see the diversity there, right? You can have some very deep tracks on how to use machine learning to bring better insights from your observability platform. Some very deep tracks about how do we build networks for resilience and how do we handle this massive global scale? How do we deal with hybrid cloud?
TJ Gibson (18:10):
But you’ll also hear a lot of higher level things. How do we do problem management? How do we do root cause analysis? How do we actually understand where our customers are coming from? How do we determine business logic failures differently from systemic technology types of failures? So I think in a lot of ways, the industry is still trying to figure out what SRE looks like. I think there’s still a lot of definition of discovery that’s happening. It’s much more solid than it was in years past. But I think, again, even if you look at some of those conference schedules year over year, you will see kind of the focusing or the narrowing down of their scope or the solidifying of the way that they’re viewing that particular industry career field. So I think when it comes to SRE, if that was of interest to some particular technologists, I don’t know that I could point you in a particular direction.
TJ Gibson (18:58):
I think what I would say is breadth at this point in time, I think is extremely important. I don’t think it’s enough to have only an application development background or only a networking background and be able to step cleanly into an SRE career field and be successful day one. I think you really have to understand all of the sort of constituent parts and how they play together towards the business outcome of reliability, of operability, of performance. And so I think what I would suggest is go get some breadth, right? And this is happening a lot in our data centers already. We’re already looking at how do we build our on-prem data centers to be more abstracted, to have the hardware, the physical infrastructure more abstracted for the users, very similar to how you would see an application deployed in AWS.
TJ Gibson (19:46):
So I think if you can kind of start to spread out a little bit and start to look for how can I bring in some of these elements of development and some of these elements of business awareness and some of these elements of observability and response. I think if you can bring those things into your purview and into your skillset, that’s where you really start to have kind of the building blocks for being a good site reliability engineer. And I think the questions that I would be asking for, the opportunities I would be looking for internally, is how to leverage all of these things together to deliver reliability as a service or operability as a service or observability as a service kind of thing. I know that’s very fluffy, but again, I think the industry is, at least this particular career field, is still in a bit of early stages.
Steve Cummins (20:33):
I actually, I think it’s spot on. The pendulum the last couple of years seems to have swung back a little bit. Everybody wanted to be a specialist in something right. Wanted to be very niche-y. And the word guru and ninja was used far too often for people. But it does seem as though the pendulum has swung back. And there’s an understanding of the value of a little more of a generalist approach where you can bring in parts of different skillsets or different backgrounds to combine them and get something done. And I think the world of SRE is probably a perfect situation when that make sense.
TJ Gibson (21:13):
Yeah. And I don’t know if I would go so far as to say SRE is really a generalist career field, but I do think that having that generalized background that’s really the context you need to be successful in SRE. It’s not enough to understand how to build the most reliable, most resilient most scalable network, if the application on top of it doesn’t know how to consume and use that benefit. And so really understanding at least at a base level, how all of these things come together and where the strengths of one particular domain compensates for the weaknesses in another domain. So I think you have to have that generalist mindset, even though SRE, I think, is a very specialized skill set or a very specialized role. So maybe in a way it’s kind of a mix of both of those worlds and it survives those massive pendulum swings over time. Again, maybe the correlation back to information security.
Steve Cummins (22:05):
There’s a really interesting book called Range, I can’t remember who wrote it, but I’ll put it in the show notes, that talks about exactly this. And the point is get a broad background first because it’ll inform on anything you do in the future. And it will also let which is the area that you really should focus on, whether it’s something you good at or something you have an interesting or a passion for. As opposed to specializing too soon and then realizing you can’t broaden out. So it’s an interesting book I think might be worth digging further into that topic. So you mentioned resilience a couple of times and an open gear that’s really our focus is this idea of network resilience. So as someone who really spends their days, I would imagine thinking about the reliability of the operations, what does the phrase network resilience mean to you?
TJ Gibson (22:57):
I think to me, it really means that we’re capable of surviving faults and that’s an overly simplistic answer, but really it comes down to our ability to meet the business needs to meet the customer expectations in a way that is efficient and effective and allows us to continue to innovate. And then there’s a lot wrapped in that. But really to me, that’s really kind of what it comes down to is just being able to respond and react and absorb and grow to what we need. I think sort of the anti-pattern there would be a very specialized I think about the olden days of mainframes and the AS400 sitting in the basement that I’m sure a lot of banks still have sitting around, you build this thing that does one thing so well that it really can’t evolve and grow with the needs of the business or with technology as they continue. So to me, it comes down to flexibility maybe is the best way that I can describe it.
Steve Cummins (23:55):
I think you may get the award for the shortest definition of network resilience we’ve had on this podcast, surviving fault. I mean, it says it all. I like it. Makes a lot of sense.
TJ Gibson (24:06):
Like I said, maybe overly simplistic and not what you were driving at. Maybe that’ll drop my stock value a little bit.
Steve Cummins (24:12):
No, because then like old good interview guests, you then went into an explanation of what you meant by those three words. So that’s spot on. I like that.
I’ve got to believe, in your role, you have a hundred “oh crap” stories where stuff went wrong and it’s kind of why this podcast called Living on the Edge. Right. Because everybody has those stories of living on the edge. So do you have one that you’d like to share with us?
TJ Gibson (24:40):
Yeah. And I’ll set it up a little bit. I was actually putting quite a lot of thought of this. And like you said, there are many, many, many examples. I had some advice early on in my career when I was doing some consulting and I was very nervous about standing up in front of a boardroom full of executives and trying to tell them where their vulnerabilities are. Not so much that I didn’t know or it didn’t have the data to back up what I was saying, but I just had no idea where the questions were going to come from what the angles and the agendas were of the people in the room. And so I was really kind of stressing out in this big readout that we were going to do. And one of my colleagues sat down and he said, «Look, consulting is like 10% technical and like 90% people.»
TJ Gibson (25:21):
The more I’ve kind of internalized that and I’ve grown in my career, I feel that’s pretty broadly true for technology in general. So a lot of my sort of oh oh moments or my hairy late nights really come down to people things kind of mixed with technology. So the one that comes to mind and maybe I’ll get a little bit of trouble for even sharing this story, but was actually in the Air Force. I had my boss at the time had pushed a configuration file to every machine on our network, relatively small network, but very geographically distributed. So 180 nodes or so spread across 40 countries. That file set the same IP address on every device on the network. And this is in the mid ’90s, so a lot of our automation that we take for granted today just wasn’t there. It just did not exist.
TJ Gibson (26:08):
And so spending three days trying to talk pilots and people loading airplanes all over the world into how to change the IP address to something that matched, what was, we’re going to work on their local network. That to me was one of the hairiest moments, not just because it was so complex and so people focused to resolve, but really just highlighted how fragile sometimes some of these things we take for granted really are a one simple human mistake and essentially shut this thing down for three days. And I think maybe the reason I bring that story out is just the irony at the end, that may resonate with a lot of your people, but my boss got a medal for fixing that problem. So I think that was kind of my funny story I wanted to bring is, we made a mistake we spent a whole lot of time trying to correct for it. And ultimately as the cause of that mistake somebody was rewarded.
Steve Cummins (27:00):
Yeah. Unfortunately that’s how life works. They forget the mistake, but just remember, oh, yeah. But hey, everybody bounced back. And let’s be honest, I guess, that’s also what we get paid for. Right? It’s not about mistakes happening. It’s about how you react to it, but it seems a shame you didn’t get the medal instead of your boss.
TJ Gibson (27:19):
Well, look, he was definitely involved as much as I was in putting things back together. And I think in a lot of ways I’ve carried that with me. I think for a long time, there was a little bit of bitterness. Darn it, you caused a problem and we had to go jump through hoops to fix it. And you got a pat on the back, but in so many ways that leads to where we are today, right? This blameless culture that we’re looking to learn, looking at every failure as an opportunity to get better. I think that maybe was an early step in that direction. Maybe I didn’t appreciate it for what it was in that moment, but there’s so much value in being able to fail miserably. And even because we screwed up, but when you can recover from that and you can learn what happens and you can look for opportunities to build automation or gates or controls that would prevent that mistake from ever happening again, that’s a win.
TJ Gibson (28:08):
The fact that we found it when we weren’t getting shot at is a tremendously great thing. The fact that my site went down last week or maybe even we took it down last week at 3:00 AM. That’s amazing because it didn’t happen during the holidays when we weren’t expecting it, we actually pushed the button. We knew exactly what happened. We’re able to fix it quickly, that kind of stuff. So I think there’s even some lessons in that kind of on the edge moment that I think we’re putting into practice 15, 20 years later in the industry and it’s making us better.
Steve Cummins (28:36):
Well, and it does put it in perspective a little bit. Right. We all think that our problems are big, but you just used that phrase. «Well, at least it didn’t happen when we were getting shot at,» which is not normally what people have to worry about. So yeah, I guess there was a life lesson than that for yah.
TJ Gibson (28:55):
A little bit, yeah.
Steve Cummins (28:55):
So talking about life lessons, I always like to give people a chance to give a shout out to somebody that’s helped them in their career. Somebody who’s influenced them or been a mentor. So anybody you’d like to recognize?
TJ Gibson (29:09):
I’ve actually got two, if that’s okay and I’ll try to be quick.
Steve Cummins (29:12):
Yeah. Go for it.
TJ Gibson (29:14):
The first one was my boss in my first people management role, it was a role that came to me through a little bit of duress, a little bit of emergency and a need in a crisis. And I had told them, I don’t know about people management. I don’t know if this is the right step for me. I don’t know if I’m going to like it. I don’t know if I’m going to be good at it, but I’m going to give it a shot. Just promise me that if I suck or I don’t like it, I can have my old job back. Yeah, yeah, sure. No problem. So I took this job and about six months into it, I went into my boss’s office frustrated and I just couldn’t, there was too much, I just couldn’t get it all done.
TJ Gibson (29:47):
And he sat me down and he gave me some of the best career advice I’ve ever had. Overly simplified, but it means a lot to me. And he said, «I’m not paying you to get to the bottom of the pile. I’m paying you to put the right things on the top.» That’s what management’s about. And so that idea that the value you see from me is in how I make decisions and how I prioritize and how I respond. That’s the value I’m driving, not in my task list, not in getting to that bottom of that pile. That’s been some of the best career advice I’ve ever got. The man’s name is Roy Santarell. And I hope at some point he gets to listen to this and know how much that’s meant to me. But I think secondly and maybe this will sound or feel a little brown nosy, but it’s actually my current boss.
TJ Gibson (30:26):
I’ve known the guy for 10, 11 years now. He’s been a cheerleader from the sidelines forever. He’s been in my direct reporting line for about the past four or five years now, but he’s really just showed me what it means to… just boundless energy. He showed me what it means to care for people. He showed me what it means to set hard expectations and challenge people, to give them the autonomy they need to be successful. Give them the support they need when they fallen down or when they’ve hit a challenge or a roadblock. And he does all of this in a way that’s productive. It’s not ever in a way that’s letting me get off the hook, right. There’s still strong accountability there. There’s still high expectations, but it’s done in this supportive way where I know that he’s got my back.
TJ Gibson (31:16):
And I think for any leader, that’s really the most important thing I could ever ask for from a boss. And I think any technologist should ask for, from a boss. Give me the autonomy to do hard things, give me the challenges that are going to help me get better and stronger and grow. But give me the support. Don’t throw me out there to fail. Don’t throw me under the bus and all of that. So [inaudible 00:31:38] as my current boss and longtime mentor is another one that I would tip my hat to.
Steve Cummins (31:45):
So both Roy and West will be very happy to hear that. Last question for you because people are always trying to work out where do I go look to learn more about whether it’s general networking or cyber liability, whatever it might be, any resources, any websites, podcasts that you would recommend?
TJ Gibson (32:05):
I wouldn’t even be able to point to one or even a handful, but what I will say is where I get the most value is from very obscure sources. And what I mean by that, the Freakonomics book had an impact on me. And it wasn’t in the things that they were talking about. It was in the way that they viewed a problem and the way that they went about finding solutions or finding insights from that problem. I think anything you can do to look for ways to think differently, to think broader, to bring a different perspective to your work. I think that’s where you really get the most value, the bits and bytes, the particular configurations, the new technologies, that stuff will just continue to evolve and change. If you have the framework, if you have the baseline, you can continue to stay on top of those things just naturally, at least in your role as those things come to you.
TJ Gibson (32:59):
But I think that unique perspective or that insight that you get from looking at things from a different angle, I think that that’s rarely where I see people shine and succeed. And if I could make just a quick plug, I would say, this is one of the big reasons why diversity is so important in the technology career field. Just because I think for too long, we’ve had a bit of a myopic view. We’ve only had a limited number of perspectives brought onto our problems. And that’s resulted in a lot of things that we probably take for granted today that in my mind are actually fundamental flaws in the way that the internet works.
TJ Gibson (33:32):
You could look at DNS, you could look at IP space, there’s a lot there that constrain us going forward and we don’t even realize it. So I think that would be my call to action, or that would be my suggestion is let the technology specific pieces just come to you as they need to, as you bump into them, as colleagues mentioned them, as you learn about them in conferences, but where you’re going to really find your success is being able to bring a unique perspective. Either from conversations with other people or from completely obscure sources that really don’t have a ton to do with technology. And that’s the best way I can non-answer your question. I hope that’s okay.
Steve Cummins (34:09):
Yeah. You should be a politician, not answering questions. I actually, I agree. And I have to say Freakonomics is one of my favorite books. I don’t know how long it is since I’ve first read it, but this idea of unintended consequences is something that often rattled around in my head of, «Okay, we’re going to do this, but what is going to happen that we never planned for. Right. And how do you be ready for it?» So, yeah, I agree with you, you have to step outside of reading just the regular stuff and bringing some broader perspectives to it. So, I’m not even going to say it’s a non-answer, I think it’s a really good answer. So I like that.
TJ Gibson (34:47):
For what it’s worth, that mindset of unintended consequences or a mindset, again I come from a security background as well, so that mindset from security of always thinking like the hacker, that’s what makes a good site reliability engineer is how is this going to bite us? How can this be better? Where are the flaws and the weaknesses in this whole flow? So, yes. I mean whether you get that from Freakonomics or you get that from colleagues, or you get that from other experiences, I don’t know that you get that exactly from industry training or industry technology learnings that are pretty typical.
Steve Cummins (35:22):
Yeah. So on that and in terms of diversity, would you hire a former hacker?
TJ Gibson (35:31):
I think it would depend upon the way in which they were communicating to me. I maybe have this flaw for a technologist that I really try to get inside the mind of people, where are they coming from? What’s driving them, what are their motivations? But I think your background really has nothing to do with who you are today. We all change. I mean, if we were held to account for things that we did when we were 17, Steve, I’m not sure how old you are, but if we had social media when we were in high school, you probably are second guessing a lot of decisions you made. So I think your background has very little to do with who you are today. And I would be looking more to assess the talent and the capabilities being brought by any particular person.
Steve Cummins (36:12):
Yeah. That’s a great point.
TJ Gibson (36:13):
Them being an ex hacker has nothing to do with it in my mind.
Steve Cummins (36:15):
No. That’s fair. And it does fit nicely with your point about diversity, right? You have to bring a lot of different pathways to really get a broad understanding of things.
TJ Gibson (36:27):
Absolutely.
Steve Cummins (36:29):
I tell you TJ, this has taken us in a couple of directions I wasn’t expecting, which is always fun for me when we’re doing these interviews. I do really appreciate you taking time to share your thoughts with us on the podcast and look forward to talking again soon. Thanks very much.
TJ Gibson (36:46):
Thank you so much. It’s been a pleasure.
Transcript: Living on the Edge Podcast with Jason Gooley of Cisco
Steve Cummins:
Jason Gooley is a technical evangelist in Cisco’s Enterprise Networking Group. And he’s the author of several books from the Cisco Press. He’s also the host of the MetalDevOps Talk Show, which you can find on YouTube and he’s actively involved in helping network engineers through the CCIE and DevNet certifications. Today, we’ll be talking about his work with those Cisco programs and how he sees the industry changing with the growth of DevOps, SD-WAN and other technology shifts.
Jason, thanks for joining me on Living On The Edge.
Jason Gooley:
Hey, thank you so much for having me, Steve. I’m pretty excited about this.
Steve Cummins:
Good deal. So people probably know you best from a couple of your recent books on SD-WAN and DevNet as part of the CCIE program. But before we dive into that, tell me about the Enterprise Networking Group at Cisco and your role there.
Jason Gooley:
Certainly. So my role is kind of interesting and unique in that it changes quite frequently (laughs). So technical evangelists actually came up, which is kind of a cool title because you get to talk to folks. And that’s one of the biggest things I love to do is talk to folks about technology. And it depends on what type of technology, because it all, for me falls under the enterprise networking umbrella, I get to talk about things such as wireless, SD-WAN, software defined access, you name it. So that’s been really interesting. And then as well as my other part of my role is enablement. So we build all the content, training and curriculum for our field and our partners as well.
Steve Cummins:
So you get to talk about all the cool stuff while it’s still cool. (Laughs)
Jason Gooley:
Yes, yes. And some of it before it even becomes known (laughs).
Steve Cummins:
Even better! So you’re out there, telling people before they even know what they should be listening to.
Steve Cummins:
So this podcast is all about Network Engineers, Network Resilience, all of that good stuff. So I just couldn’t think of a more relevant topic than CCIE and the certification program, for people to hear a bit more about. So just let me know, how long have you been a CCIE and how did you first get involved in the training and the teaching side of things.
Jason Gooley:
Wow. Yeah, so, I got my CCIE originally, it would have been March 29th of 2013. So I got the routing and switching back then, coming up on eight years, which is crazy to me.
Steve Cummins:
Yeah.
Jason Gooley:
But I’ve been involved with networking and, and certifications even prior to getting that certification. And even before coming into Cisco I was actually working with Learning at Cisco and some of the certification teams to help design and develop these certifications, which was kinda neat. And then as far as teaching and everything like that, it just was kind of one of those natural things to me. I started studying for all these different things and as you study for it, I would implement it and use it in practice. And next thing you know, you find yourself kind of working in groups with other folks who are studying for similar certifications or want to know more about your experience.
Jason Gooley:
And I just started talking at different places and kind of helping and coaching and mentoring folks, and it kind of took off. And then somehow that unfolded into writing books and doing video shows and it’s been a real cool ride so far (laughs). Hopefully I got a lot left too (laughs).
Steve Cummins:
Let’s hope so. Yeah. So did doing that work sort of behind the scenes with CCIE, did that then lead you into your role at Cisco or, or did it happen the other way round?
Jason Gooley:
Actually, it was interesting because so I don’t know that it led me here because when I first took the role at Cisco, I was a systems engineer coming on, covering a certain group of customers. And essentially if they had any questions about what they wanted to look at for Cisco from architectural perspective or design, I was the one who would go in and help them with that. And talk about enterprise networking and routing and switching and wireless deployments and things of that nature and best practice design. And then all of a sudden I started really kind of getting more and more into the, what we called at the time IWAN or what would be the original version of Software Defined WAN for Cisco.
Jason Gooley:
And I ended up joining the worldwide adoption team for IWAN and started helping customers with that and kind of going around the country and global at that point helping folks with their IWAN deployments and explaining how to set things up and get it to work. That kind of organically turned me into a position that opened up for what would have been a solutions architect on the Worldwide Enterprise Networking Sales Team, which is a mouthful. But if you get through the title, then hopefully (laughs), hopefully you can follow me along. But it was one of those things where I joined this team. And then all of a sudden I’m working on some of the latest and greatest technologies, software defined access before it became software defined access and some of these things.
Jason Gooley:
And it was just really interesting because, you know, I have a passion for technology, and obviously I have a passion for working with folks and helping folks succeed. When you start putting those together, I kind of was fortunate enough to create my own role in that I would talk about these things, but then I would go, next thing, you know, I’m presenting at Cisco Live, or I got invited to write my first book for Cisco Press you know, for programmability and automation. And it just kept growing and growing and growing to the point where I, it’s sometimes, it makes me wonder, like, how did I even get here myself?
Jason Gooley:
Because I turn around like, wait, I wrote how many books, like, wait, what am I doing? So it was kind of like an inorganic and organic thing at the same time, I guess. And then I was encouraged by a lot of folks along the way who are just fantastic people. Who’ve encouraged me to keep going and, and see where it leads me. And here I am talking to you today.
Steve Cummins:
You know, it’s funny. It’s one of those things that eight years ago, when you sat for the CCIE, you got that done. If someone had said to you, «Hey, you know, eight years from now, people will be reading your books and that’ll be helping them prepare you.» You would never have believed it. Right. So it’s-
Jason Gooley:
Not at all (laughs).
Steve Cummins:
-It’s always interesting how journeys sort of unfold.
Jason Gooley:
I never thought I’d write a book, to be honest with you. And I knew, I always had this dream that one day I was going to get my CCIE and I was going to go work for Cisco. And, when I did, it was, it was funny. I passed, March 29th in 2013. And by July 8th, I was working at Cisco.
Steve Cummins:
Wow.
Jason Gooley:
And it was a whirlwind. It was something that I always kind of hoped would happen, but never expected to happen that quick. And at that scale. So keep with it, if you’re studying for certification, if you’re going for anything like that, just buckle down and keep going, because there is potential and positives to that journey.
Steve Cummins:
Yeah. I like that. It was one of the things I noticed on LinkedIn you’re forever encouraging people when they’re posting about their experiences studying for CCIE, or they’ve just passed, just got the certificate, I see you on there all the time, congratulating people, encouraging people. So that’s obviously something that other people did for you as you started out and you’re paying it forward.
Jason Gooley:
Well, I mean, exactly. And that was one of the things that I thought was so awesome about that journey was that back when we were doing it, there wasn’t a lot of videos and things out that were available and you pretty much, you have to rely on other folks to help you. And it was really interesting because, you know, I recall going back and saying, «Man, I wish I had somebody who can help me through this, or just show me this one thing.» And I’m trying to make sure that it’s evident that that is available now and that there should be no reason to not get started or not try to pursue your dreams or do anything like that because there is so much help available nowadays.
Jason Gooley:
And although not everybody lends a hand like that in my mind, that’s part of the reason I’m here. So I it’s the only, it’s the least I can do, I guess, is what I’m saying.
Steve Cummins:
That’s great. And it is important cause it’s pretty daunting when you first start along these paths. So to know the people who’ve been there and that are there to help you out. And I know there’s a couple of other podcasts that are really focused on network engineers helping other network engineers. So there’s definitely some resources out there. But I would encourage people to follow you on LinkedIn and help tap into some of those things. So that sort of leads us into this idea of how certifications have changed. So I guess it was about a year ago, there was some pretty big changes with the CCIE, CCNA and which obviously you were involved in and very aware of. So maybe you could just talk us through what those changes were. What you think drove them. And I guess how you’ve seen that work out over the last year.
Jason Gooley:
Oh yeah, certainly. So February 24th is what we like to call Cert-pocalypse is what the date was last year. And that was the date that everything changed from a Cisco certification perspective. The addition to all the Dev Net certifications, as well as the revamping of the current certifications and some retiring of some certifications too, as well. And really what you see there is it’s the response from our customers. I mean, ultimately the industry and what our customers are going through every day, largely drive what happens from a certification perspective and what’s top of mind and things of that nature. And there were a lot of certifications that, you know, we’ve heard over the years that either were redundant or didn’t exactly target the specific technologies that we hope they would because so many things have evolved. So we retired some certifications and then we really reduced the amount of tracks and certification courses as well as actual certification exams that existed.
Jason Gooley:
And the effort to do that was to make it simpler for our customers. I mean, you may recall that we had, I think, I believe it was like seven or eight different CCNA tracks and CCNP and CCIE, and we’ve really brought that down significantly. I believe we’re at five now, maybe even six. And when you think about that, we went from a separate CCNA for every single track, which is like eight CCNAs down to, we have a CCNA, which is, is kind of interesting. That’s how it was when I very, very first started back in like 97 going for all these different things. There was just a CCNA, there was just a CCNP and it was just the CCIE, you know, and that was it.
Jason Gooley:
And there’s so much new technology out there that yes, you do have to have some sort of specialization or way to showcase that you do indeed know these new technologies. But at the same time, we have to make it simpler for you to get them, right? (Laughs). We have to make it easier for you to not only get the certifications, but also for the industry and our partners and our customers to know what it means when you have that specific certification. Because if I told you I was CCNA, you know, three years ago, you’d have been like in what?
Steve Cummins:
Right (laughs).
Jason Gooley:
If I told you I’m a CCNA nowadays, or I’m getting a new CCNA, you would kind of have a good idea that it was something around the latest and greatest networking technologies. And then some of the things such as intro to programmability and some of those more, I guess, dominating technologies such as Software Defined WAN and things of that nature.
Steve Cummins:
Yeah. And that’s actually nice. It’s rare that things get simpler in the tech world, right? So I give Cisco a lot of credit for trying to work out how to pull back from something that was sort of evolving in many directions. And then, you know, the other part you mentioned that was DevNet. And I think that was one of the big changes. So there’s obviously a lot of talk these days about NetOps and, and DevOps. And I know DevNet was one of the books that you wrote. So just your thoughts on that DevNet path, and if there’s any advice you have for people that are considering going in that direction.
Jason Gooley:
Yeah. And honestly, that’s probably one of the biggest questions that I get these days and, and the reason being, I think it’s so critical is that a lot of the folks starting off in certifications, if you’re going down to the networking path and someone throws at you will now you have to learn programmability or automation. Most of the time, the reaction to that is either A, that they’re kind of concerned that it’s a, it’s going to be a really daunting hill to climb and technology that they have not necessarily had a lot of experience with. As well as depending on what your job role is, you may be concerned that someone’s going to write a code and you won’t have a job the next day (laughs). So it really all boils down to you evolving in the room and keeping up the pace with what’s going on in our world, in our industry today.
Jason Gooley:
I mean, there’s so many devices out there. We’ve all heard of the IOT or Internet of Things and onboarding all these different devices and connected lights and fish tanks and all these other things. There’s a lot that goes to that. And when you, when you start thinking about all the different devices that are coming on, if you are a customer and you have, you know, 10,000 routers and 30, 40,000 switches and heaven forbid 50, 60,000 access points, it’s very difficult to go from a box by box perspective and hit all these different devices and manually configure things the way we used to do. And that’s only because it’s just so hard to scale. I mean, a key example is that I remember being at one of my last companies and, you know, we had, I mean, thousands of switches and we’re going through and we had to update the local username, the SNMP strings. And then we’re adding on a VLAN for a specific management function that we wanted.
Jason Gooley:
So you’re touching a VLAN. You’re adding in IP addresses for that. You have to add routing capabilities for all these different subnets that you’re adding, and you have to do it on a box by box basis. And depending on what location you’re doing, it in, the subnets all changed. And so you find yourself kind of trying to script it out, quote unquote, in an Excel spreadsheet or something like that, and copy and pasting configs all over the place. And for the time, that was pretty much the best we can do. I mean, we had some capability of EEM scripting or Embedded Event Manager scripting and things of that nature. But for the most part, you’re touching all these devices just to be able to get the config on there that you want.
Jason Gooley:
And if you didn’t have a good tool such as prime or something like that, to roll out those configs, you’re essentially still locking into all these devices by hand and pasting in all these configs and rolling back the configuration and, or backing up the configuration in case you have to roll it back, I should say. So it’s very time consuming. And I think that’s what, in my opinion, programmability and automation gives us is it gives us that time back as network engineers or network developers, to be able to say, «This is what I want my network to do for me and our business. And now I can focus more of my time and energy on realizing that value versus config T interface VLAN.» You know all of these good configuration. It’s still great valid configuration commands and things that we need and love, but it’s very difficult to do it on that many devices at scale.
Steve Cummins:
Yeah. And I think that’s one of the interesting things is coming from this automation. Yeah, there’s always that fear that you’re automating yourself out of a job, but actually reality is there’s not enough network engineers to go around. So it really is about how do you do what you’re doing more efficiently so that you can do more of it. And I think the other interesting thing that’s happened in the last year, not that I want to talk about the pandemic as being interesting, but the inability of people to travel to sites, right? So that automation now has made it much easier for people to do it remotely, or at least, you know, do it at home base and then ship something out and, and plug it in. So it seems as though it’s just gonna be a trend that keeps accelerating.
Jason Gooley:
I mean, even the prevalence of video interactive video for some of that, I mean, we’re starting to see customers deploy some of our cameras and different things in some of these locations or MDFs and data centers. And one person might be able to go into that data center because of you, especially because you mentioned the pandemic and not having that many folks be able to be on staff and in the facility at the same time, that can be walked through remotely via remote expert capabilities or something of that nature that traditionally, we, we really didn’t think of that as a use case. But now if you need to inspect something remotely or you need to showcase something or show somebody how to do something from an upgrade perspective or move a cable, you can literally call it out visually now, which is something that I think you’re starting to see a lot more of.
Steve Cummins:
Yeah. That’s a good point. I hadn’t thought of that. You know, we’re so used to using Zoom or Webex or whatever it may be for day-to-day conversations, no reason you shouldn’t turn that camera around and, and be sort of talking somebody through it. So, yeah, it’s an interesting point. It just goes to show whatever you think is going to change something, something different will happen. There’ll be another trend coming along.
Jason Gooley:
Exactly.
Steve Cummins:
(Laughs) So you mentioned a term I haven’t heard it in a while the IWAN, which you know, predecessor to SD-WAN. I guess you could say that you wrote the book on SD-WAN, so-
Jason Gooley:
Yeah.
Steve Cummins:
Let’s talk about it. How do you see SD-WAN affecting the way that the companies go about their business?
Jason Gooley:
I think when you start thinking about SD-WAN and what we really want out of SD-WAN and why it’s a thing is that we as customers or even, you know, everybody out there who’s using a WAN, wants more control and granularity over what that WAN does. Most of the time, a lot of customers have finite resources when it comes to the amount of bandwidth you have, the applications are not necessarily getting smaller and they’re becoming more and more abundant where you have more and more applications that are driving the business. And that are critical to making sure that not only their customers are happy, but the business itself can still thrive and function. So protecting those applications becomes number one. And one of the things that I’ve said quite a few times is that the network is absolutely useless if there’s not anything to run on top of it.
Jason Gooley:
You know, so if you have an application, that is the reason that the network exists is to make sure that, that application to the best of its ability. And I think with SD-WAN, it gives us the ability to not only engineer the network in such a way that it is beneficial to the business based on their business objectives. But also it gives you a form of, you mentioned network resiliency, when we started the conversation, it gives you network resiliency, redundancy in such a way that we’ve never really had before, without quote unquote kind of tricking the system to make it seem like we had fully functional diverse, redundant WAN links.
Steve Cummins:
Yeah, for sure. So interesting that you brought up resilience. It is the name of the podcast. So that’s, that’s probably how we get down that alleyway every so often you know, with things like DevOps and NetOps and SD-WAN even the meaning of resilience has changed over the last couple of years. So what does the idea of network resilience mean to you?
Jason Gooley:
So that indeed has changed quite considerably before in your mind, you’re thinking as a network, as a hardware person, as a network engineer the network itself, the physical devices have to have resiliency and redundancy within the chassis, across chassis, across locations, all of this. But now we’re starting to think about it in a totally different way. And earlier I mentioned the phrase network developer. So not only do you have to know the networking, you know, you need to know how to software develop against it. That’s based on a lot of the DevNet certifications we’re talking about. And so now you’re looking at resiliency from software perspective from not only code, but the process or the procedures that you follow in your environment to execute something.
Jason Gooley:
If you want one location to upgrade and backup another one, or if it’s you know, your customers are relying on some sort of service that you’re providing over the network, you have to figure out ways to make sure that that essentially always stays up. And everybody’s always looking for the 5×9’s and software resiliency and programmatic resiliency is also pretty huge now.
Steve Cummins:
Yeah, that’s an interesting angle on it. You’re right. It was a few years ago. It was all about redundancy. Right, and then more stuff got pushed out to the edge. So then there’s other things you have to think about. You can’t have a second of everything out of the edge and now so much is software driven. You’re absolutely right. Resiliency sort of runs a thread all the way through that. So when we were talking actually before this discussion you mentioned about one of your previous roles, some experiences you had with out-of-band management. I’m just curious if you could talk through that a little bit.
Jason Gooley:
Yeah. And it was kind of interesting that one of my previous roles, we used some of the Cisco routers that we had as kind of like a backup device. And we would pull in something like a commodity internet link, whether it’s cable or DSL or whatever it may be. And essentially we started that off as our out-of-band network that’s completely separate from the MPLS network that we had. And what ended up happening is (laughs) we were kind of sitting there like, «Well, we have these two networks now, why don’t we just use it because it’s always on, we’re always paying for it. Why don’t we start using it?» So through a combination of a whole bunch of different scripts and, and funky router tricks and things of that nature, we, we managed to find ways to essentially reroute and route traffic over both of these networks.
Jason Gooley:
And we ended up using it as a, not only out-of-band management, but also a backup video network and a redundant network for the primary MPLS links. And what you come to find is that, you know, you’re really looking at the business and trying to figure out what is crucial to stay operational. And from that you devise some sort of a plan on, well, then from that, I need to take some sort of action on the technology to make it work. And I think if you start with that, it really gives you a good direction as what you want to do and how you want to configure the network to work. So we started doing that. And next thing, you know, we started dropping off secondary MPLS links in a lot of places and going to the hybrid approach with commodity cable internet, and as well as the primary MPLS.
Jason Gooley:
And essentially we were doing SD-WAN before it really existed, you know? And you still get the gain, a lot of the benefits out of that. But now it’s so much more advanced where you can really tune in and making sure that everything’s working the way it’s supposed to, policies and security and all these different things that you used to have to manually think of or install separate secondary firewalls or something of that nature to pro-, protect that, that new threat vector or the internet link that’s, that’s there now there wasn’t before. And it’s kind of interesting how things evolve because, you know, really when we look at how these different technologies have come about, it’s basically in response to some sort of issue or trouble that our customers are having.
Jason Gooley:
And we kind of adapt to that and make it happen, I guess (laughs).
Steve Cummins:
Yeah. It’s interesting the way you describe it, right? That you have a problem, you start tinkering around to solve it. And by the time you get to the end of it, it’s almost as though this new segment or this new technology has grown up around it. I mean, out-of-band itself is old, right. 20, 30 years. But there’s always new approaches to it. And SD-WAN sort of lead you into a different way of looking at it. So one of the reasons, just sort of switching gear a little bit, you know, so this podcast it’s called Living On The Edge which that might tie in nicely when we talk about your MetalDevOps later. But the point behind Living On The Edge is for network engineers.
Steve Cummins:
There’s always, you’re always waiting for that thing to go wrong, right? Where it hits the fan, you get that phone call middle of the night and you have to go fix it. So I would imagine from your varied background, you’ve had one (laughs) or two of those situations. Any interesting story you’d like to share with us?
Jason Gooley:
Actually, I have two that kinda come to mind and, you know, one of them at a previous company I was at, you know, this is back in the days of the 6500 EAS, you know, when we’re doing a lot of different technology there and in the different modules and stuff. And, you know, I remember a call center that we had going down. And essentially what it was, was that the configuration I wanted the supervisors did not sync over to the secondary supervisor, essentially rendering all the VLANS useless ‘cause the VLAN database didn’t sync over. So they didn’t exist. So none of the layer three interfaces for the VLANs came up and, you know, you’re trying to remotely troubleshoot that because it was in a different state and you’re sitting there trying to remotely troubleshoot it.
Jason Gooley:
And, you know, it’s one of those things that you don’t intentionally overlook, but when you find something like that, you look into it and say, «Well, obviously, there’s some steps or a knowledge base that we should probably create based on these different issues and things of that nature. So if we ever run into it, we can fix it right away.» But even that, I mean, you’re talking about an entire call center with thousands of agents, all of a sudden, just down immediately because of a hardware failure. But going back to what we were talking about in software resiliency, the software wasn’t properly set up to ensure that, when the hardware failed over to the secondary it would work. So, I mean, there’s all kinds of different aspects of that, that you have to think of.
Jason Gooley:
And I think that’s one probably huge use case that affected a previous company of mine. And then one I did myself with this one was just a couple of weeks ago. So one of the things that I get to do at Cisco now is I’m in charge of a lot of the, what we call test drive program, where we’re able to develop these different hands-on approaches for our customers to try out different technology. And I’m working on a wireless one currently today. And one of the things I did was I’m like, «You know, I got the whole lab up and working, everything’s good. Well, I need to start showcasing some HA and doing some high availability things. And I’ll just plug in the secondary controller.» And whoops! The entire thing disappeared (laughs). The whole thing died.
Jason Gooley:
And you know, it was just one of those things where I overlooked the priority, the (laughs) standby controller. And it overwrote the primary, very much like an old VTP adage that you heard for so many years that, you know, you want to make sure that (laughs), you set your priorities properly. Same thing applies for wireless too. Just so you know, and so that was a multi hour rebuild from scratch because it blew away all the configs on all the APs and all the other controller. And had to pretty much (laughs) start everything over again. And unfortunately even as, as somebody that, quote unquote, has done this quite a bit in my time I forgot to back up the config.
Steve Cummins:
Oh. (laughs).
Jason Gooley:
So (laughs) it was just one of those things that even, even at this level, some of the most rudimentary things are still crucial and important, just like out-of-band management is just as important today as it was 20 years ago. When you need it you need it. So (laughs) I guess that’s a couple of little ones that have come up top of mind anyway (laughs)
Steve Cummins:
Well, I give you credit for that because yeah. Here you are the man that “wrote the book” and is encouraging everybody (laughs). “Mr. Cisco”. And yet you’re still willing to admit that every so often you forget to back it up and, you know, things go wrong.
Jason Gooley:
You’ve got to learn from your mistakes. And that’s one of the things it’s like you know, it’s one thing to kind of just play the macho role and kind of put it all aside. Like it doesn’t happen, but the truth of the matter is for our customers and, and a lot of our partners out there, and it, it happens, it happens a lot and everybody is just human. So that’s, again, another one of the things that automation of programmability bring up is that, that reduction in human error. So if you have a script that automatically backs up your config, (laughs) the second you log in, it’s less of an issue, right (laughs).
Steve Cummins:
So I’m guessing you have that script now. Right?
Jason Gooley:
I am working on that script,yes (laughs) and I also have a nice TFTP server with all the configs on that too (laughs).
Steve Cummins:
Perfect. There you go. Yeah. It’s, it’s nice to know we are all fallible. Right.
Jason Gooley:
Yeah, I mean. (laughs).
Steve Cummins:
And, you know, it could have been worse. It could have been, you could have had a call center of thousands of people looking at you going, «What the heck did you do?» So…
Jason Gooley:
Yes, luckily that one was not my fault. I was just the person who was fortunate enough to bring, get brought in and fix it (laughs).
Steve Cummins:
You were scrambling to get it done. I got you. Well, you know, talking of things that nobody anticipated. So you mentioned Cert-pocalypse, February 24th, and I’m sure as soon as that was announced, everybody was revved up and ready to do the sessions at Cisco Live and get everybody on these new tracks. And then of course, Cisco Live went virtual.
Jason Gooley:
Yes, it did.
Steve Cummins:
So which, and I think everybody in industry hated. I mean, that is the event every year. Everybody looks forward to it. I personally don’t like going to Vegas, but I will go to Vegas for Cisco Live. So what are your thoughts on how that worked as a virtual event? I give Cisco a lot of credit for pivoting as quickly as they did to, to make it virtual. And then I’d also be interested if you have any thoughts on how you think it’s going to work this year, obviously, you know, having had six months or so to plan for it this time.
Jason Gooley:
Yeah. So that’s actually another role that I just took on about mid October is I am now a software session group manager for Cisco Live for Enterprise Networking Solutions. So I get the, and for anyone out there that doesn’t know what that means, I am one of the folks who goes through all the sessions, accepts or rejects the speakers and make sure that speakers are set up to do everything that they need to do as well as planning it and, and hosting the event. So what’s really cool about that is that I am indeed involved with what’s going on from a Cisco Live virtual perspective.
Jason Gooley:
And I think that, you know, going from being at all of these things physically in person I’ve been to all of these and going from that to being virtual, it is the human interaction piece, I think, is the biggest piece that, you know, a lot of us enjoy and may miss a little bit of. But what we’re doing from a virtual perspective and without letting a lot, I can’t really get too in depth with a lot of it yet, because it’s still kind of a NDA, but it’s, we’re trying to virtually resolve that. You know, whether it’s different sessions with a group of folks where you can do after dark or after hours where you’re having conversations with engineers or where you’re working with executives, or having some of the same kind of sit-downs that you would have had one-on-one in person in, in one of these locations to essentially just bringing back the whole community feeling of everyone getting together behind a common goal.
Jason Gooley:
And I think that, you know, regardless to education, the sessions, all the technology, it’s still second to none, obviously that is primarily the biggest thing of Cisco Live is for our customers and our partners to get education and to really enjoy what we’re doing as far as the different technologies that we have. But I also think that although it is unique this year it’s going to be, I think, a global event that everybody would get some benefit out of. And as far as I can’t really get into the pricing and things like that but it is definitely going to be different than what it would have been in in-person. And we will still be doing certain things as far as musical guests and celebrations and the different type of labs, hands-on labs and, and walking self-paced labs, things of that nature.
Jason Gooley:
That even though you’re not physically walking in, you are essentially going to walk to a council of your choosing to be able to get in and get the same hands on you would have if you were sitting there. So we’re trying to leverage that different types of methods to essentially make it as close to live in person as possible without you know, keeping it, you know, with keeping it safe, I should say, right. I mean, we, we still had, the fact that we pivoted so fast and, and some of our global sales meetings did the exact same thing where, you know, we had to make the call early on, you know, a lot of time and effort goes into, to developing and, and hosting these large scale events. And if you put all your eggs in that basket and we get to the point where, you know, whether, whoever it is, you know, the local government or somebody says, «This is not a good idea.» Or just based on what’s going on currently in the pandemic that we talked about before, if it’s not safe for our customers, employees, or our partners, then we have to make that decision right away.
Jason Gooley:
So we can try to make the event as best as we can for the attendees. And I really think that although it will not be live and in person, I think that it will be pretty awesome in the sense that you will still be able to get a lot of the back and forth that you would have gotten if you were there in person we’ll, we’re looking at a lot of different, like networking type events where we can still bring together different areas of the business and customers as well as even, you know, some of like the Cisco champions that we have, bring them together and do different things where we can host meetings and stuff. Still live, but not necessarily in person. So I’m pretty optimistic in what it’ll turn out to be now, I think everybody in the back of our hearts still does indeed, you know, want to see 2022 as a live and in-person event.
Jason Gooley:
But obviously we have to consider our customers and the health of everybody attending that. Hopefully everybody will enjoy what we have in store for them this year. And I think they’ll be pleasantly surprised this time. And next year, I’m still fingers crossed that we’ll be seeing all of you live in person.
Steve Cummins:
Yyou’re absolutely right about that. I think everybody’s desperate to be able to get back out and do these things. But you’re right this year, obviously didn’t make sense. And I’m intrigued. I am quite hopeful from the way you described this, because you know, a few other conferences have tried to do virtual things. And I think where it’s failed is they try and take what happens at an in-person conference and just kind of virtual.
Jason Gooley:
Yeah.
Steve Cummins:
And it sounds as though your approach is more about, you know, embracing the idea of what a virtual event could be and doing things differently. And you’re right. The big value of Cisco Live is that interaction, which I don’t think you tend to get at a lot of the other events. So it’ll be very interesting and I’m very much looking forward to hearing the announcements and seeing what’s about, and of course seeing who the musical guest is. ‘Cause that’s always the highlight.
Jason Gooley:
I, you know, and, and everybody out there who knows me, you know, I’ve secretly got my fingers crossed for like Metallica, Megadeth (laughs) or something like that, but it may not happen, but you know, who knows (laughs) that would be, that would be awesome, but, uh…
Steve Cummins:
It would but you know what? You should, you should hope for thatfor next year when you can actually see it live.
Jason Gooley:
That would be even better (laughs).
Steve Cummins:
For sure. For sure. Yeah so…
Jason Gooley:
I thought so, but hey, it’d be worth it (laughs).
Steve Cummins:
It would. So you’re saying Bruno Mars, wasn’t quite your style, huh?
Jason Gooley:
I love all music, I do and Bruno actually put on an amazing show. We’ve had Maroon 5 I think once or twice was amazing, Elle King and Aerosmith. I mean, Aerosmith was unbelievable. And going back to Living On The Edge, I’d never seen Aerosmith live before. And I was literally right on the rail, right in front of Aerosmith. And it was one of the most amazing experiences. And it was it was pretty cool because they had a band opened up for him that had the lead singer Stone Temple Pilots before he passed away as a singer, which was amazing. And it was kind of interesting to see the dynamic between the two bands who had, they had former members of Guns N’ Roses there.
Jason Gooley:
And it was pretty amazing. And you know, it was one of those situations where if you get 14 or 15,000 people from Cisco Live even 30,000, it’s not like you would go to normal Aerosmith concert with 50,000 people or something. So it’s a lot more easy to move around and kind of wiggle your way up front, so to speak. And it was a lot of fun. So I’m hoping that we get back to that. But I also think that whoever we have for our musical guests, I mean, we’re probably not gonna be able to get as good as seats as you would if you’re digital right now, anyway, so might as well embrace it the best we can.
Steve Cummins:
Yeah, for sure. I agree. It’s one of the best parts of Cisco Live and it’s kind of a nice way to round things off and get everybody together. So you mentioned your love of music. So you are host of MetalDevOps, which is about bringing your two loves, I would say of music and networking together. It’s a video talk show on YouTube. Tell us a bit more about it.
Jason Gooley:
Yeah, it was so funny because it’s really a kind of a random story and, and it involves, it goes back to involving Megadeth. And now I had gone to a Megadeth bootcamp, is what it was called in March of 2007. Or I’m sorry, 2017. And we went to Dave Mustaine’s house, lead singer of Megadeth, and you hang out and there was like 40 or 50 of us total. And you’re hanging out with Megadeth all weekend and learning song clinics, how to write songs, how to play the drums, guitar, all this different stuff. And I just happened to hit it off really well with them, all of them, which was really nice. And we stayed in touch.
Jason Gooley:
And one day, you know, fast forward some odd years later a good friend of mine, Thom Hazaert, who is partners with David Ellefson from Megadeth, as well as the singer in the band called Ellefson, which is David Ellefson on the bass and Thom Hazaert, singing. And he reached out to me and they said that they wanted to do something with technology where they can do, quote unquote, video lessons for kids who are, you know, because of COVID, are out of school because of COVID or whether, whether they had it, or because their schools were closed. They wanted to figure out a way to do that. And I loved the idea so much.
Jason Gooley:
And one of the things that we started doing is I started doing the show and I started getting interviews with different members of the band, and it just kind of kept growing organically from that. And then fast forward, that conversation turned into essentially a way that I brought that to Cisco and Webex and managed to get a grant for what would be called the David Ellefson Youth Music Foundation, and got them a really nice Webex grant with all the video conferencing units and everything that they needed. And all these different stars joined in to the cause.
Jason Gooley:
And we have folks like Nita Strauss and Chris Kael from Five Finger Death Punch, and Bumblefoot who was a former guitar player in Guns N Roses, who is now guitar player and singer for Asia. It’s all these wonderful artists coming together to help give back. And it literally a scale that I never thought it was going to hit when I brought it to Cisco. I said, «Look, you know, I do a lot with giving back and obviously I have the MetalDevOps show.» But this was something that we really wanted to try to bring together and bring forward. And it just took off like wildfire and they’re doing amazing with it. We’re still rolling out Webex units because there was so much to get done. And obviously with COVID, you can’t travel and help install and do things.
Jason Gooley:
So we’re still working out some of the logistics on it. But it’s taken off like wildfire and it’s been covered by Rolling Stone and all these other magazines. And it’s pretty amazing. So if you wanted to check it out, check out the EllefsonYouthMusicfoundation.org, that’s the website for that nonprofit, which is amazing that we partnered with Cisco on. And then obviously MetalDevOps.com and MetalDevOps on every social media platform, including YouTube is where I have my stuff and interviewing a lot of those artists. And those interviews will continue going on. As we start getting their units and stuff installed. I’m going to start doing the interviews with a lot of these different artists and talking about what that nonprofit means to them and, and how giving back is not only encouraged but welcomed by folks that you would’ve never really thought to think about those sorts of things.
Steve Cummins:
That’s brilliant. What a nice way of bringing, you know, a couple of things together that you care about and you’re interested in and, you know, putting the pieces together and great that Cisco is a supporter of it as well.
Jason Gooley:
Yeah, it was tremendous. And another partner that I partnered with was the Old Bridge Militia Foundation. So the Old Bridge Militia Foundation gives back as far as instruments and lessons to kids who cannot afford them. Which is another one that’s huge, it’s a nonprofit. And they’re actually endorsed by Metallica and Slayer and Anthrax and a lot of these big bands. And it was one of those things where it just seems to make sense. And with Cisco, we have a way of giving back where, you know, if an employee matches or, or donates to one of these nonprofits that Cisco will match it. So I do that quite frequently to both those nonprofits to get them matched and try to really help drive those because they’re near and dear to my heart. And that’s OldBridgeMilitiaFoundation.org as well.
Steve Cummins:
That’s great. And I’ll put those links in the show notes. So if anybody wants to take a look at them, there’ll be right there. So you mentioned early on in the conversation that a lot of people helped you out and encouraged you as you went through the CCIE process yourself. So anyone in your past, a mentor, you know, someone who really helped you out that you just want to give a hat tip to.
Jason Gooley:
Yeah, actually there’s a couple and they might sound a little strange, but one of them I have to say is a gentleman by the name of Jim Cook. Uh, Jim Cook was my original networking instructor back when I first decided to go to school to get to get some certifications. And at that time it was CNA for Novell and 4.11B. So, not to age myself too much. But he has not only been a good friend of mine, pretty much my entire career. He’s also now my financial advisor, the guy’s a financial wizard (laughs). So it just, it’s interesting how things evolve and change. And you find people that you trust that you really know that they have your back. And he’s definitely one of them.
Jason Gooley:
And I have to say, thank you to, I mean, a handful of folks, Andre Laurent has helped me out considerably in my career at Cisco. Anthony Sequeira, Bryan McGann, all these folks have been really, really big in influencing me and helping me out. My buddy Rohit Pardasani has helped me out quite a bit. And, you know, I think that when you think about some of these different folks, it’s when somebody’s willing to lend their hand out, especially when they’re (laughs) probably underwater as well. I mean, these are all busy people. Myself included it, you know, but when you make that time, it really does mean a lot.
Jason Gooley:
Not only does it make you feel good for doing it but when you’re on the receiving end of that, I remember messaging a couple of folks way back in the day. I mean, we’re talking 99, 2000 timeframe. And when you get a response back, you just light up like, wow, this person, who’s a CCIE. You know, what you’re aspiring to be. And at the time it was like a thousand or 2000 of them in a world. They responded to me (laughs) you know. It means a lot. And it meant a lot to me too. And I think that, you know, through the books and the videos and, you know, the Cisco Live stuff and everything I can do to kind of give back, it’s been one of those things that, it’s not expected.
Jason Gooley:
But it is definitely warranted and should very much be the norm. And, and I think that if you can take a step back from your own career and realize that no matter how hard you fought, because you do most of the work, right. I mean, if you’re the person going for something, you’re the one in the trenches, you’re the one studying, you’re the one that’s learning, you’re the one doing the, the work. You still didn’t get there on your own. It’s just not possible in any form or fashion, right. It’s somewhere, somehow, somebody gave you some hand. Right. And I think to acknowledge that and realize that, you know, it’s okay to give that helping hand and help people along in their career and their journey.
Jason Gooley:
I think not only does it make (laughs), it makes me feel like a better person. But it really does encourage the next evolution of engineers or developers or whatever they are to do the same when they get to that point. And there’s a quote that I saw and, you’ve see it in a meme on Facebook, you’ve seen it on these all these different websites. But it says, «If you’re fortunate enough to get to the top, be kind enough to send the elevator back down.» And that, to me, it just really does hit me in a way that, you know, we’re all very fortunate for being that we couldn’t even do what we’re doing here.
Jason Gooley:
I mean, I started this ‘cause I love networking and I loved working with technology. I never dreamed that it would get to be this far for me in my career and that I would have been as successful as I even am. And helping folks I think has really gotta be the root and the foundation of what we do moving forward.
Steve Cummins:
Yeah. I love that sentiment. And I do think the Cisco community, you know, a lot of it is focused around that. And I think it’s a very supportive group of people, but it is nice to look at it in that way. So you (laughs) mentioned that you’re a busy man as are many others. And I know within Cisco, you wear a number of hats. You’re a radio show host, you’re involved in CCNA, you’re involved in Cisco Live. You have the Ellefson Foundation. Dare I ask what’s next for you. Is there another book on the way, are you expanding into other things?
Jason Gooley:
So I know this sounds crazy but actually I’m taking a break from writing for a little while. And I think what I’m going to do is I’m focusing on music. So I’ve been playing guitar for most of my life, and there has been a lot of things I wanted to learn more of the theory. I started picking up the piano, I’m actually playing drums. So these are things that, you know, I noticed that over the years of the constantly grinding and constantly, you know, crunching and doing what you gotta do for your career, that I sort of overlooked a lot of the things that I enjoyed to do woodworking, for example, hiking spending more time with my kids and things of that nature.
Jason Gooley:
And it’s something that I’m taking some time for me to kind of restart. You know, I’ve said this on a number of occasions, burnout is definitely a real thing. So I think that, if you can pace yourself and don’t forget to give yourself some time and really make that a priority, I think you have longevity in you, if you can do that. And I think I don’t really want to burn out. So I’m gonna take some time to do some stuff that I really enjoy to do and see where I end up. I don’t know. I don’t think I’ll probably be cranking out albums or anything like that (laughs). But you never know, right?
Steve Cummins:
You never know.
Jason Gooley:
I didn’t think I was going to be writing six books (laughs) and speaking at Cisco Live and even doing this so-You never know, I guess.
Steve Cummins:
-And I’ll check in with you in a couple of years and, and see how that album’s going. Good for you. I think particularly with what’s happened in the last year, sometimes taking that step back it is important. So I’ll be interested to see what you get into.
Jason Gooley:
Awesome.
Steve Cummins:
So, just wrapping up here, I would encourage everybody to follow Jason on social media to keep up with everything you’re doing both in the Cisco universe. And I guess as you expand your music career! On Twitter, I know you’re at Jason_Gooley. I’ll put your LinkedIn profile in, in the show notes as well. And, and as you mentioned, I think MetalDevOps any where on social media, will find you as well. So Jason thanks very much for spending some time to share your experiences with us on Living On The Edge.
Jason Gooley:
Absolutely. My pleasure. And I really do hope that this helps someone.
Steve Cummins:
Fantastic. Thank you.
Jason Gooley:
Thanks…
Transcript: Living on the Edge Podcast with Michael Wynston
Steve Cummins:
Michael Winston is the director of Global Network Architecture at Fiserv, having previously managed networks at a number of financial institutions. Today, we’ll be talking about CCIE certifications, SD-WAN, and Network Function Virtualization. Michael, thanks for joining me on the Living on the Edge Podcast.
Michael Wynston:
Thank you Steve, for inviting me. I really appreciate the opportunity to speak to you and your audience today.
Steve Cummins:
Great. So let’s start with the simple stuff. Can you just give us a quick rundown of what Fiserv does and how your IT organization is set up?
Michael Wynston:
Sure. Fiserv is the combination of a company that was called First Data. We merged with Fiserv a little over 18 months ago now. We are one of the world’s largest financial services or FinTech companies. We’re responsible for a number of different countries, continents, everything that you do that does not involve cash. So if you’re shopping online and you’re buying something with a credit card, or if you’re financing a vehicle, or if you’re using an online banking app for some of the largest banks and some of the smallest banks. Some part of that transaction, or in some cases the entire transaction passes through Fiserv’s global infrastructure. So we’re the big finance company behind the curtain of all of the banking and credit card and merchant companies.
Steve Cummins:
So a lot of what Fiserv does, we rely on every day, we just don’t realize it?
Michael Wynston:
Yup. That’s a good way to put it, right. So when you go to the store and you put your card into the chip reader and it says not authorized, and you say, I know my card is good. Why didn’t that work? That’s probably a failure somewhere inside of our infrastructure, which, yeah, it’s funny. I go shopping with my wife and I actually count off in my head the number of seconds it takes for that authorization to come back because I know what our SLA is for our clients. So to me, that’s something that’s actually really important.
Steve Cummins:
I like that. Nice bit of professional pride you’re showing there. So based on that, it’s obviously mission critical for a lot of companies. Just give us an idea of how the IT organization is set up of Fiserv and what your role is within it?
Michael Wynston:
Sure. So, at Fiserv we truly embrace our global nature. My team has a global network architecture team has architects located in all of the different continents. We’ve got architects in APAC. I have architects in India, North America, South America. So we make sure that we collaborate on different technologies that we implement. So we create a unified homogenous environment, but we still take into account the specifics of a particular region. So things like GDPR or any type of privacy legislation, because a lot of that impacts us and how, and where we store our customer’s information is something that we always think about. And driving our organization on a global scope, we try to always like the best of breed technology. One of the things that we really try to emphasize is no shiny object syndrome, I’m not buying something because the vendor said is what you need it.
Steve Cummins:
Yep. The shiny object syndrome. I think we all fall foul of that. It’s whether we’re buying things personally or buying them for work, there’s always the new thing that’s coming along, right?
Michael Wynston:
Yeah. And within my team, we call each other out when we do it, because we all fall victim to it, you know, constantly just you always hear, Oh, this is the newest best thing. And it’s a hundred times better than the other thing. So you need this now. Hell, yeah. Maybe you’re right. So we always make sure to call each other out on that kind of stuff too.
Steve Cummins:
Yeah. That’s good. It’s funny I was having a conversation with a friend of mine yesterday about the new iPads and which one to buy and he said, «Well, you know what, if you wait until March, they’ll probably upgrade them again. And it’s like, yeah, but there’s always the next thing. Right?
Michael Wynston:
(laughs). So, you know, what’s really funny about that. I kept waiting for the next iPad pro because I wanted the newer processor. I know. I said, I’m always going to be waiting for the new iPad pro with the newest processor. So on black Friday, I went out and I bought myself an iPad pro with the previous processor generation from the iPad air. Because like you said, you’ll never have the latest because they’re always going to announce something better the day you buy it.
Steve Cummins:
That’s it that’s, that’s how it goes. All right. So you’ve inspired me. I’m going to go make the purchase now. So at the beginning I mentioned here, you’ve worked for a number of companies. You know, people like Cisco, Pfizer, Merrill Lynch, you’ve kind of got a Who’s Who, and you’ve been in multiple roles, right. I know we’ve talked about, on a previous call, in some cases you’ve been a consultant you’ve been at value added reseller, you’ve been on a corporate team. So, just give me some thoughts on some of the skills that you’ve developed, that made it possible to switch between those roles. And if there was anything that you see that’s sort of uniquely different between them?
Michael Wynston:
Sure. So I got my start actually in this industry, not directly in this industry about 30 years ago and because… yes, I’m that old…. in a lady’s handbag warehouse actually in Long Island city in Queens, where the local IT team was never able to give me the information I needed when I needed it. And I annoyed them to the point where one day they just dropped a bunch of books on my desk about Docs Pro and Dbase and Novel and reading those, I thought, Oh, I like this a lot better than warehouse management. Funny enough, if anyone watches the show Young Sheldon and how he spent time in Radio Shack, showing people that operate those computers, I did that. I did that, and I showed people how to operate the trashy TV and the radio shack computer.
Michael Wynston:
Anyway, so as I started out, I found that the most important piece of being in this industry is to never stop learning. And part of that mindset of never stop learning is to not pigeonhole myself into just one particular, type of infrastructure. So it’s not only about the network, it’s not only about the compute, not only about security. You have to look at… now, I’d like to reference the OSI model. You have to really make sure you understand all the layers to really be a subject matter expert in any one of them. You have to understand your actions. So as I’ve gone through my career, I’ve tried to always make a point of understanding, not just a layer that I’m focused on, but the layers above and the layer below, so I understand what’s actually needed and what the requirements are in driving that infrastructure.
Michael Wynston:
Now, when I first started out, I actually got my start early on in my career as an instructor, which I think is probably the best thing that could’ve happened to me, because it gave me the opportunity to stand up every week in front of 24 new people and try to explain to them something that they had no idea, or in some cases believed they knew far, much more than I did about something to teach them something. And for me personally, it helped me develop a skillset of enrolling people into what my vision was. Enrolling folks into learning new technology and enabled me to be able to speak to all of the different levels. So whenever I talk to junior engineers or someone who says, Oh, well, what technology do I need to learn next? Or now where should I focus my energy? One of the things that I always emphasize is, don’t forget your interpersonal skills, because we can always hire an engineer to lock in a closet that we have configuring devices all day or writing code all day.
Michael Wynston:
The engineer, the architect, the developer, that’s really hard to find is the person that you can put in front of a customer or a consumer or the end-user, the requirements team, and actually communicate with them in a way that they understand, so that you can really make sure to deliver what’s needed. And a lot of that foundational information for me came from my certifications. So my CCIE number, 5449, I got it January 9th of 2000 when I had to I actually cable my own rack. I had to know IPX, AppleTalk, BACnet ISPN, OSPF, all of those fun legacy technologies. But I still up to this day, encourage all the engineers I work with to start out with the most basic networking certification.
And for me, I think the CCNA is really a great place to start, because whether your company adopts a fully Cisco strategy or not, I have not personally seen any other networking certification path that really embraces the full end to end routing, switching, telephony, wireless. And now with the, Cisco developer certification path, all of the different things from a skillset perspective you need in order to be successful in this industry. So while I don’t think certifications prove you can do what you need to do, I do believe it’s a great way to open a door and a great way to start out.
Steve Cummins:
Yeah, that’s interesting. So, you know, there’s obviously a lot of talk these days about Cisco certification and, maybe not being as relevant as it may have been in the past. And I know they’ve made a lot of changes to the programs. So from your perspective, it sounds as though you see it as a good foundation for somebody and then maybe there’s other areas you’d encourage people to dive into once they have the CCIE?
Michael Wynston:
Yeah, absolutely. Because when I first started in this, Cisco made routers and switches and that was it. And then when, you know, acquisitions like Selsius that they bought for IP Telephony and Arrowpoint that they got for the load balancer technology, you know, as that started to grow, it was real divergence where you could really pick a particular area and become a subject matter expert. But I think starting off with just the foundation of basic routing and switching is a really good way to get your feet wet so you can get insight into the other different pieces that are part of the network infrastructure landscape we know today. And, you know, maybe you’re looking at that and you go, wow, I really like the part about writing code.
And you go into being a NetDevOps developer, or you say, well, public cloud really feels like where it’s at for me, while your CCNA certification, translates very easily into some of the basic AWS or Azure certifications in the networking space, because at the end of the, all we’re doing is moving IP packets.
Steve Cummins:
Yeah. That definitely makes sense. And you made an interesting point a little earlier on about, it’s not just the technical certifications, it’s the personal skills as well. Any particular training or experience you think is valuable to that, or is that just sort of, you know, life skills?
Michael Wynston:
I think life skills are absolutely important. One of the other things I also encourage, , you know, these strange times that we’re in now, this is more challenging, is to look at things like Toastmasters, not sure, you know, every one of your listeners are familiar with that, but there are a number of places where you can go, just community things and, other engineers stand together or even the user groups that are formed for the different technology sets, where you can put yourself out there and really do your best to try to enroll and explain and convey the different thoughts and ideas that you have about the infrastructure. So you can develop your speaking skills, your interpersonal skills, that again, for me at the end of the day, that really becomes a differentiator.
Steve Cummins:
I agree, there’s this stereotype of the network engineer working away in a darkened room and not talking to anybody. But I think we all know in reality, being out in the world is as valuable as having the technical knowledge. So, that makes a lot of sense. Talking of being in the real world, the podcast is called Living on the Edge, partly because of the clever play on words, partly because every network engineer has a story about when everything went wrong and they get the call at 3:00 AM and they had to fix it. So anything from your experience, any interesting or funny story you could share?
Michael Wynston:
Yeah. So if I go all the way back to November of 2001, right after 9/11 happened, I was working at Merrill Lynch at the time and they had a project to build a data center in 30 days. And this project was actually, we worked out from Staten Island and we worked in the data center literally for 30 days straight. You know, we slept in there, they gave us cots, maybe it wasn’t exactly 30 days, but certainly felt that way.
Steve Cummins:
Yeah.
Michael Wynston:
And the funny story around that is quite often, we would go out for an hour or two at night, because there was a restaurant around the corner where we would eat at. And quite often we would come back and find that everything we had done, some of the electrical engineers had moved stuff around they had done cable, they had re-cabled. And what I found out at the end of the day was every time we would leave, there was someone on the structural engineering side who wanted to be a network engineer, who believed that there were better ways to do things. And he went ahead and he told them to move stuff around after we left. And that was so frustrating. But what we were able to do is find that person and talk to them. And instead of having them, I guess you could say work against us, enroll them into delivering and configuring and making sure while we were gone, nobody broke anything that we had already built.
Michael Wynston:
And it was always, you know, at the last minute, right before we were supposed to turn up some new piece of that data center and turn it over to the client that nothing would work and we’d scratch our heads and not know why. And because we were actually sitting there in the data center with our coats on, because it was always cold, we would walk over to the equipment, and go “This is not how we cabled it before what happened here?” And, yeah, you know, strange things.
Steve Cummins:
So you’re thinking there’s a gremlin in the works. And it’s actually somebody who just desperately wanted to be a network engineer, and you can’t blame him for that.
Michael Wynston:
Yeah. Somebody who wanted to be a network engineer who thought he could rebuild it himself
Steve Cummins:
Oh, that’s funny. You know, at Opengear, we’re all about this idea of network resilience. But as I talk to network engineers, there is obviously, that phrase means different things to different people. So I’m just curious from, from your perspective, what does network resilience mean to you?
Michael Wynston:
So to me, network resilience means adaptability. When we look at the different ways that we try to put network resilience into the infrastructure, whether it’s via they HSRP or HCNP or building redundant routers, at the end of the day all of those different features and functions are really driven towards getting the infrastructure to be able to adapt to change. And if we look at it that way, it really opens up the idea that maybe the way we’re doing resilience today is not the best possible path for delivering the resilience that we need. As long as we’re willing to open our mind to the idea of adaptability. And for me personally, what that means is not doing something today, simply because that’s the way you did it yesterday. And having that open mind and adaptability helps us build the constantly improving, constantly evolving infrastructure that serves our clients today.
Steve Cummins:
Yeah. This idea of adaptability I think is an interesting way to look at it. You know, some of the things I hear from people is this idea of well resilience used to be redundancy, right? Because it was in the data center. So you could afford to have an extra router and an extra server and, you know, a generator and, and spare air conditioning, all that good stuff. Things move to the edge and suddenly you can’t do that. So you have to adapt, right. It can’t be redundancy anymore. That all of these moves with the pandemic-
Michael Wynston:
Sure.
Steve Cummins:
… same thing. Right. What, what worked a year ago suddenly it’s very different. So I think that’s an interesting angle, the, the idea of just making sure you’re adapting your solution as the environment changes.
Michael Wynston:
And, and, you know, to speak to that point to living on the edge. Right. We have a very, very forward thinking, very future designed NFV infrastructure that we use. And one of the first things that we said was, well, we have to put two routers in front of it, so it’s redundant. And then we said, well, we should really virtualize those routers because everything’s about, you know, network function virtualization. And then we said, well, if we virtualize the routers on the infrastructure, that’s supposed to host the virtualized routers, how do we get to the infrastructure to actually build the virtual- cutting out very chicken and egg kind of thing. So know that kind of adaptability and, and being, you know, willing to look at the ideas that you might’ve had about how to create the reliability in the infrastructure. And I don’t like redundancy because to me, quite often redundancy, wastes money, right? Cause you’re building something only in case of fire, right.
In case of emergency, break glass. I would like to build infrastructure so that in case for emergency resiliency comes into play, because there are resources that are available that could be consumed to provide that resiliency and redundancy, rather than dedicating something to just sit and wait for something else to fail. And that’s really what we’re driving to, you know, from an adaptability standpoint.
Steve Cummins:
Yeah. It makes a lot of sense. So it’s actually funny. Because we, we hadn’t talked about this prior to the recording here. But you know, one of the things we’re seeing with Opengear, and our focus is really out of band management where traditionally that’s been installed as emergency access. And to your point, you’re really only putting it there, for that one moment when you need it. We’re seeing folks now are actually spending the money to implement it and then using it as an independent management plane every day. Right. So you’re not spending that money just for the once “what if”, but it becomes part of your overall solution.
Michael Wynston:
Yeah. Yeah, absolutely. Well, and especially since we’ve moved past, you know, ribbon cables for, for console conductivity, so. (laughs)
Steve Cummins:
See, things change. You know, something I wanted to pick up on there. You mentioned network function virtualization. Could you just give some thoughts on how you think that’s impacting the way IT infrastructure is being built?
Michael Wynston:
So as we’ve moved more towards a dis-aggregated footprint for network infrastructure, where network infrastructure exists in public cloud, it exists inside of your colo facilities, it exists inside of your traditional data centers, it exists inside of your client environment. Building purpose-built pieces of infrastructure routers, load balancers, firewalls, and doing a one size fits all kind of footprint, quite often leaves you with a lot of abandoned infrastructure. And what that also, in addition to your abandoned infrastructure, which wastes CapEx, which no one has CapEx to waste, it also creates an environment where you can’t scale quickly in the event of a change in circumstance and coming back to adaptability. So for us, we really looked at network function virtualization, not because we wanted to get rid of all of the routers and firewalls and load balancers, but because we wanted to decouple the function that they provided from the actual hardware that you use, to provide that particular function.
So when we look at NFV it’s so that we can go ahead and do things like dynamically scale, and add more load balancers, ADCs or firewalls, when the resources that we need to consume are not there immediately. We have applications and monitoring and performance analysis that helps us to create this infrastructure on a predictable basis. So it makes that all infrastructure dynamic. You can’t do that with a router or a firewall that you hold in front of you, you order from a provider that takes, you know, somewhere between 30 and 90 days, just to ship, let alone the time to install and put in the change request. The other advantage to NFV for us is it allows us to constantly evolve the network functions that we use. So we like vendor X today because their firewall does something that the other vendor doesn’t do. And then three or six months from now, that vendor that we were using has a competitor comes along, that does it better, faster, cheaper, also its software, swapping that piece out, that virtualized software piece is a lot simpler than swapping out a physical load balancer or a router or a firewall. So it makes the infrastructure a lot more adaptable based on the features and functions that you’re trying to deliver.
Steve Cummins:
Right. It continues that move away from being locked into a particular ecosystem, right? You can pick and choose from whatever the… well, the, the danger is it’s the shiny object of, well, Hey, we’ll switch to this-
Michael Wynston:
Yeah.
Steve Cummins:
… this vendor, but it does give you a lot more flexibility.
Michael Wynston:
Yeah. It does give you a lot more flexibility. So one of the things I always like to emphasize about NFV is that, without collaboration with your compute partners, you can’t do it. And the reason why I say you can’t do it is because you really don’t want to. So when we went into network function virtualization, I made sure that I was joined at the hip with my peer on the compute and virtualization side, because the last thing I want any of the network engineering teams to do is own patching, So ESX hypervisor, or own patching or maintaining the Linux KVM system that’s providing the ability to run all those VNFs. I also didn’t want to have to, although we did, get every engineer to understand what NUMA pinning means and, and how you allocate CPU sockets.
I wanted to be able to focus our energies on how we wanted to service chain all of the different VNFs in a logical fashion. And then work with our compute partners so that they could deliver the compute infrastructure we needed in order for us both to be successful. Again, it went with what’s our strength in that area, and what’s your strength in that area? And that’s worth the other to build this solution in a more unified way, rather than just sitting inside of our silo and saying, well, we can’t do it cause the compute guys, they don’t understand networking. They shouldn’t have to, and the same in the other direction.
Steve Cummins:
Right. And then you and your team can focus on the parts that make sense to you and rely on the others to bring their expertise into it.
Michael Wynston:
Yeah.
Steve Cummins:
So one other subject I want to bring up when we first met, you came up with a phrase that I don’t think I’ve ever heard before, or may never hear again. You said to me, «I love SD-WAN.» So, obviously a hot topic. And one of the things you mentioned to me is that with the pandemic, a lot is changing. So maybe you could just share your thoughts on what you see happening with SD-WAN, right?
Michael Wynston:
Sure. So prior to the pandemic, SD-WAN’s primary focus was enabling the branch or a campus location, the ability to deliver applications and connectivity, in a way that made more sense than hairpinning everything back to the data center or leveraging just really, really expensive private line connectivity. You know, whether you were still using TDM connectivity, or Ethernet over MPLS or Ethernet, or something like that. What SD-WAN enabled us to do was to use multiple transports that made sense and direct the applications over those different transports in a way that was actually a consumable. So I mean everybody’s tried to do policy-based routing at some point in their career, and then you forget you’ve done it. And then the night that it’s broken, somebody goes to troubleshoot and says, I don’t understand.
I look at the routing table and the traffic’s supposed to take this interface, but it doesn’t. It takes that interface. Why is it doing it? Oh, yes, I forgot. We turned off Policy-based routing for that destination address, which because of address summarization, you don’t actually see on the routing table. So you think it should go that way. Anyway, what SD-WAN enabled us to do was apply more logic to the environment. Now, what that allowed us to do, in time is because we could apply more business logic to how the infrastructure was consumed. We could get away from consuming the most expensive transports for everything because our infrastructure wasn’t capable of differentiating, when what should take what path. So now, because we’ve migrated to a combination of MPLS and broadband or DIA, depending upon the site we can easily make sure that traffic that is internet bound, goes out to the internet instead of hairpinning from the data center out to the internet, and then back into that particular branch location.
So, it’s a wonderful thing, but it’s also very complex and, and deploying SD-WAN is not simple, especially on a large global level. But then the pandemic came and once the pandemic came, everybody went home and we had all of these offices that we put SD-WAN into and, you know, we tried to figure out, well, what do we do now? And what we do now is we look at how we can leverage those same lessons we learned around SD-WAN and developing our public cloud connectivity, developing our backbone connectivity. So what’s interesting is I’m starting to see a number of vendors show up that talk about public cloud infrastructure and software defined network infrastructure in public clouds with a lot of the same features and functions that you see inside of your traditional SD-WAN environment. And one of the reasons they’re able to do that is because SD-WAN being purely software defined, and because you can’t ship an SD-WAN box to Amazon, they won’t do anything with it.
You have to solve these problems through the software. So a lot of the SD-WAN companies were actually very well positioned to transition into a public cloud network infrastructure type scenario. That’s one evolution I’ve seen. The other one is I’ve seen a lot of the SD-WAN vendors start to tackle the home users. So whether it’s through the executive VPN small form factor devices or through a software based VPN that has SD-WAN like functionality to a traditional VPN would create a tunnel back to the data center. And again, everything would go out via the data center. But since everything we’re doing today, even this GoToMeeting, it’s using the internet. My SD-WAN, executive VPN inbox that I have at home can say, Oh, this is internet bound. I can tell based on policy, this is internet traffic, I can allow it to go directly out to the internet. I just send it directly out to the internet. So it doesn’t go through my data center or edge networking path first.
So again, a lot of the things that were done for SD-WAN at the branch can be easily translated to public cloud and an individual home user.
Steve Cummins:
Absolutely. And there’s that word adaptability again, right? It was set up for one thing and then with everything that’s happened this year, we needed to adapt. So you’re obviously a significant way through the SD-WAN journey. What’s the caveat that you would give to people or the one thing that they should keep an eye on if they’re just starting to look at SD-WAN?
Michael Wynston:
So if you’re just starting to look at SD-WAN, make sure that you start with the understanding that you need a minimum viable product and feature set, to deploy for your first SD-WAN deployment. And the reason I say that is because if you look at the list of features and functions and stuff that comes with SD-WAN, all of them now have integrated security, layer seven firewalls. They all have application-based routing. They all have the ability to do not just quality of service and quality of experience. They all have the ability to do multiple trials. So there are so many features and functions, and you can try to deliver all of them on day zero, you’re going to end up delivering nothing, simply because it’s just too much to consume.
And we use this same mindset with all of the new technology we put in place, make sure you identify a feature set that at least replicates what you have today, if you’re trying to transform, or at least replicate the features you need today. And then, as you become more comfortable starting to layer on the newer features and newer functions that the SD-WAN environment provides. For example, analytics. Well, I can tell now which application at any one branch or across all the branches, is most heavily utilized where it’s going to the top consumers and this information is available in real-time. It’s great. But one of the things that we’re now focused on, now that we have it out there is how do we actually consume it? We didn’t think about that when we first rolled it out, because if we had made that part of our initial minimum viable product, we would never have finished applying. Because again, there are too many features in SD-WAN to build it all at once.
Steve Cummins:
Yep. That’s solid advice. And again, many features means many shiny objects. So you’ve got to focus on what you really need.
Michael Wynston:
Yes.
Steve Cummins:
So clearly, you know a lot about network engineering. You’ve shared a lot of your knowledge here. You’ve been in the industry 30 years, you said. I would imagine over that time, there was one or two people that have been a mentor or inspiration to you. Anyone you’d like to give a shout out or a hat tip to here?
Michael Wynston:
So I give a hat tip to the very first director of network architecture that I worked for back at Merrill Lynch, Vincent Patrizio. He’s the one who really demonstrated to me, with his very colorful personality, that you should make sure when you talk to the vendors, you tell them what you need, rather than letting them tell you what you want. And don’t be afraid to look at a vendor and go, yeah, that’s not something we would ever use. And I know you’ve convinced senior leadership it’s really important, but there’s no value here. So what… we actually need results. And, really, making the team focus on what was a business requirement rather than what was the newest, coolest, most exciting thing, and how that actually translated into financial business model, is something that I’ve carried through with me as I moved from one position to the next, is making sure whenever I talk to someone about any technology is keeping in mind at the end of the day, someone’s paying for it.
And because someone’s paying for it, you’ve got to make sure they can clearly identify why they’re paying for it and what the value is for them.
Steve Cummins:
Yeah, that is very solid advice. I like that. So one final question for people that are always looking to find the latest trends and learn more about network engineering, Where are the places that you go to keep up with what’s happening in the industry?
Michael Wynston:
So I am an avid reader of SDxCentral. For me, SDxCentral and the other networking podcasts are always very helpful. But to me that’s a good place to start. And that’s where I get a lot of my initial reads and initial ideas into what new technologies are hot and what vendors we should start looking at on those particular technologies.
Steve Cummins:
That’s great. And, and actually, strangely, the guest on our first episode of this podcast was Roy Chua. He was one of the founders of SDxCentral. So he’ll be delighted to hear you recommending his old channel. Well, this has been great, Michael, thank you very much for taking some time to share your thoughts and experiences with the audience. For anyone who’s interested in finding out more, Michael is available on LinkedIn. He has a couple of videos on there of some presentations that he gave at recent WAN summits. So again, thanks very much, Michael, it’s been a pleasure talking to you.
Michael Wynston:
Steve, this is fun as always, as you mentioned, anyone who’s interested feel free to reach out to me on LinkedIn. Happy to talk about all the different technologies that are interesting, and look forward to speaking again sometime soon. Thank you.
Steve Cummins:
Great. Thank you.