Episode Transcript
[00:00:00] Speaker A: You're listening to protect it all, where Aaron Crow expands the conversation beyond just OT, delving into the interconnected worlds of it and OT cybersecurity.
Get ready for essential strategies and insights.
Here's your host, Aaron Crow.
[00:00:19] Speaker B: Hey, welcome to the podcast. Today I wanted to have an episode to talk about OT cybersecurity products.
It's important understanding the unique challenges and solutions in OT.
I've been in industry for 25 plus years. I've spent time in it. I spent time in OT. And there's definite differences. The technology isn't always vastly different, but the implementation and the impact is. So that's why there's a lot of differences in how things are done in OT and it. And I want to take some time and really talk around what makes, what makes a product ot versus it. When can I use what? Right. So first, let's say what is OT, right. OT is.
OT environments are power plants and substations and trains and manufacturing. And it is really technology that has a physical touch, if you think about it, in like, home automation, it's, you know, you've got a light switch that's automated. Like, I can turn it on from my cell phone. It's different than it, it is normally, you know, email and, you know, web servers and my, you know, cloud and all that kind of stuff. It's more back office and sales type stuff. Right.
The key differences with OT and it, we talk about the CIA triad, confidentiality, blah, blah, blah. Right.
The CIA triad. And OT is really flipped. And in confidentiality is not the most important thing. It's really availability. And when I say availability, I usually like to say availability slash safety.
It's available like you want your power to work. When you turn your light switch on, there is no outage, there's no downtime. You don't have a Friday night at midnight where you take the server offline or the power plant offline. You want it to work all the time. And it's the same thing. With a safe environment, you have to have safety being the priority over everything else.
Power is a great example in that there's no proprietary way to generate electricity. Power plant a from vendor a and power plant b from vendor b. They use the same technology. Right? It's, it's, it's not, that's not their secret sauce, that, that's not how they differentiate themselves from others. It's. It's more in how they spread and their, their assets and all that kind of stuff. But a power plant heats up water, creates steam, turns a turbine turns a generator like it, whether it's fuel from solar or wind. Well, obviously solar is a little different, but, you know, wind or hydro or gas or coal or nuclear.
The high level thing that it does is pretty much the same. It's a tea kettle turning something to steam to turn. Turn a turbine to turn a generator to generate electricity. Can it products be used in OT?
I've been doing this a long time, and all the way back to, gosh, I think 2010 ish is when I really started focusing in OT, cybersecurity, in critical infrastructure, specifically as like my primary job.
It was never kind of called that before. That NRCSiP really changed the world for the power utility world. But there were no OT specific products back then, so we had to adjust and adapt. So I took commercially off the shelf products, you know, traditional firewalls, palo alto firewalls, and Cisco firewalls and fortinets, and all the different things that were, that were in our iT data data centers and took them and just implemented them in a different way. Cisco switches and gigamons and ixias and packet capture devices. We were using things like, what's up gold for uptime monitoring and SNMP monitoring. We're using nacs to do monitoring on the network layer.
But the way that we implemented a NAC was completely different than the way we would implement it. In it, I was using a NAC almost like an asset discovery tool. I was doing no blocking. I was not segmenting networks. I was really just alerting out, hey, this new thing came on, and these are things that I see on that device, right? So we set up SNMP traps. We set up a lot of different things. We were also doing full packet capture on the entire network. And again, this is back in 2010, we were doing active scanning. I know that's a controversial subject, but all the way back in 2010, we were doing active scanning in a production OT environment, power plants, etcetera. Now, were we just going and randomly scanning the same way I would do in an IT world? Of course not, but we were doing active scanning in an OT world. We had to do it carefully. We had to be selective in the devices and the networks that we did it in, but we were absolutely doing it. So you. It can be done if you know the environments that you're going into and you are careful and really understand. It really takes an understanding. And working with the sites, I never went in and said, hey, this is what I'm doing to your environment. It was more of a, hey, these are the requirements that we're trying to do.
How can we do this in your environment safely without bringing your environment down? All that kind of stuff. Right.
So there's a lot of ways that you can use it. The other part of the IT, the it products that don't always necessarily translate into OT is OT uses a different set of protocols. Sometimes they have OT specific protocols. So profibus and ether IP and DMP, and there's these OT specific protocols that don't show up in an IT world.
Now, again, this 2010 is when I deployed this. Even next gen firewalls, like Apollo Alto, for example, back then, they didn't understand OT protocol. So even if I put it through that layer seven application smart next gen firewall, it didn't know what the, what the protocol was. It saw it, but it didn't classify it. They didn't have that dissector. These guys have come a long ways. The Forrester report I put out the other day, like, there's a lot of our commercial IT products that have gotten into the OT space, and they have OT lines and capabilities in their IT products. So that line is actually getting closer. We have more and more products from the it side that are getting that capability in OT. Fortinet and Palo was top right quadrant in that, that Forrester example. And that was a great example of how they've done OT protocols in these products. So that I already have a firewall, I can now start monitoring these OT protocols in that space and I get some capability out of that.
There is also an impact on operational uptime. OT, a lot of the times is very time sensitive. So if I think about right now, most, most Ot is unencrypted.
It is wide open. The main reason for that is because, well, there's a lot of reasons, but one of the reasons for that is time sensitive. Right. They don't want to add additional latency by encrypting and decrypting when again, most of their traffic is on site, the physical environment is locked down. So to be a man in the middle, you would physically have to walk into a power plant or a manufacturing facility and plug in, which is not impossible. They've just done more, they focused more on the, the physical security being that reducing that risk instead of encrypting the data.
It's still something that I think we'll get to. And you are starting to see encrypted protocols out there. They're just not widely used specifically in something that was installed 50 years ago. They're not going to go back and replace all their stuff just to encrypt the data. It's a low risk point from their perspective and a high risk to availability because all the stuff doesn't necessarily support it.
So those are some of the bigger issues of things. But again, going back to, you know, nacs and firewalls and switching and you know, packet captures and dvrs or net dvrs and you know, packet capture devices, all that type of stuff is those are the same tools that you use in it. And ot, the biggest difference is going to be, like I said, the protocols. But again, more and more it products are getting ability. You see other products like tenable that has, you know, an IT product and OT product. You see Lana guard that has them both. You know, industrial defender. You can talk to it products as well as OT stuff. Like they don't only talk to OT protocol, they also talk it. They speak SNMP, they can talk to a firewall, they can talk to a, you know, a VM host and all those types of things. So a lot of the equipment that you see in the OT space is, is the same that you would see in an IT world. It's just kind of implemented and architected differently.
That really goes into why we need some of these OT specific products.
Some of these OT specific products again, may have similar capabilities to their IT cousins, but the implementation may be different.
The OT world is highly segmented in that at a single site I may have multiple network segmentations. I may have multiple active directory forests with no trusts. I may have multiple, you know, units. And each of those units is a different network environment. Same thing on a manufacturing line. Like each line could be an individual subnet and there could be a firewall between them to segment those things. You're starting to hear it now with zero trust and all these things. Again, firewall rules. All this stuff comes from it. And there's a huge benefit to that. OT was already fairly segmented and this gets it down further and gets around some of the limitations in OT. Like you can't patch the same way that you do in it. In the OT environment, you just, it's too risky. A lot of these devices are older. They don't have patches. The patches are not applicable. I mean, you've got Windows XP boxes still running in production and they're air gapped. And you know, sometimes they're not, but they're not on the net on the Internet, right. They're, they're on a segmented network. But the point is, is that it's not necessarily the right thing to just replace that XP box. As crazy as that might sound. As a cybersecurity person, it's, it's not even supported it anymore. Like it's a super high risk. But the, the alternative to that is that this environment, this control system is running and it's millions of dollars to replace it just to get a vulnerability out that I can get out a different way. So a lot of times in OT, I look at a risk and the same risks exist in it and OT, but many times I'm adjusting the way that I respond to that risk differently than I would in an IT world.
That kind of leads me to the next kind of conversation on OT. What are some of the common things that you're looking for? Right. Is network monitoring. Right. You need a lot of, right now, a lot of our OT environments don't have a lot of visibility.
Usually there is a segmentation, a firewall, something that stops normal cybersecurity. Your soc and your normal it folks, they can't see past a certain point. They can't see into the OT network. So it's really this, this void of unknown things behind that wall. Maybe they're getting out some alerts in a good, good environment. Some environments, they know nothing. So they need network monitoring, intrusion detection, because you're not, you know, patching as often. You really want to monitor, you know, use your sun Tzu out of war, use your strength for your weaknesses and your weaknesses for your strengths because you're not patching, because you're not updating those things all the time. I'm not surfing the Internet. I'm not installing apps. I'm not plugging my, I shouldn't be plugging my phone into these devices. So they shouldn't change that often. So a change in the environment should be easy to spot because it should be the same all the time. So if something is different, I should notice those things. So asset management tools are another really big one in that I want to know when an architect, when a configuration changes. Um, just like I said a minute ago, right. Is, is the more that I can see change in the environment, the better I'm going to understand. And it's beyond just a cybersecurity. No. Right. I want to know. The system went down, right. Not necessarily because China's hacking or because, you know, a bad actor did something. Sometimes it's just, hey, Bob installed something and it's not working anymore. Like, who made the last change? So you can go back to a known good state or the last known change and revert back to a previously known good. Right. So that's good from availability, from all of that. It's not just a cyber, but it obviously also gives us a lot of insight into risks and firmware and os and vulnerabilities and all that type of stuff. Right. The next goes down to, you know, anomaly detection, threat intelligent platforms. The more that we can get into those spaces in these OT environments, the better.
And another big one that's usually on my list is secure remote access.
We usually have vendors and third parties and even individual within the company that are gaining access to these devices remotely. Remotely can be, hey, this device is on the floor, and I want to be able to access it from my desk. I don't want to have to walk a mile every time I need to touch that box or, you know, in the power world, that power plant is 6 hours away or it's in another state. I need to be able to access it without having to get on a plane and go to that place. The vendors do the same thing. You know, you've got vendors that are. That are remotely tuning turbines. And it would be really expensive if they had to come in at every outage, which is the way they used to do it, but now they're so much more efficient because they can do that on the fly. They're not having to wait till outages to come in and tune those things. They can do that on the fly. So there's a huge benefit in that that really leads me to the next thing, which is there is no silver bullet. What do I mean by that? You can't buy a single product. Like, back in the day. I remember when we first started rolling this stuff out in these ot spaces, we rolled out a firewall, and most of the environments thought, okay, well, we got a firewall. We're good. Check the box. I've got cybersecurity. Like, it was a box that we had to check. And once I have this thing, I've got oxygen, I've got water. Like, it's all I need. I don't need more. I don't need more cybersecurity.
I think we've realized now that it's more than just a single thing. There's a lot of risk. You look at NIST, you look at NIST CSF or 64 for three, any of the frameworks, there's a lot of things that we have to consider from disaster recovery and instant response and secure mode access and patching and all the things firewall doesn't fix all that. No product does. So there is no silver bullet that answers all of these things. So when you're looking at your OT environment, it's important to really figure out what's most important. Again, going back to patching, you're probably not going to patch your OT environment that often. Sometimes never. Some of these environments have never been patched and again, is as blasphemous as that may sound, it is not necessarily the wrong decision.
A lot of these technologies are coming out now where you can mitigate those controls, those risks without actually having to patch. Like if I know RDP is a vulnerability on this endpoint, well, I just can make sure that nobody can talk to it on RDP. Especially now with application aware firewalls and protocols, it's not just reducing 3389. I'm looking at the actual packet and making sure, hey, that's an RDP protocol. I don't want that going to this box that's denied. So there's a lot of capabilities there. But again, it's understanding what needs to be there and know that you, it's, it's a never ending, you know, the goalpost is always moving and there is no silver bullet.
So it's, it's, it's important to have that multi, multi layered defense. Defense in depth. Yes. You need a firewall and you need, you know, anomaly detection and you need asset management tools and you need a secure remote access solution. And, and, and like it's always going to be making sure that you're covering the right things at the right level. And you're not going to do that universally across all of your sites, maybe, right. You've got lower value sites to your organization. You're going to probably focus less on some of them than others.
But then on the flip side, a chain's only as strong as its weakest link. And you see things like the target attack and they get in when the back door on one, which goes back to defense in depth. If I segment these environments, even if I'm putting one at risk, I don't want to risk the others. So it's really just understanding those environments.
Some of the bigger challenges in OT, super highly segmented networks, geographically dispersed sites.
I talked about it earlier with the segmented networks, you may go to one side and they've got, I had sites with four different units. We had five different domains, five different secure remote access solutions, five different patching solutions. Like all these things are compounded and none of them had trust. So I had to have a different account on each one. You know, it was. It was difficult and complex, but it had to be that way so that I didn't reduce the risk at the time. Not saying that's necessarily the way I design it today, but that was a reason we did it back then.
And also, that's the way these environments are designed. By default, they stand alone so that I can take one unit offline and do maintenance on the other one without impacting either either or each other. Right. So I don't want to overlap technologies so that I can't take one of them down without impacting the other. The dispersed sites. Again, that's just a geographic area in a time area, like resources, where. How long is it going to take me to send a resource out to that place? Do I have to hire more people? You know, how, you know, all of those questions. Speaking of people, it's training, right. A lot of the time you're dealing with very smart systems engineers that are really good at the OT and the systems and making sure the site does what it's supposed to do. Print widgets, electrons, whatever the thing it is. But usually they're not cybersecurity or networking or it people, though, they're very good at it because they have to be.
That is not their primary job. So many times they're not trained in all the areas that we would need them to be to be able to fully understand and secure these environments. So sometimes it's a lack of staff and or training of staff that can be a gap, you know, from operational expertise to cybersecurity knowledge.
You know, funding is always a problem.
Funding and resource limitations kind of really tie together, but really justifying the need for the right amount of money to, you know, again, this is not something I buy a widget and I'm done.
I'm going to have to continually spend money and upgrade.
Unfortunately, the adversaries are not stopping. Like, you fix one hole and they're drilling another.
So we're going to have to constantly be fighting that, which means we also need resources to help with those, right? So we need people. It's not just a technology problem, it's a people process and technology. Right? So you need all of those things. You know, vendor support is a big one as well, in that a lot of these OEM vendors, they want it deployed. They can only support so many things without naming any names. But some of the bigger ones, you know, they have, they support, you know, vendor a, and the next one supports vendor b. Well, I've got both of them at my site. So now I have to have two different solutions for the same problem, two different antivirus solutions in the same environment, two different firewall technologies in the same environment. Instead of just one firewall, that would be more than enough to handle an entire environment. I have to have two different brands because vendor a supports this brand and vendor B supports a different one. So that gets difficult. And then, you know, old hardware, a lot of these environments, they may have been installed 50 years ago, 40 years ago, 20 years ago, and sometimes the hardware that in the software and the applications that are running in them are that old to maybe not 50 years old, but definitely 20. You've got Windows XP boxes out there, which leads to how can I patch that? I can't. So patching is not necessarily the answer. And I also can't just go replace them. So I have to come up with other ways to do that.
Sometimes those ways are training and awareness. I need to train my people the importance of educating people on the risks and the vulnerabilities and what to look for.
Traditionally they're going to look, you know, if something happens, they're going to think of, you know, availability. But sometimes we need to be thinking about, hey, did, did we just have something? How did that happen? Is that a cyber incident? Is that something we should look at?
You know, we should be looking at regular audits and assessments, bringing in third parties, or even just your internal audit teams bringing in that it, your it brethren into these OT spaces to take a look, not to touch, not to architect, not to change configurations, just to look and give an opinion. Like, hey, we manage a lot of firewalls and this rule here doesn't make sense. Can you explain that to me? Hey, maybe if you did it this way, it might be better looking at that kind of thing.
I'm a vast advocate for vendor collaboration. The customer has the ability to bring the vendors to the table and say, hey, vendor a, b, c, and daughter, we have you all. We want to have a unified cybersecurity program, and I'm not going to have 15 different products solving similar problems. So these are the ones that we're looking at. Let's come to the table, let's find a solution in architecture and find a way that we can reduce that tech stack a little bit. Maybe it's not down to one or two, but maybe it's down to ten or, you know, eight so that we can work collaboratively with the right tool set and be able to understand them. Because the problem is the more tools I have, the less competent, I'm going to be in any one of them. As an organization, which is a compounding problem.
Leveraging technology is huge. Use technology, find advanced cybersecurity tools that will help in OT environments. Use these things to help you with asset inventory. And all of these problems will help on the operational side, not just the cybersecurity side. So when you're having a problem, you can see the last time configuration change. You can look at, you know, the new, hey, there was a new asset that showed up. Hey, this box tried to talk to somebody else, you know, somebody remotely logged in. Like, all these things give you visibility into what's happening in your environment. That helps on the operational side as well, beyond just cybersecurity and then, you know, building a resilient culture and that, you know, promoting a security first. Just like we have a safety first mindset in OT, we should have a security first mindset. Like we should really be thinking, you know, in OT, a lot of times it's Saturday night, 05:00 in the morning or whatever time, I just seem to get it working. So I bypass all the things that I don't understand because I need it to work and I don't know how else all this other stuff works. So I bypass a firewall or I plug it directly into something. Right.
And all those things can happen and do happen, but we need to think about, okay, if I'm going to do that, I got it up and running. Check. We need to come back around to that tomorrow morning. And I need to get the right people on the phone and we need to fix it the right way instead of just leaving this thing here and with a piece of paper or tape on it says do not touch.
So obviously that's a lot. But really what's, want to dive into what the difference is the challenges in OT? What's different between it and OT?
What is an OT product versus an IT product?
Kind of explain how sometimes they're very similar, sometimes they're pretty different.
I think that line is getting grayer and grayer and you're seeing more and more really great products from Palo to Fortinet and tenable and all of them that have these great it footholds. And they're really doing a lot in the OT space, which is really awesome because I can do, I've got an organization that uses product a, I can incorporate that into product b. And I have all these resources that already know how to use those products and can help architect and design and roll out, you know, the right policies and configurations on these endpoints.
So I encourage you to ask the questions. Bring in your it folks, ask your vendors, you know, get collaboration. Bring people to the table. You know, ask the question of, do we have a product in house that can do this? Could it work in OT? How could we make it work? Because a lot of times they're going to tell you no. Well, it's in the cloud. Well, do you have an on prem version? Could we have a separate instance that's on prem versus in the cloud? Do we, do we have a different version? Because I don't want to integrate with the it active directory. Okay, well, could we have a second version? Like, just ask those questions. Don't just take it at face value that the answer is no or that the answer is yes, because you'll also get the opposite. Oh, well, we can just use this product. Okay, but how? This is the unique environment that we have. How would we use that in our example? And make sure that you have the right understanding of your environment, the tools, the needs, and then choose the right tools and products that fit your different use case. They don't all have to say OT on them because many times we're only dealing with it. We're dealing with firewalls and switches and VM servers and Windows machines and Linux boxes. And a lot of the stuff that's in these OT spaces are really just it equipment that's physically in an OT environment that's doing an OT function. And then yes, there are also OT specific devices from PLC's and, you know, control processors and ids and all that different type of stuff. But those things even are getting more and more it aware as we go and we get these next industry 4.0 type stuff. So thank you for listening. Please like and subscribe on all the different podcast places, YouTube, etcetera. Definitely share it. Reach out with any questions or concerns.
If you want to be on a podcast, definitely reach out. We're always looking, looking for good content and good conversations to have.
Thanks a lot and have a great day.
[00:27:17] Speaker A: Thanks for joining us on Protect it all, where we explore the crossroads of it and Ot cybersecurity.
Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time.
