Episode Transcript
[00:00:00] Speaker A: You're listening to Protect it all, where Aaron Crow expands the conversation beyond just OT delving into the interconnected worlds of IT and OT cybersecurity.
Get ready for essential strategies and insights.
Here's your host, Aaron Crowe. Awesome. Thank you for joining me today on the podcast. Jory, why don't you just introduce yourself, tell us who you are and about your background, as well as maybe even a little bit about that painting that's behind you.
[00:00:29] Speaker B: Sure, Aaron, thanks for having me. So my name is Jory Van Antwerp. I am the founder of a small company called EmberOT. Long time geek.
I'm going to really show off this gray in my beard, actually.
I started off actually building PCs at home and then graduated to writing keygens and crackers and all of that fun stuff that we shouldn't have been doing, but I was doing anyways.
That led to a fantastic career that evolved into it. And as I tell a lot of people, whether you're in systems network or you've done the whole gamut, all roads eventually lead to security. Because once you build a solution, you want to protect it, you want it to run right, people want to use it. It's just a natural evolution. So I started in cybersecurity, actually. Oh man, almost 20 years ago at this point.
And that evolved into a really neat career of running around and eventually touched on some something that you and I are heavily involved in, which is industrial. And I started being exposed to industrial through manufacturing plants and kind of the IT edge that touches industrial. And then eventually I was picked up by an industrial company, I'm sure nobody's heard of it, it's called Dragos.
And I was sent out to a site to actually help with an on site assessment. And that was at a large dam here in Arizona. And it completely blew my mind. It finished that paradigm shift between IT and OT and showed me just kind of what you're dealing with, why it's so important and why it's so different. And I have been absolutely immersed in it ever since.
[00:02:10] Speaker A: Yeah, it's amazing. It's like you see behind the Oz, you know, you see the wizard behind, behind the curtain and it's, it's eye opening and it's exciting, you know, the amount of things and the need and you know, the protection. You know, as a self proclaimed patriot, right. I want to protect our country. I want to protect my kids, our way of life beyond just this country, like in general, like stopping bad actors from doing things and making sure, you know, people take for granted when they turn their light switch on that, how the electricity gets to them or when there's a natural disaster. You know, they don't really understand why it's so complex. I know in Nashville or Nashville, Asheville in North Carolina here recently, like, you know, we've, we've got all of these, these basic systems that are not working. I, I know there was a, there was a retail store in, in the area that couldn't sell anything for like two weeks because they didn't have any way to process payment. They didn't have, you couldn't go. ATMs didn't work. They couldn't go to the bank because the banks were all closed. They couldn't get cash, they couldn't accept PayPal. Like, nothing worked. So they have this stuff and there's no way to transfer value other than the old school bartering or just giving it away because they didn't have basic processes. And all of those things, as we know, once you see behind the curtain, there's all of these connections, power and manufacturing and transportation and all these critical infrastructures, they're all linked. There's a reason why they're, they're critical because something bad happens and all those things. If one of those dominoes falls, our entire ecosystem falls around it and it's very hard to get it stood back up. And it takes time and it's difficult. So that's why it's so important to think about these things as we say, left of boom and start having these conversations before something bad happens, instead of waiting until after it happened and trying to say, oh crap, now what do I do?
[00:04:08] Speaker B: Absolutely. And that actually, that keys into a really interesting conversation that I've had quite a few times recently, which is around modernization in these environments.
You know, you and I both know that it's incredibly important that we start to utilize some of the technology that we've created in the last, you know, 10, 15 years. And even the stuff that's coming out now. The issue that comes into it though, as you just stated, is some of that automation comes at a price, right? And I talk a lot about modernization in these environments and where should we stop? Well, one of the things that I think is incredibly important is keeping those manual controls around. Like actually having an operator being able to turn knobs, flip breakers, whatever they may need to do to keep that operation running, should, should they need to without the automation.
Because if we don't have that, you run into situations that we've actually seen pop up in Ukraine and other places in Europe where they have modernized.
[00:05:07] Speaker A: Right.
[00:05:08] Speaker B: And once those modern systems had an issue, whether that's a security breach or not, there's nothing you can do except go and repair that system.
And we're lucky here in the US in many, many instances we have those manual controls. So the power does stay on as long as possible or that we can get it up in an interim status. Right. As we're replacing those systems.
[00:05:30] Speaker A: Yeah, you know, I came. A lot of my career has been spent in specifically power generation, transit, you know, transm. Distribution and generation.
And we have, you know, entire playbooks on hey, if the entire grid goes down, how do we get it back up from. From. No, but people don't understand in a lot of these spaces is it takes power to start power. So if the grid goes down and everything is dark, how do I get power back going when I have to have power? It's kind of the chicken and egg thing. I have to have power to get power going. So we have black starts, there's generators like. But if any of fail. So I have to start a small plant to get enough power on the grid to get the next plant going to increase the, the phase and all, all the electrical things that go into, into this. And there's also a reason why, and I know I'm, I'm nerding out on, on the power side, but there's a reason why Texas is its own grid and the east coast is different than the west coast. And, and you know, people want to throw rocks and stones because Texas has had some issues in our, in our, in ERCOT and our grid here lately. But it was designed on purpose. This wasn't a Texas thing, this was a. How do we make sure that maintains and we don't have a, something in the east coast in New York and Manhattan that can roll a blackout across the whole country, which can take forever to repair and replace. Whereas with Texas in the middle and DC ties to everything, it is the protection from, from Texas to be able to get the east coast or west coast go back up and going same thing in Mexico etc. So it just goes to show and the bigger picture of this is when you talk about modernization is, you know, you look at a nuclear power, a lot of those things were built in the 50s or designed in the 50s at least. Maybe they came online in the 80s. And you look at those and we've digitized, we've modernized a lot of those things. But we didn't replace the analog, we put the digital in series with the analog. Right. So the Analog is still there. It is still our tertiary, you know, break glass in case of emergency. And our people are trained on it. So if all the digital stuff fails, we can still do it manually. Which is important when you're talking about nuclear power.
[00:07:40] Speaker B: No, it's incredibly important, especially when it comes to nuclear power. And you know what you're talking about, the interconnects, you know, there, there's interconnects that are spread out throughout the United States and they even lead up into Canada and, and that is part of the reason. It's also for sharing power in particular instances. You know, I live in Arizona and our neighbors to the west have a lot of power problems. So we share power a lot with our neighbors to the west on a regular basis through one of those interconnects. But those interconnects are there so that the thing that everyone fears can't happen, which is, you know, here in the news and a lot of fighting on it. But gonna turn off the grid. There's a lot of grids. Right. It's not a, we're not gonna flip a switch, you know, a threat actors not gonna be able to hack a single eyes point and just take out all of the U.S. right. And a lot of that is due to those separate grids and those interconnects.
[00:08:35] Speaker A: Yeah.
[00:08:35] Speaker B: There's actually a lot of redundancy and forethought that goes into how power is put together. And I think it's a, you know, I can absolutely geek out on it with you because it's a great way to look at how to build something that's robust. Because in the US we have one of the most stable power grids in the world.
[00:08:51] Speaker A: Right.
[00:08:51] Speaker B: And it's something that we do very, very well. And you know, our technology shows that, but it's, it is redundant. And that redundancy helps, even with security.
[00:09:01] Speaker A: Yeah. And that just goes to show that even with power, I think power, if you look at the critical infrastructures, my feeling is power is one of the more mature because of nerc, SIP and some of those other regulations on a cyber perspective.
And yet we're still behind the eight ball. Like there's still a lot of work that needs to be done and modernization and a lot of processes and proceeds. And it's not all technology. It's. This isn't a technology problem. Not only a technology problem. Right. It's. It's. We need process and people and technology all together to fix some of these things and design these things. Right. Cyber, when these systems were Designed cyber wasn't an issue. And these systems also weren't connected to corporate or the Internet or any of those things. So they were designed in a vacuum. And then we started bolting all these things on because we wanted access and capabilities and functionality. And as we did these things, we brought in these risks. In the beginning, we didn't necessarily understand the risks that we were bringing into these environments. And then once we have them there, we're like, oh, now we're dependent upon those capabilities.
But also, I have to be careful because now I brought in this risk that I want to mitigate, and I can't make it so difficult that it brings the process down or the plant down. But I also need to make sure that, you know, a bad actor can't get in or even, you know, and most of the time. And what we see is even just a misconfiguration. Like, if a misconfiguration can bring down my environment, it doesn't matter that it's not a nation state or it's not a cyber attack or, you know, there's some popular people. I won't even say their names, but they're like, oh, there's. There's all this OT stuff. Nobody's talking about vulnerabilities. There's not that much stuff here. And, you know, fear. Fear selling. I'm like, okay, let's say that's true, but you can still bring down environments. Whether or not it's ot, Colonial Pipeline is a great example. That wasn't an OT attack, but it brought down ot. So does it matter?
[00:10:53] Speaker B: Yeah, it's. It's a discussion that we really need.
We really need to have more often publicly, because the, you know, the greater public, they don't know the intricacies of the systems that we deal with. And you just brought up a fantastic point, which is, you know, we went from having 36, you know, 36 guys, 12 trucks driving out to locations, checking on things, to actually bringing. Bringing information back to a centralized area. Which one?
[00:11:21] Speaker A: Right.
[00:11:21] Speaker B: It allowed us to actually be more efficient. But we could also get metrics and telemetry status, all these things from these environments that could help us do things better. But they weren't intended to be connected. And there's. There's a fine line to walk with that as well, because as we're modernizing these environments, we introduce more vulnerabilities, different vulnerabilities. We introduce inefficiencies as well. One of the areas that interests me the most right now is the fact that it doesn't matter if you're an energy water, you know, advanced manufacturing or pharmaceutical. Many people don't monitor at the operation, which, you know, we call Purdue Level one.
[00:12:00] Speaker A: Right.
[00:12:00] Speaker B: But they don't, they don't monitor at that particular operation. It's difficult.
The only point I mean by that is that we don't have enough data about what's actually transpiring there to see what threats, if any, or reconnaissance are happening there. In that same token, one of the things that's been proposed to help with that level one problem, that operational problem, is to update the hardware with things that have more traditional operating systems, Linux, et cetera, so that you can actually house what in it we call defense in depth. Right. You'd have edr, you'd have all sorts of defense. There's an issue with that and it's actually twofold and it's very difficult for folks outside not to understand the systems that we talk about. Those, those programmable logic controllers that, that we have at energy there. I mean, I'll point this way, half a mile that way. It's literally the substation that's feeding my house right now.
[00:12:58] Speaker A: Right.
[00:12:58] Speaker B: Runs on Foxboro controllers.
It is, I want to say it's almost 19 years old at this point. Might be a little bit more than that.
And it will be running. It doesn't, it's not up for a hardware refresh. The devices in there sometimes reach temperatures. The air temperature in there is 145 degrees Fahrenheit.
[00:13:16] Speaker A: Yep.
[00:13:17] Speaker B: I mean it's, it's blistering in the summer, there's no air conditioning, there's no nothing. And one of the reasons that they run so well is because they're very specifically made for that task. So in modern computing terms we have very down tuned processors. They don't create much heat, which means they don't need much cooling. Everything is, you know, sparkless, fanless.
It's, it's all built to run that way. The second that you introduce a legitimate operating system, suddenly I need more power, we need more cooling.
And we've introduced complexity. And complexity is actually the absolute enemy of robustness. Simplicity is where you get robustness from. So just from keeping your lights on, keeping that simple and robust is incredibly important. And that may be at, not the sacrifice, but the knowing decision to say, okay, well we can't implement these types of security measures at X, but we can have a 99 point trailing nines uptime at any given point in time. The second is cost. You and I live in very Hot environments, which means we use AC a lot.
[00:14:25] Speaker A: Yep.
[00:14:25] Speaker B: It's very expensive in the summer. Could you imagine if they decided they wanted to upgrade that particular substation to the latest and greatest outside cycle, how much your power bill would be? Because it's going to go to the subscribers where the rate pays.
So you want to upgrade that substation. Suddenly my power bill is now thousands of dollars a month, and that's just not tenable for most people. So it's a difficult problem to solve when we start talking about how specialized those particular pieces are.
[00:14:56] Speaker A: Well, and it just goes to show, like we have to. You can't take a single thing. And it's really, you look at cyber people and they're good at, you know, they see vulnerabilities or they see whatever, but you have to really, truly understand the overall system to understand what that risk is. Right. So, yes, and we see this all the time. Right. Is, hey, I've got, I've got a system running Windows XP running in a, in a critical environment. Well, that's really risky. You would never allow that in your IT world. Like if somebody tried to plot a Windows XP machine into your corporate network, in theory, hopefully you have an environment that's going to block it, it's going to kick it off, it's going to say, yeah, go away. You can't use that here. We'll give you a brand new laptop. Don't try to bring that XP machine back in. Right. But in the OT world, that's the only system that can work. You can't upgrade it, you can't patch it, because it's the only thing that works. And to replace it, it's not just replacing a computer like with a, okay, I'll just buy $1,000 laptop. No, you can't do that. You got to replace the whole control system. And then you've got to bring in, you know, air conditioning because it won't set all of those other factors that go into that. And I just, by doing all of that, I reduced my, my uptime, my, my, my reliability decreased by, by mitigating the cyber concern instead of, Is there other ways that I could mitigate that cyber concern? Instead of reducing my, I put other controls around it. Can I harden the outside shell to that? I can reduce that cyber risk while maintaining my availability numbers that I'm looking for. And that's the conversation we have. And it's the same conversation we have in it. We just have different factors. And that's, to me, that's the biggest difference that we have in industrial controls and ot, it's the same problem. We just have different factors. The equation, the variables in the equation are just a little different, right? And we just have. You have to make sure that you really understand it. You can't take a business equation and dump it into the OT world and expect it to work. It's gonna, you're gonna get wrong answers. You put it into your calculator and you got 12. But it's really 5 or 105. Like whatever the thing is, it's just off. And unless you know the business, you won't know that because I did it over there, it should work over here, right? It's all technology. The T's the same. So it should be the same results.
[00:17:15] Speaker B: No, and it's a fantastic point. It actually plays into cybersecurity in these spaces as a whole.
When we talk about, like I said, Foxboro is the manufacturer of most of the controllers in this environment. Well, Foxboro could also be at an Amazon fulfillment center. It could be baggage claim. It could be part of pharmaceutical process. How do I know that? Right? And when I say I simulating an attacker, like, how do I know what that does? That's where reconnaissance comes in. It's so incredibly important. And it goes back to my previous statement of what's going on at level one.
Are we actually seeing reconnaissance? And this gets into a question of physical versus, you know, digital security. And as you said, there's so many things that we can do to mitigate that. And one of the best things that you can do in an industrial environment is properly segment your networks, right. Actually create those logical separations where it's not air gap, but it's logically separated and very difficult to get to. But that also has in itself issues around, well, how do I know what's going on in that particular environment, right. If it is as sealed off as it can be. And that physical concern comes into again, that substation down the street, if I, if I want to walk down there with a pair of bolt cutters, I can get inside the fence, I can get inside the building. And if I don't take down any type of, you know, operation that's going on, I can run reconnaissance all day long. I may be even able to leave something behind for later. Now we're crossing that realm of what's a cyber attack and what's physical, when they really should be considered, especially in ot, as being very similar or at least merged together in the threat itself down there at level one. And these things Go back to one point I really want to get into, which is asset inventories. And I'm going to be the first to tell you, I've been in the industry for a long time. I know you have, too. Something we've been talking about forever and.
[00:19:11] Speaker A: Ever and still have the problem.
[00:19:13] Speaker B: Yes, we still, we still have the problem.
[00:19:15] Speaker A: Yeah.
[00:19:15] Speaker B: And what's interesting is, and I want to give operators massive credit here, I've heard this stated on so many other podcasts, publications, et cetera.
You know, operators don't know what's in their environment. No, no, no, no, that's false. It's completely false. Operators do know what's in their environment. What's difficult for them is over the last 25 years, what's changed?
[00:19:38] Speaker A: Right?
[00:19:38] Speaker B: What's been updated, what's plugged in? Where have I had any, you know, assets that have been replaced, possibly with a newer model, newer firmware? Different, you know, different set of problems that comes with that. And it's something that they struggle with. So when we say they need asset inventory, it's not because they don't know what's in their environments, because it's a very manual process and it may be 20 years old right now, today, and they're walking around and visually checking off this item is here, but they don't know what's actually happening on that item. Asset inventory becomes a really big issue at that point. But before we delve into that, just, just going back to the physical, you know, the physical necessity of security in, in these environments is incredible because it does play into cyber, cyber attacks in these areas. Colonial Pipelines, a fantastic, fantastic example. Like you said, it wasn't an OT attack, Right. But it affected ot. And that was simply because of how that particular environment was set up and the policies that were in place in it. And in that particular instance, we're talking a jump box. So they lost visibility into that environment, which from an operational standpoint, and, you know, a lot of folks don't understand this, wasn't shut down because they couldn't build, was shut down because they couldn't see what was going on in the pipeline.
[00:20:54] Speaker A: Correct.
[00:20:54] Speaker B: They want to pump thousands of gallons of oil in the middle of nowhere and not have an idea around it. So they shut it down. And frankly, it was the right thing to do. I mean, it caused a lot of panic, but it was the right thing to do. I just don't think it was ever really explained very well. But that type of event goes back to what you and I were just speaking to around segmentation. If you're properly segmented and you have highly available or redundant systems to be able to monitor in that particular environment or a way to manually go and check those environments, you're in a safer position, Correct?
[00:21:28] Speaker A: Yeah. And that, that's the key, and I'm glad you brought that up on asset inventory and physical. Right. Is these engineers and these operators, these plant managers and all, they know their environment very well, they troubleshoot it daily. If something breaks, they know exactly where to go to fix it. Right. What they don't necessarily know is the cyber vulnerabilities, the firmware version, some of the things that we as cyber professionals are looking for. So when we say they don't have an accurate asset inventory, that's what we're talking, Right. It's more along those lines, like they know their system, they know how it works, they know how to troubleshoot it because they do it on a daily basis. They are constantly maintaining and keeping that system up and running, which is why it's running. So that's a very good point. I haven't, haven't hit on that. And it comes easily to me obviously, because I see it and I know, but others probably think about that, well, how could they not know what they have? And to your point, it's not that they don't know what they have, it's that they don't know it from that perspective, from the perspective that we are looking for or the knowledge that we want to export out to help them. On the cyber risk side, they understand it from an asset availability and from a functionality of their system. They don't look at all those components as individual pieces necessarily. They look at them, they categorize them.
[00:22:46] Speaker B: More as an entire system and it's a different language. And this is an area where you run into that ITOT communication gap and you may have areas where it's better, but for the majority of, you know, plants that I've seen, you end up in a situation where OT is talking about a cabinet and the name of a controller, right. And it is going, this is 10.160.1.3.
The operator doesn't know what that is. Right. And if your name, if the naming scheme in that particular environment isn't documented, you can't tell them what it is.
So asset inventory becomes incredibly important for bridging that gap as well in being able to tell an OT professional what's going on. And it's not just cybersecurity. I think one of the things, and I'll Follow my sword here that we failed to do for the OT folks as cybersecurity professionals is to make it accessible and provide OT folks value. It's not that OT doesn't understand security or doesn't have security in mind, it's just that they're focused on resiliency, efficiency and safety.
[00:23:51] Speaker A: Right.
[00:23:52] Speaker B: And those, we think of those things as one and the same, right, as cybersecurity. But there are, we're gathering information in these environments that could be used to help them. Whether that's preventative maintenance or just frankly letting them know if something's a little off in their particular environment. These guys aren't normally looking at traffic patterns, for example, right. Like someone would be in a sock. As you just said, they're not trained in that way. But if we could let them know that something's off with a PLC or an HMI and give them the location. This is Substation X, cabinet B, you know, device A.
That's something that I think is really powerful and really helpful. And we need to start thinking about how to help those operators because remember, they're in their day to day. The SOC analysts are involved. We have, we have security folks that are involved, but that tends to be at larger organizations. What about the co ops? What about the munis? What about, you know, your water treatment plants where there's 12 people in the entire organization? There's one person that's doing all of it. Your operators are actually out there controlling. You need to involve them, but do so in a way that's additive to their job, their daily routine, and that.
[00:25:07] Speaker A: You hit on something so key. And I go back to when I started and rolling out what I call OT or cybersecurity in the industrial space and specifically around power plants. Like I came in and it was because of NERC sip and we were doing compliance activities and we were doing segmentation and a lot of the stuff that I was doing, but I didn't have any budget. So I was going to these plants and I was saying, hey, you're doing a control system upgrade. Foxboro, GE Turbo control systems, Emerson Control Systems, Honeywell, like all these different systems we're doing upgrades on and you have to add another 300,000 or whatever that number is to your budget because we have to do it in a compliant way. Meaning I need more stuff that's going to have to be coming and I don't have any money, which means you're going to have to pay for it out of your outage budget, which means you're going to have to choose to not do something else because this money doesn't come out of nowhere. So they had to basically, you know, hey, we're not going to do this other maintenance because we, we got it, it got cut off the line. So I didn't go to them with a, we're going to do this as cyber. The way that I won friends and influence people is, you know, we talked about the. It's amazing how all these cybersecurity tools that, that splunk and all the data that you can get out and all these different types of tools, if done correctly, you can also get a lot of things that are valuable to the asset owner, valuable to the operator, uptime and knowing when systems are down or rebooted or, you know, you've got a power supply that's, that's not functioning and it's beeping at you. One of the first plants we deployed this entire architecture to, they had redundancy. All their switches, their entire network was redundant. You know, every device had dual nicks and whole nine yards, completely redundant power, everything. Like a really great designed system. We plug in the system and one of the switches in this cabinet in this back room was complaining. Nobody knew it because nobody happened to open the cabinet and see the blinking light and it was, it was not working. So it was a redundant system. So it didn't fail. But they didn't have redundancy at this place because one of the switches was off. And when we, when we pulled it up, we, we started getting all these alerts and we walk over to the cabinet and one of the power supplies wasn't working because it was overheating. We went in there and looked and there was a zip tie stuck in the fan.
So we pull the zip tie out and the fan started spinning and the switch came up and booted and started working again. So as simple as a problem. It took us 15 minutes to find the problem, but nobody knew to look. That thing had probably been complaining about that for who the hell knows how a zip tie got stuck in the fan. It probably fell when they were doing something else and nobody noticed it. But it's just a great example of instead of just focusing on fear selling around, you know, cyber and all these things, make sure that we're considering the value to the overall environment in the organization. Not just the risk, not just the fear, not just the China's attacking us or the grid's going to go down, but also what, what value can we bring to make their jobs easier? A lot of Times Cyber is bolting on and making things harder, making the process harder. It's hard to log in, it's harder, longer passwords, like all those things are more difficult. But there's a lot of benefit and capability we can give these, these operators and these, these asset owners by bringing the knowledge to them so that, like you said, hey, you. Your system in rack to Building 4 is blinking and you should probably go take a look at it. Something's not right. And that's hugely beneficial to them.
[00:28:44] Speaker B: It is. And I have a customer that I've been working with actually for a very long time now that had an outage that was reoccurring, but it was random. And now it didn't take the plant down itself, but the outage was happening.
It would take down a controller, specifically, and that controller would then have to be rebooted. Once that computer was rebooted, everything came back. It was all fine. Now, again, this is part of a redundant system. Didn't take anything down. They had been working with the vendor to try and fix this. And every time they worked with the vendor, the vendor would come in, they'd look at the configuration, they'd look at log files, they couldn't find what they needed, and they would ask for one thing, a packet capture.
Most of these environments don't have a way to do packet captures. Correct. The way that we really helped that particular customer was say, hey, well, let's turn on a packet capture and let's make sure that you have a rolling packet capture at all times so that you, if you have an incident at any point, can go back and say, hey, that happened at 9, 10 this morning. I have a packet capture from 9 until 10:30am I'm going to go ahead and grab that and send it to the vendor and tell them the timestamp on when this particular outage happened. And the next time that it happened, they were able to submit that packet capture. They were able to identify which. What the actual problem was. Which was a. It was a bad control set, actually. Well, bad command that was being sent to that particular PLC and the vendor was able to fix it. They haven't had an outage since. And that's one of those things where packet captures are incredibly important to us. You know, from a cyber perspective, I want to be able to go back historically and look for issues or hunt with new detections, right. New IOCs, IoAs, et cetera. But just for an operator, from a maintenance perspective, the vendor needs a packet capture. You've got it.
[00:30:40] Speaker A: Yep.
Well, packet captures and logs from Windows systems have a similar story. Where? Same thing, power plant. We were, we noticed that the system kept getting rebooted and we didn't know why.
And so we basically sat a person at that spot and it was always like Saturday morning at 2am and we didn't see anything going on. There was nothing in the logs. Like we didn't understand what was going on. And we came in and I had a person basically sitting in the control room right next to this computer. And around 2am, operator came in and he yelled some explicits at some point during this, this time of the day. And he rebooted the machine. And my guy was sitting there watching him, he's like, hey, why'd you do that? He goes, why did he what? He goes, you just rebooted your machine? Like, yeah, freaking happens every time. When I'm in this certain process and I do this certain thing, I have to reboot it. When it comes back up, it'll work. But for whatever reason, the first time it's like timing out. Well, it had a memory leak in the process and by Saturday it came around and when they tried to kick it off, it just, it just froze. And the only way he knew how to fix it was to reboot it. But he didn't log it in his logs because it wasn't really a problem because he just rebooted and it fixed it. For him, that was enough because he knew how to make it work. Like he knew the workaround, but he wasn't telling anybody because again, for him it wasn't a big deal. It took 10 minutes, it was annoying. But he knew how to fix it. And everybody before him had trained him. If this happens, this is what you do. So it wasn't a non known issue. Everybody knew about it, but from us as an outsider and we were trying to help the overall system and the vendor and nobody understood what was going on until we physically put a person in that chair to watch it, right? And then we started getting logs and all that kind of stuff. But it just goes to show, again, another example of how little things like that. It wasn't a cyber issue, it wasn't any of those things. But cyber tools can help with a lot of these things that we're talking about beyond just cybersecurity risk availability and information that can help the asset owner and the vendors and the vars and, and all these, these people to tie all these things together. These are extremely complex systems with so many parts and components and pieces and ties and interconnects and third party connections and all these things that all of these things have to work perfectly or it can, it can all crumble down. And these people, they do this on a daily basis and we come in wanting to change their system without truly understanding. And that's where my push for vendors, for cyber people, internal for, for business, you know, the C suite folks to make sure you truly understand the process before you start dictating changes be done in these environments. Because what you're trying, sometimes what you're trying to do to help can actually be a problem and actually cause problems and impact your environment actually cost you money directly beyond just a bill, but also lost revenue and safety and availability and so many things that can impact and you don't realize obviously you wouldn't do it intentionally. But some of the reasons sometimes that's what the only thing that stands between that is an operator that's willing to stand up against the CEO and say no, we're not doing that. And I've seen that so many times.
[00:34:02] Speaker B: That's a really good point. And I'm sure all of us have heard horror stories about when I'll say it centric technology has been brought into OT environments and has caused an outage or a major issue. And it's something that is done by someone who hasn't had that paradigm shift yet. Right. They're not intimately familiar with what's going on in an OT environment to know just some of the basic things that can go wrong inside one of those controllers. And so I'd love to actually give an example of that for folks especially if you're, if you're new to OT. So let's talk about PLCs, programmable logic controllers. Yep, they are custom built. They are usually running a real time operating system. VXWorks tends to be one that's, that's run a lot. That real time operating system for all intents and purposes is almost firmware at this point. And all it does is interpret ladder logic and that ladder logic is then sent out over basically IO at the back end to do something, something turn a valve, etc.
That is so simple that we forget there's an IP stack on that. But it's not an IP stack like a Linux system has or Mac or Windows where if you ping my particular system on, let's say I don't know, port 1096, my system's gonna, gonna actually check and see if there's anything running on 1096. So that's going to open it's going to check it and then it's not there and it's going to turn it off. It's not even going to. The stack is not even going to care.
That is not the way that PLC's work. So you could open that port now, it's listening on 1096 now you have resource contention now that actually leads to later on can add to a memory leak or even other issues that where that PLC has gone down. And it's why you hear a lot of us say we don't want active scanning, when the truth is that there are ways to do active in these environments that's safe. Right. So you can't, you can't blanket statement that. But if you were to bring in something that just did sin floods, that's a really bad idea. Right? That's a really bad idea. But from an it, an it, you know, SOC analyst never done anything in ot. They're going in there thinking they're harmless. Right? They're going, and they're going to run a quick scan, you see what's in the environment, see what they can do to help. And they may have just taken down three devices. And it is, it is, it is ignorance. But it's not willful.
[00:36:31] Speaker A: Sure.
[00:36:31] Speaker B: By any means. Right. But it's something that I want to bring up because it's a horror story that I hear all the time that makes OT standoffish.
[00:36:40] Speaker A: Sure.
[00:36:40] Speaker B: When IT solutions are brought up. And it's not that we can't utilize these tools even if they are IT centric tools.
It's that we got to understand the environment first and the people that know that as the operator. So if you're, if you're an IT person, even at a large organization, you know, like the OG and E's, the SOCO's, the Dukes, you know, you want to go sit with your operators and your plant manager and understand the operation, understand how it works and then maybe even contact the vendor and talk to the vendor about how their systems work to get an understanding of how you can protect them. Because in some cases what we would do in an IT environment that would be very successful could be extremely harmful.
[00:37:22] Speaker A: Right.
Well, and it goes to in some of those large environments. I worked for Duke Energy, right. And I did, you know, very large OT program across three years. We touched, you know, 900 substations and, and stuff. We deployed technology and OT cybersecurity across, you know, a big swath of their environment. Generation to transmission, distribution, nuclear, kind of the whole, whole gamut.
But the Part of that was to do that successfully to get business buy in. Because we were pushing all this stuff down, we got so much resistance. I pushed for building a lab and it sounds silly, you know, building a lab, like it's expensive, it's hard. But we, you know, I just find, and I explained it and I got buy in and it became one of the more valuable things that we did during that engagement. And the reason for that is because we had a like, for like representation. So you mentioned Foxboro like. So we had a Foxboro control system, we had their Emerson control system, we had their GE Mark 6 turbo control system. We had the full stack of SEL switches that are in the substation, right. We had, you know, this, all the stuff that's in distribution, all the IEDs and all the, all the components that are there. We didn't have all of them, but we had a good representation of the hardware that exists. So what that became was two things. We used it as a staging floor. So once we, once we perfected the design, then everything went through that staging floor and was built and configured in that environment before it went out. So, you know, we weren't sending raw devices. We were actually able to consolidate and build it in a factory. And then once it went out, it had the labels, it was already pre configured out, the right IP address, it had, you know, all the stuff, it got put on a pallet and shipped to wherever it was going, substation, generation site, whatever, and it was installed. So, you know, I didn't need all of my intelligence at all of these locations. Most of my intelligence could be in a central location and then they could support remotely as. Because again, this was part of this was during COVID So we were, we were working remote so we were having to do this. But the second piece to this, there are really three. The second piece of this was we were able to test what you just talked about, like, hey, I want to bring in this product and I'm not going to name any product names particularly, but we can, we could test whatever system or whatever product or capability we wanted to and see directly at least to some, you know, 80% validation that it's not going to break the system. And I know what it's going to get, I know what good looks like, I know what to steer away from. I know that I can't ping this plc, I can't run nmap, I can't do these things, but can I go active? What does it respond with if I do this? And I could have different versions of software. And then the third piece that it did is it was a good training and walkthrough. So executives or IT people that had never been in a substation, you know. Yes, Duke Energy is a large company. Everybody that's working there is not an operator. Like I grew up working in power plants. Many of those people, they came out of college and they're IT people. They've never been in a power plant. Yeah, they work for Duke Energy. That doesn't mean they've ever been to a power plant. Maybe they went on a tour one time, but you know, they never worked in an outage. They've never worn the hard hat and steel toe boots and been there. Right. So. So seeing that and seeing the equipment, seeing a plc, seeing how everything connects together was extremely powerful and letting them understand why this is different. And you know, you can see the IT stuff and you can see the OT stuff. And even though, you know, a switch, an industrial switch and an IT switch underneath, they may have the same operating system, but they're different. And there's, there's obvious differences that you can see in the form factor and the power plug in and how they're mounted, you know, on a DIN as opposed to, you know, 19 inch rack. And all these different things are just a little different that you can tell. And, and they' they're, they're for reasons like all the equipment we put in, and you talked about it earlier, had to go into a place with no air conditioning in, you know, Georgia heat or Florida heat or Texas heat and, and be able to sustain. So we couldn't just take something, you know, that would go into a, you know, air conditioned space in a data center. Like most of the IT stuff goes. It had to be able to have no fans and be able to sustain when the air temperature is 145 degrees and then it's going to add its own heat on top of that. Like all these factors that you don't necessarily think about and when you see it in person, it's just like, oh, like it was like a light switch for some, for a lot of folks as they started seeing it and why it's so complex and why it's different and why it takes all the steps and processes and, and became that training environment. So it was hugely expensive.
But if you really look at it at the total cost of ownership of, if we, if we avoided one plant from our facility from tripping or causing an issue, it more than paid for it because let's say it was, I'm Not, I have no idea. I don't remember what the cost was. Let's say it was a million dollars. Let's throw a big number out there.
How quickly would it take to, to recover to have that be a cost? If I take a plant down for a day, it's going to be way more than a million dollars of lost revenue from all that type of stuff. Not counting if there's any safety or anything like that. Like it's very easy to understand how this is a value add and there's a lot of ways that you can do this without building a physical million dollar lab. Now there's, with technology today, you know, cloud ranges and all sorts of things like that. But still it's, it's a different value statement to make to help people understand that that environment, that's different.
[00:43:05] Speaker B: I completely agree and actually I'm going to give some call outs here, shout outs and I know I'm going to miss some folks, so I apologize if I miss you. But you know, I'll talk about some of the bigger folks that we see. Duke, OG and E and SOCO are all doing similar things and they share that information and I think sometimes that's forgotten. You know, they share those information with organizations and groups and associations, but they're also very happy to share it with other energy providers. And it's something that I see much more in energy than I do in some of the other verticals. I'm sure that things like that go on in oil and gas and I'm sure the things that like that go on in, you know, manufacturing, pharma, etc. But energy is becoming very good, good about it. And there's, there's some unsung heroes in there, which FirstEnergy is one of them. FirstEnergy does a very similar thing and they, they bring people in to that lab and they teach them what they do, right? It's an amazing, it's, it's amazing. And they will openly invite you in and share that with you. And the same can be said for Salt River Project here in, in Arizona. Very big on not only having that lab, understanding what's going on in that operational environment, but sharing that information so that folks that, you know, that co op that's got 12 people can actually come in and see how they might be able to stair step their way into being more resilient and protect the environment as well. And one thing that SRP does that I'm sure other people do, but it's something that I haven't seen Much elsewhere is two of the folks over at SRP actually go in and for years would work from a location, they would go to a power plant and that these are SOC guys would go and sit and work from that location and get to know the operators. And it built a trust over many years between the operators and the SOC to know that the SOC guys were truly interested in what was going on in the operation and dare to protect them them. And on the SOC side they learned what goes on in these environments all the way down to things that if you haven't been, you know, a little bit of crawling through these plants, you don't understand how forecasting happens or you talked about startup processes. If you haven't seen a startup process and phasing a generator into a grid.
[00:45:21] Speaker A: Right.
[00:45:23] Speaker B: It is an amazing thing to see. You can't just flip a switch that doesn't just happen.
[00:45:28] Speaker A: Right.
[00:45:28] Speaker B: Especially when you have things that are like combined combustion.
There's complexity to this. And I really love that these organizations are out there and why I'm naming them more than anything is so that co ops and municipalities reach out to these folks.
Honestly, I mean Duke, OG and E, SoCo, SRP, Nipa is in there. NextEnergy, FirstEnergy, there are a lot of folks that do really good work around this that will be happy to share their time and their knowledge with you.
[00:46:01] Speaker A: Absolutely.
Excel, Exelon, you know, all the big players are there, happy to reach out to me. You know, I built the lab for Duke and know the folks that run it. It's amazing. And I've done work at Excel and Exelon and Nextera and Southern company and a lot of these different environments. My hands have been in a lot of those places. And I'm not saying that to toot my horn. I'm saying I'm impressed with the work that they're doing in these spaces and it's really awesome to see. And you know, when, when you look at the critical, the 17 critical infrastructures, I do believe that, and I think I said this earlier, but that, you know, power generation, or at least the power grid environment, utility industry is, is probably the most advanced but still needs a lot of help. Right. And we all, it's, it's the thing with, with cyber and all of this is it's never done. It's like, you know, you go to the gym, you don't go to the gym once and say, okay, I did that, now I'm done, I don't have to do that anymore. It's like, you Constantly have to maintain that. And then, you know, you, you get older and you have a knee injury and you get, you know, all this different stuff thrown at you. It's going to be constant. Or, you know, like we, we talk about shooting all the time. I can't go to the range one time. Whether it's guns or, or golf or, or tennis or whatever your sport is, you have to maintain that. You know, there's one reason why I don't play golf. It's. It's because I don't want to spend the time that it's going to take to be decent at it consistently. I'd much rather go do that, you know, throwing lead down range, because to me, that's just more fun. But, you know, I used to play golf, but now I'd just rather go shoot.
[00:47:32] Speaker B: It's all these things are it. I always say, Marshall, it's a Marshall skill. It doesn't matter if it's baseball. It doesn't matter if you're using your brain and you're doing penetration testing. It's a Marshall skill. And if you don't flex it and dust it off. I mean, I talk about being technical, but I've been on the business side for a long time now, so I go back and I try to dust off those skills when I can, because it not only does that skill atrophy, new things come out, new ways of doing things come out. A different approach to working out, a different approach to your stance, a different approach to your golf swing, all of those things, you know, and end up changing and we can learn more. And it's. And if you're interested and you want to be involved in it, you have to stay involved in it.
[00:48:13] Speaker A: Yeah, absolutely. And it just goes to show, like, you know, we're the, the people that are successful and have built teams and, you know, we talk about how there's a resource problem and we don't have people that have the knowledge that we're looking for in OT cybersecurity. And there's not that many people that do this type of stuff. And. But I built a successful team of people that had zero experience in ot. They'd never been to a power plant before. They'd been an IT person. They'd never been done cyber like, they'd done these other things. And I'm like, I don't care. I need to build a team that I can. I know I can work with and I can trust. So, like, I brought a guy that had worked for me twice before. He was A Marine. He'd worked in, in the legal field and law firms and supported, you know, their technology. And I brought him in and when I, when I sent him the job wreck, because I didn't write it. Of course HR wrote the job requirement. He's like, I don't, I'm not applicable. Like, there's so many things on this list that I don't check the box on. I'm like, dude, I'm the one that's hiring this thing. I know you would be a good fit. Just freaking fill the thing out. And if you're listening, he knows who I'm talking to. I won't call him out by name, but he knows who he is. But he was the perfect person. But it wasn't in the job wreck. He didn't align with, with, with the things that HR signed up. I can, I've said for 100 years, like, it's, it's, it's more important that the people are, are somebody that is a go getter, that is a, that can, you know, deal with ambiguity, that can, can, you know, work well under pressure, that is a team player. Like those, those, those skills are things that are harder to teach, in my opinion. And if I, if I have somebody that I know I can work with, I'll teach you the technology.
We'll send you to training, we'll. We'll put you at the plant. Like, we'll figure those things out. Those things are easier to teach than it is the other. Right? If I can build a team of people that I know will work well and that will ask questions and have a questioning attitude and, you know, we'll approach problems and, you know, be inquisitive and driven.
I can do, I can conquer the world with that. Right? It doesn't matter. On the flip side, sometimes you get the smart guy that's super smart and thinks they're smarter than everybody else, but, but nobody wants to work with them. It's like, I don't need that guy.
I'd rather have somebody that's not nearly as capable as that person, because nobody wants to work with that person. Like, yes, you're smart. Congratulations. Nobody likes you.
And I can say that firsthand because I've been that guy in my past.
[00:50:40] Speaker B: I think, I think we all have, right? You got to be like the smartest guy in the room. And honestly, at this, you've hit the nail on the head. At this point in life, it's about staying home humble, but staying ambitious and being the dumbest guy in the room. I, I would rather be surrounded by people that know much more than me so that I can learn from them than, you know, being that, that all knowing person in the room. And I think you're absolutely right about OT in general. It's something that we can teach. Especially if you've already been in technology. Yes, it's a paradigm shift. If you're humble enough, you're ambitious enough. It's not a hard paradigm shift. This is not a black box, it's not magic. Right. It's just a different way of thinking. Our consequences are different and tend to be physical.
[00:51:22] Speaker A: Right. And as long as you're open to looking at a problem from a different perspective because ultimately the problems are the same. I have vulnerabilities, I need to patch, I need to have secure mode access, I need to have availability, I need to like all these things. But the answer is not always the same. And it, I'm just going to patch. As soon as patch Tuesday comes out, I just push all those patches down to all my systems and I reboot because I have a maintenance window on Friday night between midnight and 3am on Saturday morning. Like that's just my window. I think we all want our power to work on Friday night.
We don't, we, we're not rebooting the power plant to patch it. So we don't have an outage window like that. So yeah, I can't do that. So what else can I do that's not that.
[00:52:05] Speaker B: Oh yeah, and we could, we could have a, I think that we could have a whole another podcast honestly, just on, on vulnerabilities and risk quantification in itself and how difficult it is in, in an OT environment because it changes from location, location like substation to substation. I'm not talking company to company, correction to substation. You know, a threat, that's, that's a, Hey, I need to do this right now at this substation might be something you do next at the correct substation.
[00:52:30] Speaker A: And that, that's one of the things to your point and you're right, I think we could, I know we could, could have an hour longer, probably longer conversation about this and, but you know, one of the things I always, always preached as I was pushing this out is patching and vulnerabilities and I've talked about it at defcon on the stage and I've talked about it at multiple conferences and in person and to executives is you may not ever, you may not patch this thing until an outage in some place like nuclear power plant we talked about those are 18, 18 month refueling outages. It's the only time you take the plant down is when I'm refueling it. So the only time I'm going to do maintenance on that thing is every 18 months. So if there's something, if I just, you know, had a refueling outage last month, it's another 17 months before I'm getting back in there to make a change. Unless there's something critical, right?
[00:53:22] Speaker B: Yeah, absolutely. And you get into this, this thing too, where I think all of us in IT are trying to. Let's eliminate that vulnerability. Right. I want to eliminate this exposure. I want it done. But the fact is that in many of these environments it's about mitigation. How do I mitigate that risk to a point where it's an acceptable risk at this point? Because I don't have a way to eliminate it or I can't eliminate it for 18 months, et cetera.
[00:53:48] Speaker A: Yeah, well, and some prime examples of that are, you know, you walk into an, into a control room room. There's no passwords, machines are wide open. And why is that? Like, well, that sounds dangerous. Why would you do that? That's risky. Yeah, it is. And what is more risky if I'm at a nuclear power plant and the operator forgets his password or it takes him 10 more seconds to log into his machine and he can't control the reaction, and then you have a reaction that's out of control, or had you rather just make sure have other mitigating factors that to get in that room, you've passed so many barricades of security and validation that if you're in that room, you can sit down at the computer screen and do what you need to do. Because nobody's in that room that isn't authorized to be in that room. Like I've been in a nuclear power plant, I supported one. And you don't get in the control room by accident. Like, nobody wanders in.
You've gone through a missile door and through armed guards and scanners and all this stuff. If you're in that room, you deserve to be there. And that room is 247 monitored and it's got cameras and the people that are in there. Me, even if I have authorization to be in the control room, if I were to walk up and sit down at a computer screen without asking an operator if I could sit at their station, they are going to very quickly move me and, and escort me out. Right. Even, even as a badge person, I got in the room, I'm supposed to be there. They, they, there's, there's other mitigating factors that says Aaron is allowed to be in the room, but he is not an operator and he has no right to be at this computer screen. And you are 100 authorized to forcibly move him out of the way.
[00:55:28] Speaker B: Yeah, we've, I mean, we've come full circle. It, it is the physical and cyber and cyber side combining in these OT environments that you cannot think of risk in one dimension. You have to think of it in both.
[00:55:40] Speaker A: Right.
[00:55:40] Speaker B: And when you have the, the. And that's business as well. Like you were just saying, that's policy. That's training.
[00:55:45] Speaker A: Right, Right. Yeah.
[00:55:46] Speaker B: That's not even something that you would, you would define in a system somewhere. That's frankly, it's culture. Yeah, You've. You've actually, you know, instilled this sense of purpose and empowered people to actually go out and do those things. And it's incredibly important in our OT environments. It is not just cyber, it is physical. And we, you know, I think it was a horrible time. But another example of that is, you know, when we had all of those substations shot up.
[00:56:14] Speaker A: Right.
[00:56:15] Speaker B: I could chuck a chain over this defense of this substation and cause just as much damage as a cyber attack. So am I, if I'm physically able to get there.
There's a lot of things that come into play first that I personally worry about over someone spending the time to actually breach that location remotely, understanding what it is, doing all that reconnaissance, and then figuring out how to attack it.
So they have a lot of concerns that they need to balance and budget goes where the biggest hole is. If you don't have a fence around your substation, probably fence first.
[00:56:50] Speaker A: Well, and it just goes to show, like you need to be having. It all comes back to the. We're one team. Right. And the way I say this a lot of times is you're on the same team. We're in the same jersey. We have the same goal. We want to win the game. Winning the game obviously looks different for every team, but for our team, we need to understand what that looks like. And if I'm in a, if I'm on the OT team or the IT team or the, the, the operations team, we're all one team. We're all wearing the same jersey. We all have the same goal. So we should come together and talk about it like, you know, yes, offense has a plan and defense has a plan, but overall, the head coach is driving the whole thing. The CEO is the Head coach. Right. And, and when I'm, when I'm going to a place and I'm saying, hey, these are the risks that I'm concerned about. Secure remote access and vulnerabilities and old systems and, you know, blah, blah, blah, blah. I want to also not just force down my controls on how I think they should be fixed because I probably don't know all the, where all the dead bodies are hidden in this system. My, my question is always to those operators and the plant managers and the control system engineers and those guys. Like, if you were going to take this plant down and you were a bad actor or you were, you know, somebody that made a stupid mistake, not even a bad guy, just somebody that made a stupid mistake, how would you do it? And they'll tell you and like, okay, which is more likely? Is it somebody shooting a, a transformer with a, with a deer rifle from, you know, 100 yards away, or is it that they hack in and do, you know, all this other stuff? That doesn't mean we don't care about both, both risks. Which one is more likely? Because that's what it really comes down to. If I'm going to spend a dollar or an hour of time, where is it most valuable for me to focus that time? Is it, is it more likely that, that a bad actor is going to hack in remotely and go through all these things or, you know, a bad actor is going to get into the control room at a nuclear power plant and sit down at this computer and do something at the keyboard that's really unlikely. Like, if they, if they've gotten that far, you're having a bad day because they've taken out physical security stuff and they've, they've been shot at and like, they've, they've gone way far into your environment. Environment. Like there's all sorts of other things that are going on beyond just somebody accidentally being in this room sitting at this computer. So it's really amount around approaching the problem differently and making sure you have the right people that you can have ask the right questions, be willing to say, yes, I understand that your risk is a risk, but it's not as important as my risk. So it's not going to get done maybe ever and come back to the table with another way that we can solve this because we're not going to patch this thing or we're not gonna, and we're not gonna lock the machines or whatever the thing is that you're trying to push through.
[00:59:29] Speaker B: Absolutely. And you know, I had a Professor actually tell me this a long time ago, and it's something I actually speak with cyber professionals about all the time, which is whether you're headed into a discussion or debate. You should never start a discussion or a debate unless you're willing to change your mind.
[00:59:42] Speaker A: Right.
[00:59:42] Speaker B: And that's the truth. So always enter that conversation with the door open so that you can understand where they're coming, coming from and reach a middle point. If you can't change your mind, you're never. You're never going to meet anybody halfway, frankly. You're never even going to convince them of, you know, meeting you halfway.
[01:00:01] Speaker A: Right. Yeah. Ego is the enemy, 100%. So we've talked around a lot of topics today. So excited about that. But what in the next five to ten years, what's always asked folks this five to ten years, what's one thing that you see coming up over the horizon that maybe is concerning in cyber and maybe one thing that is exciting that you see coming up over the horizon?
[01:00:23] Speaker B: So I think concerning. And we've kind of danced around the topic in the discussion here is really around the skills gap. So we talk about this a lot. And I'm 50, 50 in here because we've got job wrecks that say I need to have, you know, 10 years of experience, a master's degree, a CISSP, and every other certification under the sun to be able to go and do a junior sock job.
[01:00:51] Speaker A: Right?
[01:00:51] Speaker B: That's not realistic. That's not a skills gap. That's. That's inappropriate expectation. Yeah, it's an expectation gap. But at the same time, the thing that I've noticed and you and I started off this conversation, you know, before the podcast, talking about some of our geekery, Right. And how we started is the things that I learned fixing, breaking, and building computers in the very early days of, I mean, weren't even really a career. In my teens were what became the foundation for building robust systems and security. And one of the things I see coming for security, folks, and we're already starting to feel it, is that we've rushed to fill that skills gap. And the way that we've rushed to fill that skills gap is a lot with tools and educational processes that do not go for the basics. So they don't start someone at level zero or level one and talk about how a system works and why it works and why that's important so that you understand how to protect it. They start at level five.
And that's where we get into situations where we're now creating a Different skill gap where someone will come in with a fantastic understanding of security theory, security methodology and compliance, but have no understanding how the business or frankly the system itself, just the computer works. And I see that coming to bite us, you know, in the future. And it's something that I'm trying to get ahead of by helping really push for some of those more fundamental things, we've made things more approachable, like coding.
[01:02:30] Speaker A: Right.
[01:02:30] Speaker B: You have scratch, you have blockly, you have things that you can do that you can teach your kids to code. But are you teaching them to code or are you teaching them to use colored blocks? Something we need to kind of get around. And I think there's ways for us to do that. And on the other side of this is frankly just the jumps in technology. I'm sure you hear this a lot on this podcast. AI is always very interesting to me. But even more so around cybersecurity is what AI is doing for hardware.
[01:02:57] Speaker A: Right.
[01:02:58] Speaker B: So we're, you know, we're seeing machines that are small, affordable and dedicated to doing these amazing computational models, which is more important to me than AI because those computational models are things that we can use to defend environments and they're doing it on a budget. And I mean the things that you can do on, in some of these environments, just running a graphics card, for example.
[01:03:23] Speaker A: Right.
[01:03:23] Speaker B: Blows, blows. Things that we were doing at a speed, that's incredible. Those things that we were doing just two years ago, out of the water and I see that exponentially growing now, of course that's a, that's double edged sword. So as our hardware gets better, right? The, the threat actors, hardware get better. But I think it's going to open up a different realm of how we look at things. Because no matter if we talk about AI, machine learning, etc, it all comes down to static and dynamic or algorithmic detection. That's, it's what we do. And until we can get really good about building models and understanding how people interact with the environment, it's going to be harder and harder for us to detect the unknown.
[01:04:01] Speaker A: Right.
[01:04:02] Speaker B: So I see a lot of really interesting things coming with that.
[01:04:05] Speaker A: Yeah. And that for me that's exciting because I've preached for a long time it's really hard to protect these environments at that level, to know what good, to know what bad, or to search for bad until I really truly understand what good looks like. And that's one of the things, that's a translation problem where in it, we know it really well because we know what RDP looks like and we know what a secure HTTPs looks like and we have so many examples of that data that I can know what good looks like and I can search for those bad things, right? So blacklists, I can do all those things and I know what bad packets look like. I know what bad know program behavior looks like in memory and all that kind of stuff and processor and something's out of, out of whack in an OT environment. We're so far behind, nobody's done that. And it's not like to your point, you said earlier Site A and site B in the same company that maybe were designed the same thing 20 years ago, they, they're not the same anymore. So I can't even compare site A and site B because there's so many different components, they're so vastly different. AI can really. I believe that's one of the things I'm most excited about with this is using AI for things like that to really truly understand and map out what good looks like. So once I truly understand what normal and good looks like, which is going to take a while with existing systems, but maybe AI can fast track that, then I can really start enhancing the okay now that I know what good looks like. Anything outside of these thresholds and parameters, those are things we need. Not necessarily bad, but they're things that we need to look for. They're the needle in the haystack or the needle in the stack of needles is more accurate that we can actually start focusing on and really expedite our detection and monitoring in these OT spaces. And you talked about that level one stuff. That's where the real meat and potatoes is going to come down to when we can get to that level and truly understand again, this is what good looks like at this substation.
Anything outside of this we need to pay attention to. Not again, not assuming that it's bad, but I'm just going to assume that it's not normal and I need to look at it as opposed to right now. If I send all that data, which is what happens, I send all that data, my sock analysts are going to be like, I don't know, is it supposed to do that? Is that good? Is it bad? I, I don't know what you're showing me here. Like it's just not sure what to do with my hands.
[01:06:23] Speaker B: It's. That's where act. It's where active blocking, for example and anomaly detection fall flat on their face. Because right in an environment where we're baselining right, I'm going to baseline this environment I'm going to understand what's going on. Hopefully you never ever see a safety system fire.
[01:06:38] Speaker A: Right.
[01:06:38] Speaker B: What happens when that safety system fires? Are you going to block that because it's an anomalous activity.
[01:06:44] Speaker A: Right.
[01:06:44] Speaker B: And this is where, as you said, as we get technologically more advanced and we get into AI and building models and understanding every possible command that safety system can send.
[01:06:52] Speaker A: Correct.
[01:06:52] Speaker B: We get into actually knowing what good is.
[01:06:55] Speaker A: Exactly. That's exciting. Well, cool, man. Hey, so it's your call to action. Like what do you. How do people get a hold of you, what do you want people to know about you, your company, all that kind of good stuff. Where are you going to be at those, those fun things?
[01:07:06] Speaker B: Ah. So the next event that I'm planning to attend, of course, is S4. Hopefully I'll see you there.
[01:07:10] Speaker A: Yep.
[01:07:12] Speaker B: As far as where to get a hold of me, emberot.com is the easiest way to go and grab, you know, go and grab some information about what I do and what my company does and also seek any contact that you'd like is there. And ember, ot, just as a quick overview, is a cybersecurity company that's focused on ot and it encompasses a lot of the things that we've actually been discussing where it is a sensor that can actually deploy all the way down to level one. And it provides information not only around cyber, but around OT detections as well. So we do that asset inventory. We understand how devices are talking to each other and we do detections in those particular environments. We do this passively and we do it at a form factor that allows us to be installed directly on switches or existing hardware to make things a little bit more easy and flexible for operators to actually get into the environment and be getting insights and actionable data off of.
[01:08:07] Speaker A: Yeah, that's exciting. And that's, that's one of the problems that we have in these spaces is we need all these tools and capabilities. Then I got to deploy another piece, piece a box and I got to make sure it fits in, you know, all the power and all the things that are the problem and the expense and support and all that type of stuff. If I can deploy stuff on existing equipment that makes it so much easier to deploy and speed and all that type of stuff and I start getting value really quickly from that. So that, that's exciting.
[01:08:30] Speaker B: I say this a lot and I really, really mean it. We need to meet operators where they are today because some of their environments are 25 years old, maybe even older, maybe older. Than where they are today and help them show us where we both need to go in the future.
[01:08:44] Speaker A: Correct.
[01:08:45] Speaker B: It's a joint. It's a joint exercise and that's what Ember OT's aim is, is to actually start helping them today and then for us to journey forward in the future.
[01:08:53] Speaker A: Awesome. We'll definitely put all that in the show. Notes, folks. If you, if you want more information, reach out to Jory. If you have an OT environment and you're looking for that, that level of capability, reach out. You know, obviously that, that there's opportunity for us to dive further into that at another time, but just reach out and they'll have more than enough information on your environment and all that kind of stuff. So thank you for your time today, Jori. I appreciate the time and digging into these fun topics that we love geeking out on and hopefully the audience enjoyed a little bit of that. Until next time, have a good day.
Thanks for joining us on Protect it all, where we explore the crossroads of IT and OT cybersecurity.
Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time.
