Episode Transcript
[00:00:00] Speaker A: You're listening to protect it all, where Aaron Crow expands the conversation beyond just ot delving into the interconnected worlds of it and OT cybersecurity.
Get ready for essential strategies and insights.
Here's your host, Aaron Crow.
All right, welcome to another show. Kyle, man, thank you so much for taking the time today. Why don't you introduce yourself, tell the audience who you are and about your background and your experience in this wonderful world of OT.
[00:00:33] Speaker B: Yeah. So, I'm Kyle Peters. I'm a senior consultant for intelligent buildings. And my thing is building systems. So HVAC systems, fire alarms, lighting, all those kinds of fun things that happen in buildings. And I used to be a programmer.
I got to sit down on an upturned five gallon bucket in a dusty closet with a laptop and program buildings every day.
And I got into it, I think, the way a lot of people get into it, which is purely by accident, I didn't even know what job I was taking when I took it. And it turns out there's this great existence way outside of what I knew, and I loved it, and I've been sticking with it ever since. And as I got through that programming thing, I started seeing little issues, network things and this it mystical it stuff. And I started realizing that maybe what I was doing wasn't the best thing. And so I just started learning, taking classes and night school stuff, and it was great. And here I am today.
[00:01:51] Speaker A: That's awesome. I talked to so many, including myself. Right. That has a similar path. Right. As we started out in one place. And step a leads to step b leads to step c. And you didn't know where you were going necessarily. You just were staying where you're interested and continuing to learn, which there's a lot of value in that, right. And it's one thing to learn about OT, cyber and cybersecurity and networking and all this kind of stuff in a book. There's a lot of things that we can teach in a class. There's a lot of things that we can teach from mentors and listening to podcasts like this, et cetera. There's a whole nother thing that. Some things you just have to experience, right? There's some things that the level of experience that you got sitting on that bucket and making those configurations is priceless, right. And there's a reason why people with that skill set and that understanding and those hard hats that they've worn and earned working in the field, there's a reason I have those up there, right? It's not a trophy. Those aren't fake.
I've worn those in a lot of different power plants and gotten a lot of coal dust and a lot of different things on those things over the years.
And that's a remembrance to. Yes, I'm a CTO now, but that is the reason why I'm sitting where I am, because of the experiences that I went through and I learned. And same thing for you. Right. And that's really powerful as new people are wanting to get into this space, whether it's ot or cybersecurity or to be that engineer. Right.
Start here and then just continue to make those moves up, and you never know where it can take you. So that's awesome, and I appreciate that journey.
[00:03:26] Speaker B: Yeah, it's been fun. It's been a lot of fun.
[00:03:29] Speaker A: Awesome. Well, hey, let's dig into. So you briefly hit them on building management, and I think we know, most of us know, or if you don't, there's ot stuff in everything from power plants to trains and cars, and you name it. But specifically, one that's not as frequently talked about, unless you're in that circle, is that building management. And we walk into an apartment complex or a hotel or an office building, and we get in, and we get in the elevator, and we click the button, and we have no idea how it works. We just go to the floor that it's supposed to be on, and voila. It was magic. Same thing with the HVAC controls and the door locks and all of those things, the fire alarms, the sprinkler suppression system. Fire suppression system, all those things to a layman, we just expect it to work, and we don't really understand what's behind the wall that kind of makes it all function and where there may be some older technologies or potentially vulnerabilities or all of that type of stuff. So why don't we talk a little bit about kind of overall what that building management stuff looks like?
[00:04:38] Speaker B: Yeah, like you said, it covers all kinds of things, and you triggered a lot of memories real quick there for me of old systems and whatnot. So I live in Colorado and Denver international airports, the big airport near here, and they were one of my clients at one point in time. And, man, just the diversity of stuff floating around there. And in one building, essentially, you've got equipment still from the original build in 1993, controllers, some of them actually still controlling from 1993 that we had to have, we called the pizza box, and it was this big interface about the size of a large pizza box, and you had to have your Windows 98 that was the last update the system received, allowed you to interface it with a Windows 98 machine. So, like, bare metal 98 box sitting on top of the pizza box, plugged in. And that stuff's still running. It's still there. It's still doing its thing. Conversely, right down the concourse there, there's a brand new JCI system or new tritium system or something. So there's all kinds of stuff floating around out there.
It's really cool to me how interactive all of it is and how interlaced with things. We had a local hospital here that lost what apparently is a boiler. They lost control of their boiler and it shut the whole hospital down for like a week. Right, because they lost their one boiler. And so these systems, while they're overlooked, they're important. They provide hot water for cleaning surgical instruments or providing heat, or just the hot water that you. The domestic hot water that you use to wash the dishes and such. So, yeah, it's pretty awesome. The diversity of things that I get to run into.
And old stuff, new stuff.
It's all over the place. It's super awesome.
[00:06:56] Speaker A: Yeah. And that's another really big difference that I really see in OT. And again, it's really universal across all ot, whatever the vertical is, is that diversity. Right. In an it world. You go to someplace and you look at their laptops, and they're all dell laptops. Maybe they're three or four different versions. Maybe they've got some IBMs or some whatever, different types, but they're all windows OS, or on the flip side, they're all Mac.
Or even if that, they may have some Macs and some windows and maybe even a few Linux here and there. But again, they're all within the last two to three years, right?
[00:07:36] Speaker B: Yeah. There's not a gateway device still sitting on a desktop anywhere in corporate it. Right.
[00:07:43] Speaker A: If they find it, they're going to shut it off.
[00:07:46] Speaker B: They're going to shoot it where it sits.
Yeah. And yet on the OT side of things, we run into that.
I had a major bank that I found a windows nt box sitting there, and they swore up and down it it didn't exist and had to pull up the picture and show them like, no, we found this during an onsite assessment. It was there.
The other thing, too. I see. That's interesting. And it's a disparity between the OT and the IT side, is most it people who install things are pretty knowledgeable about the IT world because that's all they do.
[00:08:24] Speaker A: Sure.
[00:08:25] Speaker B: The OT people, they know the OT stuff. They know how air handlers work. They know how boilers work. They know when to turn what on and off. And that's what I sat down and programmed. And they just talked to the IT guy, hey, where can I plug this in? And that's it.
That may be all they know, because they're not networking guys.
They're not just boiler guys, but they're boiler guys and air handler guys and whatnot. And so then we start running into the it just works mentality. It's been working stuff's been sitting there for 20 years. I need access to it. So don't mind me putting this cellular router that I'm plugging into my little network. I own this network, right? It's not my clients, it's mine now. And we find that all the time.
Ease of access. And I want to be able to program it from my car, where it's warm, because the building doesn't have heat yet, or the tenant, the owner, rather, doesn't know how to make programmatic changes, which is probably good.
And so I have to be able to do it from wherever I'm at in the world at a moment's notice. And so we put those shortcuts in place because it makes life easier, but it also makes it significantly less secure.
[00:09:52] Speaker A: Yeah.
The analogy I like to use, or comes to mind for it, is it's like a mechanic. You take your car, let's say that you've got an old car, and you're replacing the motor or whatever, and you take it to a mechanic, and he's not the ignition guy. He just wants to work on and troubleshoot the engine, the carburetor, whatever it is. Let's go really old school, right? He's going to bypass all of the electronics to just troubleshoot the thing that he's focused on, right? So once he bypasses all of that, he's going to hardwire to the starter directly, basically, either directly with a wire that he touches together and starts to turn it over, or maybe he's got a little trigger button, but he's bypassing all of those electronics, all of the fuses, all of that stuff, because it's a car you pulled out of a barn, and there's no idea how many rats nod on what. So he doesn't want to troubleshoot that stuff. He's just troubleshooting. Can I get the motor running? Right. The problem that I see in OT, and I'm sure that you have as well, is that happens, but then they leave it there because it's working. So I don't want to undo what he did, and I don't want to really troubleshoot that. So I'm just going to bypass it because it works. So let's just keep that running. Right. So then you have this band aid, you have this baling wire and duct tape solution. And it's not that the Ot people and I talk about this a lot. This is not to diminish their skill set or their knowledge. They are extremely intelligent folks and they're very good at what they do. But it's like expecting your mechanic to fix your plumbing at your house. It's just not their skill set. They're very capable if you give them the task inside of their skill set. But you can't expect them to also understand networking and cybersecurity and functionality. Operational side of that technology stack, which is why it takes somebody, and it's why, when I was leading this off with you, is why it's so valuable that you have an understanding of that operational side, making all of the stuff work, as well as this other skill set of the technology side. That's a really powerful thing to have in the middle, to be able to understand both worlds of that.
[00:11:49] Speaker B: Absolutely. You know, I wouldn't expect my boiler guy to understand my Cisco switch any more than I'd expect my Cisco guy to understand my York chiller boiler. Or like I'm. I feel like I'm a little bit of a jack of all trades, master of none in a way, because I do straddle that line to some degree where I have to know some of both sides of it. And I'm not a boiler specialist. I'm not a Cisco specialist, but I know enough about both of them to help you get it figured out and working together and hopefully secure in a way that's appropriate for the system that is being looked.
[00:12:35] Speaker A: The. Obviously, you've probably seen a lot of them, but what are some of the common things? I know we talked about this before, and maybe this is one that you can bring up as well, but what are either a. Some of the common things that you're seeing that users in this environment, whether it be buildings, commercial real estate, whatever, are struggling with or are unaware of as well as I know you talked about one. And we can talk about the token ring side or the ring technology and how technologies are just different in OT and it. And how it can be confusing to your it guy when you say, hey, yeah, there is a ring technology in here, and it's not old. I just installed it yesterday, and it's supposed to be there, and here's why walk through that.
[00:13:16] Speaker B: Yeah.
So the biggest issue that I see actually is complacency, is a thought that who cares?
And that's multifaceted, too, where we say, who would want to? And this was a great moment in my career. I was sitting down in a building with a couple of building operators, and they're like, who cares? Come on, what's the worst that happens? Somebody gets hacks into my system and starts jacking with my temperatures, and it's cold up on the office or hot up in an office or something. And then they proceeded to go on and on and on with all of the guys they knew within, like, a ten block radius of their building who had had their systems hacked, ransomed, et cetera, et cetera, and shut them down for days or weeks at a time. I'm like, well, I think you just answered your own question here. So what I try to advocate is doing a risk based approach to things where we say, all right, how critical is your system, and what's the likelihood of something happening? And we always like to focus on the cool, sexy stuff of the Russians coming in and tearing us down. And I got a friend of mine, he says the most likely thing to happen is a squirrel chewing through your wire.
Yeah. And we also postulated then from there that if we hear of a major outage and they say a squirrel ate the wire, they probably got hacked by the Russians. If they say, well, we're not sure, it might be the Russians, it was probably a squirrel that ate the wire.
Whichever one that a company goes with, they don't want to admit that squirrel at the wire. So, yeah, we like to look at the likelihood of something happening and the consequence of that system going out and then act accordingly from there and say, I've got one financial institution right now that they manipulate. I mean, it's their job, but they manipulate global currencies. And in my opinion, they need to be top tier. They need to be protected, because they are a target of everyone. They actually have to worry about the Russians.
The office building downtown probably doesn't need to worry about the Russians as much, although we've seen some weird things in the news lately.
So the complacency and just not even looking at it, that's a big one. And then there's the little things like the sticky note under the keyboard with the username and password for the workstation.
Password managers are free, and they're beautiful things like hardware tokens and MFA apps are wonderful if they can be used, although that leads into another one, I think, where use of like MFA and building automation systems, those things don't exist.
They're just not there yet to get into your tritium system. I haven't seen a version of that yet that has MFA.
[00:16:44] Speaker A: Right.
[00:16:44] Speaker B: So I hope that's where we're going. I hope that that's coming in future releases.
And if you don't have it, then you just come up with a compensating control to get around that.
[00:16:56] Speaker A: Yes, that's a good point.
We can't gold plate it. If I've got a dollar to spend and to implement the right architecture is going to take $10. I don't have $10, so I can't do that. So what can I do for a dollar, right? And sometimes that's where these OT organizations are, and it's better to start.
And what they don't need is some holier than thou consultant or advisor telling them, well, you need to spend $1,000. That's awesome. But I don't have $1,000, so what can I do now with what I have? And sometimes that may be having a disaster recovery plan. What happens if this thing gets attacked? Like, how am I going to recover to make sure that I'm not down for a day or a week or a month, that I can recover in hours? Maybe that's my plan, because that's all I can afford. And then I start thinking about budget. How do I get that included? There's a lot of steps that go along the way. And it's not always buying a product. It's not always ripping everything out and replacing that nt and that pizza box with something brand new, because at the end of the day, it's doing its job. There's other ways to protect these environments that aren't as sexy.
They don't all have yubikeys and all that kind of stuff that they can integrate. That doesn't mean you can't make them more secure.
[00:18:17] Speaker B: Yeah, I've got my yubikeys sitting over here in the safe my own self. I love those things.
Again.
I have a friend of mine who's a rather notorious hacker, and he shares some very interesting stories with me. But when I got started in this, I asked him, what do I need to do? How do I help people the best I can? He said, it's actually pretty easy. Don't be the bottom rung on the ladder, right? Because most attacks and even the accidents, the squirrel, the misconfiguration et cetera, happen to the bottom rung, the lowest hanging fruit.
And I think it makes me think of that meme that goes around. Oh, our server crashed. Oh, where are the backups on the server that crashed? Exactly like the littlest things. Back up your system once in a while. Back up the configuration for your 300 vav boxes you've got in your tower and store them on site and off network. Right. And test them out maybe once in a while. Those little things can really go a long way to helping, and like you said, put money where it needs to be. We utilize the 62443 standard with some modification because it's primarily for industrial controls, and that's not us, but it's pretty darn close to what we need. And I love their security level and the three aspects of that that you set a target, you say, this is my goal.
I want to be Russia proof.
Good luck with that, but I want to be top tier.
And then you say, well, what are you actually capable of? And then what have you achieved? So your capability is what your system is capable of without compensating controls if it's just configured. Right.
[00:20:16] Speaker A: Right.
[00:20:17] Speaker B: And then what have you done? Well, maybe you haven't done anything. You're misconfigured. And so you. Your target is a four, your capability is a three, and your achieved is a one.
All right, we can work with that. It's possible, though, that your achieved is higher because you've got other compensating controls surrounding that Nt box. It's so behind a zero trust architecture and firewalls and gateways and armed guards and whatnot out the wazoo, it might be okay.
Of course, that Nt box has probably also been on and running continuously for the last seven and a half years and has dust bunnies in it.
Don't move. Don't blow on it. One of those dust bunnies might get lodged in there somewhere.
[00:21:12] Speaker A: That's a great point. Right in it. We're terrified of the older technology. Before we started recording, we just talked about the iPhone has a new update today, and we've been seeing a lot of those recently. And Mac, even Os and Windows obviously, has always had a lot of updates.
That's the primary response plan to mitigating these vulnerabilities, right? It's patch, patch, patch, patch, patch. Like I'm patching every week. I've got an outage window on a Friday night.
I don't have an outage window at a hospital.
It does not matter when it is. I can't just shut it down.
[00:21:51] Speaker B: That's very true that you don't necessarily have planned downtime at a hospital.
You can't say, hey, all the sick people don't get sick this weekend. We got a planned outage. Right.
But on the other side, well not, but in addition to that is if you run a patch and you've got old software and suddenly it doesn't play nice with that new patch.
Having automatic updates may not work for some things and there's no repository.
I'm not a software guy really in the sense of being able to create this, but I've been trying to come up with some idea of having some interactivity where there's like a site that you can go to and say, hey guys, I ran the latest Windows update and it crashed my Siemens system.
[00:22:43] Speaker A: Right.
[00:22:44] Speaker B: So you might want to not do that and then have Siemens interact and say, oh hey, thanks for bringing that to our, we're, we're working on that, but that doesn't exist right now. And I'm not picking on Siemens. It's everybody, nobody's got that.
We, I worked for a company here in Colorado that had a team that did that and it was uh, they'd run their update and then run the software and see what happened and release it to us after that. But in this stuff it doesn't necessarily work that way and everything's not backwards compatible.
So yeah, you might shut down your Siemens tritium Delta whatever system just by running an update.
[00:23:29] Speaker A: Well, and that's where it goes again into Ot. And I've had this conversation for 15 years, I bet at least is it may be more risky for me to apply a patch than not, right?
[00:23:42] Speaker B: Absolutely.
[00:23:43] Speaker A: Risk that's there from whatever vulnerability is there is way lower than the risk and the likelihood and the probability of me applying this patch and it causing a problem and me not being able to recover it because it's that old nt machine or it's even, not even that old windows seven for that matter. There's any number or even just the software compatibility. Maybe it doesn't crash it, but maybe it stops functioning or whatever and I don't have the resources to respond and fix that in house. So we're going to maybe try to do a backup and recovery process. And if you've ever done that before, it's usually not a fun process anyway, much less on an older system offline. On Friday night at 03:00 in the morning, I have those battle scars. I still have nightmares about some of those. Sometimes sleeping under the conference room table at 03:00 in the morning because I was running a restore and I couldn't leave until it was done. And yeah, I mean the whole nine yards, right?
It's not that simple. So when it comes to availability, it's more important. So how else could I mitigate that control instead of putting that risk? So instead of that, maybe I add a firewall rule, maybe I turn off a service. Maybe there's 50,000 other ways that I can mitigate that. That is way less risky to human life. Which again, we're talking about this, we're talking about hospitals, we're talking about somebody being stuck in the elevator. We're talking about. I had one of our team members when we were at black hat this past year, got stuck in the elevator at one of the big hotels and he was stuck there for like 3 hours.
[00:25:20] Speaker B: I think I heard about that.
[00:25:21] Speaker A: Yeah, it's crazy, but it happens. So yes, his life wasn't at risk, but he was not fun. And he wasn't the only one in.
[00:25:32] Speaker B: So let's just hope that there was a good trip to the facilities before he got on that elevator.
[00:25:40] Speaker A: Yeah, I won't go too much into that, but luckily there was a Starbucks cup in the thing is what I heard.
[00:25:47] Speaker B: Oh, boy.
Yeah, those kinds of things too. I like policies.
I'm that guy. I'm that weirdo. In case anybody's wondering. I like the policy stuff. And being able to put into a policy an articulable statement that says this is how we handle these things. Basically just planning ahead. Again, it's not being the bottom rung on the ladder, it's planning ahead, knowing at some point we're going to have to upgrade this system. This windows 98 system is not going to run forever. Eventually the gateway computer with the cow print on the side is going to die.
[00:26:29] Speaker A: Right?
[00:26:29] Speaker B: And so what are we doing? Yeah, that's part of the planning ahead. The disaster recovery is being ready for it. Or some knucklehead like me jumps in there and just starts playing with things because I'm going to fix it. You had a problem. I'm going to fix the world. And you change one Mac address and suddenly the whole network goes down and what do you do?
Okay, I was playing with this device. I'm not exactly sure what happened. Can I restore that device? Because I pulled a backup before I started making changes. So having those policies and saying this is how we do things.
And the other side of that too, along back, kind of falling back to that security level target thing. Is qualifying. By setting your own internal standard, you now have something to measure your vendors up against and say, hey, can you meet our needs?
We need to be able to protect ourselves against nation state attackers.
Are you the guy to help us?
[00:27:35] Speaker A: Right.
[00:27:35] Speaker B: And if not, then we can maybe still work with you. But we're going to have to find somebody else to cover some other needs.
[00:27:44] Speaker A: And that's huge, too, right? A lot of these environments that we see, especially, I would assume, in the building automation side of things, working with. I've had a few folks on here, Lucian, with building cybersecurity and that whole organization, which is awesome. It's a great place to pull out some of those standards that they're really pushing to push through. But a lot of this is not about. I know I've said it before, but even today, it's not about a new technology. Sometimes it's more powerful to hire somebody like you to come in and say, I don't know what to do on this. We can't patch it. We can't just install something. But I also don't have anybody with the expertise to really dig in and figure out what that thing does. Like, I need you to look at it and play with it and figure it out so that we have a good recovery plan, that we understand how we're going to come back and that it's not the first time you've seen it. Right. So when you get your hands on it and all that. Okay, now I know what it is. So in six months, when it crashes, even beyond Russia, even beyond a cyber event, when that thing just for whatever reason, craps out because the power supply fails, because too many dust bunnies got in it, right, they don't know necessarily what to do and how to get it back up and it had a hard crash, the database needs to be recovered. There's little things like that. But because you've looked at it and you've had your hands on it, you understand what it's going to take. More likely than you would have if you'd have walked in cold, I'm sure you'd still figure it out, but it's going to take you more time. So little things like that are big first steps in companies that they can bring you in, do an assessment, take a look at what's there, and have an understanding and get some feedback of, hey, these are some things that you should probably think about. And if I had a dollar to spend, this is what I would do. And then if you had $10 to spend these are the other things I would do and so on and so forth, depending on where their budget is and what the criticality of their assets and their environments are.
[00:29:34] Speaker B: Yeah, absolutely. And not just where to spend that dollar right now, but maybe setting aside $0.10, knowing you've got that old box and it's going to go, let's start preparing for that. But you're exactly right.
Again, we'll fall back in the conversation here. The squirrel is the more likely event.
[00:29:56] Speaker A: Exactly.
[00:29:57] Speaker B: The dust bunnies are the more likely thing somebody walking through because you put your server on top of the desk where everybody works, and I spilled my coffee on top of it and shorted it out is significantly more likely than the Russians hacked into your because they were targeting you.
It's significantly more likely to have those little things. So, yeah, what can we do about that? Hey, we've got a closet right over here. I can just move that machine and get a longer cable. Cables aren't very expensive and make it work that way.
So, yeah, it's seeing those kinds of things on assessments and making those recommendations that are great.
As I walk around sites, which I love, I love walking around different sites and seeing what's there is finding the things and saying, hey, what's this? And they go, I don't know. What is that?
[00:30:57] Speaker A: You tell me.
[00:30:58] Speaker B: Yeah, it's your building, dude.
And I love that. It's so awesome to find those little things. It really makes going on site worth the while to find those little things and say, hey, you know what?
Talk afterwards and say, hey, remember that system? And you've got that panel over there. You can just do this and it'll help fix your system. Or the big bank that has a 16 port, like I've got up here on the shelf, the 16 port netgear, unmanaged hub, and everything's full. Every single socket on that is full. And saying, you know, these things are like $15 at best buy, right? And your whole system relies on that one.
So that one up there on the shelf was a tear out. I pulled that out of a building and it was full. And we brought 25 new vavs online and it shut down. So for those who don't know, a VaV is just a little air damper, allows air, changes the flow of air into like, an office or a conference room or something to temper that space. So they're pretty small and ubiquitous. They're everywhere.
And we brought 25 of those little guys on and that amount of traffic through three of these hubs up and down the building shut down the entire network, and I was an apprentice at the time, and that's when I learned the difference between a hub, a switch, and a managed switch.
It was a good day, and so I pulled it out, and I just had a switch. I just had a regular switch, and I plugged that in, and suddenly the whole building came back. But it's those little things, or like I said, somebody tripping over a cable, man, we can really do a lot of work off of some very basic things before we get to calling in the big, giant network guys. Not that there isn't a place for that, there's definitely a place for that. But the average building, it's pretty manageable if you try. Get a good policy, get a good expectation, get a good team of people who are serious about it, who understand that stuff happens and they want to be ready for it.
[00:33:35] Speaker A: Absolutely. And again, I know we keep harping on it, but sometimes the solutions can be simple.
You don't paint your car if your engine is blown. Right?
You don't need to buy a new firewall or replace all your network switches if the things that you have, you have no redundancy, you have no backup power. Like little things, your network cables running across the floor and people are tripping over it, and every other day it's getting frayed and frayed and frayed, or people are running their chairs over it, right. Little things like that that you and I will pick up on. Like, hey, if you just ran a cable up and over and nobody's stepping on it, it's not going to get pulled out of the all if you also wouldn't have it in the main room. I've actually seen this, I'm sure you may have as well, where the janitor is unplugging it to vacuum, and then they plug it back in at the end of the night. I've seen that happen. It's little things like that sometimes that are super important to understand how I can better make my environment more secure, more available, more responsive, and ensuring that these things are reliable into the future and understanding how these things function. And like you said, it's not because they liked that hub. They just didn't know the difference. So they didn't realize it was going to be a problem until it was. And then they were just like, what happened? They needed somebody there to understand. Yeah, the hub can't handle the traffic, so you need to have a switch so it segments those switches, the traffic, individual ports, et cetera, et cetera, and really understand that, and once you switch that out, oh, well, the problem goes away. It was a super simple and cheap fix, but if you hadn't been there, it would have been a lot more expensive. Your team hadn't been there. Right. It would have been a lot more expensive and troublesome issue because they wouldn't have understood necessarily how to fix.
[00:35:21] Speaker B: Absolutely. Yeah. Those little things, I think we can really do a lot. And I'll step right back to what I was just saying, and this is a security thing across the board.
You had my friend Eric on a couple of weeks ago, and Eric's a great physical security guy. I mean, he's one of the best. And the mindset is paramount over everything, where if you just say, stuff can happen, we're not Superman, we're not bulletproof here. Stuff can happen. So what are we going to do about it?
Do some tabletop. I love doing a tabletop exercise, and, like, right out of the gate just kills. Who ask the question, who knows the most? Of course, now I give it away, but who knows the most about this system? Because I'm going to need your help to run this tabletop. Oh, Joe over there, he knows the most about it. Cool. Joe, you just got hit by a bus on your way into work today.
[00:36:27] Speaker A: Yep.
[00:36:28] Speaker B: Now what are we going to do? Yeah, now what?
So just think through it again. Another great guy you've had on recently, Fred Gordy. Fred's been a great mentor to me over the years and still is. And he likes to tell the story of showing up at a site and the guy wasn't there. He was sick that day, and nobody even knew where to show Fred around to, to do the know, they didn't even know where stuff.
Um, so having that over reliance on one person or any kind of single point of, again, that's, that's easy.
You, if you get the right mindset and you play. And Fred also likes to, we like to play the what if games or try to be the bad guy kind of thing. And what can we do if we have free software off the Internet and a system that's sitting on the open Internet?
[00:37:35] Speaker A: Right.
[00:37:38] Speaker B: I probably shouldn't have, but I did it one time on site with a client. The vendor was there, and the vendor couldn't get into, we couldn't get into the server room.
And so we're waiting, waiting. We're just chitty chatting, kind of getting an idea what's happening. Somebody finally shows up with keys, opens it up, they're like, hey, we don't know. We're having some trouble getting into the server. Give us a couple of minutes. So I walk down to the air handler room, plug my laptop into the switch that's there, pull up some free software I downloaded off the Internet, and I had full control of their system before the vendor did.
[00:38:11] Speaker A: Right.
[00:38:12] Speaker B: And the clients stand right there with me going, crap, what did you just do? And I'm like, hey, you want to change some set points? You want to change some temperatures, because there's free software out there that you can change the output and the input, both what it is and what it looks like, right? So you can send it.
You're running a zero to ten volt signal. You can send it 10 volts, but make the computer show that it's only sending five.
[00:38:47] Speaker A: Right.
[00:38:48] Speaker B: And that stuff's free.
It's so easy if you get access, if you get access to the system, and particularly since COVID but it was happening before that too. It just got escalated. People want remote access, and now we seem to be having a harder and harder time finding capable people who can and want to do the work. And so companies are making do with fewer and fewer folks. They're just sit in your office and remote in team view, in, God help us, team view into your ten sites where you used to have to sit in one.
Now you're remoting into ten.
So those kinds of things, we try and identify those and say, that's not really best practice.
Let's try and do a little better than that.
[00:39:47] Speaker A: Yeah. And there are ways to secure those things. Right? There is ways that you can put in secure mode access. You can add tokens and you can add some kind of process, but it's definitely not plugging that thing into the Internet or even the corporate network. Right. You need to have some segmentation in those environments that I'm locking the door. Right. It's the same thing. I don't put a lock on my door expecting it to keep special forces or the SWAT team out. I lock my door to keep my neighbor out. Right. Any random person just walks up and tries to open the door, it's locked. Okay, I'll move on down the road. But if somebody really wants in, it's not going to take them that much time to get in. They can kick it in, they can drive a car through it, they can take a sledgehammer, they can drill the lock, they can pick the lock. They've got a window right here. Windows are not that hard to break, like, any number of ways that they can get in the house, but that doesn't mean I don't lock my door. It just means that I have other controls as well. I've got a security camera, and I've got a security system, and I've got a 357 and all those different layers of security. Right. That I add on. And it's the same thing in cybersecurity. It's not a one size fits all, and every house is different, and every environment is different, every building is different. So we have to look at them that way. So it's no different.
I see a lot of entities getting frustrated because they don't know where to start, or they have very limited budget and they don't have the expertise in house, and they get frustrated and they just don't do anything. Or they throw their hands in the air and say, well, we'll just wait until something bad happens. Which terrifies me, but I think there's a lot of organizations that are doing that.
[00:41:23] Speaker B: I don't know the numbers right off the top of my head. I'd have to look them up. But I did just see something this week that was a news article, I think it was on LinkedIn. I saw that the number of cybersecurity incidents, particularly for building systems and industrial controls, are going up higher than they've ever been. And the C suite folks are planning to spend less on cybersecurity as the risk is going, as the number of incidents is going up.
And that, to me, just doesn't even make sense. I mean, I get it. Stuff is expensive, but when I walk into a building and they have no plan, they have no policy, they rely on their it guy who set up a firewall to save the world and said, that's enough. We're not going to spend anymore.
Boy, it's just putting your head in the sand, in my opinion.
And I don't think that's a wise way to run. And again, we look back at what is risk and we look at the likelihood of something happening and the consequence of that thing happening.
And that should tell you, yeah, even if you've just got a law firm. So obviously, we're not generally dealing with oil refineries, nuclear power. I've never run a nuclear power plant.
Yeah, we get hospitals. One of my biggest clients is a healthcare system up in New York, and we're going to be going in there next month. And we got to get in at the crack of dawn because we got to get out before they start surgeries.
[00:43:12] Speaker A: Sure.
[00:43:12] Speaker B: And that's a big deal.
Somebody has to put off a surgery. That's. Even if it's not life threatening, it's just an inconvenience. But, man, I don't want to be the guy responsible for putting off somebody's broken bone or whatever is causing them pain or something. I don't want to mess with that.
So we have to look at these systems and say, if the boiler goes down, what happens? Well, we shut down the hospital, as it turns out, because we only have one.
[00:43:47] Speaker A: Right.
[00:43:48] Speaker B: And then how much money does that cost that your 50 year old boiler goes out when it's negative ten outside? What a shock. When the load is the highest on it. And I know you don't get that too much down in Austin, but. No, we do get that on occasion around here. And it just happened a couple of weeks ago. They had one boiler. Well, how much did that cost? Well, okay, so it's going to cost me $10,000 to put in a redundant boiler and blah, blah, blah, blah, blah. We can't afford that. Well, it went out and now they're out. They're not making any money for a week. Yeah, I don't know how much that is, but it's probably more than I made in that week.
And they have to get a new boiler, right? Preferably two, maybe a couple of high efficiencies.
[00:44:39] Speaker A: Right?
[00:44:42] Speaker B: So that planning ahead and having that mindset, it's paramount. It takes precedent over everything, in my opinion.
[00:44:50] Speaker A: Yeah. And I think the cyber industry in general, we haven't done. I'm just looping us both into it. I don't think we've done a good enough job of really translating that from. The sky's falling, China and Russia and Iran are coming to get us, blah, blah, blah, blah. Stuck snat. Right? Most people are. Whatever. Why would anybody want that? But when we translate that to real business risk and we have that conversation, right, this is not China attacking me. This is that switch going out and it bring down your entire environment. That's the controller for the bowler going down, and there's no replacement. So the bowler is good, but somebody's going to have to come in and manually run it or whatever that may look like when we start translating that to a business risk that they can understand. Hey, if this goes down, you're going to be down for three weeks and you don't have a replacement. You have a single point of failure at this place. It doesn't matter if China hacks it, if it squirrels, or if know trips over it and spills his coffee, it doesn't matter. The point is that this is a risk, and it's coming from the technology and the does. Again, it's not all bad actors. It's not all nation state attackers. It's not all malware. Sometimes those things will happen, too, but it's really just understanding and translating and making sure that everybody is calculating that business risk. Because I think a lot of the time they're just assuming, oh, well, that'll never happen to me. Right. It's that same conversation you talked about earlier. It's like, why would they want to attack us? Like, we're just a little commercial building or we're just a little hospital, regional hospital. Nobody cares about us. We're small enough. Nobody gives a crap. That's not the point. That's not the risk.
[00:46:26] Speaker B: So two things have come to my mind. But that one, first, you might be a target just because you were found, right?
You just got found on a showdown scan. And so you're a target because some joke MGM looks like it was some kid in his mom's basement, right, that cost them $8 million a day that they were down.
And he's just some joker that may not have even known what he was getting into. Right? I mean, I'd kind of hope at that point he knew what he was getting into.
But that stuff happens. And we've seen news articles here recently where there's groups in the Middle east that are attacking control systems simply because they were made in Israel, right? And so if you installed one of those control systems or some of those devices in your hospital, bank, school, whatever, you're going to be targeted just because of that. So it's not, who wants to go after my building stuff's going to a. It's a hard thing to manage and it's a hard thing to sell because this is my other point I was thinking of there was, it's really hard to put up a LinkedIn post of, are you prepared for Joe to trip over the cable to your server and shut down your system?
[00:47:58] Speaker A: It doesn't fear that nation attacker does, does it?
[00:48:02] Speaker B: Yeah. The Russians are out there. Are you people? That's what stirs up emotion in people. And it's also probably the least likely thing to happen, correct, Joe? Tripping over the cable and unplugging the server and shutting down your refinery.
That's significantly more likely.
[00:48:23] Speaker A: Absolutely.
[00:48:25] Speaker B: But again, it's not sexy.
It doesn't create that Fud response, that fear, uncertainty and doubt. You're just like, well, don't get me wrong, my favorite thing to do is to look around my own office here and be like, how would I evaluate my office? Because I got cables all over the place and the dogs walking through and unplugs my computer from the hub here.
That stuff has happened to me. I get it. But the criticality of what I'm doing here at home, in my home office, is significantly lower than in the refinery or the nuclear power plant or the hospital or even the university.
So, yeah, it's hard to sell people on the most likely things because it's not cool.
I'm no marketing genius. Yeah, go ahead.
[00:49:25] Speaker A: Yeah. And on the flip side, I think it's equally hard to sell them on the other because they think it's so irrational or they think it's so unlikely. Right? It's so unlikely that Russia or anybody else. But again, like you said, shodan and all these different things, it is likely. It's happening. And I think we are getting some traction because we're starting to see more and more of these ot attacks hitting the know MGM and colonial pipeline, know the Pennsylvania water district and all these different things that are hitting the news. And it's happening over and over and over again and again. It's not because any number of things other than they found it on showdown or there was a new vulnerability that allowed them in that wasn't there last week. And all this time. Well, I've got firewalls and I've got antivirus and, okay, but that only fixes the ones that are known, not the ones that aren't. Right.
[00:50:14] Speaker B: And when was the last time you updated your signatures?
[00:50:18] Speaker A: Correct?
[00:50:18] Speaker B: Because we just talked about running updates. Maybe you don't have automatic updates turned on. And should you, and at what level in your hierarchy should you be doing that? Correct again, that falls back, in my view of things. Anyway, that falls back to thinking ahead and setting up a policy that works with what you're doing and having a program.
I think 62443 calls it the CSMS, the cybersecurity management system. We call it the cybersecurity lifecycle management program that we try to implement for our clients and look at the life of a system and say, hey, things are going to change. Everything's changing every day.
I used to kind of poo poo the idea of focusing on ransomware, but man, it's just starting to happen so much now.
And systems are.
I was in awesome control room in a building, just building know HVAC and such. And these guys had these huge monitors up around the room and the biggest one right in the middle of the room had Harry Potter on.
You know, what are you going to got? They're watching Harry Potter in the middle of the mean. I get it. It's been a slow day. Fine, whatever. But they're doing it on the very computer. So that tells me all kinds of, I mean, that system is in one way or another open to the Internet, that they're getting Netflix or whatever it was on it. And not to mention, they're probably not paying as much attention as they should to their systems.
[00:52:11] Speaker A: That's a whole nother conversation.
[00:52:12] Speaker B: That's a whole other issue.
[00:52:14] Speaker A: Yeah.
[00:52:17] Speaker B: That stuff happens all the time. It happens so often that I see those kinds of things or guys checking their email.
I don't even plug my computer into client systems because I know what I do on a daily basis and I try to be good.
If I have a questionable link, I'm opening up a virtual machine and opening it in that and that kind of thing. But, man, stuff happens, and I don't want to be responsible for shutting down the sterile processing unit at the hospital. That's a bad day.
[00:52:52] Speaker A: Absolutely bad day. Well, hey, we talked about a lot here, and over the next five to ten years, what's one thing that maybe you see that you're excited about coming over the horizon in cyber and all of this that we talked about in the building management world, and maybe one thing that's concerning that you see that we need to probably take some action or change or could be a problem.
[00:53:16] Speaker B: I think they're kind of linked my two points there, my two sides of that coin.
I'm excited to see the technology that's coming down. There's some guys doing some cool stuff, and I don't even know necessarily what's coming next, but I'm excited to see, because I see where we've come in the last 20 years, and I want to see what's coming next.
OT particularly building systems moves notoriously slow.
We're right now operating at about 2005 level of what it had then. And so I'm hoping that that disparity between what OT has and what it has starts getting smaller. That's what I'm excited to see is our technology start catching up with the rest of the world.
What concerns me is the rest of the world is kicking in the warp drive.
I'm not a big AI guy, but the stuff I'm seeing with AI that's got possibility is kind of terrifying.
I don't know what's coming with that either. And as I speculate on what's going to come around in the next, man, I struggle even to say five years, it might be five weeks. What's coming?
Things are changing super fast on that realm, and it's awesome and it's terrifying at the same time.
And that's where I still fall back on the fundamentals.
You got to practice the fundamentals and be as prepared as you can and be as adaptable as you can.
[00:55:08] Speaker A: Yeah, absolutely. Yeah, I agree, man. There's a lot of fun.
I've been doing this for a long time and all these changes can be scary, but they're also exciting. The reason I'm employed and you're employed and we're all employed, they're cool problems to have, but they're still things that we need to be aware of and not put our head in the sand and say, oh, that'll never, AI will never impact ot or building management. How would that impact us? Let's not wait to find out. Let's start thinking about that without closed minds in it and start thinking about how to process that. This is awesome.
[00:55:44] Speaker B: Yeah. From an attacker standpoint, we've kind of beat this one to death. But, man, if anybody's really interested to see what is possible and what might be coming, look back in time. Read the sandworm book about what happened in Ukraine, which also, coincidentally, gives some very interesting insights to the current situation in Ukraine. But what happened there?
I forget the author, but it's called sandworm. Just an interesting study of how things can happen and how quickly things can go sideways with operational technology. It's pretty cool.
[00:56:24] Speaker A: Absolutely. Well, awesome, man. So is there any closing out comments or things you want people to take away or reach out or anything like that? Call to action type stuff?
[00:56:34] Speaker B: Yeah, feel free to. Anybody can reach out to me on LinkedIn.
That is my social media of choice, as it were. My social media of choice is stop by the house here and we'll go down to the range. But that's not always a good option.
[00:56:49] Speaker A: Right.
[00:56:52] Speaker B: So LinkedIn works and strike up a conversation. But I guess my big takeaway is make sure your head's not in the sand. Just get a good combat mindset about things and I think we can do some amazing work in the coming years here.
[00:57:12] Speaker A: Awesome. Well, definitely next time I'm in Colorado, I'll come take you up on that and we'll hit the race.
[00:57:17] Speaker B: Yeah, absolutely.
[00:57:19] Speaker A: Awesome. Well, hey, thank you, sir. Thank you for your time today. I really appreciate it and enjoyed the conversation and I look forward to hopefully next time.
I know it didn't work out at Defcon last time, but maybe next time we're at the same place at the same time, we'll be able to hook up.
[00:57:32] Speaker B: Absolutely. That'd be great.
[00:57:33] Speaker A: All right, man. Thank you.
[00:57:35] Speaker B: You bet. Take care.
[00:57:37] Speaker A: Thanks for joining us on protect it all, where we explore the crossroads of it and ot cybersecurity.
Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time you.
