Episode Transcript
[00:00:00] Speaker A: You're listening to protect it all, where Aaron Crow expands the conversation beyond just ot delving into the interconnected worlds of it and OT cybersecurity.
Get ready for essential strategies and insights.
Here's your host, Aaron Crow.
[00:00:18] Speaker B: Hey, thank you for joining me today.
I want to talk about a topic that I bring up sometimes on the podcast and, but I really wanted to bring, have a deep dive into it. And it's really about, we've all heard the term it ot convergence. It's called a lot of different things, but that it ot and where the differences are and how they're getting smaller or closer together or whatever you want to look at, but really want to talk about that and the struggles, the challenges as well as the successes in that itot convergence and why it's such a crucial conversation. It's why it keeps coming up. It's also one that people kind of avoid. Some folks avoid it just because it's kind of a buzzword, et cetera. Right. So I've dove into it before, but obviously there's a difference between it and ot the technology. The t is the same. The I is different. The o is different. The roles are different.
Those roles are probably the biggest difference specifically around this topic in that it's, it's really about, it's about your experience and your coverage. An it person is usually dealing, they're working in the corporate headquarters. They're dealing with traditional it things. They have maintenance windows on the weekends. They have outages. They're patching. Like, they're looking at things from a different lens. The problems that are having, like if the server goes down, it's a different impact than if a manufacturing line goes down or a power plant goes down or a wastewater treatment plant or a train stops running, there's a difference in those things.
Again, the technology is many times very similar. It's got routers and switches and firewalls and all those types of things. But obviously, there are some differences, PLC's, IoT, different types of devices that are controlling endpoints, automation, et cetera. But traditionally, certainly the it folks don't necessarily understand the struggles in the OT space.
So even though in my experience, many times the IT team is trying to do the right thing and they're trying to do good, some of the things that they may have tried to push or may want to push down into the OT space, patching gpos and active directory, just to name a couple. They don't necessarily understand the implications of that. Right. So one of the biggest issues I've seen and challenges I've seen in that it OT conversions has been lack of trust.
Where did that lack of trust come from? It just inherently, hey, I don't like anybody. I don't trust anybody because they do trust people. They inherently usually don't trust corporate. And it.
And the reason for that distrust is because of historical reasons. In the past, usually the corporate or IT organization has provided a service or tried to help, and that help didn't work out. It ended up hurting more than it helped. And they had to fight to either undo it or go around it or yank it out. Right.
My, one of the teams that I had when I was working as an asset owner in power was we always talked about, hey, I'm going to give everybody shirts and it's going to say, I'm from corporate or I'm from it, and I'm here to help as a joke, because they don't necessarily want your help. And the reason for that is not because they don't want help. It's because they don't trust that you can actually help them. They feel that even though you may be in that organization may be very technically capable, they're not necessarily able to provide value in the space because they don't understand it. And the impact of that historical mistrust has been outages. Right. You roll out a technology, prime example, I talked about gpos. You put active directory in, and there's a computer that's sitting in a control room that is providing screens for the operator to see real time awareness of the environment. And they drop that machine into a group policy that gets patched or it gets screensavers turned on and screens locked, etcetera, in an environment that's 24/7 is always operated, that doesn't sound like a big deal, but it is. When that operator doesn't know the login and it doesn't, you know, the screen that they're, they're watching because there's a procedure going on, just went to screensaver and they can't get it out. And they're in the middle of a process or a procedure that now they can't monitor because the screen that was up, you know, in the control room in a protected area is now locked because of a group policy change that it did. Right. So those are the pendulum swings drastically. Hey, I don't want anything from you because you can't even do basic stuff like this. And that that's a big thing to overcome.
Those failures are big and they happen. And the problem is that it traditionally again, I've worked in it as well. It's Monday through Friday, 08:00 to 05:00, 09:00 to 04:00, whatever the numbers are. But you're working traditional hours. You're off on the weekends. Maybe you have an on call schedule, but you're not there.
Those plants, those ot environments, many of them are 24/7 especially if you're looking at critical infrastructure like power utility or wastewater. You, you want your water and your electricity to work 24/7 those plants are running. So there's operators there 24/7 so if that thing breaks at 03:00 in the morning on a Saturday, they're the ones that get called. They're the ones that are on call to show up and try to get that thing fixed. And usually they don't even have the right person from it or know how to get a hold of them to fix the thing. So there's, there's some, some trust gap that has to be built so many times coming in as an outsider, it's really hard to overcome that lack of trust. They don't know me. They don't trust me. Sometimes I can bridge that gap just because I've worn those hard hats and I can speak their language and they can see, at least I understand why they are hesitant. But still, there's a difficulty. One of the plants that I started working at, and we onboarded, we actually purchased the plant, and there was a local person there, and I'm sure he's listening. He may or may not be listening to this. And if he is, he knows exactly who I'm talking about. He butted heads very a lot in the beginning, and it was because he didn't trust me. He didn't trust that I knew his environment and that I wasn't going to cause problems, that I wasn't going to make his environment less reliable and make his job difficult. Right. He was the one responsible for it, and he didn't have the trust that I was going to, that I knew how to help him without causing problems. So it took a while for me to build up that trust. So that gets to kind of the next section of the piece that I wanted to talk about is building that trust and collaboration. I've seen too many times, both as an asset owner going and trying to push things down to power plants, also as a consultant coming in. And again, these large scale programs, all good things, but at the end of the day, you have to get buy in from the, from the end user, from the plant manager, from the control system owner from the engine, from the operators. Right. If they don't own it, if they don't, if they don't get buy in on those things, they're going to have, they're going to push back and ultimately they're the ones responsible and have the authority to say, no, you're not going to do that in my place. Right. So how do you push that? Like of course you can go up the chain and try to escalate and force it, but that's, that's not the best way to do it. It's like forcing your kid to eat food by prying their mouth open and shoving it into their face. Like maybe it gets the job done, but it's not, they're not going to like it. They're, they're never going to learn to like it and they're got, they're constantly going to fight and it's going to take a lot of energy.
[00:08:29] Speaker A: Right.
[00:08:29] Speaker B: It's, it's not the right way. Um, so how do you build trust in these environments? The biggest thing I could recommend is going into these environments with an open eye, listen more than you talk. Um, go in with, with goals instead of prescriptions.
What do I mean by that? Right. Instead of saying, hey, I'm going to push active directory group policies and we're going to lock all these workstations and we're going to patch all these systems because these things are insecure and you've got Windows XP over there, so we need to replace that. And going in with the 8th grade or 12th grade teacher, english teacher, writing red all over your paper, walk in and say, hey, here are our concerns. How can we solve these problems together? What are ways the screen is unlocked? Our policy with corporate is that we don't have unlocked screens.
Let's talk through why that's not an issue here or how we can approach that problem. Well, in this particular environment, that's in an, in a control room that has multiple barriers of physical security to get into and it's got a person sitting at that chair, 24 7365, meaning that, yes, they're not locked, so, but at the same time, not just a random person can walk up and start typing on the keyboard and take over because there's a person at that keyboard at all times. So if a random person walks in the control room and tries to put hands on a keyboard, it's not going to go well.
They're going to get escorted. The fact that they got that far says a lot of things, but it's very unlikely that they're going to get to the point where they're actually at a computer and they're making changes without somebody noticing them and physically restraining them, especially if you're in Texas at a power plant. So I promise that'll happen.
But walk in with an open mind and being willing to listen. Right. The biggest difference, one of the big differences in it and ot is just how I approach the problem. The problems exist. I have secure. I need the secure mode access. I need to patch systems. I've got vulnerable systems. I've got, you know, legacy equipment. I've got all the same problems that I have in an IT space, but I don't always approach the problems or remediate the problems in the same way. So going to the plants, going to the environments and saying, hey, these are the concerns that we have. Maybe we did an assessment. Hey, these are the thing. The findings from, you know, the third party or internal audit that did the assessment. How can we approach these things and work together for a solution? Because if you work together for a solution, then implementing that solution, you're going to be more likely to be successful. You're going to be able to check off the boxes that you're looking to check. It just may look different than how you did it on the it side, and that's okay. Your policy for success doesn't have to be the same playbook you're playing. It's a different game. Like, you can't take a football playbook and take it to baseball and expect to win. It's just a different game. Doesn't mean they're not both athletes. It doesn't mean there are similarities. Like, they both need to be in shape. They need to be, you know, physically capable. Like, there's a lot of similarities, but there's also a lot of differences. Maybe baseball and football is a little extreme, but, you know, the point is that there's differences in those two things, right? So building that trust, and I've had a lot of success in that. And the main success I've had with that is by going in, proving who I am and kind of my experiences and coming with an open eye and willingness to listen, hear their concerns and be able to respond to their concerns with resolution. Right. This is how I've done it in the past. This is how we've approached this problem. And if I don't know, say I don't know, right. And don't be tied to a particular way to solve a problem. Like, you've got to do this or else, like, I'm going to force you to do these things because again, it's not going to work well. Like, you're going to end up losing or you're going to end up break it, you know, causing more strife in that relationship. That's going to cause. It'll be problematic down the road. Maybe you win this one, you win the battle, but you lose. Lose the war. Ultimately, you want to have an open dialogue with the. With these, you know, different groups so that it's not a one time problem. I'm not going in and rolling out technology and I'm wiping my hands and I'm done now check I've secured the environment. It's not. It's not that easy. I'm constantly going to have to go back in. They're going to do control system upgrades. We're going to need to upgrade and replace systems. They have new capabilities and technologies that are coming out, you know, end of life. Like, there's infinite number of things. And the more that you're close with those, those groups, the better that you're going to have long term to have secure by design in the future.
Systems that are already there, obviously, they weren't necessarily secured 40 years ago. Security wasn't even part of the architecture or the engineering in a system. But again, the closer that you build these relationships, the more that you're likely to have those things. Building those relationships, you know, what does that look like? Obviously, that's. It's really simple. It's building relationships just like you do in any way. Right. You know, you break bread, you buy lunch, you go have conversations, you go there without an agenda. I'm not trying to make friends with you so that I can force you to do something. It's like when somebody friends you on LinkedIn and then the very 1st, 5 seconds later, they're sending you a sales pitch. Just like, I don't want to connect with you for that. Right. If you're not actually looking to connect and build a relationship, you're just trying to sell me on something, and then you're going to go away as fast as you came in. I'm not interested in that. Right. And these connections shouldn't be that way either. At the end of the day, the way we should be looking at it as an organization is we're. We're different teams on the same or different players on the same team. Right? We're. We're all on the same ship. We're trying to go to the same goal. Yes. I'm wearing, I'm on offense and you're on defense, or, or I'm. I'm wearing blue and you're wearing red. I've got an I in front of my, my title and you've got an o in front of yours. None of those things matter ultimately, in that we need to be realizing we're going for the same goal and we want to do it in a way that everybody wins. At the end of the day, that's the goal. We want to win. We want to be secure. We want to make sure that the site is available, that the operators and engineers and all the things work in the way that they're supposed to, and you want to be invited back. Like, if you build those relationships, the funny thing is the next time the vendor comes in with an opportunity or an upgrade or a solution, they pick up the phone and say, hey, the vendors wanting to sell us these firewalls, but I know you guys have firewalls, like, what do you think about these? And you can be there to have dialogue and say, yeah, those are great, but what if we did these? What if we did this? No, you shouldn't do that because of these things, like here. Ask them these questions, or maybe even they start bringing you to the table. Ultimately, that's the best goal. The best goal in the world would be that you bring a representative from your firewall team, from your networking team, from your server team, from all those different environments to these meetings in the design conversations when you're, when you're troubleshooting. My team was successful. One of the things I was most proud of my team being successful with is, you know, we supported 45 something power plants across the state of Texas. And we would get calls from the operators, right, from the engineers, the control system owners, the plant managers. They would have an, they would have an issue and they weren't happy with the response or maybe support they were getting from the vendor. And they would call our, my team. And the reason they would call my team is because we built that trust and they knew, hey, you guys know about these environments, you know about the technology, the networking, all of these active directory, like, all those different systems. And those are some of the things that we're concerned about. Can you come and sit at this table and ask the right questions to the vendor to make sure that we're choosing the right options?
So again, I've, I say this a lot, and it's something I feel very passionate about. And that we move business in general. We move at the speed of trust.
And ultimately that just means that the faster that I can trust you, the faster we can move forward. Like if I don't trust you, I'm not going to move forward, right. It doesn't matter what you're selling. It could be the greatest thing in the world. But if I don't trust you, if I don't have that trust, then I'm not going to necessarily listen and understand it because I'm hesitant, because I don't have that trust and understanding in that relationship. Right. So the faster that we can build those things and grow those things, the better that you're going to be able to go forward. All that to say thank you for listening.
Definitely reach out. We're going to be at Black Hat and Defcon in Vegas in August, so definitely come reach out. Say hi. We'll have some, some swag, maybe some t shirts and stickers and all that kind of fun stuff. Come, come drop by. Morgan Franklin will be at the House of Blues during Black Hat, and I'll be there as well as at Defcon supporting the ICS Village, which I love being a part of. Defcon's a lot of fun. It's hacker, hacker summer camp. So be there with and having a lot of fun conversations and seeing all the interesting people and technologies that are there. So thanks a lot, and until next.
[00:17:47] Speaker A: Time, thanks for joining us on protect it all, where we explore the crossroads of it and OT cybersecurity.
Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time.
