From Plant Operator to OT Security: Stories of Failures and Breakthroughs

[00:00:00] Speaker A: So people can overreact. So it's really just, we've got some information, let's actually apply some common sense. And again, that risk conversation of what is the real risk and that's generally how I sort of tackle those things. [00:00:17] Speaker B: You're listening to Protect it all, where Aaron Crow expands the conversation beyond just OT delving into the interconnected worlds of IT and, and OT cybersecurity. Get ready for essential strategies and insights. Here's your host, Aaron Crow. Awesome. Hey, thank you for joining me on another episode of Protected all podcast today. I'm excited to have my guest Gavin on the call. So, Gavin, why don't you tell us who you are a little bit about your background and we'll roll into this thing, man. [00:00:50] Speaker A: Yeah, as you said, Gavin, thank you for having me. I am a former operator, plant operator. It's not code for vegan, it's just generally somebody works at a factory. Yes, I'm still using that joke. And then from there I became an industrial automation engineer. And then as a hobby I studied cyber security. So when I went to the uk, everything's just sort of like merged and then came back to New Zealand and now I'm just working for myself and been doing that for quite some time. And yeah, it's, I've got full origin story of how that all transcribed if you're, if you're keen. [00:01:26] Speaker B: Absolutely love to hear it. Let's dig in, man. [00:01:29] Speaker A: Well, it's. People tell me I should tell it more often because it's actually quite funny, but so I thought I'll give it a crack. So as I said, plant operator, you know, for those that don't know what that means is you're running around the factory, you sort of, you know, keeping the production going and looking at things and they're really, really good operators. They sort of look at the scatter screen and then they walk around the plant and go, yes, yes. That reading is what I was seeing at the computer. I didn't do that. I was a pretty terrible operator. I had the idea in my head that I was going into it and so instead of doing everything I was supposed to do, I'll be reading books, looking at the SCADA screen. Yeah, everything's still good. And then how do you build like a domain controller and how do you create a file server and, and what have you. And then I, I stumbled into cyber security as an option. That stage I was already like, you know, Microsoft certified professional. And you know, I was studying all this during my off time, off shift and I really. I really found it fascinating because the concept of cyber security was just like, brand new. This is like early 2000s, you know, like, it wasn't on anyone's real radar. And I just found that this particular book, what it did was if you had an SSH server, you could connect the host to itself and then fire that protocol down the SSH tunnel and pop up. I was like, why would you do that? And they're like, oh, hackers. If you're using HTTP and you have a username and password, because HTTPs wasn't that common back then, this encapsulates it. So then your tunnel is encrypted and they can't actually siphon that information off the network. And I thought that was just, you know, absolute magic. Ended up using that several times throughout my control system engineering career. Because sometimes you get by, sometimes you get security controls in place that inhibit your ability as an engineer to do a job. So, you know, being part of the problem. Once again, I executed that and did everything you weren't supposed to. So, yeah, at that stage, I was again studying it and helping out in the local community, doing remote access and small business stuff, getting their backups and what have you. But what happened was I was changing out a CIP pipe and a lot of people who probably don't know what that is, it stands for Clean in place. So the particular machine I was working on is a dryer and it dries stuff, but it's massive. It goes up several storeys and, you know, stairs are a nightmare. Didn't like lift. So I always walked up to. Didn't trust the lift. Built in, like 1968. So you saw, like, I don't know. So I'm changing out the pipe so we could CIP this thing, clean it and dry. Being a dryer, it gets really, really hot. So it's jacketed, meaning there's thermal protection, but it's still hot to touch. And I had. My boots were worn. Okay. So my PPE equipment basically wasn't up to the task and I ended up slipping. So that made me fall forward and immediately stuck my arm up like that and my head hit my arm and my arm went into the dryer. I got a nice burn mark down there. So this is like. No, because at that time they were doing a massive revamp on health and safety. You know, like, you know, you must do this. And here's the safety pyramid. And, you know, and you got to remember I was like a teenager in early 20s, so I. I was literally just did not care. I was like, what is this health and safety thing? Oh, just they're making far too much noise, you know. The worst kind of person you want as an operator taking this health and safety not seriously at all. But I blame my impetuous young self, you know, with different ideas. Have definitely changed my ways since then though. But so with this health and safety stuff, now I've got a burn mark. I've got to go down and fill out a form. And so we get to the form section and supervisors, what's happened? I burnt my arm. They're like, oh, okay, fill out the form. And it was right on the end of shift and I'm like, as night shift too, so nobody really is around except for the supervisor and a couple other operators. I'll look at the near miss form, one page and I look at the full health and safety form and I'm like, yeah, nah, screw that. I'm going to fill out the near miss form. So I filled out the near miss swarm and I said, you know, like, nearly hit head. So that makes it a near miss. Nearly hit head, got a minor burn, everything's all good. Wrote a few other things and then submitted it. Didn't even apply first aid. I just went home, sleep and then a couple shifts later and come back day shift this time. And it's like the health and safety officer wants to see. It's like, oh God, what does he want? You know, really wasn't enjoying this, this part of the process. So he pulls me into the office and he goes, oh, here, you had an accident. I was like, well, it was a near miss. I nearly hit my head. He goes, I see that in the form. He goes, can I, can I just clarify a few things with you? I was like, yeah, sure, sure. What do you want to know? And he goes, right, so you got here nearly hit head. That's why I assume you think it's an emus. I'm like, yep. He goes, so, burnt your arm. So, yep. He goes, I see there's no first aid record. Did you get first aid? I was like, no. And he goes, this last bit, I just want to make sure it's correct. I was like, okay, yep, ask away. And he goes, so you wrote this? Doesn't really matter anyway. Chicks dig scars. So my arm's okay and so is my head. Is that 100% correct? I'm like, yep, yep, that's 100% correct. And he was just like, right, let me show you the, you know, the safety pyramid. Near misses, minor injury Major injury, you know, all leads to a fatality. So you filling out a form wrong is completely incorrect. So I got, I got thrown through the wringer. And then at the end of it I'm like, oh, okay, I get this. Now I understand why this is important and what have you. Driving home from that shift, on the way home, I was sitting there thinking like, okay, it's health and safety thing. I definitely did the wrong approach around this thing and everybody's all, you know, regulations and what have you. And then I sort of went back to my cybersecurity stuff that I was currently studying and I was like, ah, cyber security is health and safety for computing devices. And it was just that light bulb moment. But that was like, yeah, 20 years ago I came to that. So I was like, yeah, cyber security is going to be a massive deal once people figure that out. And yeah, essentially I thought, I'm going to go down the cybersecurity route. That's what I'm going to go do. I'm not just going to do it, I'm going to go straight for cybersecurity. Well, back in like 2003, 2004, no one in New Zealand was doing cybersecurity. There were no firms, there were no things. So that was that career out the window. And fundamentally the maintenance manager tap me on the shoulder and say, do you want to work in the industrial automation department? And I was like, I have no idea what that is. He's like, oh, you've been using the HMIs, the PLCs, you know, the SCADA systems and they look after that. And I was like, oh, that sounds cool. And so became Industrial Automation Engineer, control system engineer. And that was essentially it. I went, did that for a career and eventually the cyber security stuff, because I already knew like a bit of pen testing and mapping and you know, like I said, it was a bit of a hobby when it came, when OT security became a thing. I happened to be in the UK and yeah, my career just took off then. So, you know, the moral of the story is that, you know, chicks do dix class because I'm married now. So obviously, yeah, cyber security, health, safety, same thing, but for computing devices. [00:09:46] Speaker B: Yeah, I mean it's, it's so funny that you say that again. So my, my, my background coming from, you know, operation side as well. I worked in IT and, but you know, I, I was, I was working for a power utility asset owner, you know, and I, and I started out in the technology side before cyber security was really anything. And I was doing. I was rolling out operator handhelds for operators just like yourself, where they would go do those rounds and they were digitized rounds and they would scan an RFID tag. We put rfe, RFID T, all the equipment. Because these were in coal fired power plants. You can't use barcodes because it's too dirty. So we had to use, you know, intrinsically safe things that, you know, were, were dust and all of the, all of the certification. Right. But we were having problems with the rounds not being accurate and not being used and not getting the benefit out of the rounds, the operator rounds. Right. And so they, one of the things in addition to upgrading the technology, because the operators were complaining about the technology. I was at a plant and I was just doing a walk down and one of the operators were showing me like doing, you know, walking me through around. So we had one of the handhelds and we were going through and you know, looking at all the spaces and all the tags were reading and we weren't having any issues. And then we stopped by the break room because it was, it was about lunchtime. So we were sitting there and we were eating lunch and there were, you know, 10 or 15 in America, we play dominoes. A lot of, a lot of the operators would do that. So the guys are sitting there eating lunch, shooting the, and playing dominoes. And I see one of them play a domino, lean back in his chair with it with a handheld in his hand. And he scanned an RFID tag on the back wall, entered some information and kept playing dominoes. And it caught my eye. I didn't say anything. I just sat there and watched him. He played for a few more minutes, he leaned back, scanned another RFID tag, entered some information and he did this for 30 minutes. So after he was done, of course I, I'm the administrator of the system. I log into the system and look at his round and at the information he put in. He was completing his, he had a tag. He had recreated every single RFID tag on his route and put a spare right behind him in the break room. So he never had to leave the break room to do his round. And when we asked him about it, very similar to when the health and safety person sat you down and asked you about it. We were talking about it and, and his response was, it doesn' said the exact thing you did. It doesn't matter anyways. Nobody looks at the data. I can put 9 million degrees and nobody says anything. It's just, you're just trying to make sure that I'm working. It's busy work. And then it clicked with me. It was like, wait, it's not a technology problem. It's. They don't see the value in it. So they're not doing it to, to their. Every one of those guys. And this was again, this is back in early 2000s. The good operators are the ones that could walk out and they saw reading, but they put their hand on the thing and they knew, hey, there's something wrong with this device because they've seen it, they've been doing it for 40 years, all that type of stuff. Right? But it goes into exactly. What you're saying is we have to understand and cyber is that thing. Like, I can't tell you how many conversations I've had with people in, in these spaces where they don't get it. So they find workarounds, like you said, right? I'm gonna. I'm gonna tunnel in, I'm gonna plug in a wire around the firewall because I need to do my job and you don't know what I have to do. So I'm just gonna tell you, okay, go away. And then I'm gonna find a way around it. [00:13:14] Speaker A: Oh, absolutely. The thing with operators is they've got so much spare time in their shift capacity, they can come up with very, very creative ways to get around stuff. Very creative. It reminds me of a guy called Arthur, right? He was in the factory when it was built in like 1958. And he would literally walk around and everyone's sort of scratching their heads going, why is this pipeline not working properly? Like, it's meant to be the product. And Arthur was like chain smoker and, you know, just like cigarette out of his mouth and just walks up with a hammer and he's like, bang, bang. And then the whole thing kicks off and you're like, how did you know that? I've been working here since I was 16 and the dude was like 60, you know, so it's that. That crazy knowledge of they just know exactly where, how exactly, just interconnects. Well, that's, that's the thing. Like, operators will. Will do stuff. If they, if they don't see the value, they'll definitely go the other way. And I was certainly a case for that. And I've seen it many times since. It's why I always put on the control room desk, I make sure that there is a powered USB plug so that they're not meant to have laptops, they're not meant to have phones, tablets, but the game's on this Sunday, and they're going to watch it. So they'll smuggler it in and just having that ability to charge the phone just takes it away. Otherwise, they're plugging into your OT service. And that's. [00:14:40] Speaker B: That's what's happening 100%. Well, and from, from a, from a technology perspective, it's really easy from the outside looking in. This is, you know, I've built teams and brought in, you know, IT people, cybersecurity people from the IT side of the business side. And, you know, they just don't seem to grasp the concept. Well, we'll just force them, we'll reboot them. Like, timeout, stop. Like, you're not going to do that. I'm not going to reboot a system that's in the control room, that an operator sitting at controlling a without his perspective, without his permission. I'm going to go ask him, is it okay if I use this machine like every single time? I can't just do things to them because again, they'll kick you out. And that's where, you know, one of the things that you said earlier, right? It's building those credentials and making them trust me. Right. The biggest thing that I have to do when I'm walking into a facility as a cybersecurity person in this place is make sure that they understand a. That's why I have hard hats. That's why I've. I wear ppe. It's why I've done all. And I don't just. I'm not playing, I'm not cosplaying. Like, I've worked plenty of outages and done all of these things. I've earned my stripes in that. I want them to understand, not to show off or not to, you know, you know, lift myself up, but more so that I make them feel comfortable that I've been in your shoes. I'm not going to do anything to you. I'm here to help you. And I'm. I'm actually the thing that's more on your side than the IT people. I'm gonna fight for you and make sure they don't break your ye. [00:16:05] Speaker A: Yeah, yeah. Eyes. That rapport building is huge. And having had that experience, you know, like you say the PPE and going in and you've had to do things, it really helps. I think the best situation that I've ever had was, you know, OT cybersecurity guys coming to do an audit, you know, consultant, contractor. So everybody's like, don't touch my systems. And then you like, lean over and they're having a particular problem. And I'm like, oh, your PLC is doing this. And also, your ladder logic's really terrible. You should have a code standard. Guys, come on. And they're like, I thought you were the OT cybersecurity guy. It's like, yeah, but I used to be a Charleston engineer. I have programmed that particular brand of PLC like, 50,000 times, you know, and then immediately they're like, oh, this guy's all right. He's seen stuff. So we. They're like, what do you want to know? [00:16:53] Speaker B: Right. [00:16:54] Speaker A: Yeah, that experience does help. [00:16:57] Speaker B: It's priceless, right? It's one of the reasons why, you know, I worked for a big, big, you know, big four consulting firms and. And did this. And it's a reason why they would send me into these places because, you know, you send some kid straight out of college that's never even seen an OT site or been to a power plant or manufacturing facility or whatever. The thing is, they can read that off of them from a mile away. From the shoes that they're wearing to the shoe. You know, the way they have their PPE on, you can just tell they don't fit. Like, it's. They're wearing a new suit that they've never put on before, and it's itchy, and they're not sure what to do in. Just exud them. I don't know what the hell I'm doing. [00:17:35] Speaker A: Yeah, yeah. They see, like, steam leaking from a pipe on site, and they're like, oh, is that. Is that dangerous? It's like, well, don't. Don't go near it. But, yeah, no, you're fine. You're fine. That's why we're in the walk path and not in the hazardous area. That. [00:17:47] Speaker B: Exactly. [00:17:48] Speaker A: You know, things like that. You know, what your way around. But I have seen that in both Australia, New Zealand, and the uk, where it's not just Big four, just general sort of like it mssps. They're like, oh, we're gonna. This OT thing's like a thing, and we're gonna do it, and we're gonna grab some graduates. And I'm like, that business model isn't gonna work. And you try and explain it to them, and they're like, oh, that sounds really expensive. We have to get, like, a really senior OT security guy to come in and do everything. But we just wanted the cookie cutter, like, send the graduate and tick, tick, tick, because it's so easy. It's all Windows xp, right? So it should be, should be easy. And you're like, no, it's not how it works, man. So yeah, I've seen that in the uk, NZ and Australia. Businesses trying to spool up an OT department not understanding it and trying to do that security audit and then ultimately fail and then try it two or three times and then I haven't seen, I've only seen ones that are successful are just pure ot. So it's very, very hard for organizations to have that. It can be done. It's just normally what they've done is they've finally understood the commitment and they've really gotten someone that is a senior that's been doing this like 10 plus years just ot cybersecurity straight, that knows like, knows NERC, SIP and knows 6, 2, 4, 3 and this 882 like off the back of their hand type thing that can lead the department. Yeah. [00:19:18] Speaker B: Yeah, absolutely man. It's exactly the same, right? Is, you know, and I've seen it everywhere. You know, I've, I've done some work in, you know, Scotland and Ireland and you know, in power utilities over there. And it's, it's the same thing. Like it's a universal truth of, you know, it and ot. And one of the things that you know, kind of got us connected was, you know, that, that IT and OT conversation and it really leads into this. What we're talking about here is, you know, I've done again as an asset owner, you know, when I was in charge of, you know, ot Cybersecurity 4 power utility in the states here, you know, my CISO hired one of the big four to come in and do an assessment on the it side. And of course the, the, the big four said, hey, we should do ot as well. Like because we're here and it's important. And you know, same thing you just said, right? So they sent these, you know, PwC type folks, big four consultant folks, you know, to power plants and they wanted to do 45 power plants across all of my, all of my environment. And I said the same thing, what you just said. You're not touching my stuff. Like you can tell me what you want to do and I'll do, I'll get you the data you're looking for, but you're not plugging in anything. You're gonna sit in that conference room over there. You tell me what you want, I'll go do it and I'll come back to you and I'll give you your data and that's what's going to be. And we did that and we were successful with it because I had the credibility and, and I had built a rapport with all those plants. Otherwise we wouldn't have gotten anything. Like they would have just said, there's the parking lot, see you later. [00:20:45] Speaker A: Yeah, yeah, that resistance is there. It's been well earned. You know, like anything sort of, as we said, you've got to have that sort of street cred or rapport to be able to go in and actually get that sort of stuff. But yeah, the original reason I contacted you was that colonial pipeline talk with the IT terms OT it. And it's kind of funny because I was sort of on the other side of the foot where I've always explained that like a domain controller in the OT space is ot. They're like, oh, but it's an IT equipment. It's like, well, yeah, it's an IT function, sure. Like a domain controller is a domain controller. He goes, but the difference is you're applying like different criteria to what you need to do. Like you don't build a domain controller in a brand new sort of forest tree and have IT take over the other one and completely DOS yourself. You know, that's, that's the sort of nuances that you have to sort of go through and you know, like network switches, you're like, oh, it's a network switch. I could, I can just update the firmware. And it's like, no, that's connected to a plc. If you do that, you have lots of visibility now you might be able to do that, but you've got to tell the operator says that they've got an outage for like 15 minutes and you better have a spare sitting ready with the config ready to go to Drac and Chuck in if you need to get things operational very, very quickly. And you know, again, those, those slight things. The IT approach versus OT is sort of like you call it ot even though it's a file server, even though it's a domain control, even though it's a firewall. It's OT because you're applying a different sort of rigor standard criteria to IT to make sure you think about the consequences of anything you do. And the other, other thing I try to do is it's specifically for policies, procedures. And that high level governance stuff is I generally call out, what is it? What is ot? What is industrial control? Industrial automation control systems like hmi. HMI can be Windows, it could be Windows Embedded, where it's firmware that gets updated. You're not probably going to patch that. So if you've got that distinct distinction that a level two device could be ot, as in full Windows or it could be an hmi, you've got to apply different criteria to that. Same with obviously a controller plc. Anyone who's updated a PLC firmware knows how interesting that could go. I always like to do it over a dodgy Internet connection with low battery on my laptop. Just to make it extra fun on. [00:23:22] Speaker B: A Friday evening also. [00:23:23] Speaker A: On a Friday evening. That's right. Yeah. Just to make sure. But yeah, so having those terminologies like it ot ICS and what constitutes and even mapping it to the Purdue model, and I know a lot of people are like Purdue models dead. Everybody seems to teach it. So having that sort of terminology seen as it's permeated across like it's within our space all the time people talk about it, it just helps facilitate the understanding and then you can actually have better conversations. So, yeah, I find that the terminology seems to always jump around. Doesn't matter what organization you go with. And the rule of thumb is you always adapt to the organization's way of calling it. But those terms are important. But ultimately your conclusion was it's all operational risk. And that's in your previous talk. And that's completely right. That's what I've seen as well. It doesn't matter if I mean the executive board or the company. Company does not care that it's OT specific risk. It's just risk. [00:24:29] Speaker B: Did it impact my site? [00:24:31] Speaker A: Yes, that's it. That's. That's all. [00:24:34] Speaker B: That was it. That was it the trash can, the parking lot? Was it the, you know, the, the janitor? I don't care what happened. How do we make sure it doesn't happen again and make sure that, you know, we can continue doing business? Are we safe to start back up and start producing whatever we produce again? [00:24:50] Speaker A: That's it. Yeah. And it's always interesting having people that don't grasp that concept where sometimes you just gotta look at them and go, what's your interest of saying this isn't an OT attack or an OT impact? And generally it's because they're trying to sell a report or write a blog post. I found if you just pull apart a few threads, you can actually find the motivational thread. Why they're saying it's OT or not ot. But yeah, end of the day it's operational risk. So the terms matter. But all we're doing with is facilitating risk. And if you go to a board and say, here's your risks. And they're like, we accept 90% of them. It's like, well, I've done my job. It's documented. They've accepted. All right, cool out. It's really painful as a consultant and like, people that, you know, you and I have both worked in these environments where you want to protect them, but when you've got an organization that's like, you know, we're good, it's like, well, what do you. What do you do? You can't do anything. You've informed them and they've accepted it, so have to move on. [00:25:56] Speaker B: I'm sure it's like a mechanic. I'm sure it's like a mechanic when you bring it, you know, when somebody brings in a car and, like, you know, the brakes are shot and their tires are bald, and, you know, they don't have any power steering fluid and. But they don't have any money either. They're like, all these things are broken. You really shouldn't drive it. And they're like, and I'm good. I'll take the risk. Okay, but I don't think you should. [00:26:20] Speaker A: But, you know, that's a good analogy. [00:26:22] Speaker B: It's. It's so. It's frustrating to me because, again, I've. I've worked in all. I've been an asset owner. I was a CTO of a software company and, you know, providing a product in this space. I've been a consultant at Big four, and, you know, now I'm at a different consulting company. So I've wore. I've been in all of those seats. And for me, I. I want to help people to understand where they're at. To your point, right? You look at the, at the frameworks, whether it's NIST, you know, CSF, whether it's, you know, 853, 882, you know, 62443. Like, all these different standards, it doesn't matter. And the way I've always looked at these standards is it is the language that we're going to use to be able to make sure that you and I are communicating the same thing. What is the risk? What is the definition? Where is the gray area? Who is responsible? Like, what is our. Because when I'm to your point, like, if I'm going to present these risks, and many times these, the boards or these plant managers or whomever, they're accepting the risk, I. I sometimes believe they don't really understand the risk that they're accepting, because if they did, it's kind of like the car analogy. I was just saying like you're going to lose a tire and you're going to die in a fiery ball of fire a mile down the road. You're not going to accept that if you have a baby in the backseat, you know, but if you're a 19 year old kid and you're, and you know, you full of piss and vinegar. Yeah, you're probably going to take that risk and drive on down the road. We all made stupid mistakes in our teens, right? [00:27:51] Speaker A: Yes, yes. Mine, mine is definitely my operator career, that stuff. [00:27:57] Speaker B: That's right. [00:27:58] Speaker A: I, yeah, the current allegy I've always used when trying to, you know, get my point across for, for me is generally it's like I, I'm the world's safest driver. And they're like, what are you talking about? It's like world's safest driver. I've never been in a car accident. Ergo world safest driver. And it's like, oh, well, there's other factors involved. It's like, oh, is there? And then they're like, oh, I see what you're trying to say. So there it helps, you know, generate the, I guess the cogs turning to get the conversation across and get the value but jumping back to like standards and frameworks and what have you. Like I generally like to do 6:2, 4:3 purely because people are like, okay, what happens if you leave? You're like uber consultant, you know, can do all these things. Former control system engineer, you perish, you move away. Whatever happens, like how do we know that what you've done is good or how do we continue it? It's like, well you can do the training like it's online. There's an official process and structure in place for you to pick up and go. So I know a lot of people, ironically enough, still use NIST control. So you do the 6 2, 4 for free process but you don't do free free. You actually end up doing NIST882 controls. And that's fine. All you're doing is mapping controls to security levels. And so as long as you know that's what you're doing, there's again that structure, that conversation, that terminology all comes in and everybody's now communicating effectively because they're all saying the same thing and it's warm beings, they know what you mean when you say stuff. So yeah, that's, that's, that's my take on it anyway. And that's, that's why I like 6, 2, 4, 3. Because it's just. [00:29:35] Speaker B: Well and it's, it's amazing because in ot, you know, we, we do things so differently. Again, coming from the operation side, we use phonetic Alphabet, right? We use three way communication, we use stop on unsure. Like we do all of these things because of the whole safety culture. So we do these things inherently. Our IT brethren don't necessarily do those things. Like they're not doing safety moments before they, you know, run a patch on a Windows server, right. They're not, they're not doing a job, a job safe briefing. They're not doing lockout tag out, you know, when they're, when they're unplugging a server, right. They're just not doing those things because the risks are different. Not saying we're better, I'm not saying that they should do all of those things. I'm just saying we look at things because coming from a plan environment, we look at things because people die. Like literally, no exaggeration, I've been on, on an outage when multiple people have died for different reasons doing an outage. They did all the safety stuff and they still passed away, right? It's awful. Nobody likes it and it's horrible to see those people didn't go home to their families. Right. These are dire consequences we're talking about. Like we're not, we're not talking just bits and bytes and email and you know, being can't get this, you know, Facebook or watch the football game. We're talking about somebody's father didn't come home. Like these are big, these are big deals that we're talking about. [00:30:56] Speaker A: Yeah, yeah, that's, that's the thing that the stakes are higher, if you will. And you know, people who have actually experienced a death on site knows that the, it's very eerie. Like the mood just tone and the mood just changes almost instantly. You know, it's very, very bleak. And like you say it could be someone's father, mother, brother, sister that's not going home that day. And yeah, so we do take that really seriously. Especially those that have trained in functional safety, machine safety. So I did machine safety and not as sexy as process safety, machine safety slight, put a gate around it, de energize stuff. [00:31:42] Speaker B: That's right. [00:31:42] Speaker A: Relatively simple stuff. But it is interesting talking to another sort of control system engineer, cyber security pen tester. So electrician engineer, got into cyber security, pen test consultant. Right. He said, like you always seem to say, he goes, and it seems to be like a functional safety thing. Everyone in that space seems to always bring up like safe, reliable operations or safe, reliable productivity. And it's like, yeah, well that's, that's what you need is a foundation. And he had never done machine safety or functional safety, so he never sort of. He's still safety conscious. But it is interesting when you go down that path of actually hazopping and jazz op and lopa going through that process, like really digging into what is the risk then coming from that sort of engineering side and then going, what are the risks from cybersecurity? I don't know. I, I guess you're more tuned to it because you've had that exposure and so you see the, the potentials for more hazards, I guess. But. Yeah, well, you know, have you. [00:32:52] Speaker B: Go ahead. [00:32:52] Speaker A: Sorry. Oh, no, I was just gonna say, have you. Have we. Did you ever do machine safety, functional safety at all or. [00:32:58] Speaker B: Oh, yes. Yeah, for sure. You know, and it's so funny because. Or not funny, but you know, we just look at things, it changes, it changed my perspective on absolutely everything. Right. Even going into, you know, I do assessments on, you know, enterprise environments and all that kind of thing. And I just, I look at things differently because of the, the things that I saw in that space. And I'm not just talking about the, you know, the deaths and that kind of stuff because obviously that didn't happen all that often, but, but still just the, the, the possible once you understand that that light switches. Like I remember seeing a safety video because again, I worked in power utility, so we're were working with these, these giant high voltage devices and we had to, you know, wear flat arc flash protector. And when, you know, they have these machines that would rack in these big relays and they would make us watch these safety videos because the, the equipment that you have put on, like it's like a full hazmat suit and it's big and bulky. It looks like something like the dogs, the dog people wear, you know, when the dogs are attacking them, like super protective in this big shield. But then when you see an arc flash and you see what happens, it, it's terrifying. And I'm not recommending you go look, but if you're interested, go Google that. It's disgusting and not something it's very fun to see, but it, it literally vaporizes people. Like nothing left. And it's terrifying. And it happens in a split second because they had a metal ring on their finger or their, their, their screwdriver, you know, crossed a bar or something. Like something happened and they're they're gone. Like there's no existence left. They're just no, like, burns, burns the skin. Like the, the, the. The. Their clothes onto them. Like, it's just really bad. So when you see that type of stuff, you know, you, you look at things differently that you're, you're approaching risk in a different way than something that is just, again, an email server going down. So when I'm looking at an assessment and, and I'm looking at it versus ot, which is why I get so passionate, so frustrated when say, oh, well, that wasn't even an OT attack. I don't care. Like, I don't care where it started. I don't care that it was an it. I care. If it can impact this site, then it's an OT attack. Plain, simple, right? It didn't, it didn't come in from ot. It wasn't focused in ot, but it impacted ot. It impacted my print process. It impacted the potential safety and viability of this facility. That's an issue. And the fact that you're arguing over whether it was IT or ot, you're missing the forest for the trees, buddy. [00:35:28] Speaker A: Yeah, no, I concur. The, the reality is, is if you've got operators that are not making productive material, whether that is clean water or treating water or electricity or an iPhone, then you're not producing. If you're not producing, that's an operational impact. So, yeah, speaking of, you know, the, the arc flash and stuff, you know, ring, ring hands. And this is my. Yeah, my wife's okay with it, but. [00:35:56] Speaker B: It'S just mind silicone. [00:35:58] Speaker A: Ah, there you go. It's just one of those things where you've seen it like injuries happen in the workplace and you're like, yeah, no, it's not worth, not worth the risk of. Yeah. Interestingly enough, the arc flash suit, I was still an engineer when that sort of really came through in New Zealand. And the movie Hurt Locker. Yes, at the exact same. [00:36:21] Speaker B: Yeah, it looks like that. [00:36:22] Speaker A: Yeah. So we always refer to it as the Hurt Locker suit, you know, so. And suit up, because I think I've got. Forgot the TV show that. That was off. But suit up meant, you know, Hurt Locker suit. Yeah, yeah, yeah. [00:36:38] Speaker B: It really gets back to when you're looking at these frameworks and you're looking at all these things. For me, you know, when people are talking about it doesn't matter if you're wastewater, if you're a small manufacturing, big manufacturing, electricity, it doesn't matter. You're all doing something and it's using, you know, OT as physical processes. It's really about defining it. And unfortunately what I see too many times is when I walk into these places, there's a lot of gray area, there's a lot of assumptions and there's a lot of unknowns. It has a good understanding of their stuff to a certain point and then OT operations has an understanding of their stuff to a certain point. Usually there's a lot of gray in the middle that nobody really knows who owns and who supports and, and whose responsibility it is. And sometimes it's it. You mentioned it earlier, it supports the switches, they support the firewalls, they support the Active Directory, maybe, you know, some of those types of things. Those are IT type systems. But we've got to get out of the mindset of. To your point and what I talked about before on that other example in my other episode, Active Directory, if it is serving a function in ot, it is an OT system every single time. It's why I don't commingle Active Directory with IT and OT systems. It's just I don't do that. Firewalls, same thing, switches, same thing. VMware backups, you name it. It doesn't matter if it's an IT type product, it matters what it's doing, what function is it serving. And is it serving an OT function, an OT system, then it then in it becomes part of that. [00:38:09] Speaker A: Yep, yep, Completely agree. Where I've seen, you know, definitely, if you can, obviously Active Directory separation, where you have like it like looking after all the things. Generally what happens in that scenario is it's an education piece and awareness piece saying, you know, stop updating those network switches, you're going to upset operations. And you know, and that's why IT department tends to have a bad reputation from that. But there are a lot of places. And one of the fortunate things was, you know, I've come from water and waste and manufacturing and food and beverage and what have you. And I've also was fortunate enough when I got back to New Zealand to go into oil and gas, one of the things that really struck me was like, wow, you guys have budgets for firewalls like genuine. It's a line item, you know, I was like, I have to fight too for now just to get a firewall, you know, like a water treatment facility. And the parallels are just so worlds apart. So I've really been fortunate enough to learn what you can do with less. And that can also help generate value a lot sooner. The reason I bring that up is there's a lot of factories out there. There's a lot of operations, if you will, where you don't even have engineers. You've got industrial electricians and instrument techs and they are looking after the equipment. They can't even get an OT guy. And so yes, it is facilitating the OT Active Directory and the OT network switches. And realistically it's just trying to give them the best tool set and awareness so that they keep operations running. I mean, every assessment I've gone to, you come up with risk scenarios and it's like, okay, have you ever heard of Nessa Scan? It's like, oh boy, we have an ot. It has taken us down four times this year. And you're like, okay, well that's a risk scenario. Every time there's something like that or a firmware update, someone just updated the hypervisors and it's like, oh, but it's standard procedure. And once again, they haven't taken it into account. So in those really like places where you've got like electricians doing a really good job, but you know, they're doing controls even though they're not fully engineers and they're maintaining the, you know, electrical instrumentation, all that stuff, and they're expected to know tcp, IP networking and Active Directory and it's a lot. So making sure that IT guys can facilitate and help is a big deal. When you get to like oil and gas energy, generally there's a full OT team, so there's like engineers and then there's like an OT team up the, like helping with Active Directory and all those things. Firewall rules and, and you can afford switches and, and firewalls is great. [00:40:57] Speaker B: So, and tools and process and training and, and all that fun stuff. But, but to your point, like, for me it's always been, it's been, you know, training, right. And when, when I started this out, you know, again I was an asset owner and, and OT didn't exist. Like that term wasn't even a thing, especially OT cybersecurity. So I was recruiting people that had a skill set like mine, but there aren't very many people like me and you and you know, there are some of us out there, but there's not that many people that have an operations background came from working in manufacturing or in a plant environment or whatever that environment is, and also has the technology side of things too and kind of gets both of those worlds. So I was recruiting a lot from, you know, it had never been in, in a plant of any kind whatsoever and, but they had the technical chops. They were network admins or they were active directory or you know, all that kind of stuff. And then I was also recruiting from operators. Right. I would get operators from plants. This guy's been there. One of the, one of the guys that I, I hired one of the best, you know, became one of the best OT guys that I had. He'd been a plant operator and an INC technician and, you know, instrumentation and controls is. Was the title. And he'd been an Inc guy for 30 years and worked in all these power plants. So he knew every person, he knew every system, he knew every how all the control systems work. He programmed every plc. He'd done absolutely everything in all these spaces. So he got his instant credibility. And he also was a great, as we're running, hey, we want to do this, this and this. And he'd be like, this is the resistance you're going to get. And this is how we can, can talk to them and do it in a safe way they may be okay with. Right. And. And that was, that was priceless to us. So, so my, my pitch to anyone listening to this, if you have IT providing services to these things, either get an operations person on your team to help you with that or send your IT team to the sites that they're actually supporting and make them work in outage. And then they'll have a better understanding of why these guys are so hesitant for you to make changes at Friday night with your low Internet connection as you're upgrading the firmware. [00:43:04] Speaker A: Yep, yep. No, I, I completely agree with that. And in fact, I had one guy that, yeah. Same deal, cyber security. He's like, I've been doing this OT thing for a while, but I've now gone to this energy company. I've done a little bit of energy, but I don't know what to do. It's like, okay, there will be, you know, like a shutdown for a substation. Yeah. Make sure you're part of that team and learn how to bring a substation back online and into production. And I was like, the value you'll get from that would be outstanding. And yeah, he took that on board. And that's exactly what he did. And he's like, I've got such a better understanding of what goes on now and why, you know, and that's. That's the thing. But every time you grab like an IT guy and go, you're coming to the OT team. I've always noticed, like, that after a while their eyes just light up, you know, they're like oh man, this is cool. Like you know, you're explaining like a process. Like oh yeah, that machine. Yeah that feeds the retro encapsulator and retroencapulator and that kickstarts the flux capacitor and they're like oh man, that process is amazing. I've been stuck in it all this years and you had all this cool stuff sitting here. It's like, yeah, yeah. So yeah, I've seen that. And obviously massive uptick. I've noticed in the past three, four years, electricians, instrument techs, control system engineers wanting to do cyber security. It's actually jumping. And as you say, those guys are usually really, really effective, especially in their home turf. And it's definitely either option having both come in and have them teach themselves. So you've got the IT guide and the controls guy teaching controls. The IT guy. The IT guy teaching like networking and active directory really does help having that cross pollination of training. I suppose so yeah. [00:44:48] Speaker B: 100. I mean it's priceless. It's really the only way to do it right. And you know, I, I made different process. We did things differently in the OT side than we did in the IT side. I would always tell. In fact, I had an IT guy come to my team and he rebooted a machine at a power plant from four hours away and, and on a Friday afternoon. And, and my punishment was him. I told him like, get in your car, drive to that plant. He goes, well, it came back up. I'm like, get in the car, drive to that plant, walk up to that operator, explain to him what you did and why you did it. And you're gonna sit there and make sure everything is okay. Well, it's probably going to be fine. Yeah, but you're going to go make that because if you don't, we've burned a bridge and they're never going to let us touch their stuff again. Like you have to eat crow. It's not even, you know, a pond but you know, you have to go shut, put your head hat in your hand and say I'm sorry I did this to you instead of with you, you. And, and, and once I started kind of building that, that camaraderie with him and the team. To your point, the lights came on and he understood it. To your point. On, on the other side, the electrical, the, the, the electricians and the instrumentation guys and the operators. These guys are super capable. They're very technically smart. They're, they're managing these systems, they're programming these PLCs they're, they're doing networking. They may not even understand exactly how they make it work, work, but they can, they're very capable. They just don't have an IT background, no different than you or I. If we had no training and they throw us into an operation space, how would we expect to. To be successful? Right. It doesn't mean we're not capable of learning it. It's just not something we've experienced before. So use that to your advantage. We're all one team. And that's the biggest thing that I try to tell people is, is we've got to remember it's like a, it's like a husband and wife. If you're arguing about something, remember we're on the same team here. We're trying to get to the same place. We're trying to raise kids, we're trying to pay our bills. You know, we're trying to get along and have fun and, you know, build a cool life. We may have different directions or ideas on how we get there, but we're still going the same direction. So let's try to find a compromise and one that we can both agree with and go with that. Right? [00:46:56] Speaker A: Yeah, absolutely. Effective communication and reiterating that we're all on the same side and we're all trying to, you know, do the same thing. Even with, you know. So I worked for an OT network intrusion detection system company and it didn't matter what shirt you were wearing, where it was yellow, pink, green, blue, red. It's like, look, we're all competitors, but the end of the day we're trying to protect critical infrastructure so we can, we can just let that go and just get on with it. You know, it's like, yes, you won that client and we really wanted to win it, that, you know, we're here to help. [00:47:29] Speaker B: Exactly. [00:47:31] Speaker A: Yeah. It's just one of those things where I think the community can do quite well when they're all on the same page and everybody's working to the same objectives. But sometimes people can get a little emotional. As engineers, as operations, you can get a bit precious about your plant. Rightly so sometimes you just got to sort of like, let's just write what are you trying to achieve? Okay, I can help facilitate that. And that's, that's probably the, the best objectives you can. Outcome. Sorry, you can get from it. Yeah, so. [00:48:03] Speaker B: So we have very similar backgrounds and experiences and, and all the different things, obviously, and, and different ponds and. But, you know, we're, it's really no Different. I guarantee if I walked into a power plant in, in New Zealand or, or Scotland or anyplace else, they're pretty much the same. Their accent's a little different than my East Texas accent, but, you know, that's okay. But, you know, how do you differentiate and how do you. Because every vendor, every news article, every. Everything is like fear. You know, the sky's falling, Russia's gonna hack us. You know, all the FUD that comes out about all of this. My, you know, again, I was an asset owner. I was also a CTO selling a product. So obviously marketing and we're, we're trying to get you to convince you to get our thing. But how do we get through that and, and, and narrow those things down to. You can't say that, you know, to your, you use your analogy earlier and I've heard it a thousand times. Well, we've been running this power plant for 40 years. It's never been hacked. Why would it be hacked now? [00:49:03] Speaker A: Yeah, yeah, there's definitely. Well, there's a vested interest in bad actors, if you will, that want to. And you, there's. You got problems with like, as you say, fud, you know, fear, uncertainty and doubt being thrown at you constantly. And people get pretty, you know, thick skinned with it. They're like, we, we hear this every week. You know, what do we do? I think generally having a pragmatic approach and saying them to them, look, not everything's a risk, but some things are. And you do need to take this seriously. I remember it's very, very long ago now. 2017. Siso. Siso. So we call it siso here. I just found out that that's. [00:49:45] Speaker B: We call them sisos. That's weird. You probably call it a bonnet too, don't you? [00:49:50] Speaker A: Probably, yeah. It's a New Zealand thing. I always thought the rest of the world called them sisos and it's like, no, no, the rest of the world calls them cisos. You're the ones that are weird. It's like, oh, okay. So CISO was like, no, no, there's a, you know, vendor, OT vendor, their firewalls and switches. Like, there's like 20 CVEs read 999.99. You know, it's just doom and gloomy. He goes, you must patch. You must patch now. Like, this is, this is a problem because we, some of the firewalls are the gateway between it and ot. And I was like, I sent an email back saying, no, we don't have to patch. And here's why the we're not it. So we don't expose the management interface to just anyone. Like you actually have to be on plant. And it was one of those plants that didn't allow remote access. This was before COVID when you know, that was almost standard practice, standard operating procedure. Yeah, yeah, yep. So in order for you to actually access the firewalls and the network switches, the management interface, you had to get into site and then log in. If you got into the OT network or ICS network or whatever, it's not the CVEs that are going to be problem, it's the username admin, password admin that is everywhere. That's the risk. So no, we don't have to do the CVE patching and that's that again that lose bulletin CISO overreacts as you do and just having the credibility and the ability to just actually look at it. That's why you have that risk based approach to vulnerability management. I think dhs, that flowchart DHS did years ago, which is still great, you know that, that type of thing, it's just, okay, we've got these threat actors coming at us. It's like I heard people when Russia invaded Ukraine, they're like, right, Russia has invaded Ukraine. We are air gapping our OT environment. You're like, okay, that's, that's, that's a fair move, sure, but for how long? Like are you, are you pulling that cable just to go, all right, where are we? Like, do we need to look at something? Are we being targeted? Okay, but are you going to be air gapped for the next 50 years? What's the end goal here? So people can overreact. So it's really just, we've got some information, let's actually apply some common sense. And again, that risk conversation of what is the real risk? And that's generally how I sort of tackle those things. But as for the specific, oh, we're not a threat. It's like, well, my government and probably yours has been saying for quite some few decades now, it's like, yes, we are targets and yes, they are looking coming for these things. And much like the on the world's safest driver analogy, that generally helps get the ball rolling. What's your approach to the fading? [00:52:56] Speaker B: You know, same thing. Right. And I usually come back to, unfortunately, I don't believe that especially at the board of directors level, the CISO level, they truly understand the risk, especially in this OT space. I think they do a better job of understanding the, the IT space. Probably because most of those CISO type folks came from that business in that world. So they have a better way to grasp the risks and the understanding of, you know, active directory and exchange and cloud and all that kind of stuff, but they don't truly understand the OT space. So when someone, we're talking about risks in these spaces and we're talking about Windows XP and we're talking about, you know, unsecure protocols and admin admin and not patching and all this type of stuff, it just, it sends their spidey senses off because they're, it goes against everything, their entire playbook that they do in the, in the IT space. And they, they, they judge it based upon the same device in an IT space because it's, you know, in a DMZ that's accessible by the Internet. And I've got, you know, third party contractors that are logging into it and they don't understand. And again, it goes back to kind of like what I said before. The biggest, the best way I do this is I take people to those places like, okay, the only way you can get to this control system is if you're sitting in this room and there's only 10 people that have access to this room and they've all worked here for five plus years. They all know each other, they're on the same softball team together. So if you walk into this room and they don't know who you are and you sit down at that chair, they're gonna tackle you, no questions asked. I don't care who you are. I don't care if you're the ciso, if you're the CEO, you don't need to be sitting in that seat and you are not authorized to be there and they're gonna stop you. And it happens like it's just part of it, right? And, and they don't grasp that there's other mitigating factors that go into these things beyond just patch secure mode access and firewalls and all those things are good and we absolutely need to do all of those things. But it's not the same in an IT space as it is OT space. And once you understand that, and the only way to grasp that from my perspective or the best way, it's like, you know, me trying to explain to my 16 year old life lessons that I learned on why he doesn't need to do something or why he should do something, right. If he hasn't experienced it, it's just like, yeah, dad's old, he didn't know what the hell he's talking about, right. Versus when he's done it himself. And he, you know, he hits his head against the walls like, oh, that hurt. Maybe I shouldn't do that again. [00:55:23] Speaker A: Yep. Yeah. The conversations with the executives, they really don't care that, that, you know. No, no, this is OT risk and OT is different. It's just, you've got to, I find you just got to channel it to a sense where it's like, well, no, actually we've got a, we have compensating controls. You know, it's not just fix the problem. There are other mitigating circumstances. So let's look at credibility, let's look at target attractiveness. Let's look at exposure. And then, you know, and those words, they can generally pick up and go, okay, so it's like, well, as we go down, that sort of means that these things just drop off and that risk is actually quite low. And that generally has a more facilitating things. I previously tried to go, no, OT is different. In ot, you do this, this and this, and this is why. And it's just their eyes glaze over. They're like, no, no, it's a domain control, so it stuff. Do it. [00:56:15] Speaker B: Same thing, same, same. [00:56:19] Speaker A: Different, different countries, same, same problems. [00:56:22] Speaker B: It's universal. Like it really is. You know, I've talked to people on this. I had Peter Jackson from New Zealand, one of your Kiwi buddies, the other day on here. And you know, I talk to people, like I said Scottish Electric in, in London and, and in, in the Middle east. And, and you know, I had somebody from Saudi Arabia on the other day. And you know, it, it's, it's the same everywhere we go. This is a common use case. And, and part it really goes to the, the overall design. Like our systems in OT were designed to run for 40 years, right? We aren't upgrading them every year. We, we're not trying to get the new fangled technology. It's, it's, you know, it's, it's the same reason why ironically, you know, we buy, at least in the United States, I don't know how bad it is there. But you know, here, every time a new iPhone comes out, you have to get the new iPhone, right? It may not have any much difference. It looks the same. If you put two of them next to each other, you probably can't tell them a difference. Like maybe the camera's a little bit better, maybe it's got a little bit faster processor and it's another, you know, 500 bucks more expensive than last one, right? I gotta buy a new case, all that kind of stuff, right? But the four year old version of the iPhone is more than I really need. It works just fine. My kids use it, it takes great pictures, it does everything they needed to do. OT is more like that. Like it's more like my grandparents that lived through the Great Depression in the United States that they never threw anything away. They repaired stuff, they if it ain't broke, don't fix it. Right. I'm not going to replace it with a new iPhone because mine's working fine. Like I'm not going to rip out my control system and upgrade it from Windows XP to Windows 11. Yes, I can't patch Windows XP. But you know what, it's stable and it doesn't have to connect to the Internet and it doesn't have to have be on a cloud. And there's all those other problems that you bring in that yes, you're solving one problem and you're bringing in 12 more. And that's the thing that we sometimes miss in these, in these discussions is you're, you're, you're forgetting all the new problems you're adding. By trying to solve one, you're adding 12. And those 12 are way worse than the one because I can mitigate that other one a hell of a lot easier than I can those 12 that you just brought in and I can't do anything about. [00:58:28] Speaker A: Well that's, that's the old adage, you know, there are no solutions, only trade offs. And it's like you just, you're introducing new problems and you by solving others. It's just this cat and mouse sort of thing. And yeah, new technology coming in. It's like, yes, now I've solved for this. And like, you know, the grandparent analogy, this is sort of like, no, I'll just stick to my SMS text messaging. I can, I can message you. It works. And you understand that I'm going to just respond to a text or phone call or whatever and that's what I'm going to do. That's it. That's the right criteria. I mean like it's also seen industrial plants like on the old coax cable running at 1Mbps and people like, wow, that's really slow. And you're like, no, that's all it needs. So you can, that life cycle is just going to take a lot longer. I mean we haven't sold for that. I don't see us updating PLCs every five years. It brings us, it brings too many risks with it. [00:59:29] Speaker B: Well and again. But why? It's not the model that America, especially America does. Like we're this consumption capitalistic economy where we throw everything away, we buy something new. You know, I have to get onto my kids and my wife and myself because it's very easy to say, oh, I'll just go buy another one right. When, when, you know, again, 10 years ago we didn't have Amazon. I couldn't order something and have it show up same day. Sometimes where I live I can order something and literally that afternoon a freaking driver brings it to my house in a box. Right. Used to, you'd have to get in your car, you're driving. Especially if you live down the country, it could, that could be a, a two hour long thing just to drive into town, get something and come back. So you're really intentional about when you did those things. And if you could fix it without having to go to town or buy a new thing, that's what you're going to do. Right. That's reason why people had old trucks and they ran, they drove them until forever and they handed them down to their kids and their grandkids and all that kind of stuff. Like how many people have a car that they've lasted more than three years in the States. It doesn't happen very often anymore. I just bought my son, he just turned 16. We bought him a 2004 Lexus GX 470. So it's like a, you know, the Forerunner or whatever, but the Lexus version, it's got 272,000 miles on it. But that thing will run for 500, 600,000 miles if it's, if it's taken care of and it's in good shape. So. [01:00:52] Speaker A: Yeah, yeah. And it's like, it's, it's fit for purpose for, for him and you know, low cost to entry for him as well. [01:00:59] Speaker B: Exactly. Him, he saved up money, he worked all summer. He's a lifeguard and he saved up money, he bought that car. I mean, I had to help him a little bit because it was a little bit more expensive than he intended. But still, 90 of the money came from him and he worked for it. So he's super proud of it and it's a great vehicle for him. Right. But it's not a, you know, a brand new whatever. But at the same time, that car will probably outlast any brand new car we bought him if we were going to buy him a brand new thing because it probably would have Fallen apart in the next two years. [01:01:29] Speaker A: Yeah, my computer software updates over the year and you know, gets bricks and yeah, there's again like say there's some great benefits from it, but there's also risk involved. And you know, the good years of like 2004, like any mechanic could probably fix it. Well, they're gone. You have to get the old vendor plug in and yeah, so much like that in the industrial space they've tried to go down and box you in. So you have to do things their way otherwise, you know, you're non compliant and your warranty's gone. And it's, yes, it's whenever going to. [01:02:03] Speaker B: Get away from it that that's one of the big things that I always hear as well. And again, working in the plants, I understood it. If I make changes to those switches you talked about, you know, a lot of times the vendors will bring in, you know, let's say a Cisco Switch 2960, an old, old, old Switch, 2960s and, and those switches are still running in a lot of places. I've got some here in my, my lab actually. I can, It's a, it's a Cisco switch. The same thing you'd see in an IT world, in a business environment, in a data center. It's the same Cisco iOS. There's nothing proprietary about it, it's just Cisco. But if I change that configuration to something that the vendor doesn't support and it breaks it, the vendor won't support the process because you change the configuration. So they will, it's like, good luck. Like they're not going to not support it. They're not going to tell you that you, you can't use it, but they're going to say it's up to you to fix it because you changed our configuration. If you want to put it back to the, it's kind of like if you put, you know, you buy a brand new car and you throw a supercharger on it and you take it to the dealership because the engine blew, they're like, yeah, sorry buddy, that's you. [01:03:01] Speaker A: Yeah, yeah, it's, it's unfortunate but it's, that's, that's where we're at. And you know, there's, there's been, I've done assessments, I've gone, you know, hey, Honeywell, can I make these changes? And they're like, oh, we've got to go to Global. About four to eight months later they come back and they're like, yep, you can do those changes. We're happy with that it's like, oh, excellent, that took a while, but at least you got something. And then there's other vendors that are like, no, you cannot touch our switches and everything has to be completely flat. And also I found with these warranty contracts with particular vendors too, that the understanding can be quite confusing because you put in look at a new DCS system and won't name names, but put a new DCS system and new cyber security stuff. We've got logging, we've got decent remote access, we've got endpoint agents, and we're going to do application and allow listing. Okay, cool. This is all good. Come back eight months later and the control system engineer from the vendor is like, I can't get this thing to work. I'm like, what do you mean? He's like, I've had to rebuild this machine like six or seven times. And I'm like, well, what are you doing? And he's like, well, look, I update the screen, I update the image, but I'm also updating these files. I was like, have you updated the application allow listing server to say that those executables for the DCS are safe? He's like, what's Application allow listing? Like, so you take them over and then it just works again. Disconnect between the cyber security and maintaining the warranty. Some of the vendors are actually not really training their staff on the new stuff that they need to support as well. And having talked to those vendors, control system engineers, they're like, we're controls guys. We don't do that. We don't want to look after Active Directory. Give somebody else. But then the vendor company says, no, you're not allowed to look after Active Directory. We have to. So you're like stuck in limbo. [01:05:11] Speaker B: Yeah. And it goes back to that gray area where, you know, the systems engineers, the control vendors that they come out, they're controls people and maybe they have some experience or they got some training at the thing at Active Directory or switching, but usually they're running a script again. That's what happened in the. The. An example I gave last time about Active Directory. It was. It was a control system person and. And she didn't. She wasn't a technologist. She followed the script and when it didn't work, she did what the script told her to do and she broke everything again. She didn't. She wasn't doing it maliciously. It wasn't a bad actor. She just made a mistake. It was. Anybody could do it. I've made lots of mistakes. Like that. But I've also done it enough times. I knew why it was a problem and I was. And do that in a production environment. Right. But that, that's what happens in these spaces. And, and because they don't truly understand what they're doing, it's like, you know, you took it to your, you took your car to your, your plumber and you know, he's pretty handy. Maybe he can fix it, but he's not a mechanic. So if he may, if he puts in the wrong part or he breaks something, is it really a surprise when you took your car to your plumber? Like, take it to a mechanic? If you want it to be done right, it's going to be more expensive, but it's going to be worth it. [01:06:30] Speaker A: Yeah. Yes. And that's that biggest problem that we have, I think in industry is actually showing the value, like communicating the value. They're like, no, you can't just send your mechanic to plumber to go fix a car. You actually need the correct skill set. And trying to explain that to a business and show it in a business risk format in a way that you can get buy in and say, look, your business requirements say you need to accomplish X. We're facilitating that. Here's your strategy, here's your roadmap. And we're going to do that with this technology, with this process and with these people. It's very, very hard to get that structure in place and to show that value. I mean, we try, but generally, like most things, not even a cyber security thing, just trying to get like at a water treatment facility, having two OPC servers because, you know, redundancy is nice. And they're like, no, we don't want to spend the extra like 2,000. You're like, well, it's $2,000. We're not talking about a boatload of money here. And engineering effort's going to be the same. And they're like, no. And then they have a plant outage and the OPC server, you know, they can handle like 50 minutes, half an hour, an hour, two hours. But soon you're going to hit that time window where it's like, well, we don't know what our plant's actually doing. It's controlling, it's working. We've got no idea what it's doing. So we're going to have to hit that big old estop for the entire plant. And you're like, all this could have. [01:07:55] Speaker B: Been saved if you spent my $2,000. [01:07:57] Speaker A: Yep. And we're having that problem with cybersecurity as well, it's like, look the values here. But trying to get that through is still, I think, going to take time. That awareness hasn't permeated through. You know, we're slowly getting secured by design, which is nice as well. But yeah, I see it as the biggest problem we have to tackle is trying to demonstrate to business leaders why you do the things that you do. Regulation certainly helps. It really does help. I think it seems to be the biggest thing to get them on the radar. But aside from that, if you don't have it, like New Zealand doesn't have a lot of regulation around ot cybersecurity or critical infrastructure and you can see that it's definitely not a priority for them. [01:08:42] Speaker B: So it's hard. You know, again, I came from power utility and I think that's why, you know, 70 critical infrastructures in the United States and, and power utility is the most advanced and oil and gas is right behind it. Just because oil and gas, there are regulations there, but they also have the most funding. So you know, power utility just is that way because they have to be. And oil and gas is that way because they have the money and it's, it's a big enough risk that they don't want to go down because they lose huge amounts of money. So, so they put their money in it. But, but to your point, wastewater, they're gonna not choose a two thousand dollar option because maybe they don't have two thousand dollars. It's, it's, it's rolling that dice, hoping it doesn't happen. It's not changing your oil in your car. It's worked so far for a hundred thousand miles. I've never changed my oil. So I guess you don't have to change your role, right? [01:09:28] Speaker A: Yeah. [01:09:28] Speaker B: Until you do, that's, that's it. [01:09:31] Speaker A: It's always when there's an impact, suddenly there's this cash. [01:09:35] Speaker B: Yeah. [01:09:35] Speaker A: And that's, I remember a community event, one of our very first ones in New Zealand. And one gentleman, you know, he goes, look, this is all well and good for you guys because you've all been impacted, you've all had something happen. How do I, who's never had an impact, get ot security funding? And everyone just went, you know, crickets. It's like, well, until something happens, nobody really wants to deal with it. [01:10:01] Speaker B: So, so it sounds like we need to start up a new business that we go and this lightly attack people so they can have justification. And I'm completely kidding by the way, just to make sure Everybody hears this. I am kidding. I don't actually think that we should do that. But, you know, you're right. Like, it's always like, it's never happened to me. It's never happened to me. Why should I spend money on this? Like, I. Because. Because this money could go. You know, when I first started this, we were literally taking budget out of, you know, plant management, like plant maintenance during an outage. So they. They didn't do bowler tube, you know, replacement because they. I had to take $200,000 out of that budget. And they turned off one thing they were doing and gave me the money because they only had a finite amount. Like, it's not like they printed money on trees. [01:10:44] Speaker A: Yeah, a lot of. A lot of. Some of the stuff I put in was, you know, like, cheap. Cheap and effective. Upgrade a SCADA system. There's some spare servers and spare workstations. Slap the network card in there. Pfsense, you've now got firewall. There's remote access in there. We can lock down the ports. You can now access like Rockwell Automation.com to get your user manuals. Okay. It's not. Some people are like, oh no, you don't want Internet activity. It's like, well, giving the engineers stuff they need kind of helps. Yeah, it's like, again, risk conversation. It's like, well, we happy with that. So those types of things really help. And like logging service, there's free logging servers and NAS's and you just sort of like make do with what you got. So generally that's how I start when I do assessments. It's like, what are the tactical quick wins and what have you already got that's underutilized? Because we're forever hearing. Even in the IT thing, it's like, oh, we install crowdstrike everywhere. Everything's amazing. It's like anyone looking at it, anyone actually configured some of these things and they're like, oh, there's this one guy, Jeff, I think. And you're like, okay, good luck, Jeff. [01:11:53] Speaker B: Yeah. Hope Jeff doesn't get hit by a bus. [01:11:56] Speaker A: Yep, yep. So, yeah, there's plenty of opportunities to, you know, uplift ot cybersecurity using underutilized stuff, you know, like even turning on, like provided you a vendor lets you like Radius or Tacax on your network switches. You know, can. Can solve a lot of problems from. [01:12:15] Speaker B: A setting up logging, sending those logs from those switches. I can't tell you how many times. And that's the other piece to this that I say. And then we'll we'll kind of wrap this up. But I say this all the time. Like the cyber security thing. When I was pitching this in the beginning, I never pitched it as cyber security because nobody'd heard what it was, nobody knew what it was, nobody cared. I would pitch it to my plant managers of, I can get you more visibility, I can make your system more reliable. So one of the first sites we did, we turned on logging just like you said, and they had a fully redundant system. The control system had, you know, every switch was redundant. They had redundant past everything. They thought there was a switch that was sitting in the control room or in the, in the electronics room and it had basically powered itself down and nobody knew about it. Right. It was just sitting in a cabinet and it, it's so loud in there, you can't hear it. It was beeping and, but you can't. There's blinky lights and, and noise and all the things you can't usually have PPE in anyways. Like, there's just so much going on, you don't know it's there. We turned on logging and all this stuff and within five minutes we started getting this. It rose to the top really quickly. And we saw this, this, this switch was, was complaining, basically. We walked over to the switch and it was, it was, it had shut itself down for, for heat issues, like it wasn't able to cool itself. It was saying fan failure. So we walk up to the switch, we looked at the back of the switch and there was a zip tie that was in the, the, the fan shroud and was stopping the fan from being able to spin. I pulled the, the zip tie out, the fan, started spinning it, it booted up and, and now you had redundancy. So they, this was one of their core switches. It was one of a redundant pair. So they didn't have an outage because it was a redundant and the other one was doing the load. But if anything had happened to that other switch, they would have, they had a false sense of security because they thought they had a redundancy, but they didn't, didn't and they didn't know about it. And who knows how long it had been doing that. We found it in five minutes and fixed it and, and I went to the plant manager, I said, this is why we do it. We're doing this right. It's not for cyber security. Those things are good. I see those as bolt ons. If we can have these conversations of making these things more reliable, more resilient, more redundant, more, more Capable and available. And I can get more data to my engineers, I can make them more efficient, they can get their manuals online, like all that type of stuff. That's the conversation we'd have. And oh, by the way, I'll also make it more secure. [01:14:40] Speaker A: Yep. While you're there, it's just right. [01:14:42] Speaker B: It's a byproduct. [01:14:44] Speaker A: Well, yeah, very similar story. Not cable time and fan, but that's sort of that realizing, getting some logs, getting some visibility. I've, you know, I've been doing this like 8, 9, 10, 11. It was like, okay, we've got an OPC server. That OPC server has SNMP functionality. [01:15:03] Speaker B: Screwing up my problem. [01:15:09] Speaker A: Via snmp. Grab the switch details and see where we're at. And you're going to get more visibility from that and actually put it onto the SCADA screen. And soon the operators realized they could find the page and they're like, that's weird. That port's down. No wonder I'm having issues. And then just giving them that thing, the fault diagnosis just really helps. Then it became standard to have network switches with management interfaces. So you hold them for data and put that on the SCADA screen was just like, okay, well this is great. Again, not a cyber security thing. It was just like, give it, give. [01:15:43] Speaker B: It visibility because it's part of the system. It's part of what? Because if you take that network switch out. So prime example here, guys. And we're gonna. This is the, the meat for me, the meat of this conversation. That Cisco switch, that active directory server, the, the system cannot function without it. That the network, it goes across that network traffic, it goes across that route. So if I take those root switches out, the control system dies. Now, the, the. A lot of times the plant will continue to run because the PLCs are hardwired and all that stuff. It'll just continue to run. But what happens in that place when an operator can't control the system? They hit that big red button like you talked about and they punch it out because they can't control it. They can't see what's going on. So obviously they need to shut it down because it could, it could. Something bad could happen and we don't want that. Right? So looking at a Cisco switch, a Cisco switch is an OT device if it isn't an OT system. Active Directory is an OT system if it is serving an OT function like that is the moral of this story. So stop arguing about is it an OT attack or an IT attack? And look at what function it's serving because that's what matters. [01:16:52] Speaker A: That's it, that's, that's all it is. I completely agree with that. [01:16:57] Speaker B: So, so all this to say, let's wrap up with my, with my favorite question. I did prep you about it a little bit but you know, next five to 10 years, what's something coming over the horizon that's exciting and maybe something that's concerning that you see that that could be, you know, impacting us in the, in these, the things that we were talking about today or something completely off the shelf. Completely up to you. [01:17:19] Speaker A: One of the. It's both the concerning and the winning at the same time. I've been noticing for a while now OPC UA is finally getting some uptick. OPC UA and MQTT and a lot of vendors now they're like, no, no, you don't have the option of doing plain text. You by default. Now that you've installed an OPC UA server, you are doing a certificate. We've got one self signed or you can do your own, you know, so that's good in a sense that PLCs have OPC UA servers in them now. So it is no longer. Let's have a proprietary thing. And from a monitoring point of view, well, what connects if you imagine you had OPC UA is your visibility. It goes to your plant screens, it goes to the historian server. Cool. Well then what's going to use the native protocol for Modbus tcp? Let's say it's a Schneider Modicon. Okay, so Modbus TCP PLC to PLC comms. Yep, Engineering. Okay. So if you ever see an out of band Modbus TCP com, it now makes it easier to sort of diagnose because you've sort of got a separation of protocols there. So that's nice. And there's a lot more advanced stuff in OPC UA on a PLC that I just, I'm generally excited about some security features but it all hinders on the certificate stuff. And what scares me is I've seen this a few times now where they're like we'll go self sign because we don't want to do PKI properly in OT because it's a massive dependency. And it is, it really is. You've really got to get that down. Right. But then they just sort of go okay, cool, we've installed it. And you're like well what's the lifetime of the certificates? And they're like what? Surely it's like 20 years. It's like you don't know when was the first time you installed this, enable this technology? Oh, seven years ago. So it's like, well, it's a ticking time. [01:19:14] Speaker B: You don't know when it's going to go off. [01:19:16] Speaker A: It's basic asset management but I just see people not really sort of going into that level and going actually we need to take care of this. So otpki, very, very complex problem. A lot of infrastructure has to go in place to support that if you were to run it. But the disadvantages of going with like a self signed is that it's not being managed and therefore there's no automatic renewal. There's no, you know, like let's just, if it's compromised we're going to have to reissue all the certificates. There's a lot of downfalls as there is windfalls from it. So, so I'm glad that we're going down a vendor neutral path and people can talk on the same protocol. MQTT seems to be everywhere. Same with OPC ua, but at the same time there's a lot of complexity now going right back to the start. Well, not the very start but there are only trade offs, no solutions. [01:20:15] Speaker B: Right. [01:20:15] Speaker A: You solved for problem, but you've introduced some more. So anyway, that's the moral of that. [01:20:23] Speaker B: Story to me is that, and it gets back to every assessment I think I've ever done or every time I've walked into a place and trying to figure out what's going on. Most of the time it starts out with what do I have and understanding from an asset inventory, from a network architecture, from a system design, from how everything functions and do they have one control vendor or two or five or 12, or how does everything sit together. And until I really have that good understanding, I, I, I, it's hard for me to make a recommendation on what you need to do, you know. Yeah, I could tell you how many Windows XP machines need to be patched and how you need to update your firewall and any asshole can tell you that that's not the value that I bring. I, I don't want to talk, I'll talk to your IT guy. I'm going to talk to him for about five minutes, I'm going to shoot him out of the room and then I'm going to talk to your control system engineer and your ICS guys and your, and your, your, your, you know, your operators because that's where I really understand how the system works, works and which, which devices are critical in which devices if they go down, break the stuff because they know, they know which system. Don't touch, don't breathe on, don't reboot it. Or as soon as there's an issue, they run over to that machine because they know that's the thing that went down or they're looking at that screen. Yep, that port went down again. Let's go jiggle that cable because it'll come back online when we jiggle that cable. [01:21:40] Speaker A: Yep. Yes, I've definitely, I've definitely seen that. And, and yeah, there's been a few times where you've done an assessment and the operator or guy that's been there for ages, electrician, control system engineer, you're opening up panels and he's like, don't touch that. I said, well, why is that? It's like, well, no, I'll use a pen, I'll say for like, I know how to do cable tracer. He goes, no, no, I know. He goes, that thing's temperamental. If you just lightly tap it, it drops the entire rack and then the process as an upset, you know. Ah, okay, so you know that, that critical stuff, they just know how the plant and where the pain points are. And you can get a lot of value when you do those assessments for sure out of that. [01:22:24] Speaker B: All right, so, so last question. See if it's the same in, in over in your neck of the woods. In my neck of the woods, especially in power plants and things when I'm running new things, we don't get in the process cable tray, we just add new cables on top. We don't remove any of the existing stuff, we just rewire on top of it. So there's 40, 50, 60, 70 years worth of dead end of life cables that are just sitting there for the exact reason you just said. Because they're afraid if I touch anything, it's going to break things that they have no idea where it goes. [01:22:53] Speaker A: Yes. Yeah, no, there's definitely elements of that to be sure. There's even, even. Just like the, the old server is right next to the new server and the old server is right next to the old server and you've got this. Just in case we lose something, we can go back to the old system. [01:23:11] Speaker B: That guy's still there. [01:23:13] Speaker A: Yeah, you just sit there and go, what system is this? And they're like, dos. And it's like, wow, we've got a DOS machine. [01:23:22] Speaker B: It still fires up. We fire it up once a year. [01:23:25] Speaker A: Yeah. And you're like, okay, well on the plus side, hackers probably don't know how to write exploit code for it anymore. Anymore, so. [01:23:31] Speaker B: Exactly. Exactly. All right, man. So how. How do people get a hold of you? Call to action, anything you want people to know, how to reach out to you? All the good stuff. [01:23:43] Speaker A: Yeah, obviously, like I said. Name's Gavin Dwiff. I work in New Zealand, although I do consulting globally. And you can reach me at Assessment plus or on LinkedIn. So, yeah, just hit me up if you. If you have any questions on OT cybersecurity. More than happy to help. [01:24:00] Speaker B: Awesome. Hey, man, this was an awesome conversation. I think we could talk for another couple of hours. Like we said in the beginning, maybe we'll do a round two of this in the future and. And dive into some more, so. But I really appreciate you reaching out. I appreciate you taking time today and. And, you know, kind of sharing your message with. With the crowd here and, you know, the whole point, and I think I see it in so many individuals like you and I, is that we just want to get the message out there. We want to help, you know, if you don't choose me, you know, I do consulting. You do consulting. Technically, we're. We're competitors. I don't see us that way. Like, there's plenty of fish in the sea. If I can help you, if you can help me, if I can help people on that, like, that's the reason I do this podcast. It's a reason I speak at conferences and I'm always willing to answer questions, whether I'm your customer or your vendor or whatever. I don't care. I want. I want to better the world than I. Than when I came into it. And anything I could do to help that, that's what I want to do. So. [01:24:56] Speaker A: I feel the same. Likewise, and thank you for having me. It's been. It's been a blast. And there probably will be a 2.0, so. [01:25:03] Speaker B: That's right. [01:25:03] Speaker A: Look forward to that. [01:25:05] Speaker B: All right, man. Well, have a good one. And I appreciate your time. [01:25:07] Speaker A: Thank you. [01:25:08] Speaker B: Later, man. Thanks for joining us on Protect it all, where we explore the crossroads of IT and OT cyber security. Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field until next time.