Episode Transcript
[00:00:00] Speaker A: I think patching specifically is one of the best things you could do to mitigate a vulnerability.
[00:00:07] Speaker B: You're listening to Protect it all, where Aaron Crow expands the conversation beyond just OT delving into the interconnected worlds of IT and OT cybersecurity.
Get ready for essential strategies and insights.
Here's your host, Aaron Crow.
Hey, thank you for joining me again for Protect it all podcast. This time I've got my friend Oren on, on the line with me.
It's so amazing that the community is so, it's so vast, but it's also so small, right? So you know, every, it's like, you know, seven degrees of Kevin Bacon. Like it's always where either you've worked directly with someone at an entity like EY for instance, or you've worked with them as a client or, you know, secondhand. Like, I know, I know Bob through Susie or whatever, but I always enjoy having conversations, you know, years after we work together. So. Hey bud, why don't you throw. First of all, thank you for joining me. Thank you for taking time. I know you just got off stage like last week of, of doing a talk and super awesome to see how that, how well that went. So tell us, tell the, tell the audience who you are and, and a little bit about, about your background, et cetera.
[00:01:17] Speaker A: So my name is Oren Niskin and I work with Aaron for a couple years at, at ey.
I live in Houston, Texas. So I don't know anything about baseball though, so don't quiz me. Somebody offered to send me a trash can. They were like, why would I want a trash can? I have trash cans. They're like, it's Astros thing. You don't know that. So. Yeah, I don't know anything about baseball, but I live in Houston. I love it. From Miami originally.
I probably should have gone to college, I'm told, but I skipped college and instead I joined the Navy right out of high school, right after 9, 11 actually. So it was a, interesting time to be in the Navy or in the armed forces in general.
I was electrician on the Harius Truman, which is like a nuclear powered aircraft carrier. So I would stand watch and operate the electrical system and qualified the shutdown reactor operator, which is like the, when the, when they shut down. They only let me touch it when it was shut down. But there's a different job for actually operating it. But yeah, so that was a really interesting. I think. I don't know if it's so unique. I thought it was unique. But then at OT Secon, everybody Seems to have the same story, but that was kind of like. I like to say it's like the bottom of Purdue level. Like, I was crawling around in bilges.
Not like, as a real. Like, as an engineer designing the systems. I was maintaining them and operating them. And when I got out of the Navy, like, ship so much, I went to offshore drilling, and they were building a brand new ship in Korea, and it was pretty awesome. The Ocean Blackhawk, it's probably my best years there.
I heard somebody say it was a Scottish guy and he's like, she got the best years of me.
And so, yeah, I spent better part of a decade on the Ocean Blackhawk.
Probably the highlight of my career, just helping maintain these control systems. And they were brand new.
Just does like 2015-19. And it was just automation was just starting to kick off in a really big way.
We went from having no remote access to everybody having remote access at the same time. And we actually implemented Clarity there. So we were one of the. Clarity wasn't even a company. It was still in teammate. But we put Clarity on the rig. And that was really cool. And as soon as I saw that, I was like, I have to have to be a part of this OT cyber thing. I read Kim Zetter's book, Countdown to Zero Day, and I looked in our control cabinets, and we had the same PLCs that were in those Iranian centrifuges.
They just all seemed really, really cool. And moved ashore to manage the OT Cyber program.
I took over from Greg Vilano, who was kind of a luminarian in that regard, putting all this stuff on when it was just starting to be known OT as a risk.
And then I got tired of making an honest living and moved over to consulting.
[00:04:29] Speaker B: And just a caveat, you know, that was really. You know, the 10 years he was there was the best. Except when he worked with me at ey, of course.
[00:04:36] Speaker A: Right, of course. Yeah, that's right. EY is.
I think that it was really good, actually, because I was kind of working with one company all the time, the same people. You get like a very narrow view. But ey, it's like a condensed experience machine. I think working with you and Roy Solis and Doug and all these people are just amazing sources of knowledge. And all of a sudden, when I was. When I was offshore, it was just me. No, I was trying to figure out everything on my own, except for when the vendors would tell me how their product was, solve all my problems. I didn't really have any help.
But, yeah, just that all that intellect at ey really helped me.
[00:05:24] Speaker B: Well, you know, you said something, I guess. Yeah, yeah. And you said something earlier like about, you know, maybe you should have gone to college. Like. So I, I see things differently now. Especially, you know, I, I went to college and you know, engineering, electrical engineering type stuff, but that, that wasn't, I've never used that stuff that I, I went to college for.
I'm not saying it wasn't valuable and I'm not saying I didn't use anything that's not obviously true.
But you know, I, I've, I've had so many people that have worked with me and for me and I've worked for that. You know, some had degrees, some didn't have degrees. You know, I, I really see college, you know, and I have this conversation a lot. I had, I did a happy hour last night in Austin and you know, there were, there were a couple of students there and you know, they're going through cyber programs and one guy's doing a change of career, he's currently in cloud and he wants to get into cyber.
And, and you know, there's, to me, college is that thing that, you know, it's the, it's the proof that you can complete something. It's a proof that you've done some training. But it's not like you can take a college, you know, somebody that just graduated from college and drop them into a role and expect them to be successful any more than I could take anyone and drop them in. Like college is not the answer to everything. It's a great thing. I'm not talking bad about it. I think specifically especially in the technical and the, in the cyber and AI as fast as things change, even if you go to the best curriculum in the world on cyber, whatever you're learning was printed in a book and more than likely is obsolete now. Again, the tactics are the same. A lot of those things are going to be there, especially if you're an ot Obviously we know, you know, we're really, a lot of times we're dealing with really old technology that's been around for 20 plus years anyways. So the stuff in the textbooks is still accurate, but you're not going to have a, you can't expect any training, whether it's college certifications, etc, to ever be all you ever need. Like you're going to need that on the job training like you talked about, right. Is crawling around in, in the, in the belly of the ship and you can't teach that in a book. Like you can't teach what to look for, how to troubleshoot a device in a book. Like you can give them the ideas of things to look for. But you learned way more being. And I'm just assuming I'm speaking for you and I'm speaking from my personal experience as well. Like, I learned more, you know, when the thing failed and having to figure out, because it was Friday night, 3 o' clock in the morning, and I didn't have anybody else around me that could help me. Like, I had to figure it out. And I learned so much doing that, and mentors and other people that have been doing it, et cetera. It's not that I wasn't intelligent. It's not that I didn't learn anything, but there's something about getting in there, and you mentioned DY and why it's such a multiplier. Working for these big firms and places in roles like that doesn't have to just be a big four, but in those roles we have access to, you know, we're working in different use cases. We're able to see so much in such a short window of time. Like when I was working for the power company, I only saw my problems. I only saw the problems of my facilities, my environments. When I got to, you know, as a consultant, I was able to see it across 15 different organizations, and they were similar, but I saw differences. So we grow so, so much faster with that. Like how, how much do you feel that you, again, not to, not to toot the horn, but, you know, how much more do you, do you think you, you, you kind of progressed because of that ability to see those different things? Obvious foundation of your thing came from the Navy and crawling around in the belly of the ship. But as, as you started to grow in your career and go through that consulting side, being able to see so many things, how much value has that added to your, you know, trajectory? Right.
[00:09:14] Speaker A: Yeah, I think.
Well, I would say first. So when I started in the Navy, they put you through two years of school, which is kind of like, like college. I wouldn't, I wouldn't say it's quite college. It is the military vocational, how to be electrician.
And you learn a lot of theoretical stuff there that I think equates a lot to college. So, like, if I was in a cyber program, I'd probably learn details about how encryption works, like at a, at an academic level. And once you go in the field, it's really, really hard to go back and learn that stuff. So I think that stuff is really useful and if you look at people that were just in the field, a lot of times they struggle with some of the theoretical deep stuff. They have a lot of experience, but it's hard to go back and get the theoretical stuff, which I think that adds benefit in some way. But you're right. Like, you don't know anything. And I remember. So in the Navy, you do a lot of diagramming and stuff, but you don't. Like, I had 80 people in my department. So the people that actually did real work on, on a carrier is. It's not as much as you would think. Like, you don't. You're not. Like, you can't have 80 people working on things all the time or. Or nothing would work. But. So then when I went to an oil rig, oil rigs only have like two, maybe two electricians on the whole rig, and you're fixing everything all the time.
So when I first applied for the oil rig, I had very little knowledge of working on oil rig systems, but I knew how to draw it out on paper. So I draw it out in the interview and totally fast talk my way through that interview. And I get on the ship, on the rig, and I'm just like, why? Eyes wide open, all these 480 volt panels and trying to. Yeah, I think having the practical experience is really important. Having the academic experience is really useful.
And then you go to consulting and yeah, like you said, you get all this experience. But I think what really helped me is you have all these people that are masters of their craft and you're working with them and they're kind of teaching you all this practice, helping you through the practical stuff with the eye of, like, how a master craftsman would do it.
So working with you, or. I worked a lot with Roy as well, or Sonia. These people have been in industry solving problems for clients at a very high level.
And then what's interesting there is move to GuidePoint. And GuidePoint had its advantages because there aren't a ton of Royce. There are a lot of people that know various things. But at ey, sometimes it feels like you're a little fish in a big pond.
Then you go to a smaller consulting firm where, where you have a more like varied experience, but it's not. It doesn't have as many levels. And all of a sudden now you're responsible for being that person.
And that was another step in my career, which is hard to get at ey, but it was, it was really useful.
[00:12:09] Speaker B: Yeah, yeah. And you know that.
I know we're spending a Little bit of time, but I think it's value add. A lot of people that are listening to this are either trying to break into cyber, break into OT or, you know, kind of grow their career. Maybe they're an analyst or whatever. And that's part of it, right, is sometimes you have to get into those places to grow your career. The other, the other side of that conversation, and as we started this, you know, you just got off a stage at, you know, at a, at a pretty big conference, right?
And that's not something you learn or they teach you when you go to, you know, a certification class or, or you, you go to your college degree, necessarily, obviously there's electives in that, but, you know, to be an electrical engineer, you don't, you don't take public speaking, you know, but especially in today's world, that there's value to that. Right. You know, with, with social media, LinkedIn, all the things like your personal brand is, is who you are, right? It is.
I really see your personal brand as, as really your, your resume, right? Yeah, you, you, when you apply for a job, you have to give them your, your CV or your resume. But what are the first thing that those people do? They go to your website or they go to, you know, your LinkedIn and they, they see who you really are. Right? It's one thing I can put whatever I want on a resume, but if I go to LinkedIn, I can see exactly where you've been. Like, did he really speak at that conference? Because there should be something tagging him at that thing.
It should be really easy to find that, like, go to the website, like all that type of stuff. So the more that you do these things. So to anyone out there that's, that's listening, obviously you need to deep dive into the technical details. Yes, you need to get the degree and the certification and the experience and all those types of things. But also you need to, you need to really make sure that you're thinking about your personal brand. Because all that stuff, not only is it valuable to, wherever you land, whether you're Guidepoint or, or, you know, ey, or wherever your personal brand is part of your, your value to the business. Like, you know, I work for Morgan Franklin and I do consulting, and it's kind of like, you know, it's a smaller version of a, you know, a big four consulting type thing. Right.
My podcast adds value to me as an employee of that company because I get the exposure because I'm speaking at conferences and speaking at defcon, RSA and Wherever I may be, it's beyond just what I know on the technical side and the cyber side, it's also building that trust. And that's really what your personal brand does, is, is you're, you're building that trust of you as, okay, he can handle himself or she can handle herself or whatever. Right. She can be in front of an executive. If you can speak on stage, then you can probably speak to an executive or to that plant manager and be able to communicate and not, you know, be frustrated or, you know, that that communication skill is always something that can get missed, especially in a highly technical field.
[00:14:57] Speaker A: That's true. And something I've learned the hard way is if you can't communicate the value of your cyber, you can have all the defenses in the world. And, and it's, it's, they'll just cut it out the next budget round or whatever. It is like, you have to be able to, it's like everything hinges on that ability to communicate. Yep.
[00:15:15] Speaker B: So, yeah, 100%.
[00:15:18] Speaker A: And I think it's interesting that I, I was super nervous, so, and I, I felt like I had no business, but somebody in the industry was like, you should totally do it. And, and I just jumped in and kind of closed my eyes and jumped in, and, and then I got all this. It was really cool to see how supportive the community is and people that I really looked up to and I've been watching on YouTube, they would come up to me and they say, man, that talk was great. And I said, what you think it. So it was, it was a, it was such a cool experience.
[00:15:49] Speaker B: And, you know, it, it's human nature. You're going to be nervous. I'm nervous when I get on stage. I, I, it just happens. Like, you don't do it every day. You know, when I first started this podcast, I was anxious and stuff. Like, now I've done, you know, two. I've been doing it for, you know, I just hit 52 episodes a couple of days ago on this version, but also did the previous version. So I've got more than 100 episodes of this, but doing a long time and, and you get better. It's just like with anything, you get better with repetition, you know, so it's okay to be nervous. Right. But the cool thing is, is you did it anyways. Like, that's the, that's the difference, right? Is everybody nervous? But so many people just sit on the sideline and never throw their name in the hat. And then you got there and, and, you know, you didn't die, like, yeah, maybe you sweat a little bit, you were a little anxious or whatever, but you made it through. And. And now the next time it'll be a little less anxious and it may never get to zero.
You probably don't want it to get to zero because you know that that's a whole nother conversation. But, you know, the, The. The point that you just made, though, is that all people are sell. You have to sell your thing. Like, you're a salesman. You have to be.
It doesn't mean you're having to go out and sell to a, you know, a third party or a customer and bring in revenue, necessarily. But even if you're internal, if you're an employee nine to five, you're working for. For, you know, an asset owner, whatever, you still have to sell your manager on why your ideas are correct or why this thing, you know, you're. You're selling, why you need a promotion, why you should get a raise, like, why you're. You should go to this conference, why you should get the day off, like, whatever those things are. We're constantly. So I have to constantly sell my wife on why we need to do certain things or whatever, right?
Goes across everything that we do. We're constantly trying to win friends and influence people. I want to influence my wife to do something. I want to influence my kids to do something. I want to influence my boss, this customer. I want to influence you to come on my podcast and. And bless me with your knowledge and experience, all that kind of stuff, right? We're doing it. Everything that we do, we just don't necessarily think about it.
[00:17:54] Speaker A: In a sales role, I've been the only person I know that has been part of a Cyber D transformation.
[00:18:01] Speaker B: Right?
[00:18:01] Speaker A: So I took, at one point, I won't say where, but I took a fully functioning cyber program. And because I didn't know how to sell it, how to show the value, I should say, like, it was really good stuff and I should have learned how to show the value. I was early in my career, and they would, you know, we would go through these things, like, do we really need this? And I would say, well, these are the risks. And they say, oh, that's fine, cut it out. And then by the end, I was standing with nothing, and they were super pleased with me, but I didn't do them the service that. That I should have because I did not know how to sell. And now that, yeah, trying to turn that around, that's really important skill. You could have all the technical skills if you want in, in the world, but it won't matter if you can't show the value.
[00:18:42] Speaker B: Yeah, I agree man. 100. So let's dive in for a second. Do you mind sharing what your, what your topic was and what the, the, the, the presentation you just gave at the, at the conference was?
[00:18:54] Speaker A: So yeah, the, the topic I talked about at OT Secon was how to make patching easier, patching more efficient. The title actually was how to reduce your vulnerability management effort by 80% which I, it was 2:30 in the afternoon so I tried to make something jarring to keep people like do you have to pay attention if you're going to call me out on, on whether it's BS or not.
[00:19:15] Speaker B: Exactly. So patching in an OT environment, like who does that?
[00:19:20] Speaker A: Yeah, exactly. So what I found is like you have the five critical controls and vulnerability management is all the way on the right and that's because it is the worst. Like it is tedious and risky and at the end of the day like a new vulnerability comes out the next day. Have you really accomplished anything?
And I think that's a pity because I think patching specifically is one of the best things you could do to mitigate a vulnerability. There's other things you could do. You could monitor it, you could have what do you call it, a virtual patching with a firewall.
But that vulnerability is still there.
I think beyond that, if you're not patching, your system is slowly growing out of date.
I think there's a misconception that patching is a security thing and it is. But if you've ever called Cisco Tac to try to get support on an outdated switch firmware, they're going to tell you go upgrade the firmware and come back. Like in order to keep a well running system you need to be updating it.
And I think the factory floor or the, you know, the bowels of the ship, this is these, these teams are not exactly the most equipped to do that patching. And I think it's something that we can offer to the business.
[00:20:41] Speaker B: Yeah, well, and you hit something there too. Right. Is, is we think about patching from a. Oh well there's a security vulnerability that I've got to patch and a lot of patches are that, but it's not always that or it's not only that. Right. So maybe there's a vulnerability that it's, it's plugging as well, but sometimes it's functionality like they, they found a bug in their system that it doesn't run as effectively or There's a, there's a, a memory leak or, or, or whatever the thing may be. And if you're not updating those things, you don't get the benefit of those, those, those enhancements in the way that the thing functions, especially at a firmware level. Right.
Same thing you mentioned with Cisco. It used to be the same way with Dell. I remember I was working at AT&T and supporting all their servers and they had, you know, Dell servers. And every time we called the support line, the first question they ask what is the firmware and BIOS level of the server? And they wouldn't even talk. I don't care what your problem is. What is the firmware and BIOS level of your problem? And if you said it was something more than, you know, one or two revisions from what it should be at the newest, they'd be like, update it, call me back. Right, that, I'm not even going to talk to you until you've done that because that usually fixes 70, 80% of the problems because there's underlying things that are integral in that thing.
So you know, obviously in OTT we may have systems that have been 20, 30 years and never been updated. Right. And yeah, it's, it's one of those and I'm sure you've heard it many times as well, but it's one of those. Well, it's not broken. It seems to be working fine.
Is it though fragile?
[00:22:19] Speaker A: Yeah, it's working for now.
But I would say like as I, I think we're in a great industry, first of all, I'd like to say that like, if you think I was watching, I was watching something on YouTube. It was Tucker Carlson interviewing the Secretary of the Treasury.
And Tucker Carlson says, how do you expect us to fill the gap in manufacturing? How are you going to find the people? And the Secretary of the treasury looks as a given, he says, well, I think if you look at all these smart factories they're building with AI, with automation, we're going to be the world leader in manufacturing because of all those things. And if you think about that, that's all ot.
So OT is like the place to be if you think about green energy. Green energy, it's all based on connectivity to ot. AI is also, I think the OT network itself is becoming more business critical. And I think because of that we're going to want to maintain these systems. We're not going to want to have, for these critical systems, you're not going to want to have 30 year old equipment. You're Going to want to have equipment because it's important. You want to have equipment like you do on your IT side that's in support, that is functioning well, that is maintained, that is resilient. And I think the more you connect, like I think as McKinsey is going into all of our boards and telling them about all these benefits and you're connecting it and OT more, you're going to want much more resilience and security and up to date patching methodologies.
But of course the problem is if you start patching an ot, you can quickly, you can hurt people, you can bring systems down, cost people money and probably lose your job.
[00:24:06] Speaker B: So yeah, it has to be done differently. You can't just go willy nilly, you know, patch on, you know, patch Tuesday because the patch is there. Like we, we have to do it. You know, the example I always give is, you know, you're flying in an airplane, we all fly, you're in the airplane and there's a, you know, a vulnerability or a patch, a new patch for the control system that, that controls navigation or, or you know, flight controls or whatever. Do you want them to patch it while you're in the air? Probably not. Even if I've tested it on 15 other planes, I'd probably rather you wait until we get on the ground to, to do that patch. Right. Doesn't mean I shouldn't patch it. It just means I need to find a, a maintenance window. Like it's going to look different and that's okay, but it shouldn't be, never. It shouldn't be like it works fine, we're good. Like no, it's, you know, the other analogy I always give is, is the, the oil change. You know, you buy a brand new car from the dealership and you never change the oil and it goes five years, 100,000 miles and you've never had a problem.
So obviously you, you saved a whole bunch of money and you, you know, ch. Changing oil is a waste of time and then the motor blows.
How much do you, how much does it cost to replace a motor, transmission, all that type of stuff. Like it's way cheaper to just do the maintenance along the way and, and you know, have a well, well running machine. You know, I've got a, I just bought my son a Toyota or Lexus GX470A 2004 as his first vehicle. So it's a, you know, almost what, 20 something year old vehicle and but it, it's reliable, it's got 270,000 miles on it, yeah, it needed some maintenance. I had to put quite a bit of money into it to, you know, update things. But that same motor, if I, if, if we continue to maintain it will last another 200 plus thousand miles.
But only if we do the maintenance. If we don't do the maintenance on it, it's going to, you know, it could die at the end of the week. Right. It's just like anything. And that's, that's where I think we've gotten in this bad stigma in OT of because the way that stuff used to be, you installed, touched it again and, and that has to change to, to your point, like if we're going to do these smart manufacturing and, and I agree, I think that's the way that we, we get back manufacturing in this country is we're going to do it smart. It's not going to look like it did in the 50s. It's going to look like a lot of robotics. You know, I, I did a large project for a, of electric vehicle company while I was at EY doing their cyber OT program on their manufacturing floor.
They still had employees, you know, people down on the, on the manufacturing line, but they were really just running the automation, they were running the robots like they were checking in on things, quality assurance, a lot of that type of stuff. The robots did most of the heavy lifting and the welding and all that kind of stuff. And the people were there with a tablet and they were validating the tolerances on things and you know, doing visual checks and you know, more like an operator round than they are actually. The craftsmen of, you know, build in a piece of sheet metal or something like what we would have done a hundred years ago.
So there's still jobs, there's still high end jobs, but they're more technically capable, they're more, you know, driving automation and robotics and even cybersecurity obviously is a big part of that. Right. So it's just going to look a little different. But even with that, to your point, like we've got to be more on top of, because, you know, little inefficiencies in firmware or whatever can, can have a, a domino effect on our process and profitability and, and reliability and availability. Obviously safety like you mentioned, you know, people's lives could be lost because you know, a robot goes out of control and hit somebody upside the head or break something or whatever. Like there's, there's all sorts of risks that we need to consider when we're, when we're, when we're having these conversations.
[00:27:53] Speaker A: And I think, yeah. So my experience with that on the rig, like I said, we, we were one of the first people to get clarity. So we had visibility and it was amazing to see, like, oh, wow, all these vulnerabilities. Right.
And so I, I got excited and I got together. We're really good at risk assessment in the oil field. So we did a risk assessment of all these vulnerabilities and we determined that we want to patch the most critical vulnerabilities first. Right. Your highest risk, to get your most risk, highest risk reduction.
And that happened to be on the driller chair where the driller is drilling the hole.
But in order to patch that driller's chair, that you have to wait for downtime. And like they say, patching happens on Christmas. Well, it turns out the drilling rigs run on Christmas too. So you can't even patch on Christmas. You have to wait for the opportunity.
You have to involve the vendor, vendor flies out, you have to pay them.
And it's not just installing a patch. So the first thing you do is test the safety systems, then you patch it and then you test those safety systems again to make sure you didn't mess anything up. It ended up being really expensive. And at the end of the day, you patched one pattern, now you have another patch. You know, so I think what I came away with is it's a lot of effort and I'd rather focus on other, other stuff. So I put patching on the back burner, which I think what everybody does, and that's a shame.
So then later on ABB came to me and I make my accent in the talk. I think it went over well. But they were like, tell me, Oren, what do you think if we patch all your vulnerabilities, patch all your Windows systems like automatically? And I was like, what do you mean automatically?
You're going to kill people. And they're like, no, these Windows systems are for diagnostics. They are not part of the process.
And it was kind of a big eureka moment. Like, that's true. They were, it's called drive monitors. And they're just there to, for remote access. You could, ABB could log in and look at your drives and see if they're in good working order, transfer data.
So yeah, why can't they just patch? It's an ot. I had just blanketed OT with a don't patch thing.
And it's not really the case. Some things you can patch, like in your plane analogy. Some things on the plane are not anything to do with safety, like.
[00:30:21] Speaker B: Exactly.
[00:30:22] Speaker A: And it's not tied in now. It's, it's nerve wracking, I'll be honest. Anything on the plane, like you say. But if, if you're going to run that plane for eight months straight without ever landing, then patching something that is not safety critical may be the right decision.
And they just, they implemented this patching system which they tested in their lab. Now they don't have a drilling rig, so I don't know what that testing involved, but it was automatic. I didn't have to worry about it. And all those systems were patched with and it reduced, I think, a huge amount of effort. Because if you think about your attack surface, those, these diagnostic systems, they're the exposed systems, they're the systems people are remoting into and they're Windows based, so they're susceptible to all the Windows malware.
And so it was automatic patching and it was patching the assets that were the highest risk to cyber.
[00:31:16] Speaker B: Right.
[00:31:16] Speaker A: Yeah. While at the same time lowest risk to safety. And that, that's kind of, that was the gist of the talk.
[00:31:22] Speaker B: Yeah.
[00:31:23] Speaker A: Is that it's a huge return on investment.
[00:31:25] Speaker B: Yeah. I mean that, that's huge. And that I love that point. And I want to double down, make sure that everybody heard that every device that's no t is not the same criticality.
Right. Some of them. You know, the analogy I've given on this before is I've got two PLCs. They're the same make, model, firmware, everything. One controls the turbine, the other controls the ice machine in the break room.
Right.
[00:31:49] Speaker A: Very important Ice machine it is.
[00:31:52] Speaker B: But if I'm good, if I, I can't just treat both of them the same, I can patch the ice machine because worst thing that happens, the operators get pissed off at me because we don't have ice for a week or a day or whatever. Right.
I don't have the same thought on, on the turbine because if I patch that and it causes issue. You know, turbines are super expensive, they can kill people. I can't produce electricity. Like all those types of things happen. Or you know, if it's on a ship, then, you know, the ship's not moving. It's, it's stuck at harbor or in the middle of the ocean, which is even worse or scarier, you know, so again, but if I, if I, if I'm in the middle of the ocean on a, on a big ship and I patch the PLC that controls the ice machine, it's an inconvenience it's not a safety issue. It's not a, you know, production value issue. People aren't going to die because they don't have ice. They may be upset because their water isn't cold, but they're, they'll, they'll live. Like you still have water. Right? So that's a really good thing that it's very easy as OT practitioners to say nope, this is ot, stay out. I'm not going to patch it, we're not making changes. It has to be done in an outage window. And, and I think that's a good place to start.
And as far as to be safe. I think the thing that a more advanced and the way that we, we move towards, you know, crawl, walk, run in that walk and run space where we're saying okay, these systems, yes, they're an ot, but I'm okay with doing these. I'm still going to do them safely. I'm not going to automatically, I'm still going to notify people. I'm still going to do my due diligence, I'm going to test my patches. Like all those things are true, but I don't necessarily have to wait for a year. I don't necessarily have to fly the vendor in to do this update. Like there's certain things that I can do differently. To your point, the drilling chair, that, that one, that one thing that controls that. Yeah, I'm still going to do that. I'm going to, that's, that's an off limits one. But the Windows machines, the, the ice machine, like whatever those things are, we need to make sure that we're looking at our program holistically and considering what are, what are actual business critical, what is actually something that I can, that's less critical. That's a monitoring, that's a diagn that I have redundancy in. That's the other piece is a lot of times is I'll have three or four engineering machines in different locations. They all do the same function, right? And so maybe I can test it on one, validate that it works in worst case scenario I'm down to 2 instead of 3. I can still do my job, I don't lose access, that kind of thing. So there's, there's all sorts of things like that that we can consider.
Have you seen that from a, from a client perspective or even from an asset owner perspective of them, like that light bulb moment of oh, I kind of like what you had with the Windows machines. It can be automatically done.
[00:34:32] Speaker A: So I have one Favorite client that we all have our favorite client. I won't say who it is, but we're building this really cool architecture. And yeah, they're implementing. And it's cool because it's. It and OT working well together, which I'm glad to see, which is not common.
We're trying to do something where it.
I think the IT OT convergence is it's not like a solid line. It's like a spectrum.
So you take, for example, the idmz, the industrial demilitarized zone. Nothing in there should be critical to your process or to safety, especially.
Right. Like, if you cut the IDMZ off, you should still be able to do whatever your industrial process is supposed to do.
So that at the same time, like I was saying, the IDMZ is the most exposed.
So when you patch the idmz, you're reducing the most amount of risk for the least amount of risk to your operations.
So it is going to take care of the IDMZ patching, which is.
They're still going to notify you, of course, like you say, like, you don't just want to. You want to notify people.
You have patches like that Rockwell decom patch that could mess people's day up.
But then you go down to level three, and level three is a little more picky. So that's where you go kind of host by host and say, this one's okay, this one's not, and do a little more.
A little less frequent patching, but still patch it, you know, quarterly maybe, and then down at the lower levels, you're going to patch. Like to keep it in support, I think, is what Dale Peterson says, which I think is good it. And this is a. He talked about it in 2016.
[00:36:15] Speaker B: Yeah.
[00:36:16] Speaker A: But, yeah, as far as a client doing it, I think we're pretty slow. So hopefully it happens. But yeah, I have not seen it yet at a. In full production.
[00:36:25] Speaker B: Yeah.
[00:36:26] Speaker A: Outside of my own experience.
[00:36:27] Speaker B: It's. It's.
It's not, it's not rocket science. It's not that hard. Like, this is not new tech concepts or anything. You know, they do. We do the same thing on the operational side all the time. Right. As we're. We're replacing or upgrading, we know how our redundancy works, especially in these critical places and systems. Like, we usually have redundancy. We have, you know, recovery plans that require, you know, replacing a PLC or replacing a valve or a motor or all those types of things. We have this, this operational availability mindset and the way that we design architect and engineer these environments. We just need to make sure that cyber and the tech, I hate to even call it cyber the technology, the tech stack side of these environments are also part of that resiliency. And we're thinking about that. It goes to, you know, the, the whole, you know, secure by design and, and you know, the, the CIE out of Idaho National Labs. Right. And really thinking about cyber informed engineering as I'm designing these systems. Cyber and technology stack needs to be part of the same, you know, resiliency that build into redundant, you know, having redundant PLCs and paths and power sources and, and all that kind of stuff. Like we need to think about that as we're. You just talked about it and we didn't, we didn't call it cie, but that's really what you're talking about is like understanding these systems are critical. They get done once a year. These are not. So then, and then the idmz, we should patch those as soon as there's availability because other than making sure that people are notified that, you know, I don't have somebody remote accessing fixing something because that's the biggest problem as long as that's happening and I get a permission, I should be patching those regularly. Almost like it, like you said, like it doing it. They still need to get approval, but it should be done very frequently. And then further down the stack that you go less frequently but still needs to be done.
[00:38:24] Speaker A: Yeah. And as you go farther down the stack, there's less value to patching and it's really hard. So it's kind of like it's, it should be an easy business decision to make for us, you know, as you like especially I think people like to say it's pretty. You hear it in every talk to like we're insecure by design at the lower levels.
[00:38:42] Speaker B: Yeah.
[00:38:42] Speaker A: So why, why compromise a PLC vulnerability when you could just tell it to change a parameter correct. Without any authentication.
[00:38:50] Speaker B: Yeah.
[00:38:51] Speaker A: So I think that was really interesting too. And something else too, like you were saying, like making sure your tech stack is, is, is resilient is using what you have. So I think most IDMZ Networks and Level 3 are virtualized.
And so that means if you have a patch go bad or if you have endpoint protection that decides to try to destroy the world, if you remember that one, you could restore from a snapshot.
So not only is it low risk to operations, but it's also easy to recover from if you have those protections in place.
[00:39:28] Speaker B: Well, and that, that goes to the, the you know, your playbook and your runbooks and, and how I do my SOPs, Etc. Right. Is, is, you know, especially when you're talking about virtualization because again I designed systems like this all the way back to 2010 and using virtualization is a game changer and, and you know, as my team would be patching or making changes before you touch anything and not just patching before I make any changes. As soon as I log in validate, there's a backup validate, there's a snapshot. So if you fat finger something and you change something, you don't know what you did. I can always snap it back to where it was when I got here. Right. I'm just taking a known good space and I, and before I make any changes other than logging into the damn system, take a snapshot and then you can make any change you want. And if something happens, I can always go back to where I began and I don't lose anything. The problem is is people assume that the automated process does that and then when they log back in, it was from three weeks ago and there were a thousand change changes that happen between then and now and I don't know what they all are because operations made changes etc. So yeah, I can get it back online but all of those changes got wiped out and that's where the, the real issue comes in. It's not necessarily that I'm, I'm crashing systems which absolutely can happen, but it's also that hey, I made 3, 4, 5, 100 changes and I've got to go back through and redo them. Hopefully they documented all those changes along the way and they can redo them but it's still tedious that they have to actually go through and recreate all of those changes. Updates to Ladder Logic or whatever they're talking out in that system.
[00:41:00] Speaker A: That's true. Yeah. Not consider that. Yeah, definitely got to make sure. Yeah. I've seen plenty of snapsh like backup infrastructures that they thought were working but actually had failed long time ago.
Some little error.
[00:41:13] Speaker B: Trust but verify.
[00:41:14] Speaker A: And I really like the CEI thing you were talking about. I think tying cyber to actual impacts is really important.
Bringing founding it are grounding it in solid engineering is well you know, most.
[00:41:28] Speaker B: Of the time ot cyber security to your point earlier, like it's growing, it's a great place to be and most of the time because of where we're at, the, the, the architecture or the, the needs are not super complex. Like we need to do foundational things like basic blocking and tackling and patching and backups and, you know, segmentation and firewalls and just basic stuff like we're not talking AI and, you know, crypto and really fancy fun things that are going on in the cyber world.
Not saying that they're invaluable in, in this space, but many times they don't have an asset inventory. So, yeah, how can I do, how can I do zero trust if I don't even know what assets I have? Like, I gotta, I gotta solve my asset inventory problem before I can design a, a zero trust network in my space. Like, it's just common sense.
[00:42:18] Speaker A: That's true, that's true.
That is true.
And I think a lot of it comes from we in ot. When I, when I was working on rigs, I was it for ot, sure. And I've heard you talk a lot about that in, in your previous podcast. Like, you had a team of just a few people, you had all this compliance work to do, and then you look over at it and they just have hundreds of people over there and they have all the skill sets and you're sitting over there trying to figure out everything on your own.
And one of the things that really unlocked your capabilities was leaning on them more, which I think is super important to do. Like, you can't. And so in my experience, when I transitioned from rig to shore, I actually had never worked in it. Of course I knew what IP address was, but I had never worked in it. So I was thrown into that kind of by force.
I worked with IT security guy pretty much hand in hand to onboard all of our OT systems into a real program.
And I think that accidentally helped me bridge that gap between OT and it.
Because you can't do it all by yourself.
[00:43:35] Speaker B: No. And there's huge value in that. And the other thing that I like to say a lot, and I don't mind if you've heard this good because, audience, listen to this hammer at home. It ot, we're all, all on the same team.
Like, we shouldn't be enemies. That doesn't mean that you just let them do whatever they want. You know, you and I are friends, but, you know, I want you to knock at my door before you come in my house. You're always welcome at my house, but I do ask for the courtesy that you knock and let me know before you come in. Don't just walk in my house. Like, that's just different. Like, same thing from an itot perspective. Like, I always, you know, I supported 48 power plants at a Power company.
And when I was at, you know, clients and I, we, we were rolling out at 3,000 sites across, you know, the, the country.
We would not just go there and walk in the door and start making changes and be like, well I'm supposed to be here. I work for the company. Like, no, that's, that's not how you do business again in Texas. You'll get shot. You walk in my house. I don't care if I do know you. Like, what are you doing in my house? I didn't know you were here. Right. And a lot of these places are that way. Like they don't want things happening to them. They're not aware because I may assume everything's fine. What I didn't know is they had an outage last night, they're in middle of startup up, they haven't been able to, to catch their breath. And I just walked in and made a change that could impact what they're doing. And I don't know it or maybe it doesn't have anything to do with it, but I'm just in the way. Like there's all these factors that go into it and it comes down to again when friends and influence people, right? It's, it's. I would never show up and start just walk into your house without letting you know I'm coming. Just little courtesies like that. Like I'm not going to design, like I may be the smartest, you know, whatever in the world, but I'm not just going to design something and hand it to you and say hey, you should do this. Like I didn't ask you for that. Like what, what is this? Like I don't know if I want that like you, we need to bring people along with us. So one of the bigger successes I've seen and it goes across all industries but you don't, people don't want things to be done to them. They want to be part. They want to feel like they have voice even if it's just to say okay, I understand, yes, or here's my concerns. This is the, the things that we've seen in the past, the issues that we've had in the past, etc, and when you fact, when they feel like they've been heard and their, their concerns have been put into the conversation, you have an answer or at least a, a solution for those problems then, then they feel more comfortable allowing you to do the things you're going to do. If you just walk in and say, you kick in the front door and I'm from it. I'm here to help, and I'm gonna make a bunch of changes and get out of my way. You're stupid. You don't know what you're talking about. Yeah, that ain't gonna work.
[00:46:10] Speaker A: That's true. And I think you've probably seen this before.
A quick way to build that trust is to help them out with stuff they're doing.
[00:46:19] Speaker B: Correct.
[00:46:20] Speaker A: So, like, they were implementing a new.
A new vendor on the rig. And they say, hey, can. Can we get some help?
It's not answering the phone, whatever. And I would walk over there and I would help them out. And right after a while of doing that, we became pretty close. I think the. And the operations side, the engineering side, they would call me and hey, we're thinking about putting in this new system. What do you think?
[00:46:44] Speaker B: Think.
[00:46:44] Speaker A: And that's huge. Like, get involved in that early stages of designing these systems. It's really cool to see.
But that was all slowly building trust. Because at first, yeah, I was like the enemy. I was like Mr. No. I was the person that said no to everything, made them give up their USB drives, all that great stuff. But you're so mean. Yeah.
We would stop it at PC procurement, actually. We would look for people or. And people get upset about it. And I understand that. Like, they're trying to do their job and we're taking away their USB drives from a thousand miles away.
But, like, slowly. We're both humans and we both have a lot to offer each other. So slowly building that trust back. It's cool. They bring me into their secret room and say, hey, look at this cool project we're working on.
And yeah, it's super helpful. I think I heard Patrick Miller talking about it to the. To Congress, I think, saying, we gotta secure new systems.
[00:47:44] Speaker B: Right.
[00:47:45] Speaker A: In order to do that, they have to invite you into the room to see those new systems. Yep. So building that trust I thought was really cool.
[00:47:51] Speaker B: 100% right. Because if they think you're going to be a roadblock and a problem, they just won't invite you and they'll find workarounds and. And then it'll be too late for you to make a change.
You know, it. It's so important. Guys, guys and gals and folks listening like you, we. We've got to make those connections. We do business at the speed.
Just what you just said, right. Is the faster that you can build trust with the person sitting across the table from you or across the. The voice, you know, the zoom call, or the team's call or whatever it is. The sooner that they're going to allow you, the, the more likely it is that they'll allow you to, you know, come into their protected space and, and you know, look at the problems and all of that. If they don't trust you, it's, it's very unlikely. Even if, you know, comes down from the top and it's mandated that you do it, like they'll, they'll do just enough so they don't get in trouble. But if you don't build that trust, you're not going to get the full, the full picture. Right. And that has to be part of everybody, especially in ot. It's very, very important that we build trust first before we try to cram down technologies or solutions or anything else. Before you start talking a lot, you should be listening a lot more.
[00:49:05] Speaker A: Yeah.
Make sure they know you're not an auditor.
[00:49:08] Speaker B: Correct.
[00:49:09] Speaker A: I would never let anybody from the IRS look at my, my accounting exactly.
Like, you don't want to be an auditor. You want to be a friend and somebody, somebody helping them.
[00:49:19] Speaker B: Correct. And not just in words and actions and, and building that, that relationship. Right.
It's more than just saying, I'm here to help, it's actually doing it. So when they call you and say, hey, we've got this problem, can you help? Yeah, absolutely. I'll be right there. Right. It doesn't mean you have to drop everything, but they, you need to build that relationship. It should be a give and take, not just, not just a take.
[00:49:40] Speaker A: And I always laugh when people say to scuff your hard hat. I think that's funny.
[00:49:43] Speaker B: It's true.
[00:49:44] Speaker A: If they're not gonna under, like, they're not gonna see through that. Like, yes, I walk in, if I start, if I walk into a facility, even even though I worked on rigs, I was never very good mechanic. Like, if I start picking up like hydraulic equipment, they're like, whoa, what are you doing? Like, they know, they can see it.
[00:49:59] Speaker B: Yeah, they can, they can read that body language. But also they're gonna judge you. You know, I remember doing walk downs of power plants and the whole team is there and you can tell they're like, you know, very easy to spot. Who has experience in, in the field and who doesn't. You know, when the, when the executive shows up wearing slacks and, you know, dress shoes or, you know, there's, there's a, there's a woman wearing, you know, high heels or whatever, it's just like, okay, first of all, you can't even Walk in the plant with that stuff, you're not allowed to go. And secondly, why would you even show up at a power plant wearing that stuff? Like, what do you think this place is?
[00:50:36] Speaker A: Like the hard hat balances delicately above the hair gel.
[00:50:40] Speaker B: Exactly.
[00:50:42] Speaker A: Very cool.
[00:50:43] Speaker B: 100. So all. All this, you know, dude, awesome conversation. I love it. You know what? Next five to ten years, what is something that you see coming up over the horizon? Maybe that's good, exciting. Maybe something that's concerning in this space.
[00:51:00] Speaker A: So, I guess, yeah, I think. I think we've had a lot of.
I think a lot of our conversation today has been around it and OT working together, and I think OT is becoming super critical to the goals of our companies, to the goals of the country, the world, everything like global warming, climate change, AI manufacturing.
It's all kind of hinging on connecting our industrial systems to become smarter. And so that's either going to happen in a secure way, which is opportunity I'm excited about, or it's going to happen in an insecure way where they're just going to make it happen by themselves and.
Oh, you want dart now you're good. There we go.
They're just going to make it happen by themselves without our help. And that is concerning, I think, because then you're opening. We need these systems for both the safety of these. Of our people that work at these plants and just to operate our society. And so building these secure industrial systems that connect information are smart, I think is super important.
But, yeah, that. That's what I would say. And that's from my experience, like, working on the plant floor, scared to death that something's gonna fall on me.
[00:52:13] Speaker B: So, like, it happens, man. There's a reason why we have a safety culture, why we wear hard hats and PPE and all the things, right? Is. Is bad things can happen and do happen. They didn't. They didn't get that way by accident. So I agree. It's. It's 100, you know, critical that we protect. You know, obviously, greenfield, new stuff that's coming out, we still have to protect the old stuff that's around. Like, we're not just going to rip and replace everything. There's too much infrastructure out there to do that, so we have to protect it all.
Hence the name of the podcast. Right.
We have to find a solution and, And a way that we can. We can get in all of these spaces. So awesome, man. So how do people.
Is. Is your recent talk available for people to check out? Like, what, What. What's your call? To action for folks.
[00:52:58] Speaker A: Yeah, thank you. I would. I think they're publishing it on YouTube soon, so as soon as they are, they publish it. I'll send it over and put it on LinkedIn. Please connect with me on LinkedIn.
I'd love. I'm really wanting to get out there in the community and kind of share my experience and help other people that are trying to share their experience and gain experience in general. Like, I think we all have something unique to offer and I think this community is really special.
The community you're building and OT in general, just because you have the cyber community, which is super friendly, but you also have the industrial side, where even though we may be the fiercest competitors, we all want to see each other go home at the end of the day with all of our hands and fingers and toes. And I think that's a special connection that we have as people that work in industry and really looking forward to sharing that.
[00:53:53] Speaker B: Yeah, man, absolutely. We'll definitely put up, put your link to your LinkedIn and the show notes as well. I do want to just say one, one last thing. It was really cool last year. Went to Husetcon and, and he brought his kids with him, right? His daughters were there and they got to experience it all and, and kind of, you know, see the lock picking and the different cool stuff and you know, it's important to, to, to bring, you know, the next generation of, of of practitioners into the space. Now obviously it doesn't mean that if you take your kid to work, you know, dive into that space, but at least they're aware of it, right? And introducing those things to, to, you know, other folks. So whether that's mentorship, you know, I did a happy hour last night and there were, you know, students from, from, you know, university that were there, you know, working towards cybersecurity. It's super important that we give back. It's why I do this podcast. It's why I have, you know, conversations with folks and really love, love this thing. But man, I love the fact that you brought your kids to that event and loved meeting them and all that. So really cool. I appre. And, and you, you even mentioned that there's going to be a, a youth setcon coming up during hun. Is that correct?
[00:55:00] Speaker A: That's right, yeah. Youth seccon. It's September 30th, so my daughter signed up. Super excited and it's hard to get teenagers excited about anything. So the fact that she's excited is, is really great. And, and all the, all the people at the conference. Really enjoyed having them there. And. And Sam Ryder tried to give her a bottle of whiskey at the raffle. That didn't, but she ended up getting a fire TV instead. So she.
[00:55:23] Speaker B: There you go.
[00:55:24] Speaker A: It was, it was a really cool experience and I, I think, yeah, it's. It's great to see them excited about something, so.
[00:55:31] Speaker B: Well, tell your daughter she got a shout out on the podcast and that, you know, we look forward to, you know, potentially hearing. Hearing her speak at one of these things. Maybe about her. Her Instagram conversation that we talked about before. Before we started recording.
[00:55:45] Speaker A: I hope so. I hope she speaks. And she, She's a huge fan of this podcast, actually. I force her to listen to it in the car.
She makes fun of your. Your mic. Like when somebody doesn't have a good mic and then you come on, you're like, correct.
[00:56:00] Speaker B: So make sure that you have a good mic when you come on the podcast. So. So give her a shout out. Tell her, tell her hi. And when she listens to it, she'll. Everybody, I'll hear her name if you want to share that.
[00:56:09] Speaker A: That's right. That's you, Celia.
[00:56:11] Speaker B: There you go.
[00:56:12] Speaker A: All right.
[00:56:12] Speaker B: All right. Thanks for listening, y' all. I appreciate. Hey. Or thank you so much for coming. I really enjoyed the conversation, sir.
[00:56:17] Speaker A: Yeah. Thank you for having me. This has been an honor, so.
[00:56:19] Speaker B: Absolutely.
[00:56:20] Speaker A: You've had a lot of star studded guests, so I'm honored to be part of that.
[00:56:23] Speaker B: I appreciate it, man.
[00:56:25] Speaker A: Thank you.
[00:56:26] Speaker B: All right, thanks for joining us on Protect it all, where we explore the crossroads of IT and OT cyber security.
Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time, Sam.
