Episode Transcript
Aaron Crow (00:01.413)
All right. Thank you for joining me again on another podcast for protected all. today I've got Justin with me, Justin. I'm very, thank you so much for joining me. We, we, we always talk a little bit before the call, before the call. I love that, you know, the area that you're in, I've been there, spent quite a bit of time there. Next time I'm there, I'm definitely going to take you up on your offer. So Justin, why don't you introduce yourself to, to the guests or the podcast and let us know who you are and kind of your background.
Justin Searle (00:09.951)
Sure.
Justin Searle (00:24.558)
Oh, sure. Yeah, not a problem. So my name is Justin Searle. So I'm the director of ICS security for a company called InGuardian. So we specialize in doing penetration tests and architecture work for critical infrastructure, right? So anything inside of OT, industrial control systems, and also all the way up to the cloud and Kubernetes kind of specializing Kubernetes because it's kind of one of those weird organizations that kind of specialize in those two different areas. But so kind of more on my background, I've been in the field now for 25 going on 26 years, my
very first full-time job was working as a technician for an engineering firm building control cabinetry for water treatment facilities. And after that job, I then kind of shifted over to the dark side and went into the IT world and started focusing on doing IT security. So this was back in the early, or I should say the late nineties, early, early two thousands and really started focusing on more network based security controls. So a lot around Cisco, Cisco firewall, Cisco routers,
getting into intrusion detection systems, and then getting into the host-based side and probably primarily on the Linux servers is kind of where my area of specialization is on the endpoint side. And yeah, worked up the IT ranks for a number of years. You know, probably the pinnacle of that part of my career was as the security architect for JetBlue Airways trying to redesign their infrastructure to become PCI compliant, but also dealing with things like baggage handling systems at JFK airport.
and dealing with synchronization of data sets between the aircrafts when they actually pull into the bays from licensing to payment cards, but all the way down to weight and balance for the pilots. Kind of a funny side story there. I was the only employee in the entire company that was a non-pilot that actually had permission from the FAA to ride in the cockpit with the pilots, which was kind of fun.
And then after that job, kind of then shifted my career again and focused, started focusing more on the offensive side of security instead of the defensive side of security. I had always been big into testing my defenses to make sure I knew that the defenses were adequate. And so I kind of started focusing on that, joining a consulting firm in Guardians. And the very first job I got thrown into was for an electric utility company that wanted us to do some testing for
Justin Searle (02:41.654)
some of their smart meter systems, well as some of their substation automation equipment as well. And because of my background inside of the engineering field, I got thrown into that engagement. And 16 years later, I'm still doing penetration tests and work inside of the OT field. And probably on top of that, all parallel to that, big into education side. I've been teaching for the Sands Institute now for I think 13, going on 14 years.
I've also been teaching for Black Cat at all the Black Cat events for 16 years now. In fact, I think I personally hold the record of over the entire history of Black Cat teaching the most number of personally teaching the most number of students in the history of Black Cat.
Aaron Crow (03:23.893)
That's awesome, man. That's an amazing career. You and I sound probably about the similar age and kind of got in this around the same time and honestly have similar trajectories in our career. And one of the things that stands out to me and all the things that you just said is you have that diversity in your career, right? So you did some IT stuff and you worked for an engineering firm and moving into IT and then working on networks, the security side of network, even...
Justin Searle (03:28.942)
Yeah.
Aaron Crow (03:51.765)
because I was in that same thing back in that day, CCNA, all that kind of stuff. And we didn't really call it network security. was just Cisco. It was networking and all that kind of stuff. we really, yeah, we were really doing cybersecurity stuff. We just didn't call it those things. And to the same point, I got that same tap on my shoulder when I was working for a power utility and I was in their technology group within the business. And this whole NERC SIP thing came up and they needed
Justin Searle (03:57.932)
No, yeah. Yep. Routers and firewalls.
Aaron Crow (04:20.789)
cyber in the space and I was the only one that really spoke, you know, networking and that, that security language. So they're like, you're the OT guy now. I didn't know what that meant or, you know, they were like, congratulations, you got moved to this team and you're the only person you and this guy. And like, I don't know what that means and what we're doing, you know, this is back, you know, mid 2009 to 2008, maybe 2010, something like that. Yeah. was a SIP three time period. Right. So.
Justin Searle (04:28.407)
yeah.
Justin Searle (04:43.248)
yeah, so sip tooth at three time periods.
Aaron Crow (04:48.301)
We were segmenting units and worrying about high, medium, and low, and you were self-classifying things at that time. There were no OT products to be heard of. They didn't exist yet. I was architecting things, taking the experience that I'd had working in IT and forcibly bringing those, what's up gold, WSUS, Splunk, a NAC, things like that, and I was putting those in power plants.
Justin Searle (04:59.022)
Yeah.
Justin Searle (05:17.474)
Yep. Yeah.
Aaron Crow (05:18.201)
Gigamon, know, all that kind of stuff, you know, and NetDVR. So we were doing full packet capture, really advanced things in these OT spaces, but that was because we were just, it's the only thing I knew. I'd done that in the IT side and I was bringing those same skill sets over here. So what I love about that story and for people that are listening, I get this question all the time, how do I break into OT? How do I do cybersecurity? my cyber, I never took a cybersecurity class. never, know, eventually I ended up taking, you know,
Justin Searle (05:43.086)
Agreed.
Aaron Crow (05:45.861)
the CEH, you know, was CEH certified and I took some of those hacking classes and things like that. But most of my experience came from working in infrastructure, working in switching, working in firewalls, working in, you know, the OS and active directory and all of those things. All those are components of a security posture. So all those things are get really good at something that you enjoy, whether it's switching or networking. For me, when I was building out teams and this is I'm leading to a question, I promise.
Justin Searle (06:14.286)
So now you're fine.
Aaron Crow (06:15.221)
All of this leads to, for me, when I was building out teams, especially when there aren't a whole bunch of OT people in the world that, again, this is back in 2010, it's not like anybody had OT experience on their resume, because again, it just didn't exist. So I was trying to find people that I could teach the operational technology side of things, but I was having to grab them from the IT world. And I really honed in on the biggest thing that was beneficial to me was finding people with networking experience, because
You know, if you look at the OSI model, everything transports across that networking layer. And it doesn't mean if I can impact one system, that's great, but it has to get across the network. It's going to try to pivot and get someplace else and all that kind of stuff. And that networking, if I can understand that networking layer, I could really understand how to secure it. You know, security in depth and a lot of the things that we do, both from an offensive side and a defensive side really comes down to that networking layer.
So how much has that been a big impact in what you do on a daily basis and doing red teaming and penetration tests in these environments?
Justin Searle (07:16.738)
it's absolutely huge because when we start looking at our industrial control systems, right inside of an OT environment, you get down, you know, you have your supervisory levels, then you end up having all your individual processes themselves. Once a process has been commissioned, we usually can't make modifications or changes to it. And so quite often the bulk of what we're doing inside of an OT environment from a security perspective is wrapping the processes with network security because we can't modify the endpoints or add things to, you know, you can't install.
you know, EDR solutions onto a PLC for the most part, right? Even though it's starting to change, which is kind of cool. Good and bad. But yeah, so really trying to do some of that segmentation, right? Wrapping the whole OT environment with security layer, but then also trying to do some separation east and west between the processes and north and south between your site-wide supervisory or region-wide in the case of electric utilities, right? And all the different sites that you have down lower on where the processes are. So I think that's huge.
Aaron Crow (08:10.431)
So how much do you see? Walk us through it just a bit again, on this podcast, we have everybody from C-suite executives to brand new people trying to get into this and seasoned experience. kind of all the gamut. What is a pen test in an OT environment look like? Because I know it's vastly different than it is in an IT environment. So walk us through what that may, what is a typical, again, you don't have to call names out or anything like that, but just a typical, you know, pen test look like in an OT space.
Justin Searle (08:29.8)
shoot you.
Justin Searle (08:40.43)
So quite often, so I think the most important thing to think about penetration testing in an OT environment, is don't run with that assumption that we're doing what we do inside of IT and just going and compromising and pivoting through everything, right, and gaining full control over everything, right? I think the important thing to think about is when we do penetration testing, break it into two major buckets, right? One is going to be production sites that are, you know, live machines that have been commissioned, either actively running production or preferably not running production, right, depending on the environment.
Sometimes you don't have a choice. But then the other side of that coin is going to be penetration testing, which is more product testing or component testing in a lab. And that's where we actually really get down to the, you know, down into the weeds and we can really find those zero day vulnerabilities and identify them and try to work with vendors to address them. But in a production environment, which I think your question was more kind of focusing on, right, because that's where we have production concerns. Quite often what we're what we're doing inside of these production environments is we're going in with very, very
Aaron Crow (09:30.773)
Mm-hmm.
Justin Searle (09:38.914)
delicate touches inside of these environments. And I would say the primary thing that we're actually doing is we're looking for network traversals going from one subnet to another subnet and just identifying if we can actually cross those boundaries. And then depending on the environment, depending on how much time you have there, depending on how much time you've worked with the engineers, because you can't do any of this work without partnership with those engineers.
Right? We generally at a new site, we'll usually start at that site-wide supervisory level and usually test from either IT down to the OT environment to check that external perimeter or from the internet to that perimeter. And then as a second engagement or a second phase, maybe we'll go through an actual look at that Purdue level three or that site-wide supervisory area to try to identify vulnerabilities and issues there. And in that Purdue level three, you can be a...
it is more like an IT penetration test. We can be a little bit more aggressive. We're doing a lot with Active Directory, because most Purdue Level 3s have Active Directory. So we can do a lot there, but you still are being very, very delicate, very, careful. We're still being very, you know, very cautious in calling out specific IP addresses, which are going to be some of the major management machines that are there to be very careful with them.
So, you know, like inside of electric utility, right, our distribution management systems, we're going to be very, very careful with, right, anything like state estimators, any of that stuff, super careful with. But a lot of the workstations, right, we can do a lot more with those workstations. We're looking for data that's on those workstations that an attacker can use because that's where usually all the details are that an attacker wants to do low level attacks. And then maybe we'll actually test perimeter or test from that, you know, as a third phase.
maybe test for that Purdue level three down to a Purdue level two, right? So the supervisory inside of a process. So transitioning from supervisory site-wide to supervisory more local inside of the process and try to get to the perimeter or just barely inside of that perimeter of that process. But usually to get to that phase three, right? That's usually going to be several engagements down the road after I have a comfort level with those engineers because if I don't have a comfort level, I'm not going to take that liability on myself or for my company.
Aaron Crow (11:50.537)
Yeah, I mean, it's a big risk, right? We're dealing with production things. And how many of these are you doing while they're online running versus in an outage perspective?
Justin Searle (11:50.636)
Right? Yeah.
Justin Searle (12:01.454)
So it depends on what we can do based on the environmental requirements themselves. So for instance, we did a penetration test about a month or two ago of one of the larger airports here inside of the United States. And of course, with that airport, they kind of are operating 24 seven for the most part. And so we actually did that during their production hours. So we were testing specifically in this airport, we were actually testing three different segments for them, I should say.
maybe say security zones because each security zone had multiple segments inside of it. But we were testing the baggage handling systems, we were testing the CCTV, and we were testing all the physical door access for the airport, right? And so once again, super, super careful, super lightweight. We don't wanna cause any baggage outage inside the airport. We don't wanna cause any problems with locking people out from getting to the planes. So, exactly, all the above.
Aaron Crow (12:54.867)
are opening doors that shouldn't be open.
Justin Searle (12:58.894)
And so we're being very, very careful. And once again, we're primarily looking for traversal to get into these environments, to be able to identify what paths are available to get into these environments, and then trying to identify what are the key assets we can get control over. In one case, we had Active Directory. At least in two of the three, we had Active Directory deployed. And so we were able to get into the Active Directory and show a compromise in Active Directory. And then from there, you basically surmise what can you do with those credentials, right? We don't actually practice or demonstrate that.
unless an engineer wants it and an engineer was willing to take on that liability and they're working with us to make sure we're containing whatever we're doing. But for the most part, we have this access and we have the engineer basically validate what can you do with this access, right? And so that's gonna be part of it. And then another part of it is we will lightly probe some of the field devices depending on the criticality of the field devices, but we will simply lightly probe them to try to identify what types of connectivity, what types of protocols there, if there's something that...
provides authentication, maybe a light probe of what that authentication is, but we're not gonna brute force them, right? We're not usually going to run and exploit anything that would be inside of a process. The only exploitation we generally do is using our workstations and servers with very few exceptions outside of that, I would say in the production environment.
Aaron Crow (14:11.573)
What are some of the real life examples of misconfigurations or security flaws or things like that you found in these OT environments? can imagine. I know the ones I see, but what are some of the bigger ones that kind of stand out to you that were surprising or just like, my gosh, I can't believe this?
Justin Searle (14:29.902)
So getting into an environment where they thought they had some type of perimeter there, they said, yeah, we have these firewalls actually set up and we have this perimeter that's there. And so we go in and start trying to test that perimeter and find out there's no perimeter there. There's firewalls, but yeah, the firewalls literally are routing everything in. That's a pretty common one. Probably the most hilarious thing I ever saw in a perimeter was a firewall that was misconfigured that ended up doing port forwarding.
Aaron Crow (14:42.581)
It's just a router.
Justin Searle (14:57.384)
and they were port forwarding from the outside one single port to all 65,000 ports on the inside, which I didn't even know that you be able to allow to do. But what that caused is anytime we start doing port scans on that, every single port scan that we would actually send with Nmap would be amplified 65,000 times. So it would very quickly fill up the memory buffers inside of the firewalls the firewalls would stop functioning. That's probably the worst configuration. And they did that, think, three or four different times on that same firewall configuration.
Aaron Crow (15:28.19)
Wow.
Justin Searle (15:28.582)
But then to flip the other side of the coin, right? Because you asked when do we actually do this, right? In production when not production. Another kind of the opposite side where we don't do it in production is we have some large generation facilities for electric utility company that are both coal burning and gas burning generation plants. And we'll come in and usually test those annually for this company. But we only test those when they are offline or not providing power to the grid. We power everything up.
but we're not actually generating power at that time, but all the controllers are actually on in an idle state. And so we'll do that. So that would be kind of the example on the other side. And we commonly do the same thing for manufacturing as well, because usually there's going to be some times where we can find some downtime for manufacturing sites to do the same.
Aaron Crow (16:12.169)
Yeah, a percent. mean, a great thing in a power utility, you know, I'm an outage because I'm not providing electricity to the grid, but my control system is up and still running. All the equipment is there. still getting all, everything is running and available. So absolutely. So when, when I've done this in my, in my past, you know, we, we, we hired a, know, I was, I was the asset owner and we did this across 40 something sites. and, and to the point that we didn't trust them, them being the, the,
third party contractor. So I was doing all the hands on keyboard and they would just tell me what commands and where, what they wanted to do and all that kind of stuff. And then I would, because the, the engineers trusted me to your point. And I say this all the time. We do business at the speed of trust, right? It doesn't matter how great or how smart I am in my bubble. When I show up to this new place, they don't know me from Adam. They don't know who I am, who, who my credentials are and how much I know or don't know. And they're going to,
be hesitant because in the past they've been burned because their bonuses are based upon availability. If something breaks, they're the ones that have to fix it. Like all of these things are true. And knowing this, walking into these sites, half of, I know from my perspective, so I'm assuming it's probably the same from your experience, more than half of my job is just proving to the people on site that I'm not going to break their stuff and that I understand I'm only going to do things they allow me and that they're comfortable with. And I'm going to do it with them, not to them.
Justin Searle (17:38.926)
Yeah, agreed. No, I think 100 % true. Anytime I'm actually doing even something as simple as scoping where IT or management's bringing me in to do some type of an assessment, be it a penetration test or be it something more benign like a security architecture review, my very first one step is that very first call on scoping when I'm talking to the engineers to get an understanding of how much time it's going to take. My first goal is not to necessarily understand their environment. My first goal is to actually
win over that engineer and let that engineer know that, safety and reliability are the very first things on my plate and we need to work together to make sure that we're ensuring this. And I want you to be a part of this as much as you're willing and able to be.
Aaron Crow (18:20.639)
Yeah. So, so all the time that you, so you said you've been here 26 plus years doing, doing this type of thing in and out in different places. Like what are some of the trends that you've noticed that in, in, in better, good or worse, but that you've seen in these spaces of, whether it be preparedness or architecture changes or struggles and, and, and results of pin test results and things that you're finding, misconfigurations, et cetera. What are some of those trends that you're seeing, in the area?
Justin Searle (18:48.142)
I think that the biggest trend I've seen over the history of my career is the fact that we are making huge progress inside of OT. Where we are today is dramatically different where we were 25 years ago or 15 years ago, or even 10, five years, five to 10 years ago. The companies that I'm working with, they actually have dedicated security people for the OT environments now, which is awesome. I am seeing more commonly that there's actually a team of people doing cybersecurity inside of OT.
where five to 10 years ago, I would expect or hope that maybe there was one person dedicated to OT. 15 years ago, there was never anybody, right? I'm also starting to see what we've been trying to recommend through the Black Hat course and my SANS course, to constantly recommend to everybody that the number one benefit that we see for these environments is having dedicated human resources for the OT environment, right? And that's dedicated cybersecurity professionals that are only working in OT because...
That way they learn the constraints and the limitations of those environments. And they're working with the engineers on a daily basis. And there's a partnership that's formed there. And more importantly, I think the engineers actually see these individuals as their personal resource for cybersecurity instead of some outside foreign third party that's coming in and mandating things to me. So I think that's the biggest change I see. because of that shift, I think that's what's really started making a lot more effectiveness change. Because we have more visibility now.
Aaron Crow (20:02.069)
100%.
Justin Searle (20:11.214)
15 years ago, was zero visibility inside of the OT environment. Now I generally go in and I expect to see at least something on the perimeter for visibility. I expect to see some type of decent authentication and some type of decent remote access, hopefully, that I can actually monitor and see what's going in. But yeah, we're still very immature from where we need to be, but there is huge progress that's actually been made across the industry.
Aaron Crow (20:37.237)
Yeah, I 100 % agree. mean, the amount of things I've seen change since 2010 till now are drastic and we're still way behind the eight ball. Both things are true at the same time. We're not where we need to be. We're behind where IT is, but there's a reason why we're behind. can't just, you know, I say this a lot. It's like the GM Ford versus the Toyota model, right? Toyota is one of the most reliable vehicles. You you look at car and driver, you know, independent third parties say this all the time, but you also, you go look at a brand new
You know, there's a reason why the Toyota 4Runner, for instance, it was the same model for 15 years. They didn't change anything about it for 15 years because it just worked. Now, it didn't have all the bells and whistles of the Tahoe, but you look at reliability, the engine, the internals, the radio, everything about it, it's bulletproof. And that's the reason we don't change things as fast in OT, very similar to the Toyota 4Runner as they do in IT, where it's not going to be as sexy, it's not going to be as fancy.
but it's reliable and reliability and availability is vastly more important and a bigger risk to this business than it is having the latest bells and whistles, which the other thing I wanna pull out of what you just said and I think is hugely impactful. When leaders or businesses are asking me what's the best thing, what can I do in my environment? I don't have an OT environment, I don't have whatever, right? They're looking so build out this OT cybersecurity thing.
And I think you hit on something that doesn't always get lit on, right? We always talk about it from an IT side as there's people processing technology. You can't just roll out a firewall to your point. If you put a firewall in, but it's not actually doing what it's supposed to do, then you didn't actually gain value. It's just a router. So you have a really expensive router that you're not actually getting any benefit from. It's a false sense of security. I think the biggest investment that companies can make is that people, right? Having that OT person, that champion,
Justin Searle (22:30.382)
Yep, agreed.
Aaron Crow (22:32.883)
that both understands the OTP, OTP side, the systems, the operation side, and is that trusted person. Because as soon as that person is there and trusted and understands those environments, the more likely you are to be able to get technology and processes to change, that person is the key though. Without that person and some champion that the business trusts, the other stuff is that they may do it as long as it's just gonna be a hard road to get implementing change in process and technology without that trusted.
Justin Searle (23:03.406)
Oh, 100%. Yeah, 100%. Well, and the other crazy thing is if we're 100 % honest with ourselves, the way we do cybersecurity inside of an OT environment is 80 % overlap for what we do inside of IT. The problem is that 20 % that's different is absolutely critical. And you'll never identify that 20 % unless you're living in that environment on a daily basis.
Aaron Crow (23:23.861)
Yeah, I mean, that's why, you you see the hard hats behind me, right? You know, I worked, me and my team, we worked outages. We were in the control system outage. We were on site working the 12 hour shifts, you know, for six weeks or eight weeks or whatever it was. And we were doing our work during those outages for the same reason you just talked about. We were pen testing. We were doing patching during that time. We were upgrading, you know, the VMs and the patching all of the systems and, you know, running backups and all of the maintenance that we needed to get done. We had to get it done.
in that window because we were so cautious about doing any of those changes. We would only patch during those outages. So we patched twice a year. We had a spring and a fall outage and we would only patch during those times. The rest of the time, unless there was something super critical, I would not mitigate a vulnerability with a patch outside of an outage unless there was something hugely wrong. And sometimes, you know, outside of those two windows, sometimes they come down for unplanned maintenance. They have something break.
And if they're down for a week, I may send my bat team out to go do some work while they're there doing other things because the system's offline. But it's just vastly different in these OT spaces than in these IT spaces.
Justin Searle (24:34.52)
It is.
Aaron Crow (24:36.383)
So how much of this people hate this word and I'm going to say it and they hate the I T O T convergence, right? To me, that word, the convergence to me is because all of these control vendors and all these spaces and all this technology even mentioned it before PLC is you can't put EDR stuff on it. Well, actually they're starting to have stuff that you can. And as we start converging and to me, the word convergence means we're bringing those commercially off the shelf technologies into the space. We have Cisco routers, we have
Palo Alto firewalls, we've got Fortinet, we've got VMware, we've got Active Directory, like you mentioned, I've got WSUS, and I've got all these same exact technologies that are running in these IT spaces that are supporting my OT environment. And I just did a post, a video the other day about this, and I was talking about Colonial Pipeline. And I had multiple people attack me because I said it was an OT impact. I was not implying that, and I absolutely know, it was not an...
It was not an attack on an OT system. But my point in the whole conversation was it doesn't matter. It doesn't matter. Like where does the line get drawn? To me as an asset owner, when I was an asset owner, it didn't matter who, like in Maximo, my EAP system, it didn't matter whether it was an IT system, it was owned by IT and they paid for it and it was under their support, or it was one of my systems that the plant paid for. Could it impact production? Would I have to shut down a power plant?
It didn't matter if it's a PI server and IT system. If the firewall gets impacted and I, for whatever reason, can't run my power plant. And the example I gave was Active Directory actually is Active Directory and OT system. It's Active Directory. It doesn't matter what it is. It matters what function it's serving. So these are so many people get hung up on this. Well, that wasn't an OT attack. Okay.
Yes, you're right. didn't, it wasn't Stuxnet and they didn't change the code on the PLC. It doesn't matter if it impacts production. When you walk in and do a red team. Yeah, exactly. When you walk in and do a pen test, they don't care which system that you hit as long as can it impact my environment. Like, well, I was only in the firewall. I didn't expect that it would take down your system. They don't care.
Justin Searle (26:38.062)
Yeah, we had a preflop let down for a week. That was an impact. I agree.
Justin Searle (26:53.378)
Yeah. Well, and it's still a cyber security issue as well, because in Colonial, there was a weakness, right? They had inadequate controls between their ITOT perimeter boundary. And while they did have some level of control there, right, they didn't have the full visibility. They had never actually tested islanding, which is what we generally recommend inside of those situations. right, going into an islanded mode without ever testing it would have been the wrong decision to make because we don't want to destroy our process. Right.
But yeah, mean, 100%, right? That was an OT impact and it was a management decision to impact OT because they were afraid of it, you know, actually losing control. And that's a better way to do it. You know, gracefully bring down the process instead of letting malware rip it down, right? But guess what they can do today? They can island today. Yeah.
Aaron Crow (27:39.507)
Amazing. have, it's, it's, it's the same thing as, as, you know, you see the same thing when, when you're in an area and, know, I grew up in a small town in, in, Texas and, we never locked our doors, you know, left car keys, just sitting on the dash of the truck, you know, all that kind of stuff and never had problems. and you, you see in the news, people's cars getting stolen and houses getting broken into and you're like, well that's somebody else until it happens to you or it happens to your next door neighbor. And then you're like, wait a minute, that.
that my car got broken into. I have to look at things differently. Unfortunately, especially in things like cybersecurity, I can't tell you how many plants and plant managers I've talked to and were like, well, I've been here for 40 years and we've never had a cybersecurity attack.
Justin Searle (28:22.786)
Ha ha ha
Aaron Crow (28:24.533)
You're probably right, but also you may not even know. You may have had somebody break into your environment and you don't even know about it. Maybe they didn't do any bad or maybe they didn't break something. That doesn't mean that you weren't impacted at some point, right? So what are the...
Justin Searle (28:30.456)
No clue.
Aaron Crow (28:40.853)
As we start looking at these, how do we start changing that conversation? How do we stop making IT a bad guy and OT a bad guy and fighting over whether it was an IT or an OT attack? you know, what is the bet? How do we get to a place where people are looking at this and realizing we're on the same page, we're on the same team and using process? Because I can't tell you how many people that I've been in OT and they would, they're
terrified to do a pen test in an OT environment. How are y'all able to kind of see that and validate and show the value and that it's not just something I want to do in my IT world. There's a big benefit in the OT side as well.
Justin Searle (29:16.91)
No, definitely, definitely. I think the biggest benefit to doing inside of that, inside of the OT area is once again, right, focus on the things that are important to OT, right? If you go in and you have a penetration test in the OT environment and you generate a report with 500 missing patches, and that's 95 % of all your findings, right? That's not gonna be helpful for the OT people because they know they're missing those patches, well, at least I hope they know. Generally the most common when it comes to patching,
The most common recommendation that we have for our customers is usually that you need to have a patching program to actually review and document and make these decisions in an organized manner. That's the number one recommendation. And so if I do know they have a lot of missing patches, usually I roll it into that one recommendation and then say, hey, look at appendix A and I'll give you the full list inside of appendix A, unless there's like very specific assets that are missing patches, which are absolutely critical. Then I'll call those out separately. But I think the...
in order to show value of a penetration testing side of an OT is show the things that are the most important, which is how can attackers actually get to the assets? How can attackers get across the boundaries that we actually have? How can we actually demonstrate that we can't see that the attackers there in the first place and show a lack of visibility inside of that area? These are things that generally I think engineers are going to have a better appreciation for as well as whoever's running that program, because that's where we're putting most of our time and energy inside of these systems is on that perimeter.
and I think that's, that's kind of the key of it. And kind of to your, your, your first part of your question is how do we make that IT OT work together, right? A little bit more cohesively is I think that Casa come down to a partnership and an understanding that we have two separate groups here, right? Dedicated resources for cybersecurity inside of OT. And as a side note, anytime I'm talking to like CIOs and CTOs in these organizations, I usually make the same recommendations for all their IT resources. So network engineers, I recommend dedicated network engineers for the OT area.
Right? Same thing with Active Directory, dedicated Active Directory engineers for the OT area. And then from a management perspective, what are the best ways I've seen to do that? Right? You can have them each go up the engineering line or up the IT line for reporting, but you can also go ahead and have the OT team, a sub team of the IT team. Right? And I really like that method because being a sub team, they actually have the permission to be able to make changes and modifications that are necessary for that environment.
Justin Searle (31:39.502)
but they're still part of the greater unit. And so there's a little bit more cohesion between the two and we're still going in the same direction, which comes back to your initial point for the question, which was ITOT convergence, right? I actually like the term ITOT convergence, but I think it has been misused because most of the technology and most of our vulnerabilities are because of these IT technologies that we're actually adding. But the thing I try to talk to people about when we talk about ITOT convergence is, hey, let's reclaim the term, but reclaim it in the right way.
Aaron Crow (31:53.821)
too.
Justin Searle (32:09.184)
If we think about two lines, so go back to math, right? High school and college math, when we talk about two lines that are converging, those two lines or those two rays never become the same line. They are just becoming closer and closer together as we go. And that's what we're trying to do. We're trying to gain some of the benefits we have in IT inside of the OT environment, but it is a separate line. It is a separate ray. It is a separate entity, a separate group.
And so if you want Active Directory, great, deploy Active Directory separately there. If you want backup management, deploy backup management separate there. If you want identity management solutions tied to Active Directory, great, deploy a separate identity manage there and actually use human business processes to do the synchronization between IT, OT. Don't use technological means to do that synchronization, because any technological means that you use management, co-management between IT and OT at the same time, that will always be the gateways that I'm using to be able to get into your OT environment.
Aaron Crow (33:01.201)
Absolutely. You've got to have that, that segmentation separation of duties, even of, know, my, I, unfortunately it means that my environments, my OT environments can be super complex. And I almost, especially again, my, a lot of my, my experience comes from, you know, working in power utility for, you know, much of my career at power plants and substations and things like that. Right. And, and, and because of that, almost every one of those units is almost like its own business.
So it has its own Active Directory. It's got its own antivirus and patching and all of the tech stack. That may mean at one site, I have five Active Directory for us and I've got five sets of firewalls I've got five sets of backup and five sets of antivirus and five sets of everything, which can be a little complex, but some of those things are needed. Now, some of those things have a nested my WSUS. Like I have one server that's reaching out for patching and bringing the patches in. You can do some of those things,
The thing that we struggled with, my team was dedicated in the business side and I had six people that worked for me and another probably five or six contractors that assisted. And that was supporting 45 power plants across Texas. So that was a really big scope with a very small team. And we were supporting everything from the firewall to the VM host to the applications.
the networking layer, everything at a power plant from a networking layer up, you know, all seven layers of the OSI model. Like we supported all the technology stack and my team had to be, you know, an inch deep and a mile wide on all of that stuff. Like I had people that were better at VMware, other people that are better at switching and networking, et cetera. But I didn't have a dedicated networking person. I didn't have a dedicated active directory person. Now I was an active directory administrator. I was an exchange administrator. I was a firewall administrator. So I had a lot of those skillsets.
But on the IT side of things, they had a dedicated networking team and a dedicated firewall team and a dedicated, you know, VMware team. And you get the point. I think you hit on something really important is, is having that, that, that ability to build those bridges across the IT and OT where yes, I should be the final decision. My team should be the final decision on firewall changes, cetera.
Aaron Crow (35:19.721)
But we should be running any changes we're doing through the firewall team that that's all they do every day. They know the latest and greatest. They look and speak firewalls 24 hours a day, seven days a week. Why am I going to expect that I'm going to be able to see everything that they can and that they can't provide value? I bring in the OT operations and exactly. Yes.
Justin Searle (35:37.006)
especially if you're an engineer, right? Especially if your primary job is engineering, right? We want our engineers to actually do engineering. We don't want our engineers to do IT management. And that's why I'm such a big fan of doing the sub teams inside of IT, right? So I have five or six network engineers. Let's go ahead and assign one of them to be the primary OT person. And anytime there's something to do network related inside of OT, that's the person that goes and does it. So that way we have that consistent connection point.
Aaron Crow (35:44.533)
Correct.
Aaron Crow (36:01.397)
Correct.
and I would train those people, I would send them to plants. I would make them work in outages. would make them have the hard hats and the steel toe boots. It's going to do, A, it's going to, and this gets back to, there's way more than just the technology side of things. So we talked about people, process, and technology. They're going to understand the processes at a plant. They're going to understand, they're going to make relationships and connections and build trust at that plant because it's one thing to...
to give somebody a call from the ivory tower and corporate and never actually be there. It's another thing when you're standing side by side with somebody in an outage at 10 o'clock at night, troubleshooting a problem and they see that you can fix the problem and you helped them because you're wearing the hard hat, you're wearing the PPE, you're there with them, you're not leaving, you're not abandoning them. it's five o'clock, I'm out of here, see you later, I'll see you tomorrow when they continue working, right? There's a lot of trust that gets missed.
by people just staying, showing up and wearing, we will do, when I bring my IT people in and I'm training them and they, the dress code in an industrial environment is drastically different than it is at corporate America. I'm like, do not wear slacks, do not wear dress shoes, like all of these kind of, you can't do that because as soon as you show up at a power plant in East Texas and you're not wearing, and you're wearing,
fancy looking shoes and slacks and a brand new hard hat that is clean and I would literally take my new guys and I would take their hard hats and scuff them up in the dirt in the driveway so that it wouldn't look like it was the first time they just took it out of the plastic.
Justin Searle (37:36.94)
make up the target.
Justin Searle (37:40.78)
Nice. Very nice.
Aaron Crow (37:42.441)
But that perception makes a difference, And experiencing those things helps you have an understanding of why we don't patch, you know, remotely, why I'm not updating things remotely without letting somebody know. I'm not just going to reboot a machine in the middle of the day because there could be an operator sitting at that machine controlling something. To your point, we don't have outages. There's 24 hour things. They run all the time.
Justin Searle (38:01.71)
Yeah.
Yeah, 100%, 100%. And I would say even on a more simplistic level, right? Because we do have limited resources and so we might not be able to get them back on the plant themselves. But if we just have them do the safety training, right? That safety training is going to be an eye opening for them, right? And I honestly believe that anybody doing any type of configuration of technology in a plant should have to have the same safety training of people working at that plant. I think that's something that should be a hard, fast rule. I mean, that in and of itself just gives them some insight into what could go wrong.
Aaron Crow (38:16.789)
Correct.
Aaron Crow (38:31.401)
Yeah. Well, and the last piece to me was the fact that we were a power company generating electricity. The budget for the IT team was drastically higher. mean, 10 X what my budget was and my thing. Yeah. And my thing was the thing that made us money. And if my thing went down, we were having a bad day. If the email server goes down, somebody got annoyed, right?
Justin Searle (38:45.806)
Yes, still is generally.
Aaron Crow (38:59.029)
And I'm not saying that we should take money away from the IT organizations because absolutely we need to, they are the first line of defense before, in theory, before anything gets to us. Like I absolutely want those things protected. At the same time, we can't just assume that that's enough. We have to continue to put budget and there's no goal line for me. I know you do a red team. You mentioned a use case where you guys come yearly. Do you not?
like you just did a a a pimp test last year at site a whatever this company is when you come back the next year do you expect that you're not gonna find anything wrong
Justin Searle (39:34.928)
no, we expect to find most of the things that we found the previous year probably still there. So, but we hope that the bigger findings, especially talking about access and traversing the network, we hope that those are addressed, but we do expect to see other configurations or critical patches still missing for that environment because there are constraints about doing some of those things.
Aaron Crow (39:53.609)
Well, in a year there's gonna be new vulnerabilities and zero days and all sorts of new things that have come out that weren't available last year. So you're also a new attack vectors and new problems that exist. again, like you said, in these spaces, we're not patching all the time. We're not upgrading our systems. We may have Windows XP running in these environments intentionally and they're not going to change it. So knowing that, okay, you know that's there. So how can I protect against these?
That's another difference I assume in your red teaming is sometimes you know that these things are going to be there. How do I protect these environments knowing that I'm going to have less patching, knowing I'm going to have older systems that are out of patching availability or support like Windows XP or sometimes even where Sun Microsystem stuff that's still out there running this stuff. And you still have to red team these things and decide, okay, I know these things are there. I'm accepting that risk, but I still have to mitigate it in other ways.
Justin Searle (40:45.068)
Exactly, and that's why we're primarily, the primary benefit for penetration testing in OT is that access and that traversal and the moving around the environment is one of the biggest benefits that we actually have for those environments because we are overly dependent upon network defenses in OT because we can't have the endpoint defenses that IT has, which is a huge, powerful defense, right? In most instances, especially in the process, we can't do that. level three, can do that a huge percentage of pretty level three generally.
But when you get down into the process, we can't do endpoint security for the most part. So we are overly dependent upon network defenses. So our penetration test should actually show that emphasis on where we're getting through those network defenses.
Aaron Crow (41:27.221)
for sure. So we've talked all through this. I love the fact that you have customers that are doing pen tests. Would you say that how many of your customers are mature and do an annual thing like what you're just talking about, have a robust program and have maybe a good grasp on what they have versus folks that are just like, I don't know anything about my OT environment. Maybe this is a first stab at.
trying to figure out and implement some things, because I don't even know what my problems are to be able to implement some changes.
Justin Searle (42:00.418)
Yeah, I would say a small percentage of our customers are actually are having us come in and doing an annual penetration test with us. Unfortunately, we still have probably 50 % of all of our customers being the very first time they've ever done a penetration test in their environment. And they're often not coming back that following year, mostly because they're still trying to wrap their head around what we gave them the first time and actually build a program, right? Because that's huge. Yeah, and it's crazy all the areas that we work in, right?
from poultry manufacturing and growing chickens and processing chickens to going to pharmaceutical companies, chemical manufacturing companies, automotive manufacturing, aerospace manufacturing, satellite communications, substation automation inside of transmission substations and distributions. Yeah, things like microgrids for military bases, right? All sorts of.
crazy, crazy little things. And there's just a wide variety. And it's interesting to see kind of across the board where the different maturity levels are, where electric utilities tend to be the most mature, mostly because of the mandates with NERC SIP and all the regulatory requirements. And I would say behind that probably oil and gas. And then in manufacturing, would say maybe pharmaceuticals tend to be a little bit more mature because they tend to have more money, bigger revenue values.
Aaron Crow (43:20.319)
Yeah. Yeah. And then you get into wastewater and things like that, that have no budget, no people, no resources like, you know, and, and, and similar things like these are new risks that, that we didn't, you know, 40 years ago, we weren't thinking about again. And the vendors started bringing this convergence. They started bringing commercially off the shelf switches. started networking everything. We started wanting to get data out of these things. We started bringing in windows and active directory and all these things. And we didn't necessarily understand what we were doing.
Justin Searle (43:25.08)
Yeah, that's hard. Yep.
Aaron Crow (43:49.149)
on the business side as we were making these available, but we also brought these risks in with all of these things that great. And I'm not saying we shouldn't have done it, but we also didn't do the other side of the coin, which was securing, which we did in the IT space 20 plus years ago, we started this process. We're just behind the eight ball on the OT side.
Justin Searle (44:06.828)
Well, and I think another perspective that's really good, if you are new to this environment, you're new to OT and you're coming from an IT background and you're trying to wrap your head around, why do we have so many problems? Why do we have so much insecure communications inside of these environments? I think the best perspective that a new person to OT cybersecurity can bring in is anytime we think about the OT environment very simply, where we have our supervisory systems that are servers and workstations, very IT-like, but not IT.
And then we have all of our processes, right? And the process are going to be a manufacturing line or something similar, right? When you think of that process, think of that process as an endpoint, right? We designed that process, that manufacturing line, very similar to how we design a motherboard on a laptop, right? We have inherent trust. We are just working for basic functionality. Most of the communications that we have on the motherboard between our chips on the motherboard have zero security in them, right? Basic serial protocols going back and forth.
It's the same thing we have inside of our OT environment when we actually get into the process. So that's where we're coming from.
Aaron Crow (45:05.652)
Yeah, 100%. Yeah, I mean, it's vastly different and it's okay to be different. We just have to understand those. And that's where we start building that trust is when we understand those things and then we're not trying to just cram IT processes and policies down in an OTSpace, because all it's going to do is cause havoc. It's going to break things and then they're going to push you away. They're not going to trust you anymore and they're not going to let you on your site.
Justin Searle (45:23.566)
has to be modified.
Justin Searle (45:27.758)
We have so many examples of that over the last 15 years. I would say 2010 to 2015 was probably the nightmare years of that occurring.
Aaron Crow (45:30.335)
Yes.
Aaron Crow (45:35.925)
100%. Yeah. I have so many war stories around that time that the, firsthand experience of, uh, of that breaking and all that kind of stuff. The other thing that I always tell people is, is when you're, especially when you're an outsider and anybody that doesn't work at the site is an outsider. Um, so always say, you know, how do you win friends and influence people, especially in a manufacturing and OT environment, bring donuts, bring food, bring, you know, break bread, like find ways that you can connect with those people on a personal level.
And then when they start trusting you and seeing you as a person and that you care and you understand and you want to help them, then they're more likely to let their guard down and kind of maybe listen to you more on the technology side and the process side because you've built that rapport. I had a mentor tell me a long time ago, all business is a people business. And once you realize that, yes, you're doing red teams, yes, you're doing penetration tests, at the same time, you're having to deal and engage with people. And when you realize that that is the
probably the most critical part of being successful in your thing. And you focus on that and really dedicate time to building that trust and that you're gonna be quicker, more responsive, more capable, and the outcomes are gonna be more successful because you focused on the people side of that.
Justin Searle (46:54.614)
Agreed. And remember, I would think as well, right, if you're coming from an IT background, remember these OT engineers, they're technological geeks just in a different way than you are. And so one, another way, besides focusing on reliability and safety and always starting with that and always rolling everything back to that in my conversations, right? I think another thing that really helps win over some of the engineers is let them shoulder surf what you're actually doing them, showing some of your tools that you're actually using because they often like to see what's going on. And if I ever do component testing where I'm pulling things apart,
They would love to see what's inside the box, right? Bring them in, show them.
Aaron Crow (47:25.535)
Yeah. Yeah. Yeah. These are very, very capable humans that are very intelligent. They're very good at what they do. Yes, they don't have a 20 years experience doing networking, pen testing, but they're very capable and they will catch on really quickly. And usually the way that the reason they're hesitant is because they don't understand. And when you can show them, how can you, how do I know that you're not going to break my stuff when you can show them? Well, this is what I'm going to do. We've done this in a lab. I've built many labs for that exact reason, because I wanted to show them.
I want to show people and prove that I can do this. What the outcome is going to be and how it is safe and not going to break something. So they understand it. And once they get that aha moment, they're like, yeah, well, I'm okay with that. Yeah.
Justin Searle (48:08.418)
Yep, exactly. be humble and ask questions to right? I mean, because you need to show that you're dependent upon them and they have answers that you need. And at the same time, right, share back to them, right? Help fill in some of their gaps as well. And if you can get that part where you're actually equally sharing knowledge and you're actually collaborating together, right? That's where we want to be.
Aaron Crow (48:17.738)
Yes.
Aaron Crow (48:27.901)
Absolutely. 100%. So next five to 10 years, what's, what's one thing that you see come up over the rise and that maybe is exciting and maybe something that's concerning.
Justin Searle (48:36.11)
boy, of course, everybody throws out the whole AI thing. You know, yeah, it's it's it's there. Right. It's I think AI helps and helps us and and hinders us equally on both sides of the coin. So I'm definitely not going there. I honestly don't think AI is going to I honestly don't think AI is going to have as big a change as we I just see AI literally as the next step in automation. Right. IT.
originally came from our field inside of the OT, right? It's just another higher level of automation. I think AI just is that next step of automation and the evolution of the automation world. So honestly, I think that we just continue to progress forward. And I think the biggest thing that we see is just more maturity inside of these areas. And I hope that we continue down that path and gain better visibility. We start actually having better dedicated resources inside of the OT, right?
environments and we actually start understanding that yes, this is a different entity and if you are worried about trying to maintain the highest level of OT cybersecurity integrity, what you would want in an oil and gas or electric utility or manufacturing, which would be very different than business management systems where you don't necessarily care about that level of separation. But if you want the highest level of separation, highest level of security, think of that OT environment as a separate entity, like a separate business partner.
and you want to exchange data back and forth between IT and OT like your company does to their business partners across the internet. And if you can get to that point where you aren't sharing VLAN space, you're not running OT VMs on a physical server sitting inside of IT, you don't have Active Directory spanning the whole area, right? If you can get to that point, right? I think that's what we're gonna see coming into the future is that we get more and more companies taking this very seriously and getting better resources inside of that.
that OT environment. And then on the flip side of that coin, what do I think is kind of the worst thing? Well, this is something that I've been worried about for, you know, ever since I've been inside of this field. And honestly, 15 years ago, I would have predicted way before now, right, that we would have these more problems. But I have to say that I think the worst part in the next five to 10 years is I think we have to have at some point, some type of cyber attack directly on OT that has a direct impact to human life at some level of scale. And
Justin Searle (50:50.924)
Honestly, having been in this field and being the one doing the penetration tests and actually finding how easy it is to break into so many of different organizations, I am honestly dumbfounded that we have not had more really any significant, right, documentable direct loss of life from a cyber attack. It really does blow my mind that we haven't seen that yet. So unfortunately, I think that is something that still is down the road that's going to happen.
Aaron Crow (51:17.781)
I agree. Unfortunately, I think you're right. And sometimes it takes something bad happening before people, before the lights get open and money gets pointed. You you talk, we talked about Colonial Pipeline. They have a huge cyber budget now and they fixed a lot of those problems because there was this big incident and it was in the news and it was embarrassing and all the things. Same thing with Target and you name it. You see a cyber event and then you see a bucket load of money be thrown at cybersecurity and then they have a really mature and secure environment. Not that it's ever secure, but it's improved security.
And unfortunately, I feel that you're right. Too many, we've been in this space long enough to see that it's not going to change until they see, you know, a reason why, and a significant reason why that they have to.
Justin Searle (52:00.014)
Yep, agreed.
Aaron Crow (52:01.577)
So all this, thank you so much for all this conversation. How do people call to action? How do people get ahold of you? We didn't even talk about, you know, your website and all that kind of stuff. So why don't you share a little bit more about you and how people could get ahold of you. Maybe they wanna do a red team. Maybe they've never done that kind of stuff, a pin test in their OT environment. There, maybe they wanna become a red team pin tester, all that kind of stuff.
Justin Searle (52:22.946)
Yeah, no, definitely. I think the best way to get in contact with me is just through the either directly social media. Feel free to reach out to me anyway. LinkedIn is probably the easiest way to be able to find me. But going over to my website, ControlThings.io, ControlThings.io has a whole bunch of open source that I've actually written. basically everything except for the courses that I write, pretty much everything else that I do, any of the research I do, any of the tools I write, I try to release open source publicly to the world. And the only thing I kind of hold back to myself really are the courses themselves.
So feel free to reach out there. There's actually an invite on controlthings.io that will actually take you to my Discord server. And there's about 3000 of us that are, you know, doing penetration tests and different types of active assessments inside of an OT environment. And that's what that Discord server really is kind of focused on. And if you really want more training, right, definitely check out, I have two courses out there. One is through the SANS Institute, which is ICS 410. It's called Security Essentials for SCADA and ICS. And it's really just talking about how do we do
good architectures and good defenses inside of an OT environment. And then the other course that I have is an offensive course, so kind of flip side to the defensive. And that's my assessing and exploiting control systems and IIOT course that I had been running at Black Hat for the last, I guess I've been running at Black Hat for going on 12 years now, that course. But I teach it at all the Black Hat events, every single Vegas event, every single Europe event, which is currently being run in London and about 50 % of the Singapore events, I'm actually running it there.
But I also offer it as a self-study online too for anybody that can't make it to the public runnings. So, yeah, I'll throw the website.
Aaron Crow (53:52.479)
That's awesome. Guys, this is a great way to get into, learn more about these spaces, be able to get these skill sets and have that in your quiver of capabilities in these spaces. We need more and more folks that have these capabilities and understanding and learning. Learning for people that have been in the trenches for 25 years doing this is a great way to exponentially speed up your learning process.
Yes, you can tinker on your own, but learning some of the ins and outs and dos and don'ts and all that kind of stuff is hugely valuable in these spaces. And it's also that building that trust within your environment. Hey, I've taken this, these things, and I have these certifications. I know in my career, a lot of my career was I'd go get a CCNA or I'd get an MCSE or I'd get the CEH and all the different certifications that I had. Novell Netwear, that tells you how old I am.
Justin Searle (54:40.662)
Yep, yep, that was there. I never finished that one though.
Aaron Crow (54:44.797)
Exactly. But all these certifications, A, they show proof that I can do this thing. I've got the certification. It's kind of that credential on the wall. You have a degree and all that kind of stuff is great. the great thing I love about these, and I'm not discounting higher education at all, but the great thing about these courses like this is that they're so easy to change on the fly. If a new vulnerability comes out, you can adjust your training like that. And next time you do it, it's updated. Whereas,
at a four-year university, it's a lot harder to adjust the curriculum in these spaces. There's all sorts of bells and whistles and things that you have to go through, hoops you have to jump through. So it's a lot easier for you to do that on a smaller course.
Justin Searle (55:22.012)
yeah, my Blackhat course I updated probably four times a year for my Blackhat course and my SANS course I update once a year.
Aaron Crow (55:25.333)
Correct. Right. Exactly. So definitely check that out. If you're going to be at Black Cat in Vegas, definitely check out that training. that. SANS courses are amazing as well. Like all of these are great resources and opportunity. Justin, thank you so much for your time today. I really appreciate it. It was a great conversation. I love diving into red teaming and pen testing in these spaces.
I know it's not something everybody talks to. People are scared of it, but I absolutely think it's value add. And I think you can do it in a safe way if you have the right people. Like, don't get somebody that's never been to an O.T. and exactly right. And don't have somebody that's never been to an O.T. site doing a pen test on your O.T. site because they don't understand what they're doing. It's a different process. Well, thank you, sir. I really appreciate your time. I'm sure I will see you in Vegas and grab you a beer or something like that.
Justin Searle (55:54.606)
Yeah, agreed. And work with the engineers.
Justin Searle (56:05.738)
Definitely true.
Justin Searle (56:15.65)
Sounds good, anytime. Thanks, Aaron. Bye-bye.
Aaron Crow (56:16.999)
Alright, man. Thank you.
