Episode Transcript
[00:00:00] Speaker A: You're listening to Protect it all where Aaron Crowe expands the conversation beyond just OT delving into the interconnected worlds of IT and OT cybersecurity.
Get ready for essential strategies and insights.
Here's your host, Aaron Crow.
[00:00:18] Speaker B: Thank you for joining me today. Protected all podcast. I'm your host Aaron Crowe. Today I'm excited about having Adam on. Adam and I have worked together for, for quite a while now. He just happens to work at one of my quote unquote favorite firewall companies. I have a lot of experience actually using palos in, in in OT and actually transitioning from other non next gen firewalls. And again this is, I'm dating myself, I haven't been a firewall admin for a long time. But you know, doing that transition back in the day, we rolled it off I think, I think we supported I know 150, 200 firewalls across, you know, power generation sites. So it was really a breath of fresh air to be able to use some of the, the capabilities and the firewalls and honestly I'm diving a little bit deep before I even let you talk. But you know some of the things that the PALO really did that was, that was so different for us is I had a team of six and they none of us were necessarily deep level firewall people. Like we had to support the computers and the virtualization and all of the tech stack and a lot of the problems that we were having was just troubleshooting like hey, this thing isn't working and we don't know why and it would go into the firewall and we wouldn't know the other firewall, not the palo. So whenever I was able to put in the palos we were able to track the traffic and actually see hey, the PALO did this with the thing like it accepted it on this rule and it went out this way so we could very easily troubleshoot the packet and see it come in and go out. Okay, it's using this rule and all that. So it really made up us beneficial just from a troubleshooting perspective even beyond the next gen firewall capabilities and application aware rules and all that other cool stuff that it does. But just that basic level capability was night and day difference in an OT world where we didn't have that capability before. So with all that to say Adam, why don't you introduce yourself and tell us about you.
[00:02:15] Speaker C: Hey Aaron. Yes, I am Adam.
I'm the head of the OT3 research at Pentagon Palo Alto Networks. And for disclaimer that what I will Say here it's my personal opinion, not company opinion.
I know you love Palo and I love Palo as well. And we did a really have a good time working together.
And I also, I agree with you. There is a lots of things that I like about the firewall and how the firewalls can be in the OT environment and bring like it's a game changer when give you more details and more packet analysis.
And I'm more than happy to talk more about these, some of these capabilities or how we actually used it also for threat research as well.
[00:03:05] Speaker D: Sure.
[00:03:06] Speaker B: So what is it, what is it that you do in your role at a 50,000 foot view? If you're somebody, if you're talking to somebody in an elevator and they say, hey, what do you do for Paloma? What is it? Like it's a cool title, but what is it that you do?
[00:03:20] Speaker C: So this is a really good question because every day I wake up, I ask myself the same question, what that means, what do I do? What do I do today? And basically I'm trying to help identify threats that impacting OT environment.
[00:03:37] Speaker D: Sure.
[00:03:39] Speaker C: How can we do that? There's so many different ways we can do that. And that's why every day when I wake I was like, how can I do this today? How can I help finding new threats? How can I help inform our industry about new threats or evolving threats or even old threats, like what are the information they need? And I usually like to start my presentations, my classes, any meeting that we are the cyber security professionals, we are coast, we are not revenue and that's why the CFO don't like us.
[00:04:17] Speaker B: Like very true.
[00:04:20] Speaker C: Yep. Been in lots of meetings and I know the CFO's very well. Like we are friends outside of the meeting, but inside the meetings, like nope.
[00:04:29] Speaker D: Yeah.
[00:04:31] Speaker C: So when I go to meetings was that mind it helped me also to understand what information are important for our industry to know about these threats because these threats help them to make decisions.
[00:04:44] Speaker D: Sure.
[00:04:45] Speaker C: Right. So what information that can help make these decision? It's what I focus on every day. And that date by day to day can change sometimes. Like there's a new malware. Okay, so we need to focus on new malware, malware analysis, reverse engineering, new vulnerabilities. Then I'm focusing on CVEs and signatures and alerts and so on and so forth. New technology. Then I'm. I'm looking for new attack vectors at attack surface and how that can impact and do some validation and use cases, so on and so forth.
More Information or insights like they want to know like what happened in the past year or what's the new trend. So like publishing white paper, collect thread telemetries and doing analysis and correlation between data to get more insight and more meaningful information. So as you can see, it's, it's a very different angle to, to do some threat analysis. It depends on when you wake up, what's going to happen, what, what, what comes to my email.
[00:05:54] Speaker D: Sure.
[00:05:55] Speaker B: Well, what I love about that though is that you are focused on OT and, and you know Palo Alto obviously is a very big company. You know, they have firewalls and everything. I'm sure from Amazon to go book and any, any big name company you can think of and the IT problems and vulnerabilities and threats and how I attack things. I talk about this all the time, but it's different. Yes, the tech stack is the same, a firewall is a firewall. But how I use IT and what problems I'm looking for and the, the attack vectors are different like so it's, it's different enough. And that's what I love about Paulo, is that you guys have a dedicated environment and you're not the only ones. But I love the fact that you guys are dedicating and investing because every company has ot. Obviously the, the duke energy of the world's OT is a little bit different than you know, Amazon, but it's all ot and if you own a building and like OT is everywhere but it's enough of a difference that dedicating time and resources to looking at what are these problems, how do I solve it in it? It can be the same problem, but I solve it differently in OT instead of how I do it in it.
[00:07:05] Speaker C: Yeah, I totally agree with you. And when I originally joined I came in just to focus on ot. And then I start my rules start to grow and then start to hire people in my team and I start to work cross functional also to impact other teams as well. Like okay, I may not, I cannot do everything by myself or like I have a very small team and we are still growing.
[00:07:29] Speaker B: Yep.
[00:07:30] Speaker C: But I'm taking advantage that it's a huge company and we have a lots of other experts in different fields. And so I tapped to those experts to say like hey, let's collaborate and work together. I will bring the OT side, you bring the IT side and we can work together on building that model or building that research and so on and so forth. So it's, it's definitely an advantage that to have see like a Dedicated team just to talk about ot. And it comes with its own challenges as well. Like it's, it's, it's a good thing to have. And it's also a challenge too that you need more flexibility.
Sometimes you can feel isolated because you talk OT while everybody's talking it.
[00:08:16] Speaker D: Right?
[00:08:17] Speaker C: And then you take that rule of I need to keep educating everybody around me about why it's different, how it works and making like, you know, any decision making.
So it is a challenge that I like to have every day. So for, for this opportunity.
[00:08:38] Speaker B: Well, you know, some of the, some of the very simple differences that you know again from my, you know, implementing firewalls like, like the palos into these spaces. The difference may be as simple as, you know, when I see a threat or a vulnerability and the firewall detects it in an OT world I may just acknowledge or notify as opposed to on the IT world I'm going to block, kick out, you know, be a lot more aggressive on my response. Whereas in an OT world I'm probably just going to raise my hand and say, hey, look over here. There's something going on. Right? To be really clear. And that's because of the availability and safety and all the things that go along in an OT world and why it's, it's a little bit different.
[00:09:20] Speaker C: Yes. And also like we recently build actually also an OT lab, like a specific environment and a rail and OT environment, like the full Purdue model stack, like from level 0, full automated factory system and all the way to level four. And just by putting myself on someone else shoes like, okay, I'm trying to install this firewall here in a real environment. What are the challenges?
[00:09:49] Speaker D: Right?
[00:09:50] Speaker C: And I had an automated engine like automation engineer, like a electrical automation engineer. I have a lab assistant, have, you know, all my team. And there are it, there is ot, so on and so forth and doing commissioning for the lab and something is not working. The first thing the automation engineer will say, oh, it's the firewall.
[00:10:23] Speaker B: I've heard that before.
You have to prove it's not the firewall before they'll believe you. Exactly. Which goes into my very first story. But go ahead.
[00:10:32] Speaker C: I was like, dude, we are working the same team, like, right?
So it was really like, I was like, okay, now, now I get it. Also like to prove that's not the firewall, we put like any, any, any, any, any just here, prove it. And it will be surprising if it worked.
[00:10:54] Speaker B: Right.
[00:10:54] Speaker C: It may have nothing to do about the firewall, but it just incidentally and Then we want to run, like, finish this project quickly. We need to operate it so we can, you know, doing our research. So we don't want to like, okay, the firewall rules. Maybe something is challenging. Just put any, any, any. So we can get the traffic and get things done to move on. Right. So these challenges, like between the time and the resources and like all that pressure, it's so understandable why there's a lot of misconfigurations.
[00:11:27] Speaker D: Yeah.
[00:11:27] Speaker C: In the OT environment. So understandable why like someone will choose not to have a firewall or even to not to enable the right rules on the firewall to get their factory running. Right. Yeah, it's, it's, you know, it's like it's cost benefit analysis.
[00:11:45] Speaker D: Yeah.
[00:11:45] Speaker C: Is it more beneficial for me to worry about security or just get the product production line going so I can make money?
[00:11:52] Speaker D: Yeah.
[00:11:53] Speaker C: So it's. Yeah. Even, even just like the detailed of putting the cables on the PLC or connecting like the iOS to the PLCs and how long that takes or like labeling. Like there's so many details that, like from an IT perspective that someone may not understand that at all.
[00:12:15] Speaker D: Right, right.
[00:12:17] Speaker C: And that also like every PLC work completely different and may need different configuration. Like, even like something like in a hardware, like when a PLC to move it from a remote to. To local. Sometime like you may do it this conf to configure it on the project file or you need to go on the hardware and manually push the button from remote to manual. It's a small detail, but it makes different. Yeah, small detail, but someone like, oh, like, well, would remote access to this PLC be more challenging than other plc? Well, it's. The mechanism is different, that tech surface will be different and the operation also will be different and giving also like during COVID and a lot of challenge that happened during COVID like when people needed to work remote, that also opened the loss of solution. Okay. We need to make sure like our engineers, our technician can access remote to troubleshoot and fix and you know, all of that. And that's just open the remote access like a huge door.
So I'm sure you know all of that. So I appreciate.
[00:13:34] Speaker B: No, but it's good to go over. Right. Is, you know, as we talked before this, you know, there's people that listen to this podcast. Some of them are getting into ot. Maybe they've recently become responsible for ot and they come from an IT background. You've got leaders that are, you know, they're responsible for ot. Maybe recently or even if They've had it, but they didn't realize how big of an issue it is. You know, and little things like, you know, again, going back to basic stuff, you know, a lot of the problems that we would see when we're rolling these things out is, is as I'm transitioning from a, you know, a stateful firewall to a, to a next gen is, the rules were so vast, they weren't quite any innies, but almost like they would have, you know, port 44,000 all the way to 65,000, just like, do you, do you really need all those? Well, it changes. Okay. But when you get to an application aware, then you don't have to do that. I don't have to have that stateful rule in there. So we would have to put in the old rule and then the new rule, the application aware right above it with the goal of it should never get to that original rule, which is the old way, so that we can decommission it. But we had to do that and you hit it on it before. We had to prove that the firewall wasn't the thing that was going to break it. And we made the, the firewall be the hub and spoke so it was the router and all traffic went through that firewall, which again, I'm dating myself, it's, it's more commonplace now. You know, 10, 12 years ago, the last thing they wanted to do in an OT world was depend on technology. They didn't understand and the vendor didn't install. Like the vendor being the control vendor, they put all of the, the weight in that, that control vendor basket and anything you did outside of the configuration that the control vendor put in from factory, they were very hesitant and leery on and you had to prove that it wasn't going to break something before they would let you install anything and change anything. It's an uphill battle sometimes.
[00:15:33] Speaker C: Yep, I totally understand that. And I also, I think my career gave me a really good benefit that I experience both like life of OT and it and believe it or not, I started as an ot, became a senior, went back to grad school, and then I start I t from like from an entry level.
[00:16:01] Speaker D: Right.
[00:16:02] Speaker C: So I have two cycle of career like 10 years and another 10 years. So.
[00:16:08] Speaker B: Yep.
[00:16:08] Speaker C: And it was a challenge like to be honest, because when I started I t from from the beginning as like entry level, I started as like a sys admin in high school or help desk high school.
[00:16:21] Speaker D: Sure.
[00:16:21] Speaker C: And just I was a manager in OT and now I'm changing batteries for kids. That's the working or the projector in the class is not working. I need just to restart it or like a student laptop is not working, just restarted.
[00:16:39] Speaker B: Right.
[00:16:40] Speaker C: It. It did it. It make me humble.
[00:16:44] Speaker B: Sure.
[00:16:45] Speaker C: They say okay, I need to start from beginning.
But I get exposed like to the servers. Like I think this was the first time I was exposed to Active directory.
[00:16:55] Speaker D: Sure.
[00:16:55] Speaker C: Like I've been in OT for 10 years. Yeah, we don't, we don't know what Active directory is like.
[00:17:01] Speaker D: Right.
[00:17:02] Speaker C: It's a thing. Yeah but and also like even I remember like from an OT life like when we even have servers. I do remember we did we work with a lots of Microsoft servers back then.
[00:17:17] Speaker D: Right.
[00:17:18] Speaker C: There was a lots of Linux or even a proprietary like operating system which use command lines to configure it. And I would go with our IT guys to when they configuring the server. It's like this is boring. I just like I don't want to see that.
Then I go there, I see operating like active servers and Active Directory and see like how creating policies and rules and security policies and how creating like a usernames and domain controllers and just get involved in the entire environment. It just opened my mind like oh wow, now I understand.
Like it is missy. Like there's tons of stuff happening in IT and way faster OT is like we are very structured. Like we like things like you know, you have to be very precise. Like you. You work on like in a microsecond like to. To avoid delay and like how long the like the signals comes in how like the counter and send it back. So like we get very, very structured and detailed and organized and you know, you go to it it's just find a sport open just put the cable and.
[00:18:41] Speaker B: That's right. Yeah, well there's, there's something to that. And, and you know I think my, my career is very similar to that. And I had you know some, some lateral moves. I had a few that I took steps backwards and I did it. You know, I was. I grew up in the you know, MCSC and you know going after CCNA and all the different certifications and I would go get a job that I had zero experience in and I would get paid less and it was a title bump and down in the wrong direction. But I would do it because I wanted that experience. And it's kind of like, you know, I get this question all the time of how do I break into cyber security? And it's not rocket science, but it's not easy either. Like sometimes you have to take a job where you were a manager and now you're the, you're the janitor or you're the entry level person.
But the good thing is is like when I did that, I remember I did that right after the Internet burst of the dot com era back in like 2000, right? And I got laid off like so many people did during that time and I was, I was like a technical architect and I was making big money and then all of a sudden you couldn't get a job and I found a job working at AT&T Wireless I think it was at the time as a, as a systems administrator and I, and I, I think I cut my salary in half. Like I went from like a six figure salary to not a six figure salary. I could barely pay my bills. But I was only in that role because I got the job and then I was in that role for like six months and I was, I did a great job. I didn't complain. I was you know, knocking it out and doing everything they asked me to do. And very quickly I got moved to the active directory and domain team and I got a big promotion and all this kind of stuff but I had to, you know, eat crow and humble myself to say I know this is going to go somewhere. And I learned a lot during that time and it jumped me to the next place but I had to go through that and there were so many during that time that that wouldn't take that job because well, I was a technical architect. I'm not going to take a step down and I'm not saying my decision was right but for me it was the right decision and it helped me get experience. So now in an OT role I've been the ad, the active directory admin on an enterprise level but I've also been the OT guy at a power plant, right. And I've had to have both those experiences so I can put myself in both of those avatars perspective and understand that which helps me to be able to communicate both both of their concerns and kind of be that mediator because that's really what we need. A lot of the times in the OTIT is someone that can translate even more. You're using the same words but they don't mean the same things in both in the different environments.
[00:21:31] Speaker C: Absolutely. They don't mean the same thing at all times. It's like the same word like an automation engineer, right? Like I try to make like to say like an industrial automation engineer. So people no, like there's so many kind of automation engineer. Like no, I'm talking about the guy who do the code for the blc. Like who? Right, that's one.
And I need to remind myself when I go to meetings it's like okay, I need to understand who is in the room and what language do I need to bring up and how can I, you know, you know, interpret that easily to everyone.
And also the more that I know, the more I know that I don't know.
[00:22:16] Speaker D: Sure.
[00:22:16] Speaker C: Like there is tons of knowledge and, and that makes me realize like okay, this is something we cannot do it alone. Like we need to work together to be able to accomplish any mission or any vision that, that we have.
But the good thing is we did it. Like we don't have like I hope that I don't need to go through this again.
[00:22:45] Speaker B: Me too, me too.
[00:22:49] Speaker C: It's like once or maybe twice a lifetime thing. But there's, there is a moment where okay, I, I learned as much as I can. I now I need to lean on other people that learn something else that we work. Work together.
[00:23:04] Speaker D: Yeah.
[00:23:05] Speaker B: Well hitting on that it's ever changing and that's something to you know again I'm dating myself. But you know, when I, when I was deploying all these firewalls in an OT space and I've done it since, but I wasn't the hands on. But that started back in 2012. I mean that was a long time ago. Technologies even on the palos like they've, they've leapfrogged. They're so much more capable than this was the PA.
Oh God, 220s I think way back in the day. And they were, you know, you'd hit commit on them and it would take, you could go get a coffee and come back and they'd be done running. They were great. But anytime you made to change you're like oh man, I forgot to make a change. Well, come back in 15 minutes, it'll be ready.
[00:23:52] Speaker C: Yep. Yep. I, I have some experience some of just not the firewall. A lot of devices that just takes long time to reboot or some.
And it's, it's impressive how like how technology evolved very quickly and convergence between technology is like a real issue now that.
[00:24:18] Speaker D: Yeah.
[00:24:19] Speaker C: Not just it increases on its own, but it converge with others as well.
[00:24:24] Speaker D: Sure.
[00:24:24] Speaker C: That the, I think that's one of the evolving challenges that we're going to see.
[00:24:34] Speaker D: Yeah.
[00:24:35] Speaker B: Yeah. So. So what are some of the things that you, you guys are, are really fighting for and, and you know, kind of building new capabilities. And like I said, obviously OT is, is, is different but similar. The skill sets and, and some of the, the, the the use cases will be similar but again the response may be different. What are some of the things that you guys are focusing on really solving in the OT space that is unique and that you guys are excited about.
[00:25:02] Speaker C: So there are a couple of things that my team is working on from threat research. One is a new OT white paper specific and it was a very interesting journey because there's millions of firewalls out there for palo. Right?
[00:25:25] Speaker D: Sure.
[00:25:25] Speaker C: And a lot in they could be in an OT or non ot and we only have access to limited information. So we don't have a full visibility to say if this firewall in the OT or it's not in the ot. Like how can we do that?
So we had to be very creative on how we can identify firewalls that's involved in the OT network. And one of the first things we used is I think you may reference to it is like app id which is the capability of analyzing the packets on the application layer. Not just like the port number or the IP address.
We have a list for application that categorized or tagged as an ot.
[00:26:20] Speaker D: Yep.
[00:26:20] Speaker C: And, and said okay, let's utilize that. If any firewall trigger traffic that tagged ot, most likely that that firewall either touching or like on the edge of the network of OT or inside the ot either or I don't have enough visibility to distinguish. But I think that's more than enough to say I need data from these firewalls.
[00:26:47] Speaker D: Right.
[00:26:48] Speaker C: So we started from there and then we start to click threat telemetries and we looked at the signature fired on those firewalls and usually when a signature is fired there's a high confidence there's a malicious traffic that fired these signatures and we took those signatures and we did our analysis. Basically two major outcomes of this analysis is interesting. The one we mapped these signatures to MITRE ICS and we were able to identify what are the top ttps are impacting this environment. And we're not so surprised. But remote access was number one on the list. Sure. Yeah, exactly.
So that was really interesting. We we dive deep more on the white paper about the other ttps and also like what other like what a threat actor have used this technique in the past as well. Right. So it just to give you more holistic view of that threat then the second piece of information which is if anybody listening I want your attention to that particular part specifically.
So we talk a lot about CVEs. We talk a lot about vulnerabilities. We talk a lot about, oh, there's a new CVE that's critical. Or we did a vulnerability scan and we find old CVEs.
I am not talking about any of that.
Okay. I am talking about CVEs that has been exploited.
[00:28:36] Speaker D: Sure.
[00:28:37] Speaker C: That's like the worst of the worst. Right. Like I'm telling you, they cve, we've seen it and it's been exploited.
[00:28:45] Speaker D: Right.
[00:28:45] Speaker B: That's.
[00:28:47] Speaker C: So we looked at the ages of these CVEs that has been exploited.
Can you guess the age of more than 60% of these CVEs?
[00:28:59] Speaker B: I would guess 10 plus years.
[00:29:01] Speaker C: It's from five to 10 years.
[00:29:03] Speaker B: Yeah, that's correct.
[00:29:05] Speaker C: Yeah. And that was just like mind blowing.
[00:29:10] Speaker D: Yeah.
[00:29:10] Speaker C: That. We are not talking about CVEs in the systems. We are not talking about. There's a CVE and critical, but the chances of being exploited is low.
[00:29:20] Speaker D: Yeah.
[00:29:21] Speaker C: Right. Now I'm telling you these CVEs are being exploited. So if you have it, you must do something about it. I'm not gonna.
[00:29:32] Speaker B: You gotta remediate in some way. Yeah.
[00:29:34] Speaker C: There's something you have. You need to do something about it.
[00:29:37] Speaker B: Right.
[00:29:38] Speaker C: Because. Yeah. So. So this is, this is the piece of information like one, like when you get access to the white paper, I strongly recommend go to some of these cve. Just use them. See if this, if any of these CVEs are in your environment and remediate immediately about around there.
So those were like kind of like the top. Oh. And then the third piece, which was also interesting.
So usually any traffic have source and destination, like where it's coming from, where it's going to. We looked at the traffic that from an internal network to an internal network. We're not looking for inbound or outbound.
[00:30:23] Speaker D: Sure.
[00:30:24] Speaker C: So it's from internal to internal. Meaning it's a lateral movement. Most likely.
[00:30:29] Speaker D: Right.
[00:30:29] Speaker C: Like it's something. It's like, you know, exploited or something compromised that's going to somewhere else.
[00:30:36] Speaker D: Right.
[00:30:36] Speaker C: And we looked at what our number one industry for in like internal to internal traffic. We have seen and manufacturing industry was number one by a significant.
Sure. Ratio.
[00:30:51] Speaker D: Yeah.
[00:30:52] Speaker C: So that just gives us like a holistic view a little like if you're in the manufacturing environment or an ot.
What's going on?
[00:31:02] Speaker D: Yeah.
[00:31:02] Speaker C: Right. Remote access, old cve, lateral movement.
[00:31:07] Speaker D: Yep.
[00:31:08] Speaker C: With that. Start with, start from there. Like that's, that's, you know, a starting point. But those are Kind of like the, you know, the top three pieces from the IPS threat limit.
[00:31:25] Speaker D: Yeah.
[00:31:25] Speaker C: We also looked at the malwares. So we have a wildfire which basically any like file that transfer over the firewall, we can capture them, analyze and see if they are known malwares or not. One of the also interesting information we have seen that I think over 70, 60 to 70% of these malware categorized as unknown.
So with a high confidence we know they are malicious.
[00:31:56] Speaker D: Right.
[00:31:56] Speaker C: But have we seen them before? Most likely not.
[00:31:59] Speaker D: Right.
[00:32:00] Speaker C: So that what that tells us tells us that a lot of new variants of malware are out there. So our passive detection of like, oh, you have an antivirus or firewall, like you turn it on and gonna do your job. Most likely it would do like 30% of the job.
[00:32:17] Speaker D: Sure.
[00:32:18] Speaker C: There are 70% of effort that need to be done which is proactive defense.
[00:32:25] Speaker D: Right.
[00:32:25] Speaker C: Whether this is a SOC or monitoring or using like more advanced solution, so on and so forth. So this is like what we have been seeing so far.
Yeah.
[00:32:41] Speaker B: So all that is. All that is awesome in that.
And you and I've talked about this extensively a lot. Right. Is you, you. It has to be an ecosystem. There is no silver bullet. You have to work with your vendors. Because to your point, I've got, I've got antivirus, I think I'm good. But when you detect, when it's not a detectable, it's not on the blacklist, then your antivirus isn't going to detect it because it's not looking, it's not in the signature. So it doesn't know that that thing is bad. Whereas you need other products and segmentation. Right. We talked about that briefly. But why do I need to segment? So that one of those things that gets off in the wild in theory would be isolated to a smaller environment. It wouldn't spread to all of my environments, to my other sites and all that type of stuff. Right. So that defense in depth and having, you know, the other piece that you said earlier and I wanted to circle back to is we're all on the same team and I hear this a lot of oh, I'm the Iot guy. And I'm the IT guy. It doesn't understand what we're doing. Well, you need to make them understand. Right. We need to be on the same team, we're wearing the same jersey, we're trying to protect the same assets. We both, we all get our paycheck from the same company or you're my customer and we have the same End goal of protecting your environment.
We need to stop being adversaries and start being allies and, and utilizing the tools. And that's what I did in my space is because I had a small team. Yes, we manage the firewalls but I would reach out to my IT team that had their, an entire firewall team and they had hundreds of people and all this stuff and I would use them. Hey, these are our firewall. Would you mind doing a review? Am I missing something? Are my rules too broad or you know, do I have things turned on? And I would do an internal audit almost with. I still held the control but I would get somebody else with a different set of eyes and more, potentially more usually more experience than I or my team had. And I was okay with that. I was okay with them giving me suggestions. It was my responsibility and in, in it was my choice to implement what I wanted. But I, I was stupid to not reach out to and lean on expertise that was in house on my team. It didn't cost me. I didn't have to go hire a you know, consultancy or anything else. I could just start with my internal teams and get their, their feedback.
[00:35:04] Speaker C: Yep, that's, that's true indeed.
[00:35:07] Speaker D: Yeah.
[00:35:09] Speaker B: Well, awesome. So dude, I, I know we did some cool stuff with mtu. I don't know if you want to talk about that a little is and kind of you kind of briefly talked about the, the lab that you guys have and you know, kind of the purpose behind building it. But I think you guys have a kind of a multi use for your labs and kind of what you're doing both internal and you know, kind of customer facing as well.
[00:35:34] Speaker C: Yes. So this is an invitation for anyone that's going to S4. I will have a talk about risk assessment. Basically more about how to apply game theory in ot which in summary we all get like all this cool solution, you get all this security solutions. The question is how strategically you can apply them. How can you avoid overspending or underspending for the threat that you are facing? And so I will go more in deep in this presentation from a little bit mathematical but more simplified. How can you make a strategic decision combining the threat that you are facing and analyze it or map it to MITRE's MITRE framework and also use the game theory model to identify your solution. The cool part of that, if mass is not your cup of tea and you want to experience this in a really like a game format like as a tabletop exercise, we will have also surprise there. So you come over like join the talk. And then also you can sign up for this like, tabletop exercise format that will ex, like help you to understand these concepts. And it, it's, it's a really, really interesting. And I, I really was curious to see people from different backgrounds, professional backgrounds, how they run this tabletop kind of exercise. And it's very fascinating when you see like, oh, how it personnel respond versus an OT versus someone knows both of them.
[00:37:35] Speaker D: Yeah.
[00:37:36] Speaker C: So this is an invitation for anyone to come join us and I'm looking forward to see you there as well.
[00:37:41] Speaker B: Yeah, absolutely. I'm excited to see it. You know, it's been a long time, but, you know, you hit on something right there too, right? Is, is your, the, your perspective and your experiences go into how you answer things or how you view things. So when I've done tabletops, hundreds of them, if not more, with different audiences from, from, you know, plant people and manufacturing or power generation or, or you name it, all the way up to it people, I've done them individually and I've done them with, with all the parties in, in, in, in the room, and I've done them with people that are outside of cyber and are wanting to get into. And it's amazing the perspectives. And sometimes you're like, oh, well, you have no experience. What do you know? Sometimes the most just off the wall, amazing ideas or thoughts come from people that have zero experience because they look at a problem differently. They're not jaded by previous experiences or past failures or, you know, all of the things. Well, we've done it this way for 40 years, any of that stuff. Sometimes, sometimes having those interns in the room or whatever they are to ask the, the dumb question that people like you and I may not ask because we're afraid we may look stupid.
Those questions sometimes make you be like, oh, you know, I didn't, like, I just assumed we couldn't do that. Like, and you look, you see the heads looking around like, I've never, I never voiced that out loud. And you just did. And sometimes those are more powerful than the smartest people that have 30 years experience because they're, they have this bubble or the, these, these guardrails that they inherently bring to the conversation, especially when their leadership or when something, their mentor or their boss or whatever, they're, they're afraid to answer or look stupid. We've all been that kid in the back of the room that doesn't want to raise their hand, even though I don't get it. Like, I don't want to raise my hand because people look at me and then I have, everybody's going to know I'm stupid. At least that's how I used to feel.
[00:39:42] Speaker C: Well, I can relate to this as well.
[00:39:47] Speaker B: So it's just fun to see the differences in there. There really is no stupid question, right? And getting that in front of all of these people, there's value add. And you talked about, you know, the white paper and the fact that it was 10 years. It didn't surprise me at all that it was five to 10 years of the age of those. Because of the age of the equipment, the environments, and the way that things are rolled out and implemented and as slow as they are to change and in these OT spaces, they didn't surprise me at all. It also didn't surprise me about secure remote access because, or the lack thereof, because that is one of the common problems that you have. A lot of these environments are in the middle of nowhere where I don't have my staff, so I have to make it where an engineer can get to it. So what do I do? A lot of the times I just SSL VPN into the environment and put my network on it, right? It makes it easy, but it also brings all of the issues. I just bypassed all of my controls. Not all of them, but many of my controls by segmenting these environments, putting in these great firewalls with all these rules. And then even worse in an OT environment, which I was going to bring up before, but now that we're talking about it, you also have these weird architectures where I've got a machine that's dual and triple homed and They've got a 3G card that goes straight to the vendor and bypasses all of your security things and you don't even know it's there.
And it's impossible to understand that. And even down to an asset level, I don't even know what the equipment is in the corner and it's not on this asset inventory and nobody in the room really understands it. And they say, oh well, that's vendor X's. And it. What does it do? We're not really sure, but it has to be there.
Does it, does it bypass it? Does it go the Internet? Does it go through our stuff? Like many times I'm walking in these environments and, and I'll get an asset. I'm just doing a, an assessment, right? And I'm, and I'm walking down using the asset inventory they give me. And many times like, oh, there's, there's 15 devices. And I'm walking around and there's 300.
Where are the rest of them? They're not on here. Do you know about them?
[00:41:45] Speaker C: Oh, yeah. I have seen those kind of stuff as well. Like, yeah. Yep. It's not a surprise, but I was like, wow. Well, we need to deal with the reality, and this is the. What we're dealing with and how can we start from there and. And improve? And I think one of the good things is, like, someone like you or the projected all. It's. We. We start these conversations, we talk about it, we bring it up to the surface so people can listen and hear. Because also one of the challenges that some. An expert may have the right information, but they may have a challenge to educate up.
[00:42:23] Speaker D: Sure.
[00:42:23] Speaker C: Right. And hopefully, like, you know, now, like, people start to listen and when you go to your managers or your C suite, they can align quickly. Quickly, and they get, you know, react or respond to the request faster.
[00:42:40] Speaker B: Yeah.
[00:42:41] Speaker C: I think sometimes just some people may have the right request and just get stuck on the process or the, the leadership or the C level. Like, we don't do that. Why do we do that? Or specifically, if there's a CFO there, don't hate me, but you guys are usually a big blocker.
[00:43:00] Speaker B: The no, man.
[00:43:05] Speaker C: We love you, but you sometimes make our life hard, which is reasonable. I understand that.
[00:43:11] Speaker D: Yep.
[00:43:12] Speaker C: So I hope that this knowledge and information is helping to facilitate this kind of conversation, to make things move faster and in the right direction.
[00:43:22] Speaker B: Yeah, I mean, that's. That's one of. As you know, with me, that's one of the reasons why I have this podcast, is I want information to get out there. I want you to be able to take this to your boss and say, hey, just listen to this. Like, this is a problem we're having right now. We're not the only ones. Right. One of the things that I, I wanted to ask you, especially because Palo is so big and y'all are global, like, what. What problems do you see or how are people? I guess it's kind of a multifaceted question. What. What differences are you seeing in geographs and different countries and, and, and. And, you know, whether it be the UK or Asia or Indo, you know, wherever. Australia handles things differently. Europe has, you know, regulations that are different than, you know, NES2 and US has Nercsep and CMMC and all these different regulations. But, but in addition to that, people in other entities in other countries handle, like, obviously China, you know, the government controls the things that they do. And like, there's just all of these different problems and solutions and each you know, geographically, area, country, etc, you have to handle them. And some companies, you look at FedExes of the world or whatever and they're in all of those places so they have to have a different policy and capability. Maybe a firewall rule difference in China as they do in Mexico.
[00:44:44] Speaker C: Well this is a really good question because we did came across geographical analysis in our white paper and that was a deal paid actually should we include it? Should we not? Because there are some data biases that sometime you run into it and we have to normalize the data to keep it integrity as much as we can. Because maybe we have a thousand firewall in one country and we have 10 firewalls in another country that's like you cannot compare this. But we can like normalize the data and get average payer 1 firewall and then we can try to compare the averages to get that picture. The other thing also beside the bias is it doesn't mean we don't see much malicious traffic from one country that they are more secure. It most likely could be because they are. They don't have enough information or they are not enabling all like the right security measurement or they bypass it. Right. So geographical as much as it's interesting piece of information to bring up but it's. It's. It's really hard to make a story like. Or getting precise information what really that that means. Right.
[00:46:04] Speaker D: Sure.
[00:46:06] Speaker C: That's. That's from just like a very, very high level.
If we want to go like okay, what are the top threats impacting per country.
We did not go that granule.
I think like global infrastructures like it's. They mostly face this some similar issues.
The compliance you mentioned just add a different dynamic.
[00:46:38] Speaker D: Sure.
[00:46:39] Speaker C: I think the threats are the same.
[00:46:42] Speaker B: Yeah.
[00:46:44] Speaker C: And even in different regions. I think different regions. The only thing I would say was. Was interesting or like I think it makes sense to see different the volume.
[00:46:55] Speaker B: Okay.
[00:46:55] Speaker C: Right. Like some country that has more manufacturing environment makes sense. The volume of attack will be more sure there. Right. So that's something you know I will pay attention to it in. In specific geographical location.
[00:47:11] Speaker B: Right, right.
[00:47:13] Speaker C: Country that known for mining that industry will be the top one we see there because they are known very well. I think like in Australia it's very known for mining. So that what we see a lot for manufacturing like US and Europe, something like that.
Yeah. We maybe we don't get a lot from like. Like from China just because different regulations or different.
But those are like from a geographical.
We give a high level but because of the data biases, we try to, you know, I don't. Yeah, like how true the story or how much we confident. Yeah, it's, it's a, it's, it's a tricky.
[00:48:02] Speaker B: Yeah. And it just really gets to the complexity of these, these larger entities and, and again, you know, how I do things. And it's funny because I can go to a US company that has, you know, again, use my example or my, my experience. A power plant. I could go to a power plant or a company that has power plants and they've got, you know, one in this city and one in that city in the same state and they, they may have the same vendors, but I go in there and the architecture is different, the implementation's different, how it's done is different. Like where the firewall sits, the rules are different. Like, everything about it is different. Like again, I'm dating myself. But when we rolled out Palos, we weren't even using Panorama. And that was because that was a conscious decision because of NERC sip. And the way that we read the NERC SIP is if I had Panorama that can control assets that are providing access control to these environments and that would have been a NERC SIP asset. And we didn't want. So we were individually controlling all of these 200, you know, firewalls manually. So when I wanted to make a change, I had to go to all 200 of them and implement those changes locally on those firewalls. Now we had HA and backups and all that kind of stuff, but I didn't have the benefit at that time, or again, at least, right or wrong, our compliance attorneys thought that that was our response to compliance. So we did it differently. Segmentation, you know, it, you're going to connect everything and you're going to block people off, whereas an ot, you're going to segment and put as many barricades and, you know, layers of protection as you can. So I'm going to hyper segment zero trust all the, all the newer capabilities. But you have to understand the environments that you're going into before you roll out these new technologies. So it just, it adds complexities, but it's just awesome to see, you know, big companies like Palo and others that haven't forgotten about ot because, you know, one of the frustrations in, you know, again, working at a power company is the power plant is the thing that brings the revenue. But, you know, if I looked at my team, I had six people, we supported all the power Plants, every bit of technology from the firewall to the control system my team was responsible for. I had six people that supported 45 power plants. It had hundreds of people.
They had an entire dedicated firewall team. They had a virtualization team, they had an application team, they had a server team, they had a backup team, a networking team, cloud team, like all of these people. And I'm not saying they shouldn't have. I'm just saying that my team was only six, because again, this was a while ago and it was a struggle to get budget and team because it was new, it was different. Right. And used to. They leaned on the control vendors to do all of that work. And it just really showcased the need and the tech stack. Like, again, my team of six supported 45 sites and hundreds of firewalls because we had firewalls at multiple levels. And the virtualization and the servers and the applications and the networking gear and the switches and the logging and monitoring and antivirus and secure mode access and all the things with a team of six. So there was no way we could be great at any of that stuff. We were just whack a mole.
[00:51:16] Speaker C: Yeah, yeah, Yep, I totally see that. And when you add compliance on top of all of this, it's just like you're losing 50% of your resources. Just.
[00:51:27] Speaker B: Absolutely.
[00:51:28] Speaker C: Yeah, that's. It's. Yeah, I underest. I feel your pain.
[00:51:32] Speaker B: Yeah. Well, and that, that's why it was so important for me to leverage our internal teams. And, And I knew my people were good at. I mean, I was, I was good at palo. Like, I could deploy them and all that kind of stuff. But was I the same level as the team that dedicated and that's all they did? Of course not like, they were way better than me. Like, I actually went to and got a job offer to go work at PALO for a while way back in the day, before I went to work someplace else.
And, you know, it was. I went in on the interview and I thought I was a great PALO person. And they were just asking me questions and I was like, I have no idea. I've never done that with a follow. But I was using it in a very specific use case. So I didn't do some of the things that they were asking me about because that wasn't applicable in how I use the firewalls in IRA space, even though I had years of experience and I deployed hundreds of them and support them in production, but I just didn't do the. Some of the things that you guys were asking for and Rightfully so. Because most your customers do use them that way.
[00:52:30] Speaker C: Yep, yep. It's.
It is ongoing challenges and the more we see, the more we learn. And I remember one of the thing that was interesting I saw reviewing someone network architecture or like IP planning and then I saw a public IP address assigned to a local device.
[00:52:56] Speaker B: Yeah.
[00:52:59] Speaker C: What that is doing here originally it's like it took me off guard and it took me a while to, to analyze like what's going on here, like why we have this public IP address, what's doing, blah, blah, blah. And after long investigation we found that was a misconfiguration.
[00:53:16] Speaker D: Right.
[00:53:17] Speaker C: Someone just know they needed ip, they just put random IP so they get things done.
[00:53:23] Speaker B: Yep.
[00:53:24] Speaker C: And it's like, okay, that's, that's good to know. But we need to fix that.
[00:53:31] Speaker B: And unfortunately I've seen that in many places where they're just like, well our stuff doesn't touch the Internet, so I can use whatever IP address I want. So they take routable IPs on the public space and put them into the private space. But then when you see them now, when we see them in the firewall space and we start seeing these public IPs showing up in our OT space, you're like, why is this here? Is this trying to get out? Like what is going on?
Sometimes it's just stupidity or not stupid, but ignorance. And they didn't realize and it was a. They shouldn't have chosen those, but they didn't know that they shouldn't choose them. And others, it's just like they know, but they did it anyways.
[00:54:09] Speaker C: Well, I was one of those people like an early in my career.
[00:54:13] Speaker B: Sure. Yeah, I was too. Yeah.
[00:54:16] Speaker C: I was in a position where I was just like first graduate and my manager went out for vacation and it's like, just do your best.
[00:54:25] Speaker D: Right.
[00:54:26] Speaker C: And the commissioning team were going down to the side to commission the device and they said we need an IP address. I was like, here's one.
[00:54:35] Speaker B: Right.
[00:54:36] Speaker C: Maybe it worked, maybe it's not. I don't know.
[00:54:41] Speaker B: And, and especially in ot, many times those band aids get put in and the reason they're still there is because it was put in 10 years ago and it was. It'd be so hard to change it, to rip that out and re IP it and rename it and all of the things, interdependencies that are tied to it would be really difficult. So sometimes it's easier to just leave it in place and document it of why it's that way. And eventually you'll upgrade it or roll it out, but it, you can't just go out. And that's another, you know, vast difference in my, in my experience in OT is you're not just going to go rip and replace. It's too complex. There's too many dependencies and problems that that can bring. So many times I may just document it. You know, I've got xp, I've got Windows XP running in a critical system in my environment. I see that all the time. You would never allow that in an IT world. But in OT it's extremely common or running IP addresses that are publicly routed in a private space. It's not ideal. I don't recommend it. But if it's there and it's working, there's other ways to mitigate those, those problems other than just re ip.
It's not the popular answer, but it is reality.
[00:55:52] Speaker C: Yeah. Yeah. It does require time to fix and change and just the cascaded issues.
[00:55:58] Speaker D: Right.
[00:55:59] Speaker C: Of trying to take this bandit off.
[00:56:00] Speaker B: So yeah, absolutely. All right, so ask this question to everyone.
I've given you a warning warning on it. So it shouldn't be too much of a surprise. But over the next five to 10 years, what's one thing that you see that's exciting coming up over the horizon? And maybe one thing that's concerning that you really think that we need to, we as a the collective, we need to, you know, really do something about to make sure it doesn't impact us in a negative way.
[00:56:27] Speaker C: So one thing I think like exciting which is I was predicting and I think AI came and took, took the all the trend for now. But I think after that augmented reality.
[00:56:42] Speaker D: Sure.
[00:56:44] Speaker C: I, I have a strong belief that virtual reality and augmented reality will grow exponentially.
[00:56:53] Speaker D: Yep.
[00:56:53] Speaker C: Giving the lots of challenges that we see in the OT environment for in terms of personnel and expert. Expert. Having this easy way to operate a factory that's just like amazing. Yeah. Cost efficient and help for troubleshooting and so on and so forth.
[00:57:14] Speaker D: Sure.
[00:57:15] Speaker C: Which actually just heads up like I have done this years ago when I predicted this is coming. I said well I need to start to use them now because I need need even physically to be trained. Like my eye needed to build a stamina.
[00:57:33] Speaker B: Right.
[00:57:36] Speaker C: It's like when like PCs was generated like 10 like 15 years ago. You will see someone who have not grow up with computers. They're just struggling using PCs sure tops. Right.
So I didn't want to be like struggling when like all the virtual reality and augmented reality they come out and become More prominent out there.
So that's something I know it's not security per se, if that's what you were expecting, but it's technology that.
[00:58:11] Speaker B: You.
[00:58:11] Speaker C: Know, I will keep eyes on. And the other thing that's concerning is the conversion of technologies.
[00:58:20] Speaker D: Yep.
[00:58:21] Speaker C: We are not just talking about it and OT, we are talking about IoT. We are talking about AI. We're talking about so many technologies that gets converged and just increase the attack surface of the, of the impacts that can happen in our OT environment.
So that's a real challenge. Specifically something for like IO Iot. That's something I have done research on like a couple of years ago and I had discussion around it with other people, but it, it wasn't reality until like few months ago.
[00:58:55] Speaker D: Right.
[00:58:55] Speaker C: The I controller came out and they show how they use the MQTT to do a command control and so on and so forth. Right.
[00:59:02] Speaker D: Yeah.
[00:59:03] Speaker C: So it is. Yeah, it's like that the conversion is one of the things that's a nightmare, that it's really, really important. You cannot stop it, but also open the challenges.
[00:59:17] Speaker B: Well, and I think that's the point is. And the key is it's coming so you can't just. And unfortunately, I've seen this too much in OT is. And just, you know, controls and operations and all that kind of stuff is, well, we're not going to do that here.
We need to start planning for it. How would you do it? Like, we need to start thinking about it, getting our minds around it. So to your point, we're not 10 years down the road and now it's getting crammed down my throat and I don't know how to do it in a safe, good way. It's gonna come.
And I used augmented reality, you know, to roll out all that stuff at power plants across 800 sites. And you know, it was really good because I could have one really smart person centrally located and they didn't have to go to every single implementation because they could sit in a, in a war room and see all the things going on because somebody was wearing glasses and they could see, oh no, it doesn't go there, it goes over here or oh, I know what that thing is. Plug it into this. Like. And they could get that expertise by sitting in a war room. Almost like, you know, the control room and they're controlling and seeing all that data. Kind of like the engineers and, you know, NASA, you know, sending it up to Apollo 13 to help them solve the problem. I don't need all of those engineers on the spaceship. I just need them to be able to understand it enough to be able to help them solve the problem. And augmented reality is way better than using words. You know, we've all done the telephone game where I whisper in your ear and you whisper in the next person's ear. When you see it on camera at high definition in real time, it makes it a lot easier to describe and, and you know, troubleshoot for sure.
[01:00:54] Speaker C: Yeah. Even like, like Mark, like change this one. Like you don't even have to say like this. The cable, the red cable to. From this bottom. Like. Yes.
[01:01:05] Speaker B: Right. Nope, nope, not that one. Not that one.
[01:01:10] Speaker C: And it's too late. They pulled it out.
[01:01:12] Speaker B: Exactly, exactly. Well, I mean I've done that in virtualization again way back in my career. Not to dive too far down that. But you know, you think you're on server A and you're on server B, so you reboot it. You're like, why did you re. What happened? I don't know. I just reboot the server. Did you reboot the right server? I don't know. I'm on the same kvm. Oh crap. I reboot the wrong server.
It's. It happens to all of us. Right. So. So having that ability to, to have somebody peer. Peer check you watch over your shoulder. Am I on the right thing? Three person communication, like a lot of these things that we use in an OT space. Having tools like, you know, augmented reality would make those things so much vastly better. And my hiring's better. I can get, you know, resource allocation. I can hire really good people that. And they don't necessarily have to live in the small town where the plant or the facility is. They could be in a more geographical location and I still get their expertise on site without having to put them on a plane or a car or whatever to get them there. So that's, I could see that being a huge impact in industry 4.0.
[01:02:13] Speaker C: Yep.
[01:02:14] Speaker B: So. So what's the call to action for y'all? I know you, you, you already mentioned S4 and your speaking thing there. Anything else that you want people know? You also mentioned the white. White paper and we'll, we'll definitely put that in the show notes as well.
[01:02:25] Speaker C: So yeah, like the white paper that read the CVEs and the white paper because of the exploited one and come see us as S4 for the talk. And also we will have the cyberwall as well as S4. If you haven't seen it, it's also a good time to come and see it. So come play without the tabletop exercise for MTU. And I'm looking forward to see you at S4 and I'm very happy that I was able to join you today.
[01:02:57] Speaker B: Yeah, man, thank you. I enjoyed it. It's. It's been a long time. We've had a lot of these conversations offline, so I think it's beneficial to have these conversations so others can hear it. And it's just getting different perspectives and, and, and seeing, you know, asking questions differently or seeing things a little differently and you know, who knows who this could help, you know, to, to level that up to their boss or, or think about a problem differently. Definitely don't hesitate to reach out to myself. I know Adam is very active on, on LinkedIn. Reach out to him and the PALO team. They're a great team to work with and they have some awesome, awesome stuff. So definitely check that out. So excited to see you in at S4 and Tampa. Not too long from now, actually. Until then, sir, have a good day and glad you glad you were able to, to spend time with me today. I appreciate it.
[01:03:42] Speaker C: Thank you, Aaron. Same to you.
[01:03:44] Speaker A: Thanks for joining us on Protect it all, where we explore the crossroads of IT and OT cybersecurity.
Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time.
