Building Resilient Tech Environments: Lessons from Dennis Maldonado

October 07, 2024 01:09:01
Building Resilient Tech Environments: Lessons from Dennis Maldonado
PrOTect It All
Building Resilient Tech Environments: Lessons from Dennis Maldonado

Oct 07 2024 | 01:09:01

/

Hosted By

Aaron Crow

Show Notes

In this episode, Aaron Crow engages in an insightful conversation with Dennis Maldonado, Director of Technology for Harris, Fort Bend ESD 100. The discussion emphasizes the importance of resiliency in technology environments and how strategic planning can safeguard against unforeseen disasters without necessitating a complete technological overhaul.

From his extensive experience, Dennis shares how effective communication and collaboration were critical during events like Hurricane Harvey. He also provides his perspective on future trends and concerns in cybersecurity, including the rise of ransomware and nation-state attacks targeting critical infrastructure.

The episode illuminates the significance of networking, with Aaron and Dennis underscoring its value in career advancement and sharing personal stories to illustrate how being well-known and trusted can open doors to unexpected opportunities. 

Additionally, Dennis discusses the zero trust model and the intricate balance between maintaining cybersecurity and ensuring system availability in critical infrastructure.Listeners will gain practical insights into building resilient tech environments through real-world examples and expert advice. 

The episode is a treasure trove of learnings on keeping organizations secure, responsive, and prepared for any eventuality. Join as "Protect It All" dives deep into building resilient tech environments with Dennis Maldonado's invaluable lessons.

 

Key Moments: 

09:15 Networking is crucial for success in cybersecurity.

13:46 Volunteer firefighter boosted dispatch center through IT.

18:52 Transfers emergency calls to fire and EMS.

22:06 Quick response with information saves lives effectively.

26:22 Implemented lessons for resilient project development.

42:14 Sharing lessons learned from threat modeling experiences.

48:04 Zero trust model effectively mitigates cybersecurity incidents.

57:32 Public safety adapts by reverting to manual methods.

01:02:51 Cybersecurity's mainstream rise sparks widespread interest.

 

About the guest : 

Dennis serves as Director of Technology for Harris Fort Bend ESD 100 (WESTCOM) managing and maintaining the technology needs of 911 call taking and emergency dispatch services for multiple public safety agencies.

With over 15 years of experience in information technology and over 12 years in cybersecurity enterprise environments and consulting, Dennis’s experience includes cyber resilience, network penetration testing, full-scope red team engagements, adversarial simulation, and physical security assessments.

Dennis presented at multiple security industry conferences including DEF CON, InfoSec SouthWest, BSides conferences, Houston Security Conference, Houston OWASP, SANS HackFest, and several local meetups and organizations around the United States.

As an active leader in the Houston cyber security community, Dennis is responsible for founding two cyber security meetups in the Houston area: Houston Locksport, founded in 2014 and Houston Area Hackers Anonymous (HAHA), founded in 2016.

How to connect Dennis: 

LinkedIn: https://www.linkedin.com/in/dennismald/

Twitter/X: https://twitter.com/dennismald

Houston Area Hackers Anonymous (HAHA): https://www.meetup.com/houston-area-hackers-association/=

Connect With Aaron Crow:

 

Learn more about PrOTect IT All:

 

To be a guest or suggest a guest/episode, please email us at [email protected]

View Full Transcript

Episode Transcript

[00:00:00] Speaker A: You're listening to protect it all, where Aaron Crow expands the conversation beyond just ot delving into the interconnected worlds of it and OT cybersecurity. Get ready for essential strategies and insights. Here's your host, Aaron Crow. Awesome. I'm excited for this conversation. Dennis, why don't you introduce yourself? I met Dennis last week in person. I love meeting people in person at conferences. He was introduced to me by another friend, and we did some chatting, and he actually was speaking on stage about some really cool stuff that we're going to dive in today. So, Dennis, why don't you introduce yourself to the audience who you are, kind of what your background is? [00:00:38] Speaker B: Hey, everyone. So, yeah, I'm Dennis. A lot of people know me in the Houston area, but I'm kind of a person here that wears many hats. I serve mainly as director of technology for a local government entity. Harris, Fort Bend, ESD 100. They're an ESD here, and I can kind of explain what that means in a little bit. But mainly I operate, built and operated a 911 dispatch center. So we answer 911 calls for fire and EMS, and we dispatch those to their respective fire and EMS departments. And so just managing all that technology, make sure it stays reliable and resilient and stuff like that. I also do technology for the local fire department as well as kind of consult for some other fire departments and dispatch centers in the area. I have about 15 years of information technology experience. Twelve years in cyber security, another ten years in public safety. I mean, they all kind of overlap, so I kind of did them all at the same time. [00:01:38] Speaker A: Sure. [00:01:39] Speaker B: And I've spoken at quite a few conferences. I've spoken three times at DeFcoN, DeFCON 23, 24, and 25, if I remember correctly, some other local conferences. Infosec Southwest was the conference that used to happened in Austin. That was my favorites. Houston security Conference where I met you. I spoke at this last one last week as well as some previous Houston security conferences. It's a great conference, by the way, if you're ever in the Houston area or you want to come stop by, it's growing. And then just other small, little local meetups in the Houston and also Austin, Texas area. And lastly, I founded a few local meetups I love to do. I do a lot in cybersecurity. So maybe about ten years ago, I founded Houston Locksport. It is what exactly? It sounds like. We just hang out at a restaurant, pub or bar and pick locks. We've been doing that for over ten years. And then my favorite is Houston area hackers anonymous. So I founded that almost nine years ago. It's still extremely popular. We had our meetup, actually a monthly meetup yesterday where we had about almost 50 people show up. It's back to pre Covid numbers, but yeah, it's just a. It's a meetup where a bunch of hackers, geeks, nerds, cybersecurity people and everyone else show up, hang out. We share knowledge, network talk, and eat food and drink. [00:03:01] Speaker A: So that's awesome. You know, before we dive in too much of the details, it's really important for that networking side of things, right? So I had a mentor, and I say this frequently, but I had a mentor a long time ago that told me all business is a people business. So doesn't matter if you're the CEO, the janitor, the cyber guy, engineer, whatever, right? We all have to interact with people. That people element is what makes us work faster. And another thing I say all the time is we do business at the speed of trust. And how you build trust is by building connections with the people that you work with or that you may interact with. So when you're coming in from outside, people don't know you. The faster you can build that trust between me and you, the more likely we're going to be able to talk about whatever it is we talk about. Right? So if I need to deliver bad news, or I'm asking you to change the way you do things, or I need to impact something that you're doing. Doing, the more likely, the more that you trust me, the more likely that you're willing to listen to what I have to say and take it. That doesn't mean you just trust me blindly, but you're more likely to at least listen to what I have to say. If we have that trust built. So that networking aspect, technology people, us nerds, us engineers, those type of. We don't always focus on that networking side because we don't necessarily. I can just sit in my basement and, you know, hack and compute and all that kind of stuff. And what's the benefit of going to a networking event? I'm not a sales guy. Why do I need to do that? But I think the fact that you see that and so many, I see a lot of folks that get it and a lot that struggle with it. Even if you're an introvert, even if you, you know, you don't like those types of things. A lot of us are like, I'm very outgoing, but also, I just, I perfectly also find just being locked in a cave and doing my thing. Right. But I have to pull myself out of that and put myself into these social situations. And that is, that is grown my career exponentially because I've made those networking connections before I needed them. So when I lose a job or I'm looking to change a job or any of that type of stuff, I have those connections, and that helps me faster, go faster. Right, right. [00:05:05] Speaker B: Yeah. No, I completely agree with you, Aaron. I feel like your story you just told you about yourself, your personal growth is exactly, you know, the same motto and the same story that I had. You know, I, many people, in fact, I don't think anyone will would agree with me on this, but I do consider myself an introvert. Yeah. But when it comes to those social situations, I feel like at least when I was starting in the career and growing my career and growing my, you know, personal growth and connections, I forced myself to go out there. That's the whole reason why I started Houston air and hackers Anonymous, or haha. I'm gonna call it haha from now on. [00:05:39] Speaker A: Sure. [00:05:39] Speaker B: That name and the way I started it was I wasn't even sure people were interested or were going to come to it, but I just put it out there like, hey, every month on the first Thursday of the month, I'm going to be here. Regardless. If anyone shows up, I'm going to be here doing stuff and talking about stuff. Show up if you'd like. And over the past nine years with COVID kind of putting a pause on that, it has grown exponentially, and it has become a great place to facilitate people talking and sharing information, sharing knowledge. And I just love it. Every time I have that meetup and there's always a record number of people, and there's always people after the fact saying, thank you for hosting it, I love it. And, and then seeing relationships get built from that meetup, like personal relationships, but also professional relationships, people getting jobs for my meetups. It just, just makes me feel warm and fuzzy. [00:06:32] Speaker A: Yeah. [00:06:32] Speaker B: And of course, we know when I talk more about Westcom and cybersecurity and public safety, I took that same mindset is I wanted to facilitate relationships, intercommunication, information sharing. And so that's kind of what I, what I try to foster in the Houston area public safety community. I started this mailing list with other, you know, emergency communication, emergency services agencies where we can share and disseminate intelligence and cybersecurity information and stuff like that. So if any one of us have an incident, we already have that relationship together, so we can share that threat intelligence or that indicators of compromise to the other agencies, and we can all kind of just grow. Grow from it or get stronger from it. So, yeah, I didn't used to like talking to people and going out there and being extroverted, but I think I've grown into it and I'm having fun with it now. [00:07:29] Speaker A: I think that's the key, is people think, well, I'm not extroverted. I'm not outgoing. It's just like anything. It's a muscle. You can. You can get better at it, right? That doesn't mean you'll ever be the most extra voted verted person. But if you actually look at a list of podcasts the other day, I can't remember which one it was, but they were talking about most of the large cyber leaders, or not even just cyber leaders, most like Elon Musk and Steve Jobs. And some of those, many of them are introverted, but they force themselves to be more outgoing because they know they need that to be successful. Right. So it's just like anything. Nobody wants to go to the gym. Not nobody. Most people don't want to go to the gym. But, you know, when you go, you get a benefit. Like, you don't. You don't want to do your taxes, but, you know, if you don't, then you're going to get thrown in jail for not doing your taxes. There's certain things that you know you've got to do. And this is one of those I think most people don't necessarily put the right amount of focus on and how beneficial it will be to you and to others, not just to you individually, and what am I going to get from this? But also, when you tell your story, you're building someone else up. Like, if he can do it, I can do it, right? If he's out there and he's introverted and he could come out and start these organizations and have all this stuff, why can't I just show up? Like, he's done all the hard work, I can at least show up and get some benefits from that, right? It helps other people see, put themselves into your shoes and see theirselves through your eyes and empower themselves to say, hey, I can do it, too. Right? It's that, it's. It's so empowering to just be there and just show up and do those types of things even when you didn't think that you could. Right? And failure is okay, but it's. It's fun to get out of your comfort zone sometimes, especially in things like this. [00:09:15] Speaker B: Yeah. And the benefit may not always be immediately realized, but the benefit is absolutely there. You know, I'm sure you get a lot of questions of how, you know, people asking you how can you get started in cybersecurity? How can you get a job in cybersecurity? And I, I have a lot of different opinions and advice, but one of the biggest ones, of course, is go out there and network, get known. You don't have to, I guess, do what I do, right, start all these meetups and then start a blog or start, you know, doing talks and everything. But if you just get out there and get known, you know, even if you, you may not be the best, most technical person, you have a Persona, you are, you're well liked by the people that you've networked, that you, that you've methemeral. And whenever they know of an opportunity or they have an opportunity, they're going to remember you and you're going to get the, you're going to probably be one of the first people to get those opportunities. And that's kind of what happened to me. I've been fortunate where most of the jobs that I can remember, if not all of them have been a result of having a connection or building a network and someone recommending me or advising that I apply for this job. And I've been able to use kind of the reputation that I've built to get those jobs and progress over the years that I've been working. [00:10:29] Speaker A: Absolutely. I mean, I've been doing technology and all this for a long time, more than 30 years. And it's, I can honestly say none of the jobs I've ever gotten in my life have I gotten blindly, meaning I didn't just blindly apply, submit my resume and get the job. Every single one of them were because I got referred. I knew somebody at the company, I knew the position, that kind of thing. And I used my network to get my sister works there or my best friend works there. And, you know, I still had to do the work and I still had to pass the interview. But it's amazing when you go in and you like, if you knew me and I'm, and I have an open job and since I know you, you say, hey, Aaron, I think I'd like to apply for that. Awesome. Send me a resume. I'm going to put you on top of the stack so you're not going through the normal channels. You're already at the top now. You still have to deliver, you still have to pass the interview and be have the capability and all that kind of stuff. But you're getting considered because of your, of your networking and who you know, right. Use that to your advantage. It's a benefit to you and it's a benefit to the business, right? So it's, it's, it's cheating. It's the cheat code, right? It's, it's back in the day, Super Mario Brothers, you know, or, or street fighter. It's knowing the secret codes to get the special thing that, that's the cheat code. How many people are applying with their AI resume and all that kind of stuff to all these jobs on LinkedIn? I talked to. I talked to folks all the time, and they're like, I've applied for 500 jobs and I haven't even gotten a recall response. I'm like, so obviously you're doing something wrong because I transitioned in January, was downsized at the company I was at, and I was an executive in the company, right? And within a week, I had ten offers week. And it's. It's not because Aaron is the sms saves all the time. It's not because I'm the smartest guy in the room. It's not because I'm the best person, any of those types of things. It's because I built that network. And once I told that network, hey, guys, by the way, I'm on the market, I had multiple people reaching out. I don't know. I don't have an open position, but I want to make something for you. I want to do something like, I want to work with you, right? And it's because of that networking that I've done way before I ever needed it and continue to do and building relationships with people like you, and you just never know how this connection can, can impact you or I or others in the future. And it's more. It's so valuable to have this. So I know we've dove very deep into this thing, but I think it's so powerful and so important for people to understand this is like a deposit and investment into your future. You put your money in now, you put your time and your efforts into this networking today, and you never know when you're going to need to pull out a, make a withdrawal from that environment or from, from this account in the future. And maybe it's for you, maybe it's for you, you know, the opposite side. Maybe it's the other person that needs to make it a withdrawal from you, but that, that account can continue to be built upon, and you, you're going to need it in the future at some point, for some, for some reason that you can't even imagine today that that is going to be helpful in the future. So I dig it, man. [00:13:46] Speaker B: I actually have a great story of that. Like you said, you never know when, when those opportunities are going to come up from, from that network that you built. So, you know, unrelated, I've been a volunteer firefighter at the community fire department here in the Houston suburban area for over maybe twelve years now. And on top of the typical, you know, firefighting, doing fire truck responding fires and stuff, I also offered and volunteered to help a lot with the technology, you know, growing the technology, managing all the technology for the fire department, radios, it infrastructure, all that stuff. And because of that, I've grown, I've obviously grown, be known to, I became known as the it nerd, the geek for the fire department. And so, you know, whether it's for the fire department or personal, people would come to me for help with it and stuff. And so, you know, many years down the line, there was, without getting too much into the story, but there was a need for the fire department to get a new dispatch center because the existing dispatch center was that there's, you know, some politics and stuff, they were going to cease to exist in a few months. And so if nothing was taken care of, then my fire department, as well as twelve other fire departments, would not have anyone to take and dispatch the 901 calls. And so because I had this reputation of just being able to handle it and build and fix and manage things, some of the leaders came to me and asked, hey, we need a dispatch center. We need it in three months. Can you build one? And I thought a lot about it. It ended up, I wasn't sure if I could do it right because although I had a lot of it and infrastructure and cybersecurity experience, I hadn't really built like a full enterprise grade, resilient, reliable network from scratch ever. Right. The most I've done was manage something at the University of Houston when I used to work over there or build something on my home lab. But I decided, I think it's a good challenge to do. And so I decided to take that challenge on. And much to my surprise, I was able to do it. I was able to build like a full 911 dispatch center that was reliable, resilient, and worked in three months. And they hired me to do that. And so that's just, it came out of nowhere, but it came from networking and having that reputation and knowing people. [00:16:20] Speaker A: And you built that trust. Right. So you built that trust. They knew you as a person and they said, hey, instead of grabbing somebody, some consultancy or some contractor company that I don't know, you know this system. I know you. I know if you tell me you're going to do something, I believe that you're going to do it or you're going to figure it out or you're going to ask the right question again, because you had that, that trust, you were able to move really quickly. So that, that is an amazing example of. It's not accidental. It's not, it's not coincidence. You're not lucky. You're in the position you're in because you made it happen and you built that relationship with, with those people and that organization, that they trusted you as a person and your skill set because you'd shown them what you can do over time. It didn't come out of the blue. It wasn't just like, hey, could you do this? It was like, hey, you've shown that you can do this. Let's, let's up it. Like, if you can do this, let's see if you can do this next level. And of course you knocked it out of the park, which is awesome. Which is where we're going to get into next when in that story. So talk about what this, what this 911 center is and, and do that's, that's super exciting to build from the ground up. Very few people get the opportunity to build a green field from scratch. Brand new environment, you know, redundancy and especially something critical infrastructure like that that is doing 911 dispatching and the importance of that to the community that it serves. So obviously it's got to be resilient because when I call 911, it's got to go through. Like I want my, I want the fire trucks to show up at my house, I want the police to show up, I want the ambulance to be here and all of that keys on going through this dispatch center? [00:17:59] Speaker B: Yeah, absolutely. So, yeah, so we're a 901 dispatch center. We're officially called Westcom. That's the name that we decided. And so on the radio, they'll refer us to us as we dispatch right now for eleven fire departments. And that's currently growing where additional fire departments are working to figure out how to come to us or we're onboarding others, but we specifically focus on fire and EMS. So we don't do law enforcement, which I kind of wish we did because I think it would be a little bit more fun. But with law enforcement there's a lot of additional things we have to worry about, like cgis and tea. Let's cybersecurity related things. Not that cybersecurity is important, but with the state of Texas, when you handle law enforcement information, there's a lot of specific procedures and policies that we need to follow. So we don't do any of that at this time. It's just firing EMS. But, yeah, so we dispatched for eleven different fire and EMS agencies here in the Houston suburban area, and we're what's known as a secondary answering point. And so what that means is when anyone calls 911 in our areas of coverage, they first go to a primary answering point, which would either be Harris county sheriff's office or Fort Bend County Sheriff's office. Wherever they are physically at, when they call 901, and when they say they need fire or EMS, that call is immediately transferred to our center, and then we take care of it from there. And so at that point, we ask for specific information from the calling party, the person that's calling 911, that is their phone number, their address, their name, and stuff like that. And then we finally ask what's going on? And as soon as the person tells us enough information, even when they're not complete with their sentence, when we get enough information to determine, okay, this is gonna be a medical call, or a fire call or something like that, we are already dispatching it, the system already took it, the fire departments have already been alerted, and we already have an ambulance or a fire truck, or whatever we need to send en route. While the call taker is getting more information and adjusting that. And as we get more information, as the call taker tells us more about what's going on, we can adjust that dispatch. So if we dispatch an ambulance and all of a sudden it becomes more critical and we need a second ambulance or a fire truck, or a supervisor unit, then the computer aided dispatch, the CAD will accommodate for that and send an additional recommendation to where the fire departments will get additional dispatches to add to those incidents. So, yeah, so there's a lot of technology that goes into that to make it work. Of course, we have the 911 call taking equipment that handles the 901 calls. We got the CAD, the computer aided dispatch. That is pretty much the brains of any dispatch center is dispatch. It takes in the 901 information and it outputs the dispatch recommendation all while, you know, the notes and stuff get taken in there. And it's all stored for historical reasons and record keeping reasons. And then you've got the radio infrastructure. We actually need to talk to the agencies in the field, the fire departments and the. The EMS services. So that's the radios that we communicate with the fire departments and then everything else that supplements that. Right. Back in the day, that's all you really needed, a 901 phone and a radio. But nowadays we got additional technology. We got, every fire truck and ambulance has a computer in it that tells them the details for the calls that they're going to, routes them with their mapping and stuff like that. And so we support that as well. We have automatic station alerting, so, you know, at nighttime they don't have to. That the firefighters don't have to keep the radios on and stay awake and listening the whole time. They've got automated alerting system that will turn on the lights, you know, put alert tones on the speaker, open up the bay doors, and, you know, do even turn off the natural gas in case they left it on so that you get alerted, wake up, go to the call, and, you know, the fire station stays safe. So all of that supplements additional technology that we put in place and support as well. [00:22:06] Speaker A: That's awesome. And all of that goes to, the faster that you can get to and respond to an incident, the better it's going to be. Time is of the essence. And the more information that the responders have of the medical condition that they're walking into the state of the building, if it's on fire, if it's a wreck, if whatever those things are, the more information they have, the more prepared they are to go on scene. What do I need to bring with me? Like, is it. Is it a chemical fire? Is it a house fire? Is it a car fire? Is it an accident? Like, do we need the jaws of life? Like, do we need one truck or two or any of this stuff? It's that triage. So they're starting that triage process before they ever get to scene. In theory, they'll be able to respond quicker and you'll be able to save more lives and save more property damage and all that type of stuff because they're more prepared, which is amazing. That's the whole benefit and value add with technology is even 1 second, even one bit of information can save a life. Right? And when we're dealing with critical infrastructure like this, lives matter, right? Everything you're talking about is specifically around human safety. Yes. Property. Yes. All that kind of stuff. All those things are important. But the. But the priority statement there is human life and human safety, right? And saving people's lives, getting into the hospital, getting the care that they need as quickly as possible. And if technology can help them do that faster. Wow, that's amazing. Right? So it's awesome for the firefighters, it's awesome for the paramedics, it's awesome for the dispatchers. Like, everybody in that can be just a little bit more efficient. And how much the domino effect of the impact of that is huge. [00:23:44] Speaker B: Yeah. And then responder safety is also extremely important, too. And I like to say that, you know, when an incident just starts, no one knows more about the incident, of course, not talking about the person who's calling, but, you know, person who knows the most about that incident is the dispatcher who answered that call. And it's important that that dispatcher is able to convey that information to the. To the units in the fields so that they can be the most situational, aware of what's going on, what they're going to, and then they can make their informed decision on how to handle that call, whether, you know, there's a potential for danger. So we need to stage and call law enforcement, or it's just a run of the mill medical call or fire call, and, you know, just, just respond as per usual. [00:24:27] Speaker A: Sure. Yeah. So dive into what does the tech stack look like? As much as you can go into it. Like, how much is there? Like what? Obviously, you've got phone systems, you've got radios, you've got this CAD system you talked about. There's got to be some redundancy and power and networking and alternate threads of redundancy and network connections and all that kind of stuff. But just kind of dive into what does that architecture look like to provide something as robust in a critical system like this? [00:24:55] Speaker B: Yeah. Yeah. So I feel like I like to fool people because, you know, I talk about 901 and critical infrastructure, and, you know, people want to know about what, like, what's so special about technology, but really it isn't any special. It's. It's like any other conventional corporate it technology. It just, you know, when it comes to cybersecurity, cybersecurity is what protecting confidentiality, integrity, and availability. I like to say that, you know, when it comes to 901 dispatch, availability is the most important thing by far. [00:25:22] Speaker A: Yeah. [00:25:22] Speaker B: Confidentiality, integrity are still important, but also, we're a government entity. Everything is available to the public anyway. But availability is the number one thing we need to stay available. So all the technology that we built here at Westcom was designed with that resiliency strategy in mind to be as available as possible. And if an outage or an issue does happen, how can we recover as much as possible? So, and that's the benefit I had going into this building. This, when I started a few years ago, was, since I have the cybersecurity experience and background, I was able to build this with cyber security in mind, with cyber resiliency and availability in mind. And thankfully, I had a board, an ESD governmental board, that approves all the budget and everything. They supported that. And so when I pitched them, here's what we want to build, here's how I want to build it, here's what it's going to cost, they completely agreed with me, and that's what you talked about earlier, is that trust. They trust me, but the trust goes both ways. I trusted them to give me the support, the tools, and the funding we need to do this right to build it. So before we took on this project, we talked with a lot of other agencies in the area. We toured a lot of other dispatch centers and fire departments and server farms and stuff, and just learned, took some lessons learned from them, learned about the incidents that they've experienced, the mistakes or the improvements that they wish they could have made, and we are able to implement that here. So when it comes to resiliency, where we try to be, I try to threat model every possible threat, man made, natural or otherwise, that I consider, and then kind of balance, how much money can we spend on that, and how can we remediate that threat? So I feel like we've done a pretty good job. I invite you to can come up with any random threats, and we can talk about whether I protected against them. But, for example, like natural threats, lightning strikes, we had a previous dispatch center that did dispatch from my fire department before Westcom, and there was several incidents over the past few years where anytime they got struck by lightning or lightning struck nearby, their services were degraded, whether some radios went down or their cabin went down or whatever. So we took a lot of steps into ensuring that lightning strikes or similar weather related incidents don't happen. So we're very heavily grounded. We have a lot of grounding all over the place. But not only that, we have a lot of redundancy in case certain systems, like the radio tower does get struck by lightning and does go down, we've got redundancies in place, so we can still maintain radio communications. Multiple Internet service providers come into the facility, so if one goes down, we're still good, and then most importantly, electrical. So for those who are familiar with the Houston area, we don't have the most robust or even the whole state of Texas, we don't have the most robust electrical grid. So hurricane barrel came to the Houston area a few weeks or a few months ago. And like most of the region, Westcom had no utility power for about four days. But because of our resiliency strategy, we have not one, but two generators on the property. And even if both of those generators were to fail, we have a large ups system that could hold us for several hours until we can either call an external generator, drive them up and hook up to that, or fix whatever problem our two generators may be having. So luckily, none of that was needed. But our generators did protect us for the entire four days that we had no power. And the great thing is, no one here at Westcom really noticed until I gave the report that, hey, we were, you know, our generators were active for four, four days straight. And that's what I tried to do. I tried to build this resiliency strategy that would be protect us from availability issues. But even when we have to recover from something, it is almost transparent to our dispatchers. So they can just completely focus on taking those 901 calls, and our constituents will get the 901 services they need. So, yeah, as far as the other technology, it's the conventional stuff, right? We have a network stack where we use various different vendors of switches, but everything is redundant. We have two of everything from our firewalls. We have two of those that are, you know, hot, warm, spares ready to go. We got switches that are either virtual task, either are stacked in one way or another, and everything that's critical, like the servers, the CAD workstations, they are connected to multiple different lines, multiple stacks of switches. So even if we have a full switch failure, or cables fail, or a network interface card fail, we're still good, we're still pretending. A lot of things have to fail at the same time in order for it to actually impact our services. And then same thing with power, we have not one, but two large 100 kva ups on the facility. So again, everything critical is plugged into each ups. One power supply on one ups, one on the other. So if one were to fail, which we actually did have happen, all the equipment is still powered, right? They'll have some sort of resiliency even after that. [00:30:56] Speaker A: So, yeah, yeah, it's amazing. So I've grown up in critical infrastructure. A lot of my time was in critical manufacturing, critical power generation, transmission distribution, that kind of thing. So for me, it's very, it's very easy to see because every system I've ever been in and designed and been part of has been that critical. Right. It's, it's redundant power, it's redundant ups, it's redundant backup, it's redundant switches, like absolutely everything. And this redundant switch has two power supplies. One's going to two different power feeds, and I've got two network connections to everything. And even my windows machines or my HMIs or the engineering workstations, they have dual power supplies and they've got dual network feeds and all that kind of stuff. But that gets expensive. So you can't have every system in every critical or every non critical system do that. Right. You only focus that. You're not trying to platinum code everything. You're trying to put it so that the critical systems can maintain an outage. Right. And the second piece to this goes back to, okay, you've designed this robust system, this network, all this kind of stuff. The other part that I see, and we haven't gone there. So that'll be my next direction for you, is in these critical environments and architecting these highly robust, capable systems. Monitoring is important because if I've got two switches and everything's redundant and I don't know that one of them went down, I think I'm redundant, but I'm really not because one of them's already failed and I don't know it's failed. Right. My ups is down, like you knew that it was down. So obviously you have this implemented. But I've seen super critical capable systems where they don't realize that a switch is failed. It's in series, they've got two of them and it's done its job. So the system, they never noticed anything went down, but they also didn't know that it was down. They didn't know to get it back up and going. So it never, it didn't actually cause a problem, but it could have if they, if it, if the secondary system also failed. You know, you've got primary and backup. If the primary fails and it goes to the backup, they never notice anything goes down. Now they're only running with one system. If that backup goes down, then we have an issue. So that monitoring is also important. So talk a little bit about how you guys monitor all that stuff and how you notice that your systems are good and healthy. [00:33:17] Speaker B: Yeah. Yeah. So we do a lot of monitoring here, but of course, there's always room for improvement. But like, for the viewers who can actually see my background, there's, I have a monitoring dashboard where I can just see the current health status of all of Westcom systems, you know, green, of course. Good. Red or orange? There's an issue that needs to be fixed, but we do get alerts for almost anything that happens. And, yeah, there is such a thing as alert fatigue, but I think we've done a good job kind of triaging and prioritizing those alerts. So whenever we get an alert, whether something is down or maybe a power supply fails or there's a cybersecurity incident or an intrusion, something like that, the it team here gets an automatic email as well as a support ticket system that then automatically prioritizes it. And we can. We can respond to that. So I'll give you a good example. And I know you've already talked about Crowdstrike, like, six different episodes. I've counted them all here. It is a shame, because I love. I love talking about how crowdstrike impacted us, because it did, because I feel like we learned a lot of lessons from it. But when Crowdstrike happened, so July 19, right at zero zero. So right at midnight, I just randomly started getting alerts on my phone while I'm getting ready for bed at home that some servers are going down. And at first, when one or two servers went down, that's okay. That sometimes happened. The server reboots, it comes back up, but they weren't coming back up. And then when I started getting additional alerts for additional servers were going offline, I noticed that there wasn't a specific pattern. They were random servers. And usually, you know, if a hypervisor goes down, then all the servers on that hypervisor, but there wasn't that pattern. So I'm starting getting concerned. So I vpn, I'm at home, right? So I remote in. I VPN, I start remote desktopping into servers that are still online. And as I'm investigating, looking at things, I lose connections. So now I start freaking out. Not really freaking out, but like, I'm starting thinking, okay, what's going on? So I do have Dell Idrac, which is pretty much, like, local access to one of my hypervisor servers, so I can connect to it. And an interface, just like if I had a monitor and keyboard. And so I remote into that, and I'm looking at the hypervisor, just checking the health, and then I see right in front of my eyes, it blue screen. And that's where I get. That's where I internally panic. I'm like, oh, dang, I'm a cybersecurity guy. I always talk about cybersecurity guys. I finally, I'm thinking to myself, I finally got ransomware. I'm finally the victim. [00:35:43] Speaker A: Let's. [00:35:43] Speaker B: Let's start working on this. But thankfully, while I'm investigating that, someone sends me a news article that some airlines are having it issues and are grounding all their flights. And I almost immediately correlate, okay, there's some global it out. This is probably been not me, so. But the alerting system going back to that clued me into that, and I was able to address that and start investigating relatively quickly. So I used to brag that in three years of Westcom operating, we've had zero outages. Crowdstrike happened. So that crowdstrike actually did take us out. But because I was able to be alerted so quickly and respond so quickly, we were able to find the root cause analysis, attribute it to the Crowdstrike Falcon center, and come up with our own temporary fix to get servers back online, even before crowdstrike acknowledged an incident and published their own fix. So even with the outage, we recovered from that within 62 minutes. 62 minutes and 19 seconds. And I'm particularly proud of that recovery period, because talking to other peers and agencies, they weren't as quick, but there was a few things that contributed to that. So our alerting contributed to that. You know, our process of quickly activating our contingency plans worked really well. And even though we were down, our CAD was down for 62 minutes. We still had a contingency plan. We were still able to take nine one calls. Our radio still works, so we can still dispatch the fire departments. And so our contingency plan was pen and paper. Our dispatchers took the calls that were already on the CAD at that time on pen and paper. And any additional calls that came in, we had about 5901 calls during our outage period. They were able to just do that in pen and paper and dispatch that to the fire department. So the great thing is, the fire departments had no idea it was still business as usual, and we never missed a 901 call. The people that we serve still got as good of a service as we possibly can, and we were able to recover that cat and go back to normal operations within 62 minutes. So the one lesson learned, though, going back to alerting, is my alerting server is on the same hypervisor that crowdstrike killed. So after that was down, the alerting was also down. But of the many lessons learned, I did an actor action report, and I shared that publicly. That's one of the lessons learned, is have different alerting platforms that are completely separate offsite. So they're resilient for, if we have a major system wide outage, those alerting systems probably will not be affected as well. [00:38:32] Speaker A: Yeah, that is huge. So the fact that you guys recovered that quickly shows that a, it was resilient in the design, but also your recovery plan. Right. So your recovery, obviously you had backups. You tested those backups. Everybody knew what to do and, and the fact that. But the systems went down, but you had the worst case scenario. Everything's down. What do we do now? We can still take nine one calls and we can still dispatch. So the fact that that happened, and again, the fire departments didn't know that that's the best scenario. Like you hope it never happens, but you have to plan for those things because you just don't know. You can't assume that that's never going to happen. No matter how resilient you make your network and how many switches and how many upss you put in and how many generators you put in, there's going to be something that happens that you can't see. So you have to plan for. Okay, I designed the best network in the world. It's still going to go down. What do I do then? Right? So the fact that you guys thought through that process, at least at some level, shows that everyone needs to do that. Now, again, if it's a not critical system, you know, you're, you're managing a self storage place. You don't need to have that level of capability, but the same thing, okay, my gates are automatically controlled. In that scenario. Somebody needs to go and lock the gate or I need to have somebody standing there and, you know, opening the gate for customers or whatever, like that's the kind of process that you need to go through. And I think that's missed a lot of the time. Whether, you know, most organizations are doing a tabletop exercise and they're probably doing that once a year. Unfortunately, in my experience, they're doing that at super high levels and they're not necessarily including all the people that are in the know, so they don't get the real true risks like what you just experienced. Like that should be turned into like you did. A lesson learned. What can we learn from this? I should learn from my mistakes. I should learn from my wins. We can always be better. Like, it's not a negative on you or anybody in the organization that you had an impact. And the fact that if you don't learn anything, it's that whether it's in personal development or anything, and there's that. That line, you either win or you learn, right? There is no lose. You only lose when you quit. You only lose when you give up. If you have something that you can't, you fell short on, what did you learn? So that next time, you don't have that same thing happen again. Right now, if you have that same incident and you didn't learn anything, you didn't change anything, and you just continue to repeat that process over and over again, then, yeah, that's a problem. But the fact that you guys learned something and you, you made it a little bit better for next time, that's an awesome win. So the next time this happens, you'll have your alerting done faster and all that kind of stuff. Like you guys, oh, we've done this before. We know what to do. [00:41:21] Speaker B: Yeah, yeah. And then going back to our previous discussion on, like, you know, communicating with people and networking and sharing information is, yeah, we did learn a lot. There was things that we can improve. So full after action report over here. Well, here's what happened. Here's the time line. Here's the conclusion. Here's the lessons learned. You know, like building an incident, solidifying our incident response plan, or using some power policies or a backup strategy. Not only did I share that within Westcom, but I also shared that to the fire departments that we serve. But I took that further. I shared that to all the other public safety agencies here in the greater Houston area and in the state of Texas, just so in case, you know, they want to read the report and see how it impact, it impacted a similar organization and the lessons learned. So, hopefully, maybe some of the lessons learned that they can apply to their agencies. [00:42:13] Speaker A: Exactly. [00:42:14] Speaker B: And, yeah, and that's what I love to do, even if I may not look in the. In the greatest light, because one of the lessons learned is probably something I should have done anyway. It's still something I'm willing to share. So, hopefully, other agencies can learn from my mistakes. And then, you know, one thing that I kind of thought about the other day was when I built Westcom, I did a lot of threat modeling. And that is, you know, considering what cybersecurity threats do, I want to consider how do I want to build for it? And when I threat modeled, whether I should bitlocker the CAD workstations that are here inside this secured facility, the big workstations, I decided, no, I don't need that. I probably don't need to spend the time and effort to physically encrypt the disks for the hard disks that are for the CAD computers sitting in the locked desks in the locked facilities because that'll save me some time and effort and maybe, you know, something that. But the unintended benefit I got is because I did that threat model, which I feel is an accurate threat model, not needing to encrypt the desk computers, that greatly improved the amount of time I needed to recover from Crowdstrike because now I no longer had to consider, you know, dealing with Bitlocker. I just had a straightforward recovery plan to fix those computers and that's how we're able to cover so quickly. A lot of other agencies, you know, they bit locker everything and that's great. The better security is, is often better, but at the cost of, you know, recovery time or at the cost of having to get more people to help, more resources, more tooling and stuff like that. So that's kind of a threat model benefit that I think I should talk about more. [00:43:54] Speaker A: No, I agree. So I've seen time and time again that examples like that. So again, my career mostly has been in critical infrastructure and places like that. And when I'm having conversations with my IT counterparts coming from corporate that can't understand why. So a lot of environments I'm in like power plants, they're antiquated, they're older technology. So we may have a Windows XP machine serving as a critical function or they're not, you know, they don't lock their workstations, there's no password, they don't have individual logins. Like things like that in a power utility are a little different. But to your point, you know, the CIA triad, and I say this all the time, in power utility and critical infrastructure, the CIA triad is kind of flipped upside down and availability is at the top. Not that I don't care about confidentiality, not that I don't care about integrity, right. It's that availability is way more important than anything else and especially in this space. I say availability slash safety, right? So it's more important that I can control the process than it is that I know who's logged into that machine. So I have other mitigating factors. In a control room, there's physical security. They had to get into the room, had to get onto the facility. It's a small group of people. So if some random person walked into the control room and somehow got that far and they sat down at the desk, that's not going to happen because they're going to know, who the heck are you? What are you doing in my chair? And get away from that keyboard or I'm going to hurt you. So I don't, I don't need to lock the workstation. I don't need to have the screens, you know, somebody individually log in. I've got cameras. I know who's on shift during that time and it's a small group of people. It's five people, it's ten people, it's 20 people. I know who has access so I can narrow that down very quickly. There's never a situation where a control room is unattended ever in these environments. Right. It's just not. So I don't need to. To your point, it would, would bitlocker and encrypting all this stuff and having secure logins, hypothetically, would that improve security? Sure. But does it reduce my availability? Absolutely. And so what's, what's the risk reward? And this is where you have to look at this more cybersecurity is not always better, right. Sometimes less security and more availability is the right answer to the question. And a cyber person, a strictly cyber person, it just blows their mind. Like why would you not want to be more secure when it, when it hinders my availability? If, if I have to choose between availability and cybersecurity without making it, you know, I'm not going to put on the Internet. Like, I don't mean obviously there is a line where I have to say, no, you're not doing that. I'm not going to allow you to secure mode, you know, remote access in from China without any authentication. Like, no, of course not. Right. There is a delineation but that delineation is not the same on OT and especially in these critical infrastructure environments as it is. In it, in it. They would never allow Windows XP machine on your network ever. For under no circumstance, right. But in it or OT, it's there all the time. I see it at every power plant I walk into and sometimes older stuff than XP, believe it or not, but we put other mitigating factors around it. So it's off the network or it's off the, you know, the Internet. I've got firewalls, I'm monitoring it. Like there's a lot of other things that I can do because I can't patch it, I can't do a lot of the things that I would do in an it world. And that's okay. Right. Understanding that difference is the key. And being able to communicate that in a way that says, yes, I'm not doing this. I looked at that, I did the threat model and this is why I think it's okay, here's how I'm going to accept that risk and why I'm going to accept that risk. And that's the most important piece, right? [00:47:42] Speaker B: And that threat model is just for that specific threat of someone physically coming in and stealing a workstation. That's not to say that, you know, just because you're inside the facilities of Westcom or inside the Westcom network that it's free reign, right? I'm a huge proponent and believer in the zero trust model, continuous verification, all that stuff, least privilege. So, you know, even if you're on the network or you get access to something, there still is just every step of the way, every chain, every rung and ladder, you're continuously verified to make sure you have the access you need to or you're granted when, only when you need it. So we've actually had a few, several, several minor cybersecurity incidents here at Wescom that luckily ended up not being a big deal because we had that zero trust model. So one of them being like, everyone falls victim to business email compromise. Email Phishing recently had a successful email phishing attack where three of our relatively new employees were successfully convinced to click on a link and type in their Microsoft 365 credentials. But because of our alerting, we were able to see, recognize that real quick and stop it. But even with, even had they maintained access for a little longer than they had, they would have hit a brick wall. Like all they would have had was the emails. For the relatively new employees, which isn't much emails, those employees don't have access to anything that they don't need to have. And for the things that they do need access to, like the computer aided dispatch software, the recorder, the station alerting stuff, all those have additional steps of verification. So someone who just has control of their password or accession token may not be able to access those resources. And then another thing, another incident, we had one of our, oh, this is fun, our Palo Alto firewalls. A few months ago there was a pretty wide, widely known public, I don't know the word, but there was a zero day vulnerability for Palo Alto firewalls. One Friday, Palo Alto disclosed that hey, there's a ten out of ten critical remote code vulnerability for the global protect VPN portal. There is no patch at the time, but here is a setting change you can do to temporarily mitigate this vulnerability until it's a patch. So it's Friday afternoon I implemented that, you know, temporary mitigation and then on Monday I came back in the Monday afternoon, I reread that advisory. I see they've updated it and I forgot the wording, but the advisory effectively said, just kidding, that mitigation didn't work. So the whole, the whole weekend, we were seemingly vulnerable to this ten out of ten critical remote code execution exploit. And I never gotten alerts or anything that we were compromised. But just because I knew we were vulnerable, I started investigating the logs, and I actually did find indicators of compromise. I was able to find that someone did exploit these chain of vulnerabilities and were able to pull our entire firewall configuration. [00:50:58] Speaker A: Right. [00:50:58] Speaker B: Whoever this threat actor was had access to our, you know, private certificates, VPN keys and credentials and all that stuff. But I wasn't too concerned, because again, we have this zero trust model continuous verification. Yes, this is something that I need to address right now and remediate. But even had they tried to take advantage of the information that they got, they wouldn't have gotten far. And from my forensics after the fact, I was right, there is nothing done from that. And we were able to completely blast those firewalls and resolve the issue. And just to brag, again, because of our resiliency strategy, our redundancy, we had, we were able to completely reset and rebuild those firewalls without taking Westcom or the network or the facility down at all. Again, one of those things where no one noticed until I published the after action report. [00:51:51] Speaker A: Yeah, that's awesome. Right? Again, like this, it's the culture of be okay to make a mistake or be okay. Things are going to happen. It's not a lack of, it's not like your environment wasn't built correctly. It's going to happen. Threat actors are really capable and smart people, and there's going to be zero days that there's no way you can know about why they're called zero days like you can't patch, you can't update. There's going to be those vulnerabilities. The fact that it's redundant, the fact that you've got the resiliency, the fact that you're able to recover and respond to those things is vastly more important and valuable than it is to have a perfect environment. Nobody's going to have a perfect environment in any situation. There has never been a perfectly secure. And there, you know, I hate it when vendors and or people say, I've got a secure environment. My environment. Secure. No, it's not. You have no idea. There is. It's impossible to have a secure environment. [00:52:46] Speaker B: I set those expectations like my board member, I report to them every month. I give them a report of, you know, what's going on with technology, any incidents that we had, and I always tell them, even when we're building Westcom, this will not be unhackable. Just because I'm a cyber guy doesn't mean I'm going to make this, you know, perfectly secure. It won't be. Incidents will happen just like the crowd strike. And a lot of us get that question, how could you prevent. How could you have prevented crowdstrike? There really isn't a good answer. When you have. When you depend on one vendor to provide, you know, cybersecurity, endpoint detection, response to all of your servers and workstations, you can't really protect against an outage like that. But what you can do is improve your resiliency. And again, resiliency. I like to define reliability and resiliency. [00:53:31] Speaker A: Right. [00:53:31] Speaker B: Reliable is, you know, protecting against failure, against downtime, that entirely. Resiliency is your ability to recover from failure and get back into operation. And that's what I focused mostly on, is there will be an additional. Additional outages, there will be incidents. People will get into our network that shouldn't be, but our ability to respond and recover from that quickly, efficiently, effectively, is what's super important. And that's what I like to set the expectation to our board and our fire departments and other agencies, you know, work on your resiliency plan, build it, prioritize what's most important so you can recover those first, but then not just document a plan, but practice it, test it, exercise, because much like backups, if you're doing backups, but you're not testing the backups, you have no backups. Same thing with an IRP, an incident response plan or resiliency plan. If you don't test and exercise your plan and it continuously improve it, you effectively have no plan. So that's what I like to push for, you know, my agency and other agencies. [00:54:37] Speaker A: Yeah, yeah. And that goes to, you know, not everybody, as we kind of kick this off, not everybody gets the opportunity to build a ground up, you know, environment that has fully redundant and, you know, does all of those things and is looking at it that way, that doesn't mean that you can't be resilient. Right? So if I can't have redundant switches everywhere, what can I do? And I dealt with this a lot in power plants and wastewater and critical manufacturing and things like that. Okay? So I can't have redundancy everywhere. I don't have the budget, let's say. Well, what can I do? Well, I know I'm going to eventually lose a switch. So if I can't have redundant switches, what's the next best thing, as far as resiliency, to your point? Well, I can have a spare switch. I can't have ten spares, but if I have one, that'll help me get back up. So if I lose a switch, I'm going to have downtime. But if I have an on site switch, and I have the configurations of all the switches in a certain spot, then I can drop the config on that switch, rip the old switch out, put the new switch in, boom, I'm back up and running. Like that's, that's hours of time sucked out. And not to mention that if I don't have that capable, then I'm gonna have to order one. That could be days of outage versus a few, maybe minutes, hours of time for me to just throw a switch in, drop the config on, boom, I'm back up and running. And now maybe I've got two of them. Maybe I've got three of them, or I just got one, and then I order a new switch to replace that one that's broken, or RMA, the one that's broken, and I get a new one in. There's lots of ways to be resilient, and it's not all complete ground up design of my environment and an environment like you've created, which is the, the goal of every engineer and networking guy and nerd like you and I, right? We all want to build that environment. Not everybody has the ability to do that. That doesn't mean that you just give up and throw your hands and say, well, they'll never let me do that. Okay, what can you do with the resources you have? That's the question you listeners should be having, is, this is the resources I have. How can I, with knowing that? How can I be more resilient? Right? Not redundant, resilient. How can I do that? And sometimes that's not a technology problem. Sometimes it's it. You talked about the planning. Sometimes it's going to be making sure I have a plan, I'm testing my backups, I've I've making sure that all the parties know what they're supposed to do in an outage. Like, how do I work around these things? If the switch goes down, what do I do then? Like, those conversations cost no money. That's just time and experience that is going to help you be more resilient, the faster you can get back up and running. And that's not just going out and buying technology. Everybody focuses on the technology and the switches and the crowdstrike and all the different capabilities because they're fun and they're sexy. That's not the only way to be resilient in all environments. So that's kind of the lesson learned that I say from my experience and what I hear and what you've done, you're not always going to get to that, that great place of perfect resilience or redundancy, but there's, there's still ways to be resilient. [00:57:27] Speaker B: Yep. Yep. And it's, it's all just the contingency plan, right. Like I said, with the crowdstrike outage, all of our technical technological contingencies failed because of crowdstrike. So we went right back to pen and paper. And, you know, every industry is going to have, you know, their own way of doing things or what's important to them. But in public safety, it's still just getting to the 901 calls and dealing with the on calls. So if we have to do that with just pen and paper and archaic, you know, rocks and whatever we can, and that's part of our contingency plan is how to operate completely with no technology at all. While we're doing that, we're still working on recovering the technology. So in this day and age, people, you know, a lot of, a lot of people older than me love to say that, you know, the kids are way too dependent on technology. And it's true in one shape or form. Even now, when, when the, the computer aided or the computers in the fire trucks are down, the firefighters can still do job, but they love to complain about that. So we still fix it with urgency, but, you know, ignoring the complaints. They can still do the job. They can still get to the call. We still have key maps, and they can still put out the fire without a computer or without a cell phone or whatever. So, yeah, just practice. Practice. That's funny, working without the technology. [00:58:45] Speaker A: Yeah. Like, like I showed you before we started recording, I've got a fire hat up there. That was my grandfather's. He passed away long time ago, but he was a, he was a volunteer fireman for 40 something years. He, his casket was delivered by the fire department, you know, on the whole parade thing. But, you know, they didn't have technology. They had radios. That's it. They had these little pager things that it would go off at all times of the night. And you'd hear all the broadcasts of all the things and he, I remember spending out of his house and, you know, those things would be squawking all night long. And when he was on call, he had to take those calls and show up and, and all that type of stuff. So, you know, he had nothing, right, except a radio and, and his ppe. Right. You know, his, his fire suit. And he'd come in from work and throw that stuff on. And sometimes I'd go with him and I'd sit in the truck and watch him put out a fire or watch him show up to an accident or all that kind of stuff. But, you know, we get relying on technology because it makes us more efficient, but we have to be able to plan for the worst case scenario. What if I have none of that stuff? And we see that right now in North Carolina with all that's going on in that environment? 911 is down, phone lines are down. Like, they can't, you know, the local forces are not able to get to those places. So it's taken people reaching out and helping and helicopters coming in and bringing in supplies because roads are shut down. You can't get in or out of, um, so we have to do that contingency planning for worst case scenario. Right. And there's going to be a whole bunch of lessons learned coming out of that incident. You guys in Houston deal with hurricanes all the time. They don't, they're not, they're not used to hurricane, not that level of hurricane. Like, there's, there's people that have lived in their houses for 50 plus years and have never flooded ever. Right. And now they're, their house got six foot of water in it. Right. And they're, they were, they had to rescue themselves by getting on the roof and wait until the water subsided. Right. It's insane the amount of damage, but that's the type of incident that allows us to say, we've got to learn from this and do different next time. Right. It's, you can't just, it's not just a failure. It's not just point fingers at people and say, you messed up. Like, instead let's focus on what we can do to get better and more resilient next time. [01:00:48] Speaker B: Absolutely. I completely agree with that, man. And yeah, we do have hurricanes all the time, and that's why we, not just us, but, you know, the whole region, we're constantly discussing and practicing how can we operate, how can we respond to major disasters without our usual comforts of the radio system working the Internet working the electricity. And Hurricane Harvey was a great example of that is Westcom didn't exist when Hurricane Harvey existed, but a lot of the fire departments did have to deal with just what they had, boats and, and maybe some point to point radios, but certainly no wide range network because a lot of things were down. And then Hurricane barrel was a recent one where the power was down for a lot of facilities. A lot of, you know, some fire stations didn't have power for a while, and dispatch centers had issues, but we were able to work through that. But also we were able to work together. And that's one thing I love to do, is foster that communication and that working together, because if one dispatch center has issues, maybe Westcom could help, can assist that dispatch center technology or operational. Likewise, maybe they can help us. And we do have that contingency. If we were to fail, we disappear off the face of the earth. Our 901 calls automatically get routed to another dispatch center. Likewise, we are backups for others. So help each other. [01:02:10] Speaker A: Absolutely. Well, so we went on a roller coaster ride today of all the different topics, from career to resiliency, which I love. I love having these conversations because it's. We're more than just the things that, the technology stuff, it's more than just the ones and zeros and the redundancy and all that kind of stuff. It's a lot of, like we said, right. It's the connections, it's the networking. It's all those types of things. So with that in mind, and I did prep you for this, that it's coming. What is one, in the next five to ten years, what is one thing that you're excited about coming up over the horizon and maybe one thing that's a little concerning around the cybersecurity footprint in your space and that you. That you've seen or thought about. [01:02:51] Speaker B: Yeah. So I guess one thing that I'm excited for, it's happening now. In the, in the past five years, I think cybersecurity has now become a household name. I know when I started in this industry, you know, it was, it was brand new. No one really knew, understood what I did. When I worked in cybersecurity, no one knew what a red team engagement is or what ransomware is. [01:03:12] Speaker A: But. [01:03:12] Speaker B: But recently, with all the latest things in the news and even crowdstrike becoming a household name, cybersecurity has become a household name. So everyone is talking about it from CEO's and CIO's down to my mom in the kitchen. Hey, what's lockbit up to these days. But I think that's great and that's beneficial because it gets more attention to the issues that cybersecurity brings to. It gets more endorsement from the decision makers to invest into cybersecurity and to train their employees for cybersecurity. And just generally, more people, more students are interested in cybersecurity, want to get into the realm. So I think in the next five or ten years, there will be more organizations who have budget and consideration for cybersecurity, but then there will also be more really good people in the industry working in cybersecurity because more institutions are pushing for cybersecurity related curriculums. More people are interested in it, more people are just learning and getting into it. And so we're going to have a good, I think we're going to have a good cybersecurity workforce that would help kind of defend all the entities and organizations we have and just defend everything, defend our country, defend the world. So, yeah, that's, that's what I really look forward to is how the industry continues to rapidly grow and evolve in the next 510 years. As far as what I'm worried about, I don't know. I didn't have a lot of time to think about what I'm worried about. I like to stay optimist positive. So really, I'm just excited to see where all of this goes. Now. I do follow a lot of, I do a lot of threat intelligence, so I do follow a lot of what's going on with ransomware groups and nation state attacks. So, you know, that stuff's only getting worse, right? Yeah, that does worry me a little bit because as, as these threat actors continue to target additional things like critical infrastructure, you know, the recent being the water facilities, a lot of PLC's and water facilities getting hacked, but, you know, it's not going to stop there. Additional critical infrastructure, my critical infrastructure will get targeted. Who knows what else can happen? And so that, that does concern me. But my previous, what I'm excited for will hopefully be a good defense against that. As more people get into cybersecurity, become aware and get interested and be involved, we will be able to protect from those threat actors, those bad guys. [01:05:43] Speaker A: Yeah. [01:05:44] Speaker B: Yeah, I think that's it. [01:05:45] Speaker A: Absolutely. I dig it, man. All right, so how can people, you talked about a couple organizations that you have. How could people find out if they're in the Houston area, they want to reach out on your, your haha. And the lock pickers as well as maybe see you speak at a conference or what do you have coming up that you want people to know about? [01:06:02] Speaker B: Yeah, so I have. I have LinkedIn and Twitter. I don't, I'm not as active in Twitter as I'd like to be, but I'll try to be more, but mainly on LinkedIn. I do post a lot of what's going on LinkedIn. I answer a lot of messages or questions. So you can. You can find me. I'm Dennis Maldonado on LinkedIn. I think my Twitter handle is Dennis line is. But as far as I don't have a lot of super public things that I'm talking about. But I know I actually got asked to be on the Hugh SEC cast podcast. That's a Hugh secon podcast. I'll be on that sometime next week. I'm not sure when that'll be actually released, but it is. October is Cybersecurity Awareness Month, so I have had a lot of engagements, people or companies asking me to give talks either for their organization, organizations or panels everywhere. So I think the next public one will be Sam Houston State University have asked me to be on a cybersecurity careers panel. So no idea if that's public, but if anyone wants some advice on cybersecurity careers, you can go to that. I'm speaking at a local ISaca conference here in Houston. That's certainly public. I'll be doing that October 25 and then various other small meetups and universities and stuff like that. [01:07:11] Speaker A: Very cool. [01:07:12] Speaker B: If you need anything, if you want to reach out, ask more about Westcom or hell, you want to tour, hit me up on LinkedIn. And I love talking about this, so I love to do it. [01:07:23] Speaker A: Awesome, man. Hey, I really appreciate your time today. Hopefully it wasn't too painful. I know you were a little anxious around it, but, dude, you did great. I really appreciate your time, and it's great knowledge for folks. This is what people enjoy. They want to hear the real stories behind the scenes of what's going on. What was your thought process and how you got there? Because I think we all know that there is no right answer to this. Right? It's an ever changing environment where we have to constantly improve and learn and grow and all that kind of stuff. And that's no different. If you're in critical infrastructure, you're a mom and pop shop, right? It's the same principles. It's just a matter of where that risk tolerance is and adjusting accordingly. Right. So thanks again for your time, man, and it was a great conversation. I look forward to maybe coming down and doing a tour of your system and seeing that in person. [01:08:09] Speaker B: Come on down. I love showing it. So my tours range from 15 minutes just showing around to like 3 hours if we talk and have, you know, ask questions, whatever you I love showing off the place. [01:08:20] Speaker A: Awesome, man. [01:08:20] Speaker B: Thanks for having me. It was fun being I was a little nervous, but when you get me talking about the topics, I'm very passionate about the time flu. [01:08:29] Speaker A: Absolutely. Absolutely. Well, thanks, man. Appreciate it. Again, thanks for your time. And definitely reach out to Dennis for more information. Thanks for joining us on protect it all, where we explore the crossroads of it and OT cybersecurity. Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time.

Other Episodes

Episode 7

March 14, 2024 00:42:26
Episode Cover

Securing OT: Strategies for Prioritizing Vulnerabilities

In this conversation, Bryson Bort discusses his background and the creation of Scythe, an offensive security platform. He also talks about the ICS Village...

Listen

Episode 24

September 16, 2024 00:52:03
Episode Cover

Evolution of Maritime Safety: From Analog Beginnings to Digital Redundancies

In this episode of Protect It All, host Aaron Crow is joined by Christopher Stein from Royal Caribbean Group to delve into the fascinating...

Listen

Episode 22

August 19, 2024 00:51:29
Episode Cover

Tackling Tech Troubles: Inside the DFW Airport Cyber Incident and Wider Industry Challenges with Evan Morgan

In this episode of Protect It All, titled "Tackling Tech Troubles: Inside the DFW Airport Cyber Incident and Wider Industry Challenges with Evan Morgan,"...

Listen