Episode Transcript
[00:00:00] If every person in California were going to have an electric car, where are they going to plug them in? Where are they going to charge them? Like every parking spot has a plug. What about an apartment complex where you've got hundreds or even a thousand parking spots? How do you have the capacity to extend that, that charging network to those places? So it's not that we can't solve these things, but it's going to impact our ability today. We don't have a solution. Right. It just isn't. It isn't there.
[00:00:27] You're listening to Protect it all, where Aaron Crow expands the conversation beyond just OT delving into the interconnected worlds of IT and OT cybersecurity.
[00:00:39] Get ready for essential strategies and insights.
[00:00:43] Here's your host, Aaron Crowe. Hey, y'. All, thank you for joining me. I just wanted to have this episode I posted about this. This week, the DOE just released a major report on the future of our power grid. And I personally think something's missing. Everybody's talking about AI, AI coming. What does that mean? And we need more power. We need more data centers. So data centers are getting built all over the country. They're getting built all over the world, really, but for our country or for the United States and the, the grid specifically, I live here in Texas and our grid in ERCOT is impacted. In fact, they actually call out ERCOT because there's a lot of data centers being built in this state and a lot of power utility or a lot of generation that's getting stood up in this state as well to meet that demand. So I'm excited to talk about.
[00:01:30] I've gotten a lot of hot opinions on this. So, you know, I've, I've spent a long career working in power utility. By no means am I an expert, so at no time am I pretending to know all the details. But I do know I've been through a lot of plant commissionings and control system upgrades and fats and sats and all that type of stuff. And I've seen plants get built and handed off. You know, the builder hands it off and commissioning to the owner and things just aren't met. And the experience that I have, and I've seen this and I've heard these stories. My father was in power utility for 40 something, 50 years. He still is dabbling in it and he has the same experience. So this is not coming from, you know, some research paper or, you know, something that I've heard about. This is firsthand experience seeing plants get commissioned and things that are missed so many Times. And I don't want to call out any names of people that build plants or the companies because it's not necessarily their fault. But they're built on a contract and the only things that get done are the things that are in the line, in the paperwork that the attorney signed. Right. The problem with that is sometimes the eyes aren't dotted and T's aren't crossed. Right. So we're talking about a 50% load growth by 2030. That is huge, low growth, being we have to build capacity, 50% more capacity than we currently have in this country on this grid because the demand is going to grow that much with electric cars, but specifically around AI by 2030. This is 2025. You don't build power plants very quickly. Now granted, before I left my, my previous as an asset owner, we decommissioned like four plants and those things are still sitting there, the substations are there. It's very possible for folks to walk into those sites and spin up a plant a lot faster than if they were doing it from just a piece of earth that doesn't have a lot of those other things there. So switchyard and gas and roads and you know, the cooling from the, the, the lakes that are nearby. So a lot of those requirements are in those places. But still, even with all of that, there's a lot of things that have changed over, you know, the last, a lot of these plants were built in the 70s and there's a lot of things that have changed in designing those plants. So all this growth is really driven by AI data centers, cloud. And in the report, the Department of Energy didn't even mention CyberSecurity in their 2025 Resilient Adequacy Report. Now granted, that wasn't the intention of it, a hundred percent agree. But when we're talking about the resiliency of the grid and the Resource Adequacy Report, there's not a word about ot, there's not a word about cyber. There's not even a footnote like there's nothing in there. This is a grid level transformation and we're building it without security in mind. And again, I know this is something that is not supposed to be part of just that, but again, it has to be there, right? So as we're building out these sites, we have to be considering what those things are, right? So let's dive in. So I'm reading the report. The numbers are just, they're staggering, right? ERCOT, my home state, we're looking at 50% increase of peak demand by 20, 30, 50%.
[00:04:16] You know, I've been in power plants for again, a couple of decades. I've been around it my entire life. You know, I grew up going to coal mines and power plants as a kid as my dad was working there. Right. So I've seen load growth and I've seen plants come online. We've built plants, we've commissioned plants, we've purchased plants from third parties, but we're retiring generation faster than we're replacing it. Coal plants, gas plants, and even some nuclear going offline. Fortunately, I just saw today, my former alma mater just got approval to extend the license on a nuclear plant that they have up north. So yeah, we're building renewables, which is great. But what happens when the wind doesn't blow in West Texas? What happens when it's a cloudy day? It's not going to meet the demand for all of these AI data centers for all of these electric cars. As we see, I think California has a requirement to, to outlaw the sale of combustion engine vehicles by like 2035. I don't remember the number, but if you look at it, if every person in California were going to have an electric car, where are they going to plug them in? Where are they going to charge them? Like every parking spot has a plug. What about an apartment complex where you've got hundreds or even a thousand parking spots? How do you have the capacity to extend that, that charging network to those places? So it's not that we can't solve these things, but it's going to impact our ability today. We don't have a solution. Right. It just isn't, it isn't there. So are we going to charge them at home? We're going to have, I mean, you go to a Tesla charging station today and you could be in line for an hour or longer, especially in highly populated areas, especially California where you have a lot of these cars. So that's a huge, huge, big deal for figuring this thing out. And we have to start thinking about cyber and not just cyber security, not just nation state attackers, which those are things that we have to consider. But you know, NERC just came out with a new, you know, what is it? SIP 15, that is East west communication. We have to monitor those things. So before we were just monitoring north and south. So everything from a firewall in a NERC asset, you know, anything that's, you know, 1500 megawatts or 15 minutes is medium or higher and generation can never be more than medium. Um, and then, you know, obviously transmission, distribution, that kind of stuff is is different classifications of different requirements. But still like in a generation site if you are medium or higher and you're going to have to do east west, meaning I have to know about all everything that talks to everything and have a tool there. But I have to be monitoring, alerting on those things and then take actions. They don't have the skill sets to do those. So we have to be thinking about and some of the switches may not have the capability to do spanning. Like there's so many issues that it goes into. So that's why if we're not architecting, especially on these new plants, these are greenfield. There's such an opportunity here as we're designing these control systems, as we're designing these third party controls, as we're designing how the IO and the endpoints, IDs, everything that PLCs, whatever systems are being used to build these things, we need to make making sure that cyber is part of that table, that we are designing these systems to be cyber resilient. The report says board actually says I'm not making this up. Versus does not contain information about operation, technology and cybersecurity aspects. They immediately say that they're not doing. And again I've already had people say well there's another report for that. Awesome. I strongly believe it should be be in this report. There should be a section in there at least saying hey, we need to get a handle around this. How do you build a resilient grid with one of your largest risk risk factors? Not part of the conversation, not part of the evaluation. So let that sink in for a minute. We're talking about the biggest transformation of our grid in history. And nobody's asking about cybersecurity implications. Nobody's talking about how to secure these systems. Nobody's talking about how to design security by default into these systems as they're being built. Which means if we're not designing them from the beginning and there's another report that talks about cybersecurity, which is always seems to be the way, what does that mean? We'll figure it out later. We're going to design the system and yeah, there's another report, we'll get to that one later. And maybe that's not the case. Maybe some of the organizations are going to bring it right in and it's going to be parallel paths path with this and we're going to design these systems. But the fact that there's a separate document in my experience causes confusion, causes issues and that's going to lead to Systems that are not designed. And again, that doesn't mean they're insecure necessarily. It just means they don't have security in mind when they're designing them. You know, ID National Labs, design security by design. Right. CIE Cyber informed engineering. When we're designing these systems, we need to be designing cybersecurity into them from the beginning. It's so much cheaper, it's so much more effective to do this from the beginning. How am I going to do secure mode access? How am I going to get operators access to these systems? How am I going to get vendors access? Everybody knows the largest turbo control or one of the largest turbo control companies in the world. They monitor their turbo control systems. Okay, how do they do that? They have their own process to get in, but they have access to the networks. And there's been multiple vulnerabilities known about that system. There's vulnerabilities all the time. There's zero days that come out that we're not going to know about. And if we're not thinking about these things and how do we design these systems so that these things aren't going to take us out, we're losing a massive opportunity to secure these systems and do it the right way the first time. Again, I've been in these sites where we've built these plants and we knew that they weren't to up to standard, they weren't going to work. We literally built a power plant in Texas. And when they took the boiler and it was in pieces in a field and it had been sitting there for years, they put this thing together, this boiler together, and they put it together wrong. Like when they were cleaning it off, they sandblasted off the things that, you know, it's kind of like put A to A and B to B and C to C. Right. When they did, they lost some of the instructions. So they put two pieces together wrong. So what that meant was, is this boiler that you're supposed to be able to have scaffolding and be able to walk all the way around. Well, you couldn't walk all the way around it because part of the section didn't go around and it blocked the path. It sounds silly, but it was too hard to fix down the road, so we just left it that way. But the man on this one floor, if you went around this way, you couldn't walk all the way around the boiler. She had to walk a certain way. And then if you're doing a round, you had to walk all the way around the boiler back to the other side to be able to get to the next piece of equipment. That also can mean an issue of evacuation. Like there's just all these domino effects that go into this problem, right? So it's such a impactful thing. I talk to people all the time. You go into these state of the art facilities, you know, beautiful equipment, and you ask a project manager that's been there for 20 years and hey, can you show me the T architecture or can you show me the cyber security? Like how are you handling this? How are you handling that? And they're, they're always like oh well that's somebody else. That's I'm not exactly sure, like we're focused on the operations or that's an operations problem. I'm dealing with the mechanical. But the cyber stuff has to be everybody's problem. It's no longer an optional thing. It has to be part of the over. Like I wouldn't not include safety as part of my conversation. I would not have it be reliable as part of my design. So I can't not have cyber to be part of this. I think it's too big of a problem. So here's what's keeping me up at night. I mean literally like this is mind blowing to me as I see this and I mean I'm super excited about AI and the future that's coming with it and all that's coming with data centers and the opportunity to build power. You can map out the success of a country based upon their power generation. That's because of third world countries don't have the power, so they can't create. So who is building all this new generation? And I'm talking about the companies that funding it. I'm talking about who's actually designing the systems. Are they including cybersecurity? Are they even mentioning the Idaho National Labs? I informed engineering process, right. A couple weeks ago I was meeting with a major tech company. They're planning to build their own generation to support their data centers, which is very common. You know, co join. We used to do it all the time with like a aluminum smelter and one of our plants supported aluminum smelter. And the company guaranteed, hey, any power that you buy will buy it from you, right? And any overage then we could sell it on the grid. But these people are building their own generation to support the data center. Smart people, they know servers, they know software, they know data centers. So I asked them about redundancy planning for the generation assets and they just said we'll handle it. The Same way they do in the data center. I'll add more servers. And I said, no, wait, like, how do you. Your power generation redundancy? And they looked at me like I was speaking a foreign language. Right. So they don't really understand. We're, we're, we're putting this in front of people that don't necessarily understand. We've bought plants from financial organizations that bought a power plant as an asset because it made money for them, but they didn't do the same maintenance. I think I've told this story before, but I'll tell it quickly again. I was working at the previous company and we had three plants and they were all designed and built, I think within the same year. But the plans were exactly the same. You could cookie cutter, you could go to one site and look at it, and it had the same exact floor plan. They basically just turned the drawing. When it was built, it had the exact same control system, exact same hardware, exact same boilers, exact same turbines. Like everything about it. Where the offices were located and the buildings and the pupping and every. The concrete layout, the parking lot, like absolute. Everything was identical. Two of the plants were owned by a power utility company. One, the plant was owned by a financial services company. The two plants that were owned by the power company, they did maintenance, they did upgrades, they did control system upgrades. They did all this normal operations because they know how to run a power plant. Financial company never upgraded the control system ever. It had never been touched. Nothing in that system had ever been upgraded, patched new functionality capability, anything like that. It was still running now to their point. It was still making the money. So if it ain't broke, don't fix. But we had to go in there and fix a whole bunch of things that we knew about it that were issues and could cause problems in safety, availability, reliability, all these things that we knew, and they were just plain Russian roulette. And nothing bad had happened yet, right? So I say this all the time. OT is different. It's not harder, it's just different. You can't control, alt, delete your turbine or your boiler. You can't patch a generator while it's running. We've learned that the hard way. You can't just, you know, be upgrading systems as they're running in the middle of the night, three hours away and expect everything to be okay. You're going to get some really upset operators and plant managers, tech folks. I'm one of them. You know, I'm a, a recovering IT technology corporate person. You know, we want to move fast and break things. I want to upgrade, I want to test, I want to push all this stuff, I want to reboot, I want to take control. This is mine, all the things. But in ot, you break things, people lose power, people could die. Like in Texas, in August, we don't have electricity. Some people can die. In the wintertime, you don't have power and there's a snowstorm, you can die. So specifically, let's talk about Texas for a minute because it hits host for me. I live in Texas. I lived through last year. Last couple of years we've had freezes and heat waves, you know, notifications about conserving electricity. We've had had rolling brownouts. Our grid walks a tightrope. This is a deregulated environment.
[00:14:26] Other states, you can, you know, they can go do a rate case and they can get a rate increase so that they can go justify building a new plant because the demand is low or the demand is high and the supply is low. In Texas that's not the case. So there's no guaranteed return on your investment here. So it's difficult, it's more of a risk for a company to build a plant here. We can't just import power from, you know, Oklahoma. When things get tough, we can, Texas is different. We ERCOT is an isolated environment by design. Right? It's DC connections into Oklahoma and Mexico and all these places. But that's to protect the grid, like the overall grid. So that one trip or, you know, bringing the grid down on the east coast isn't going to black out the whole country. So Texas can be used. And there's this whole thing about Black Star and I'm not going to dive too much into that. But you know, you have to have power to start power. That's the funny thing is, is like you can't just start up a power plant if, hypothetically if the entire GR went down, you have to be able to start. So you get these little gas turbine generators basically and you get them spinning up enough that you can start one small plant and that gets up enough and then you have to start the next one. And there's a whole electrical phase that you have to do and to get them all in phase and it's this whole process and we have this black start process for a reason and we can then jump start other grids, east coast grid and the west coast grid and even Mexico can help as well. But it's a very, very complicated system. Now we're adding all this load, all this complexity, all These people and who's owning them? And are the people that are running or are building these plants, are they operators? Are they the normal folks like the company I used to work for that runs power plants and knows all the maintenance and all the things that go into them, or are they Tesla? I mean I love Elon Musk, he's a smart guy, but he knows how to build rockets and all the things. But they don't know power plant management. That doesn't mean they couldn't hire people and come in. I'm not saying that you have to be, you know, Duke Energy or Vistra or you know, Florida Power and like to be able to own a power plant and do a good job. Not at all what I'm saying. But what I am saying is those people have been doing this for so long, they're really good at it. So we have to at least make sure that the people are going to be running these things, understand how to run them and they're doing them correctly. Right. So when we're running a grid with rays within margins, when every megawatt matters, you have zero room for error. Like zero. Now we're adding a cyber incident into that equation. What happens if we bring down a plant? Picture this. You got a bad actor who compromises the control system. Maybe they get into an HMI to substation, maybe they manipulate some protection relay settings. In the past we would have like spinning reserve that we could. Once one thing went down and we had one kind of sitting and waiting. We're getting so narrow on our spinning reserve, we don't have those things as much anymore. So that that can lead to. And again it doesn't have to be a cyber attack. It can be, you know, somebody having an accident running into a transformer or bad weather. We just had a bunch of bad weather in Texas. They water and damage, anything like that can cause issues. We just need to be considering all of these things when we're designing these systems and we're doing a good job on some of them. I think this report they're talking about a lot of the good things. But what about the cyber aspect? What about the technology design and designing that cyber resiliency in. Right. I believe it's non negotiable. I believe cyber engineering needs to be mandatory and all these new generation projects, it's not a. We'll think about it. It's not a separate process or a separate report. You know, nerc, SIP is a great thing but it's not mandating like a lot of these Will probably be underneath the NERC SIP requirement for a lot of the problems that we're talking about. Those will probably be smaller, you know, under 1500 megawatt units, which means there'll be low impact and there's very low requirements on a low impact NERC SIP site. So they don't have to do. I mean, I don't even think you have to have an asset inventory like. Don't quote me on that. Patrick. Patrick, you can scream at me if I'm wrong on that. But the point is, is like the requirements for a low impact generation side is super low and it's not enough. It's also short sighted. I love compliance. I love nercip. I think it's done. I think it's one of the reasons why out of the critical infrastructure verticals, you know, 16 or 17 critical infrastructures in the United States, I think power is one of the most mature. And I think that's because of NERC sip. I've spent a lot of time with Narcissip and it can be frustrating. Compliance can be frustrating. But I also believe it's the reason why we are so much more mature in the generation and power utility environments. But we need to be able to understand has to be mandatory. This has to be. These things need to be designed with this capability now because it's too expensive and too difficult after the fact to add it because you're changing the way the plant works many times you're changing the way. And then I'm gonna have to get budget for it and it's not part of it. It's way more expensive to do it. Like everything about it is more difficult. We are so losing out by. It's not even gonna be a huge cost difference to add the security by design in now. If we add it later, it's gonna be exponentially more expensive. And then when are we gonna do it? That's the problem. It's like, oh, I'll handle that.
[00:18:57] Is it before or after we have something bad happen? So we also need to bridge the knowledge gap, right? We need to, you know, make sure that the tech companies that are playing in this power space, they understand that what they're buying and what they're running and understand that the power industry and OT is different than a data center. Even though, you know, it's not just a data center with turbines. It's not just, you know, it's different physics and risks and consequences and requirements and compliance. We need to understand who owns the security. How many times have they Seen this like plant gets built, handed over to operations. Suddenly they disc. There's no security architecture. There's no. I talked about the walking around the boiler like it. Once they're done, they've signed and they, you've received it, you've commissioned that plant, they're done. They're not coming back and fixing things. They're done. Company I worked for, we would literally say, oh, we'll fix that after the fact. So it was too expensive to get the contractor that was building the plant to change it. So they, they just build it knowing that it's wrong and that's not how we want it. And then we spent the next three years fixing it to be the way that we wanted it. That's, that's super dangerous in this space. So I always say that we're on the same team. Whether you're IT or ot, your generation transmission, you're on the C suite, you're a vendor. Like we all are wearing the same jersey and it's, it says critical infrastructure, it says power utility, it says generation. Right? We're on the same team, but we're not always playing like we're on the same team. A lot of times, you know, we've been building critical infrastructure like it's 1995. When the threats are I'll just disconnect it from the network or I'll disconnect the modem, right? But the math doesn't work like that anyways anymore.
[00:20:23] So here's my challenge to anyone listening. If you're involved in new generation projects, I don't care if you're the owner, the vendor consultant, you know, you're just a grunt, whatever.
[00:20:33] Start asking hard questions. Where are the OT security requirements? Who's responsible for security post commissioning? Have we considered cyber informed engineering? And if they haven't heard of it, Idaho National Labs is a great resource. Reach out. It's there, there's training you can take. Like all the stuff is online. It's super, super, super powerful to use in this scenario. Do we actually understand the difference between OT and IT reliability? And if you're in operations, if you inherit these new plants, don't wait to hand over. Ask about security.
[00:21:03] Do it now because it's going to be more difficult. After you commission that plant, make sure that you have a cyber representative in the design meetings. Bring in donuts if you have to. Right Bribe your way. Get in the room, talk to the vendors, figure out who the right people to talk to are. Make sure that we're doing this stuff early as Possible. At the end of the day, we all want the same thing. I want the lights to stay on. I want my electricity to work. I want my cooler to work, my fridge. We want our critical infrastructure protected. I want my kids to be able to go to school. I want to be able to go pop gas. I want to be able to go, you know, to the movies. But here's the thing, and this is why I'm, I'm, I'm optimistic. Our community, the OT security community, we've solved harder problems. Power industry, we've solved harder problems. This is not a difficult problem to solve. We know how to do it. Sometimes it's not always convenient to do it the right way. Sometimes. Why do it right when you can do it twice? I can't tell you, unfortunately, how many times I've heard that statement in power utility over the years. Why do it right when you can do it twice? So we seem to make sure that we're at the table, make sure that our voices are heard. Because the bad guys, the nation states, the attackers, they're not waiting for us to figure it out. As soon as they see these things come online, they know that's probably the time that it's the most vulnerable. So do this now. So at the end of the day, that's it. My rant for today. But seriously, if, if you guys are, what are you seeing out there? Are you involved in new generation projects? Are you seeing cyber requirement security requirements? Are you seeing what I'm seeing? You know, if you're with Microsoft, if you're a tesl, if you're with any of these folks that are building these new data centers and new generation to go along with them, drop me a line, find me on LinkedIn. I'd love to have a conversation with somebody that's building one of these data centers. It's part of these teams. How are we doing this, right? How are we making sure that security is part of this and security by design? It's included in these conversations. That's all. Thanks, Charles.
[00:22:46] Thanks for joining us on Protect it all, where we explore the crossroads of IT and OT cybersecurity security.
[00:22:53] Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field. Until next time.
