Critical Infrastructure Security
,
Governance & Risk Management
,
Operational Technology (OT)
EPA Drills Water Utilities to Disconnect
It is 4:45 a.m. in the fictional town of Riverbend and the internet has just gone down. A tier one national cellular and telecommunications provider is suffering a service outage as a result of a cyberattack by Chinese military hackers, cutting off connectivity to millions of consumers and thousands of commercial and government organizations.
See Also: How Cyberattacks Can Turn Battery Farms Into Grid Blackouts
For the next three days, one of those organizations, the Riverbend Public Utility, will have to figure out how to keep its 120,000 customers’ water supply flowing and safe, without the ubiquitous connectivity they, and the rest of us, take for granted.
Cellular and landline phones don’t work, and neither do SMS-based messaging or alerts systems. There’s no internet access, so no cloud-based services like email, document sharing or timesheet apps. No telemetry from or SCADA connections to Riverbend Public Utility’s handful of remote sites.
That was the scenario for this week’s U.S. Environmental Protection Agency’s National Cyber Drill, held via Zoom call on Wednesday. As the federal agency regulating the nation’s water and wastewater systems, EPA oversees cybersecurity in the sector. The scenario was the latest such national exercise the agency has staged for the more than 50,000 water utilities, each serving communities ranging from many millions to a few hundred customers.
The communications outage in the scenario is based on intelligence reporting about the Chinese cyberthreat group dubbed Salt Typhoon, said the official leading the drill, EPA Senior Cybersecurity Specialist Brandon Carter. “Unfortunately, that part is not made up,” he said.
Participants on the drill Zoom call included emergency response planners, IT and OT security specialists and state and local partners, as well as utility managers from more than 200 water and wastewater utilities across the country.
The drill aimed to prepare water utilities for “severe cyberattacks,” and “mirror sophisticated real-world adversarial tactics,” by simulating “a complete loss of third-party communications,” in “a worst-case scenario,” EPA spokesperson Jacob Murphy told ISMG in an email.
Carter and his colleagues walked the participants through the various stages of the fictional outage, including discussion of who would declare the situation a “significant operational incident,” and whose responsibility it was to open an “investigative record” to “preserve enough information” for “post incident activities.”
Issues such as alternate internal and external communications channels, alongside changing staffing patterns and how sustainable they were, also came up for discussion.
“Locations would need 24-hour personnel present,” said one participant, “For some locations multiple operators would need to be there at the same time.”
“We switched from 8 to 12 hour shifts,” said another.
Several participants said that if water treatment had to be done manually, there might be a tradeoff, at least initially, between maintaining water quality and maintaining water pressure.
“A decision that’s there is, are we going to be trying to fully treat [our water], or are we just going to keep the system full, so we can fight fires and maintain pressure,” meaning water not treated in the usual way might need to be released into the system.
Given such dilemmas, and the other burdens involved, it’s perhaps not surprising that so few utilities took part. Only a literal handful of utilities appear to have gone all in for the exercise.
The EPA in advance of the exercise encouraged utilities to conduct in a live action drill, either prior to or during the two-hour Zoom call Wednesday, in which they would attempt a real life transition to manual operations.
Carter asked participants on the Zoom to “let us know if you are currently taking systems offline today, operating manually, or if you have done so prior to today.”
Of the 390 participants on the call, 67 had opted for the manual operations module, rather than the tabletop exercise. Just one of those 67 participants responded as Carter requested in the chat, “we have done it.” Another four said they planned to complete the exercise “after today.” It’s unclear if others responded via direct message or through some other channel.
“Transitioning to true manual operations could be a pretty heavy ask” for a water utility, “depending on the size and complexity of the organization,” said Andy Krapf, director of Cybersecurity for Loudoun Water, and co-chair of the Cybersecurity Advisory Committee for the Water Information Sharing and Analysis Center. Loudoun Water serves more than 325,000 customers in densely suburban Northern Virginia.
A smaller, less technologically dependent utility might find it easier to operate manually without cellular communications, if they could send “a guy in a truck” to check on a small number of remote facilities, he said. One supplying a large metropolitan county might be more reliant on being able to do remote operations, explained Krapf, who did not take part in the exercise.
“I would go directly to manual operation and notify the town manager right away,” said one participant on the Zoom call. “Our system is small and easy for us to navigate.”
Another participant, a control system engineer working in Massachusetts, said most of the utilities he was familiar with had non-internet based communications with their remote sites. “Most of them have either private fiber or radio, and so this scenario wouldn’t be significantly impactful,” he said.
Water utilities are very different from each other, Krapf explained, depending on their customer base, treatment technology, regulatory environment, even their geography. “If you’ve seen one water utility, then you’ve seen one water utility,” he said.
Krapf also said it was important to distinguish between true manual operation and local operation. A communications outage like that in the EPA’s drill would mean going to local operation – remote facilities could no longer report or be operated remotely, but the local automation that kept them working should continue to operate, even if it lost connectivity.
“It depends on the failure mode, how the systems have been architected over the years,” he said. A remote-pumping site should continue to operate safely with periodic checks he said, “Tank levels go up, tank levels go down, pumps turn on, pumps turn off, based on demand. That’s not manual operation when the local automation is managing it.”
“True manual operation is when the automation dies and I’ve got to have an operator there 24/7 watching an instrument and manually turning a pump on or off when they need to.”