CISA Alert – Threat Actors Aggressively Target U.S. Water and Wastewater Facilities

The Cybersecurity and Infrastructure Agency put out an alert this past week that highlights ongoing malicious cyber activity targeting the information technology (IT) and operational technology (OT) networks, systems, and devices of the U.S. Water and Wastewater Systems (WWS) Sector facilities¹. This alert covers these sorts of threats in detail, and provides useful guidance and resource recommendations so you can build up your defenses.

As an example of the current threat environment, one dangerous attack on a municipal water facility happened earlier in 2021. We covered this attack in detail in an earlier blog. Cyberattackers were reported to have targeted and accessed the information technology system responsible for controlling a water treatment facility located in the southeastern United States. The threat actor’s goal was reported to have been to make the town’s water supply undrinkable, or perhaps even worse. The potential harm to that municipalitie’s citizens might have occurred as the cyberattackers were manipulating the sodium hydroxide levels in the water supply. The cyberattackers had adjusted the sodium hydroxide level to 100 times more than the normal levels. Sodium hydroxide, also known as lye, is the primary ingredient in drain cleaners. Used carefully by water treatment plant staff, it helps to control the acidity of the water and assists in processes to remove heavy metals. Misused and ingested in sufficient quantities, it becomes dangerous and potentially poisonous. 

In this attack, the attackers choose to upset treatment and conveyance processes around the dosage of sodium hydroxide by opening and closing valves, overriding alarms or disabling pumps or other equipment. They could also have decided to deface the utility’s website or compromise the email system. It is a short step from this activity to steal customers’ personal data or credit card information from the utility’s billing system. Finally, they can install malicious programs like ransomware, which can disable business enterprise or process control operations. Attacks like these can compromise the ability of water and wastewater utilities to provide clean and safe water to customers, erode customer confidence, and result in financial and legal liability.

How many potential water and wastewater facility targets are out there in the U.S.? 

There are approximately 153,000 public drinking water systems and 16,000 publicly owned wastewater treatment systems. Over 80 percent of the U.S. receives their water from these drinking water systems.  Approximately 75 percent of the U.S. has its sewage treated by these wastewater systems.

The Water and Wastewater Systems Sector is vulnerable to a variety of attacks. This can include contamination with deadly agents, physical attacks (such as the release of toxic gaseous chemicals), and cyberattacks. The result of any variety of attack could limit or shut down access to public facilities and resources, illnesses, or perhaps even fatalities.

WWS facilities may be vulnerable to the following common tactics, techniques, and procedures (TTPs) used by threat actors to compromise IT and OT networks, systems, and devices. Tactics represent the goals of the attacker, techniques show how the attacker will execute activity towards a specific tactic, and procedures represent the lower level, very highly detailed description of execution in the context of a technique.

Spear Phishing personnel to deliver malicious payloads, including ransomware [T1566].

• Spear Phishing is one of the most prevalent techniques used for initial access to IT networks. Personnel and their potential lack of cyber awareness are a vulnerability within an organization. Personnel may open malicious attachments or links to execute malicious payloads contained in emails from threat actors that have successfully bypassed email filtering controls.
• When organizations integrate IT with OT systems, attackers can gain access—either purposefully or inadvertently—to OT assets after the IT network has been compromised through spear phishing and other techniques.
• Exploitation of internet-connected services and applications that enable remote access to WWS networks [T1210]. For example, threat actors can exploit a Remote Desktop Protocol (RDP) that is insecurely connected to the internet to infect a network with ransomware. If the RDP is used for process control equipment, the attacker could also compromise WWS operations. Note: the increased use of remote operations due to the COVID-19 pandemic has likely increased the prevalence of weaknesses associated with remote access.

Exploitation of unsupported or outdated operating systems and software.

• Threat actors likely seek to take advantage of perceived weaknesses among organizations that either do not have—or choose not to prioritize—resources for IT/OT infrastructure modernization. WWS facilities tend to allocate resources to physical infrastructure in need of replacement or repair (e.g., pipes) rather than IT/OT infrastructure.
• The fact that WWS facilities are inconsistently resourced municipal systems—not all of which have the resources to employ consistently high cybersecurity standards—may contribute to the use of unsupported or outdated operating systems and software.

Exploitation of control system devices with vulnerable firmware versions.

• WWS systems commonly use outdated control system devices or firmware versions, which expose WWS networks to publicly accessible and remotely executable vulnerabilities. Successful compromise of these devices may lead to loss of system control, denial of service, or loss of sensitive data [T0827].

Mitigations

The FBI, CISA, EPA, and NSA recommend WWS facilities—including DoD water treatment facilities in the United States and abroad—use a risk-informed analysis to determine the applicability of a range of technical and non-technical mitigations to prevent, detect, and respond to cyber threats.

Personnel responsible for monitoring WWS should check for the following suspicious activities and indicators, which may be indicative of threat actor activity:

• Inability of WWS facility personnel to access SCADA system controls at any time, either entirely or in part;
• Unfamiliar data windows or system alerts appearing on SCADA system controls and facility data screens that could indicate a ransomware attack;
• Detection by SCADA system controls, or by water treatment personnel, of abnormal operating parameters—such as unusually high chemical addition rates—used in the safe and proper treatment of drinking water;
• Access of SCADA systems by unauthorized individuals or groups, e.g., former employees and current employees not authorized/assigned to operate SCADA systems and controls.
• Access of SCADA systems at unusual times, which may indicate that a legitimate user’s credentials have been compromised
• Unexplained SCADA system restarts.
• Unchanging parameter values that normally fluctuate.

Remote Access Mitigations

The increased use of remote operations due to the COVID-19 pandemic increases the necessity for asset owner-operators to assess the risk associated with enhanced remote access to ensure it falls within acceptable levels. 

• Require multi-factor authentication for all remote access to the OT network, including from the IT network and external networks.
• Utilize blocklisting and allowlisting to limit remote access to users with a verified business and/or operational need.
• Ensure that all remote access technologies have logging enabled and regularly audit these logs to identify instances of unauthorized access.
• Utilize manual start and stop features in place of always activated unattended access to reduce the time remote access services are running.
• Audit networks for systems using remote access services.
  • Close unneeded network ports associated with remote access services (e.g., RDP – Transmission Control Protocol [TCP] Port 3389).
• When configuring access control for a host, utilize custom settings to limit the access a remote party can attempt to acquire.

Network Mitigations

• Implement and ensure robust network segmentation between IT and OT networks to limit the ability of malicious cyber actors to pivot to the OT network after compromising the IT network.
  • Implement demilitarized zones (DMZs), firewalls, jump servers, and one-way communication diodes to prevent unregulated communication between the IT and OT networks.
• Develop/update network maps to ensure a full accounting of all equipment that is connected to the network.
  • Remove any equipment from networks that is not required to conduct operations to reduce the attack surface malicious actors can exploit.  

Planning and Operational Mitigations

• Ensure the organization’s emergency response plan considers the full range of potential impacts that cyberattacks pose to operations, including loss or manipulation of view, loss or manipulation of control, and threats to safety.
  • The plan should also consider third parties with legitimate need for OT network access, including engineers and vendors.
  • Review, test, and update the emergency response plan on an annual basis to ensure accuracy.
• Exercise the ability to fail over to alternate control systems, including manual operation while assuming degraded electronic communications.
• Allow employees to gain decision-making experience via tabletop exercises that incorporate loss of visibility and control scenarios. Utilize resources such as the Environment Protection Agency’s (EPA) Cybersecurity Incident Action Checklist as well as the Ransomware Response Checklist on p. 11 of the CISA-Multi-State Information Sharing and Analysis Center (MS-ISAC) Joint Ransomware Guide.

Safety System Mitigations

• Install independent cyber-physical safety systems. These are systems that physically prevent dangerous conditions from occurring if the control system is compromised by a threat actor.
  • Examples of cyber-physical safety system controls include:
    • Size of the chemical feed pump
    • Gearing on valves
    • Pressure switches, etc.
  • These types of controls benefit WWS Sector facilities—especially smaller facilities with limited cybersecurity capability—because they enable facility staff to assess systems from a worst-case scenario and determine protective solutions. Enabling cyber-physical safety systems allows operators to take physical steps to limit the damage, for example, by preventing cyber actors, who have gained control of a sodium hydroxide pump, from raising the pH to dangerous levels.

Additional Mitigations

• Foster an organizational culture of cyber readiness. See the CISA Cyber Essentials along with the items listed in the Resources section below for guidance.  
• Update software, including operating systems, applications, and firmware on IT network assets. Use a risk-based assessment strategy to determine which OT network assets and zones should participate in the patch management program. Consider using a centralized patch management system.
• Set antivirus/antimalware programs to conduct regular scans of IT network assets using up-to-date signatures. Use a risk-based asset inventory strategy to determine how OT network assets are identified and evaluated for the presence of malware.  
• Implement regular data backup procedures on both the IT and OT networks.
  • Regularly test backups.
  • Ensure backups are not connected to the network to prevent the potential spread of ransomware to the backups.
• When possible, enable OT device authentication, utilize the encrypted version of OT protocols, and encrypt all wireless communications to ensure the confidentiality and authenticity of process control data in transit.
• Employ user account management to:
  • Remove, disable, or rename any default system accounts wherever possible.
  • Implement account lockout policies to reduce risk from brute-force attacks.
  • Monitor the creation of administrator-level accounts by third-party vendors with robust and privileged account management policies and procedures.
  • Implement a user account policy that includes set durations for deactivation and removal of accounts after employees leave the organization or after accounts reach a defined period of inactivity.
• Implement data execution prevention controls, such as application allowlisting and software restriction policies that prevent programs from executing from common ransomware locations, such as temporary folders supporting popular internet browsers.
• Train users through awareness and simulations to recognize and report phishing and social engineering attempts. Identify and suspend access of users exhibiting unusual activity.

Rewards for Justice Reporting

The U.S. Department of State’s Rewards for Justice (RFJ) program offers a reward of up to $10 million for reports of foreign government malicious activity against U.S. critical infrastructure. See the RFJ website for more information and how to report information securely.

DNS is an Important Part of Your Defense

DNS in the kill chain in the great majority of attacks. Sooner or later malware must reach back to command & control and DNS is used as a covert communication channel for this purpose. In some cases, attackers use  malicious domains and IP addresses  that could already have a reputation and may be  identified by threat intelligence. In many other cases the behavior of DNS queries, in context, can provide the critical clues you need to identify and stop the attack.

DNS is ubiquitous and frequently used as an attacker technique for malware infiltration and data exfiltration. Standard security controls and technologies such as next-gen firewalls, IPS, and gateways generally do not inspect DNS for detecting malicious communications. Much of the time they are unable to prevent specific attacks such as DNS data exfiltration . Most importantly, they are not able to detect the subtleties of newly created malicious addresses and domains.

Infoblox BloxOne Threat Defense enables security operations teams to leverage DNS to get the highest visibility into malicious activity so that the cyberattackers can be detected and shut down early in the kill chain of events.

Endnote

¹ Alert AA21-287A