SCADA Redundancy and High Availability: Hot Standby, RAID, 24/7 Operation

When a single-server deployment becomes a line-stop risk

A single SCADA server is a reasonable starting point for a pilot line, a laboratory, or a plant that can run for hours without a live HMI. It stops being reasonable once operators rely on that screen to keep production moving, once alarms are the early warning the shift actually uses, or once the same box also holds the history that quality and energy reports read from.

The failure that hurts is rarely a dramatic crash. More often a disk fills, a Windows update hangs, a power supply dies, or a network card drops and nobody notices until the next shift cannot open a trend. The question is not whether the server will fail; it is how long production can continue without it, and whether the last minutes of data are still there when it comes back.

• Continuous production with no idle window: a discrete manufacturing line, a process unit, a water-utility pumping group or a utilities plant that cannot wait for a rebuild
• The HMI is the operating position: start and stop, recipe download or alarm acknowledgement happens on that screen, not only on the local panel
• History lives on the same disk as the runtime: a disk failure wipes both the live picture and the evidence needed for quality or energy reports
• Communication drivers sit only on that host: PLC, OPC and gateway sessions die with the box, so even a spare PC cannot see the plant until channels are rebuilt
• 7×24 coverage with a thin night shift: the person on duty can acknowledge an alarm, but cannot restore a server image at 02:00

If two or more of those conditions are true, a single-server design is no longer a cost saving. It is an unstated decision that a server fault is an acceptable production stop.

What hot standby and clusters have to keep in step

Buying a second licence and parking a second PC next to the first is not hot standby. Failover only works if the standby already holds a usable copy of everything the operator needs in the first seconds after the switch. Four stores tend to be forgotten until the first real fail-over: the realtime database, the history database, the HMI project and the communication channels.

In our delivery work, redundant clusters and dual-machine hot standby are treated as an architecture choice, not as a checkbox on a quotation. The pair has to agree on who is active, how state is copied, and what the operator sees during the switch. Second-level screen refresh on the surviving node is only useful if the tags behind that screen are still being polled.

• Realtime database: current tag values, alarm state, interlocks and in-memory setpoints. If the standby is even a few seconds behind, operators act on a picture that is already wrong
• History database: trends, event logs and process curves. RAID1 on the disk is not the same as a replica on the second node; a history gap after failover is a quality and audit problem, not only an IT problem
• HMI project: screens, tag bindings, scripts, user rights and language resources. A runtime that comes up against yesterday's project file will show the wrong picture even if the databases are current
• Communication channels: PLC paths, OPC UA/DA sessions, gateway links and which NIC they bind to. Channels that exist only on the primary mean the standby boots into an empty plant

A practical test is to pull the primary while an operator is looking at a trend and acknowledging an alarm. If the standby cannot show the same tags, the same alarm list and the same recent history, the pair is not a redundant system yet.

RAID and the server it sits on

RAID1 on the database disks protects against a single disk dying. It does not protect against a failed motherboard, a hung operating system, a ransomware encryption of both mirrors, or a UPS that was never tested. Disk redundancy and application redundancy sit on different layers; mixing the two in a proposal is how plants end up with RAID and still stop the line.

The server that holds SCADA also needs a power path, a cooling path and a network path that match the availability the plant actually wants. Dual NICs are useful when they isolate the industrial network from the office network, not only when they are teamed for extra throughput. Database RAID1, dual-machine hot standby and a redundant cluster can share a room and still fail together if they share one switch, one UPS and one cabinet fan.

• RAID1 for the database volume: a disk fault should not take down the history store; a restore drill still has to prove that a backup can be read
• Dual power supplies and a UPS whose runtime has been measured, not assumed from the nameplate
• Dual NICs with a documented purpose: one face to the industrial network, one face to the office or historian network, rather than two cables into the same VLAN
• Physical separation of the pair: two machines in one rack is already stronger than one machine; two machines on two UPS circuits and two access switches is the arrangement that survives a single cabinet fault

Layer, redundancy measure, and what acceptance has to prove

A proposal that says "redundant" is not testable. The table below is written so that a third person, with no project history, can run a test and record a yes or a no. Fill the third column with the plant's own numbers — failover time, history gap, who is allowed to switch — rather than copying a vendor default.

Architecture-level practices we use on projects include redundant clusters, dual-machine hot standby, database RAID1, 7×24 operation and second-level screen refresh. Those measures are acceptance items, not a field availability percentage. Three-year retention into a history store is a separate data-layer decision and should be written next to the RAID and replica design, not instead of it.

Network isolation and access control — only as far as availability needs

High availability collapses if the office network can flood the industrial network, or if a single compromised account can stop both nodes. This is not an OT-security article; the points below are the ones that show up in failover tests and in factory audits.

On large projects we have put industrial and office networks on physically separate paths with gigabit fibre Layer-3 switching, dual-NIC isolation on the servers, RAID1, tiered authorization and high-risk file filtering. Permissions, roles and audit trails are configured against the plant's security requirements. Designs and documents can be supplied within the project scope and used in factory audits, acceptance and third-party assessment; conclusions rest on project evidence.

• Isolate the industrial network from the office network so a broadcast storm or a desktop malware event does not take both SCADA nodes with it
• Bind dual NICs by function: process traffic on one face, office or historian traffic on the other, with no casual bridging
• Put write rights, recipe download and engineering access behind named roles, and keep an audit trail of who changed what
• Filter high-risk file types on the SCADA hosts; a USB copy of an installer is a common way both nodes get the same unwanted software

If the plant already has an OT security programme, SCADA redundancy should sit inside it rather than beside it. The acceptance tests in the table above still apply: a well-isolated pair that cannot fail over is not yet a highly available system.