Field notes — oversight and safety-critical systems

The silence after the current

What happens when the systems keeping people alive lose human oversight — first through neglect, and increasingly through an automated layer that can fail in its own, quieter ways.

Time since last human intervention
00:00:00:00
grid nominal

Modern civilization does not rest on solid rock. It floats on a thin, continuous hum. Every glass of clean tap water, every refrigerated meal, every hospital ventilator, and every digital exchange depends on an unbroken circuit of electric current and, increasingly, on software making judgment calls about that circuit. We treat infrastructure like a background physical law, as reliable as gravity, forgetting that unlike gravity, our built environment requires constant, exhausting human labor and correct machine judgment just to resist collapse.

The classic version of this failure is a blackout: the grid drops, and the clock starts on a predictable physical cascade. A newer version is quieter. It doesn't require the power to go out at all. It only requires the systems making life-or-death decisions, increasingly automated ones, to be wrong, unmonitored, or unaccountable at the wrong moment.

The first 72 hours: the instant decay of the grid

Phase 0 · Hours 0–72

A widespread, extended power outage reveals just how tightly bound we are to real-time supply chains, and how quickly the loss of oversight becomes a loss of life.

Communications go dark

Cellular towers rely on battery banks that deplete within four to twenty-four hours. Emergency dispatch, remote patient monitoring, and grid telemetry all depend on that same channel staying open.

Loss of water pressure

Municipal systems rely on electric pumps to hold 40–80 PSI of line pressure. Within 24 hours, pressure drops and untreated groundwater is drawn back through microscopic cracks into drinking water.

Cold chain breakdown

Commercial refrigeration holds cold air for roughly 48 hours if sealed. By day three, food and vaccine storage becomes incubator ground for bacteria and spoiled medicine.

The medical cliff

Hospital diesel generators rarely last more than three to seven days under heavy load. Once fuel runs out, oxygen concentrators stop and intensive care units collapse into stillness.

How AI-related failures cost human lives today

Not hypothetical — already documented

Long before any large-scale collapse scenario, AI systems are already embedded in decisions where a wrong or unmonitored output has a direct path to a death. These are the mechanisms researchers, regulators, and incident investigators point to most often.

Automation complacency in safety-critical systems documented

When a system performs well most of the time, human operators gradually stop double-checking it — a well-studied pattern called automation complacency. It has been cited as a contributing factor in fatal crashes involving driver-assistance systems, where drivers over-trusted automated steering and braking in situations the system was not designed to handle. The failure isn't the model malfunctioning in some spectacular way; it's a human handing off a judgment call the system was never certified to make, at the exact moment a judgment call was needed.

AI-enabled cyberattacks on hospitals and utilities infrastructure

Ransomware groups increasingly use AI tooling to speed up target selection, phishing, and lateral movement inside a network. Hospitals hit by ransomware have had to divert ambulances, delay chemotherapy and surgery, and revert to paper records for weeks. Investigations into several hospital ransomware incidents have linked the resulting care delays to patient deaths. The attack itself doesn't need to touch a ventilator; disabling the scheduling and records systems around it is enough to put patients at risk.

Bias and error in life-critical algorithms healthcare and safety

Algorithms used to triage patients, flag sepsis risk, or allocate care have been found, in independent audits, to systematically under-flag risk for some patient groups because of biased or incomplete training data. A model that is 95% accurate in aggregate can still be reliably wrong for the population least equipped to catch the error. Because these tools are often deployed as a second opinion that clinicians come to trust, the error doesn't announce itself; it just quietly changes who gets seen first.

Autonomous and semi-autonomous weapons systems military

Militaries are increasingly using AI-assisted targeting to process reconnaissance data faster than human analysts can. Human rights investigators and some military ethicists have raised concerns that compressed decision timelines and algorithmic target scoring shift lethal judgment away from a human who can weigh context, toward a system optimized for speed and pattern matching. The concern isn't that these systems act entirely without a human in the loop today; it's that the human's role in that loop is shrinking, and the incentive to shrink it further is strong.

AI-amplified misinformation leading to real-world harm information

Generative tools have made it cheap to produce convincing false content at scale, including fabricated health guidance, emergency alerts, and incitement material. Researchers tracking misinformation have tied specific viral falsehoods, from fake evacuation orders during disasters to fabricated medical claims, to documented injuries and deaths among people who acted on them. Unlike a mechanical failure, this pathway doesn't fail the infrastructure; it fails the judgment of the people relying on it, at scale and quickly.

These are documented categories of harm from AI systems as currently deployed, not a claim that any of them is common relative to other causes of death, and not a claim about where the technology is inevitably headed. Each is also an active area of regulation, auditing, and engineering mitigation — the point is that the mechanism exists and has already produced real outcomes, not that it is unaddressed or unaddressable.

The pivot: from crisis to permanent entropy

A power outage is merely a pause. Permanent abandonment of oversight, whether of a physical plant or of an algorithm making decisions inside it, is a shift in the underlying dynamics. Complex systems are delicate balance acts held together by active human attention. Without that attention, the built and automated world begins to consume itself, or to keep running while quietly making worse and worse calls.

Phase 01
Days to weeks

Loss of power and active pumping

→ Flooding of subterranean spaces

Phase 02
Months to years

Seal failure, rust, freeze-thaw cycles

→ Mechanical seizure and spalling

Phase 03
Decades and beyond

Silt accumulation, structural decay

→ Dam breaches and concrete collapse

The death of water treatment infrastructure months

Water treatment facilities are complex bio-chemical processors. When pumps stop, suspended solids settle in flocculation tanks, forming dense, anaerobic sludge layers. Without active aeration, algae blooms choked by stagnant nutrient loads die off, releasing toxins into the basins.

Groundwater seeps continuously into subterranean pumping galleries. Ordinarily, automated sump pumps keep these drywell spaces clear. Without electricity, water rises, submerging motor windings, control panels, and transformers. Within months, the plant's mechanical guts are drowned and ruined by corrosion.

Chemical refineries and industrial stasis months

Complex industrial machinery relies on dynamic equilibrium: heat balanced by cooling, pressure balanced by venting, friction reduced by forced-oil lubrication. Over months, elastomeric seals dry out and crack, lubricating oils oxidize into thick varnishes, and atmospheric moisture combines with residual sulfur to form acids that eat through steel vessel walls from the inside out.

The thermal crisis of spent fuel pools weeks–years

At commercial nuclear facilities, reactors shut down automatically during grid failures, but spent fuel assemblies in cooling pools remain thermally active for years. Without circulation, decay heat brings the water to a boil, exposing zirconium cladding to air where it can oxidize rapidly and release localized radioactive isotopes. Heat production declines over months as short-lived isotopes decay, leaving cold, hazardous hot spots in structural limbo.

The long decay: concrete, steel, and water

Phase 02–03 · Years to decades

Concrete cancer and freeze-thaw destruction

Micro-cracks form in bridges and facades; rainwater carries dissolved salts inside, and winter freezing expands that water by roughly 9%, widening the fissures. Once moisture reaches the rebar, rust occupies up to six times the volume of the original steel, and the resulting pressure causes spalling — chunks of concrete fracturing away and exposing the structure's skeleton.

Silt, debris, and dam breaches

Without humans clearing intake grates and managing spillway gates, debris jams the overflow routes and water rises behind the dam until it overtops. For earthen dams, overtopping is fatal: rushing water erodes the downstream face until the barrier fails catastrophically, sending wall-sized floods down dry river valleys.

The difference between a blackout and entropy

A power outage is an operational crisis. Human abandonment, whether of a physical plant or of an algorithm quietly making the wrong call, is a transition. During a blackout, infrastructure remains intact, waiting for human hands to flip a switch. When oversight of automated decision-making lapses instead, the system doesn't wait patiently; it keeps acting, on incomplete data or outdated assumptions, without announcing that anything has changed.

Civilization is not a static monument. It is a high-energy process that must be aggressively maintained against the baseline tendency of complex systems, mechanical or algorithmic, toward failure once nobody is watching closely. Whether the watching stops because the grid goes dark or because a dashboard gets trusted a little too much, the underlying lesson is the same: the hum only continues because someone keeps tending it.