The Generator Ran. The Servers Died Anyway.

At 4:52 p.m. on a Thursday in August, a squall line took out the utility feed to a 22-person marine services company on the bay. The generator started. The operations manager saw the shop lights come back through her office window and went back to what she was doing.

In the server closet, two hypervisors had already been on battery for fourteen seconds, and they stayed there another six minutes — because the UPS never accepted the generator's output. When the string hit cutoff, both hosts lost power mid-write. The generator ran cleanly for four more hours, delivering stable power to a room of dead servers, while the UPS reported a low battery condition that was, technically, accurate.

The question the next morning was "why didn't the UPS switch over to the generator." That is the right question in the wrong frame. The UPS did not fail to switch. It evaluated the generator's output against its acceptance window, found it unacceptable, and did what it was designed to do — protect the load by staying on battery. That design assumes utility power returns before the battery runs out. Nobody told it the source across the transfer switch would never be acceptable at all.

Nobody writes the chain, because nobody owns the chain

Five links sit between the utility feed and a running virtual machine: the utility, the transfer switch, the generator, the UPS, and the shutdown agent that tells the hypervisor to stop gracefully. Each has good literature behind it, written by a different trade — battery vendors on batteries, the electrical trades and NFPA 110 on transfer switches, UPS-to-generator compatibility almost entirely at data-centre scale.

None of this is a knock on those trades. The electrician who tested your transfer switch tested it correctly, and the generator technician's monthly exercise catches real problems. The gap is that nobody is contracted to test the sequence end to end, and the three failure modes below appear only in the sequence.

Failure one: the UPS rejects the generator and stays quiet about it

A double-conversion UPS continuously evaluates its input against an acceptance window for voltage, frequency, and — the one that catches people — harmonic distortion. Outside that window it runs from battery instead. That behaviour is correct — feeding a distorted source into the rectifier is worse than not. The mechanism is a feedback loop, which is why a generator that looks fine can still be unusable:

• The UPS itself creates the distortion. Its rectifier draws non-linear current. Older SCR-based designs are the worst offenders, with fifth and seventh harmonics dominating; newer IGBT front ends switching at higher frequencies produce considerably less.
• The generator amplifies it. A generator is not a stiff source. Its subtransient reactance — typically around 11 percent on a standard alternator — turns that harmonic current into harmonic voltage. Higher reactance, more distortion from the same current.
• The UPS then rejects what it helped create. Distortion crosses the threshold, the UPS goes to battery, the harmonic-generating load comes off the generator, and distortion drops. The UPS sees a clean source, accepts it, and the cycle starts again — until the battery is flat.

Generator manufacturers address this with a sizing ratio: SCR-type non-linear load should generally not exceed about 66 percent of the generator's prime power rating. Small offices size generators around the HVAC and the lights, treat the UPS as a rounding error, and land outside that ratio without anyone running the number.

Failure two: the transfer gap outlives what the battery has left

An automatic transfer switch does not transfer instantly. It senses the loss, waits out a confirmation delay so it does not chase a momentary sag, signals the generator to start, waits for acceptable voltage and frequency, then throws. NFPA 110 defines a Type 10 system as one where the transfer switch load terminals are without acceptable power for no more than 10 seconds — covering the whole sequence, start through transfer.

Type 10 applies to legally required emergency systems: hospitals, life safety loads. An optional standby generator at a 20-person office is not held to it. Real gaps at that scale run considerably longer, longer still with a cold engine or a failed crank, and nobody has measured yours — measuring it means deliberately dropping the utility feed.

Any of those gaps is trivially covered by a healthy battery string. The problem is the word healthy. VRLA batteries — the sealed cells in essentially every rack and tower UPS — lose capacity continuously and without declaring a fault. IEEE 450 and IEEE 1188 share a replacement criterion: replace the battery when measured capacity falls below 80 percent of the manufacturer's rating. A string at 55 percent of rated capacity is well past that line, will not light a replace-battery indicator, and reports itself as fine.

Internal impedance is the early warning, and it behaves differently by chemistry: a 20 percent rise from baseline corresponds to 80 percent capacity in flooded lead-acid, while in VRLA that rise is closer to 50 percent at the same point. Which is why "battery OK" and "usable runtime" are two different claims, and why a four-year-old string rated for twelve minutes may have three. Heat compounds it — a Gulf Coast closet that loses cooling during an outage runs its batteries in the conditions that accelerate dry-out.

Failure three: the shutdown agent that was installed but never configured

Assume both problems above are solved: the UPS accepts the generator and the battery has runtime to spare. There is still a third way to lose the servers, and it is the most common of the three — nothing ever told the hypervisor to shut down.

For a VMware environment on APC hardware, shutdown requires a network management card in the UPS, PowerChute Network Shutdown installed and reachable, working credentials for vCenter or the host, an on-battery trigger threshold, and enough runtime allotted for the sequence to finish. Two things go wrong repeatedly:

• The agent shuts down the host, not the guests. The unlicensed version of PowerChute Network Shutdown supports standalone ESXi hosts not in an HA cluster, and it issues a host shutdown command rather than orchestrating individual VMs. Whether your VMs come down cleanly depends entirely on the host's Autostart configuration — enabled, stop action set to "Shut down" rather than power off, sensible delays, VMware Tools in every guest. Miss VMware Tools on one VM and that VM hard-stops while everything around it behaves.
• Clusters change the rules. VM Startup/Shutdown is deliberately disabled on hosts belonging to an HA cluster. If you built a two-host cluster and assumed the Autostart settings from years earlier were still doing something, they are not.

How do you load-test a UPS in a live 20-person office?

This is the honest obstacle. Everything above is verifiable, and verifying it means taking power away from people who are working.

• Do the arithmetic first. Read the measured load off the UPS in watts, not percent, and find the manufacturer's runtime chart for that model at that wattage. If nameplate runtime at your real load is already under three minutes, your problem is sizing and no test will fix it.
• Test the rack separately from the office. The failure modes that matter live in the server closet. Dropping only the rack — not the desks, not the phones — is a far smaller conversation, and it exercises everything except the transfer switch.
• Use the generator's monthly exercise as free instrumentation. Generators on an NFPA 110 style schedule run at least 30 minutes monthly, with the transfer switch electrically transferred to the alternate source and back. If the switch is exercised anyway, the UPS is seeing generator power anyway. Watch its event log during that window: if it goes to battery and stays there while the generator carries the building, you have found failure one without an unplanned second of downtime.

Why the UPS may refuse to run a runtime calibration

Calibration is the obvious way to learn true runtime, and people are surprised when it will not start. Schneider's guidance for its Easy-UPS line is that at least 30 percent load must be attached and must not fluctuate more than plus or minus 5 percent during the run. An underloaded UPS — a 3 kVA unit carrying two small hosts and a switch — sits below that threshold and declines. Three further constraints:

• The battery must be fully charged going in. Calibrating a partially charged string measures the partial charge, not the capacity.
• It is a real discharge. Calibration runs the battery down until the UPS drops the load — that is the measurement. Schneider recommends no more than one every six months, because deep discharges consume cycle life.
• The load is unprotected while it runs. You have no reserve for the duration, so a genuine utility event mid-test is an immediate hard outage.

Calibration is not a safer alternative to pulling the plug — it is the same risk in a different order, and it measures the battery only. It says nothing about the transfer switch, the generator, or the shutdown agent.

The annual test, and what counts as passing

Once a year, on a scheduled evening, with the generator serviced and the batteries fully charged, open the main breaker and watch. Five numbers, each with a line it clears or does not.

Measured runtime at real load versus nameplate. Record the load in watts and the time from transfer to cutoff, generator locked out. Pass: at least 80 percent of the manufacturer's published runtime at that load, mirroring the IEEE criterion. Fail: anything under — the string is at end of life whatever the panel says.

Transfer gap, in seconds. Time from loss of utility to the UPS reporting acceptable input. Pass: under 25 percent of measured runtime, leaving margin for a failed first start. Fail: a gap eating more than a quarter of what the battery has.

Whether the UPS accepted generator power at all. Read the event log, not the panel. Pass: one transition to battery, one back to line, nothing further. Fail: repeated on-battery and on-line cycling while the generator carries the load — the harmonic rejection loop, meaning that generator does not back your UPS in any meaningful way.

Whether the hypervisor shut down cleanly, unassisted. Let it run with nobody touching a keyboard. Pass: every guest logs a clean shutdown, the host powers off on its own, no VM logs an unexpected stop. Fail: any guest that hard-stopped, or any sequence still running when the battery cut off.

Whether everything came back without hands. Restore utility power and count what needed intervention. Pass: hosts power on, guests start in order, services answer without anyone logging in. Fail: any service that needed a person — because at 4:52 on a Thursday in August, that person may be an hour away with a squall line between them and the building.

Four of those five numbers cannot be obtained any other way. That is the argument for the test: not that something is probably broken, but that the alternative is finding out during the storm.