Why Our Backup Power Failed When the Lights Went Out — The Real Problem Wasn't the UPS

The Tuesday afternoon I picked up the wrong phone

September 2022. A Tuesday. I'm an office administrator — I manage purchasing for a 400-person company across 3 locations. Around 2:40 p.m. my cell phone rings, and it's our facilities lead. He doesn't say hello. He says: "Main building's out. Data room's down. Both telecom rooms are down."

Then he says the part that stuck: "The 9355 is fine. I just put a meter on it. It's fine. It's fine."

I didn't process that last part until about 8 p.m. that night.

The surface story is simple enough. We had an Eaton UPS 9355 protecting the data room. We had two Eaton UPS 9130 rack units protecting the telecom closets. When the utility feed dropped, none of it did what it was supposed to. Or so I assumed, for the first ninety minutes, while the sales team was standing around without CRM access and the warehouse crew was using their phones as flashlights.

I called the integrator. I said something like "the UPS failed under load" — because that's what it looked like. Replace it, upsize it, add a flywheel, whatever. I was already mentally writing a PO.

The diagnosis that made me sit down

The service tech came out the next morning. He put a load bank on the 9355, ran a battery string test, pulled the log. Battery health: 94%. Inverter output wave: clean. Runtime at half load: matches spec. He looked at me and said: "There's nothing wrong with this unit."

Then he walked the circuit with our electrician. That's when it started to make sense. Three things had happened in that 45-second window, and none of them were the UPS's fault.

1. The HVAC starters were riding on the UPS output

This is the one I'm still annoyed about. Back in 2019, our mechanical contractor "reorganized" the data room distribution panel for maintenance access. In doing so, he moved the three computer-room AC compressor starters onto the UPS output side. Not the redundant utility side. The UPS side.

Here's the problem: a 60 kVA Eaton 9355 has real surge capacity, but three compressor starts stacked in a 30-second window is a lot of inrush current for any UPS that size. It's not a failure of the unit — it's the unit doing exactly what it should when you ask it to feed a load it was never sized for. The inverter went into current-limit, the static bypass tried to catch the load, and there was no utility power on the bypass to catch it with.

If you've ever had an HVAC unit trip a breaker in a building that felt "small," this is the same math at a different scale. Startup current is typically 4–6x running current on a compressor. Nobody had ever added up the numbers.

2. The annual battery report was decorative

I'm going to be blunt about this because I got it wrong too. The annual "battery health check" we were getting from the maintenance vendor was a static voltage reading plus a visual inspection. It does not tell you what a battery string does under load. It tells you the batteries are not currently dead.

We had a terminal on one string with what I would later learn is called a "corrosion bloom" — the kind that hides under the plastic boot and only shows up when you cut power and run the string. The battery charger clips and terminal connectors were never on anyone's checklist. I asked the tech why, and he said something I've repeated to my own team since: "People assume clean means connected. It doesn't."

IEEE 1188 (the standard for maintenance of vented lead-acid batteries) calls for a load test at intervals, not just a float voltage read. We weren't doing that. We were doing the cheapest possible version of maintenance and charging the same invoice.

3. The generator started, ran for 45 seconds, and died

This one is on me. Our 250 kW diesel generator is supposed to pick up the whole facility load once it stabilizes. It fired on the ATS signal, ran for 45 seconds, and then went into a low-fuel-pressure shutdown. The transfer never completed.

Why? The generator's automatic test cycle ran every month and reported "normal" every month — but that cycle only checks start, run, transfer, and return. It does not tell you whether the fuel pump can hold pressure at load. The pump had been degrading for months. Nobody had run a load test on the unit with a real load on it in over a year.

If you've ever Googled "how to check fuel pump" at 11 p.m. after a power event, you and I have something in common now. What I learned: on a standby diesel, the check isn't visual. It's a pressure reading at the pump outlet under load, compared against the manufacturer's spec. Takes ten minutes with a gauge. We had never done it.

What that 4-hour outage actually cost

I want to be specific here because "downtime is expensive" is the kind of line that slides off the page without convincing anyone.

• Emergency generator rental and fuel: $4,200 for the mobile unit we brought in at 4 a.m.
• IT incident response and overtime: $2,100
• Rescheduled shipments: about $3,400 in expedited freight we ate to keep two accounts
• Payroll recovered for the day: roughly $2,100, since 400 people were effectively idle for four hours

That's $11,800 in one afternoon. And it wasn't a first. We'd had two shorter events in the previous 18 months — 45 minutes and just over an hour — that we'd written off as "one of those things." Add it up and the power continuity issue had cost us around $16,000 in three years before we took it seriously.

Here's the part I keep coming back to. Nobody lied to me. Every vendor we worked with sent reports, and the reports were accurate for what they covered. The problem was that none of us were looking at the system — we were all looking at our own piece of it. The UPS sold me a UPS. The generator guys sold me a generator. The electrician billed me for the work order he got. And I approved all of it without once asking the one question that would have caught all three problems: what actually happens to the whole building on a real outage?

Why I stopped trusting the spec sheet

To be fair, the spec numbers weren't wrong. 60 kVA is 60 kVA. 250 kW is 250 kW. Every product did what its datasheet said it would do.

The problem is that datasheets describe the box, and what I bought needs to survive contact with an entire building that keeps changing underneath it. A remodel here. A new AC unit there. A vendor who's "reorganizing a panel for access." A pump that's been slowly wearing out since the last cycle test.

In my opinion, this is the real reason backup power is so frustrating for people in my role. It's not that the equipment is unreliable. It's that we're managing a system through a series of individual transactions, and every transaction is optimized for its own line item. The UPS doesn't know about the HVAC. The generator doesn't know about the battery charger clips. The maintenance vendor doesn't know that the electrical panel was moved.

I was the same way for the first two years. I'd get a quote, compare it to the last quote, and approve it. When the annual maintenance report came back with all green boxes, I'd sign the invoice and file it. What are the odds, I told myself, that anything actually fails?

The odds caught up with us in September.

The 12-point checklist I built after the third write-off

I'm not going to pretend I'm an electrical engineer. I'm not. What I did was sit down with our facilities lead and the service tech, and build a one-page pre-approval checklist that any of us can run before we buy, rewire, or "reorganize" anything power-related. It's not perfect. It's already been revised twice.

The twelve points, in the order I actually use them:

1. Real connected load. Not nameplate, connected. Measured with a meter, documented, and updated whenever equipment changes.
2. Startup surge of every motor on the UPS bus. Compressors, pumps, anything with a motor. Pull the inrush current spec from the manufacturer's sheet and add up the worst-case overlap scenario.
3. What's actually on the UPS output vs. utility. A photo of the panel schedule, signed by the electrician who did the last modification.
4. Last full battery load test. Date, result, who witnessed it. If the answer is "last year's report," that's not a load test.
5. Battery terminal inspection. Including charger clips and every crimped connector. Cleaned and re-torqued on schedule.
6. Generator load test date. A load test, not a no-load run cycle.
7. Fuel pump pressure at load. Gauge reading, spec, sign-off. This is the one I never skip now.
8. Fuel level and quality. Age of fuel, polishing schedule, water test result.
9. ATS transfer test result. Full building transfer, not simulated.
10. Runtime calculation at current load. Not at rated load. At the load we actually have.
11. Maintenance contract scope. Does it actually include the things above, or does it include a visual inspection and a report?
12. Single point of failure review. Anything that, if it fails, takes the whole thing down. Update every time we change the layout.

Is this overkill for a 400-person company? Maybe. But it takes about an hour to run through, and it's the cheapest line item in the entire power budget. A $600 load test that catches a $4,000 failure has paid for itself six times over.

What I actually believe now

The UPS was never the problem in that September outage. Neither was the generator, or the batteries, or the electrician who moved the HVAC starter. The problem was that we treated backup power as a set of equipment purchases instead of a system we were responsible for keeping coherent.

I still buy Eaton equipment — for what it's worth, the 9355 did exactly what it was supposed to do, and our replacement plan is built around the same platform. But I don't buy it the way I used to. The quote is the last thing I look at now. The first thing I look at is the checklist.

Prevention is not a slogan. It's the hour you spend before the incident that you'd otherwise spend four hours cleaning up after. That's the whole lesson. It took us three outages and about $16,000 to learn it, and I'd rather you didn't have to.

If you take one thing from this: ask one question before you approve any power equipment PO. "What does a real outage look like across the whole building?" If nobody can answer it — including the person asking — that's the problem, not the UPS.