Recovering corrupted InnoDB databases under pressure

Context

CryptoVerse had a production database that would not come back up. InnoDB corruption, an engine refusing to start, and a business that was entirely stopped until it did.

Incident work like this is a different discipline from building. The constraint is not elegance. It is that every action is potentially destructive, and you usually get one attempt.

The problem

Corrupted InnoDB tablespaces after a critical failure. The database would not start cleanly, which meant the normal tools for inspecting the damage were not available either.

Compounding it:

• Backups that were not as recent or as verified as everyone assumed. This is almost always part of the story, because a backup nobody has restored from is a hypothesis rather than a backup.
• A running version with known stability issues, which is the reason the situation happened rather than an incidental detail.
• Operations fully halted. Time pressure on decisions that should not be made under time pressure.

Approach

Work on copies, never the patient. The first action was preserving the damaged data files untouched and doing everything on copies. Recovery attempts are destructive, and you want as many attempts as you can get.

Recover in increasing order of damage. InnoDB offers forced recovery modes that trade consistency for the ability to start at all. The discipline is to start at the least invasive level that works, extract what you can, and only escalate when you must, because each level discards more.

Extract, then rebuild rather than repair. Once the engine could be brought up far enough to read, the goal shifted to getting data out into a clean instance rather than trying to make the damaged one healthy. A repaired corrupted tablespace is a thing you will never fully trust again, and you should not.

Verify against something other than row counts. Matching counts tell you very little about whether relationships survived. Referential integrity checks across related tables were the actual acceptance criterion.

Then the part that mattered more

Restoring operations was the urgent work. It was not the important work.

• Migrated to stable versions, removing the known-unstable engine version that put us there.
• Automated backups with a defined retention policy, so recency stopped being a matter of belief.
• Restore testing, because an untested backup is a guess. This is the step that gets skipped, and it is the only one that converts a backup policy into an actual recovery capability.
• Monitoring on the failure signals that preceded the corruption, rather than only on the symptom.

What I took from it

Everyone has backups. Far fewer have restores. The gap between those two words is where most data loss actually lives. If you have not restored from your backups this quarter, you do not know whether you can.

Version currency is a reliability feature. Running a version with known stability issues is a decision, even when it is made by not deciding. The cost arrives all at once.

Incident work should end with a change, not a relief. The recovery was memorable. The automated backups and restore tests are what meant it did not happen again, and that part took longer and got less attention.