GitLabGitLab 2017 made one lesson impossible to ignore.
A destructive action hit production, and the team discovered several assumed backup paths were broken or unusable when they needed them.
6h
Trigger
Wrong target
A cleanup action intended for a replica hit the primary database.
Gap exposed
Backup assumptions failed
Redundancy on paper did not equal recoverability in reality.
Lasting lesson
Restore tests are mandatory
A green backup job is not proof of recovery.
This is the industry’s clearest reminder that untested backup paths create false confidence.