Egregoros

@sun Sadly I had to test in production a _lot_ in a previous job.

But our way of testing in production was doing a full release process for the change (which was gated by a condition), then deploying the change for a small percentage of users or requests, and evaluating metrics before going higher.

Together with realtime analysis tools for those metrics and tools for emergency rollbacks of the changes.

Database commands OTOH always had to be executed on a dev instance if applicable, otherwise on a temp database with synthetic data first. Before they hit production, a human needed to be convinced that the commands indeed are harmless (with outcome in the dev or temp environment as evidence).

Doesn't mean accidents don't happen. But none as big as this one.