Why
#I help run the servers for a gaming community. It’s an old game engine with a pile of add-ons, configuration and content stacked on top.
Changes used to be made by hand on the servers, so they slowly drifted apart and nobody knew which differences were intentional. Some mistakes only showed up once players ran into them, and there was no clean way back after a bad change.
What I changed
#All of it now lives in one repository. Staging is the base, and every other server only lists what’s different about it.
CI checks every change for the kinds of mistakes that used to reach players. Each of those checks exists because something broke once. Old issues that can’t be fixed right away are written down, and adding a new exception needs a reason next to it.
Production doesn’t follow the main branch directly. It follows a release manifest, so promoting a release or rolling one back is just a small commit. Backups get restored on purpose from time to time, to prove they actually work.
Where it ended up
#Nobody edits the servers by hand anymore, and every difference between them can be reviewed. Problems that used to reach players now fail in CI instead. The checks that turned out most useful all came from real incidents, and adding a new one while fixing the bug is cheap and keeps paying off.