Gitlab has lost it's database and realised they have no backups
Posted by Stanislav Furman on February 9, 2017There was an interesting news just in the end of January 2017.
On January 31th, 2017 Gitlab accidentally deleted their production database (git repositories were not affected though).
What happened. For some reason, replicatation started lagging (PostgreSQL). One of the Gitlab employee some tried to fix the problem by playing with different settings but it did not help. Then, at some point, that employee decided to delete everything and rebuld the replica again. He (or she) tried to delete the folder with the replica data, but mixed up servers and removed the folder on the master (rm -rf on did db1.cluster.gitlab.com instead db2.cluster.gitlab.com).
It could have been not as bad but they realised they had no backups:
Continue reading