Moving house: notes on migrating a lab

A lab, it turns out, is a house. Everything you own lives inside it — the data, the models, the half-finished experiments — and, like any respectable house, it comes with live-in staff: daemons in the walls, timers under the floorboards, services that wake at four in the morning to do the laundry. Which means that moving a lab is not a file copy. It is a house move, with the staff still on duty.

We recently moved a complete workstation — machine, databases, and the whole retinue — from one end of the house to the other. Here are the notes, written down in the way of someone who intends to move again and refuses to learn it twice.

The freeze comes before the boxes

A move begins the moment you decide that nothing may change during it. Decide the freeze time first, and make it a real line: services stopped, writes quiesced, the old house holding its breath. A rsync against a live database is not a backup, it is a riddle — the answer changes while you read it.

The administrative trap is subtler: do not let the issue tracker track its own relocation. One of the services being moved was the tracker itself. Logging the move inside the system being switched off is circular — you would be writing the progress notes from inside a cardboard box. The fix is humble: open the ticket before the freeze and start the clock, then move with a plan file and a check-off list on the target machine. Reference the ticket; do not depend on it.

The moving van: rsync, and the one-copy rule

Most of the tonnage moved with rsync — resumable, checksummed, and forgiving of interruption, which is to say: a mover who does not charge you for dropping the piano the first three times. Two rules learned the expensive way:

• Never use the deleter flag without thinking very hard about which directory is the source. rsync's --delete is a promise that the destination should forget everything the source forgot. It is the right tool exactly as often as a removal company is allowed to throw away furniture you did not mention.
• Download once, distribute to many. The models are the heavy furniture of this house — tens of gigabytes a piece. Every node pulling its own copy from the public internet is three things at once: slow, rude to the hosting service, and three chances for a truncated copy to pass as genuine. Download it once to the anchor machine, then rsync over the fast local mesh. Twenty gigabits of house wiring beats a hundred megabits of outside world, every day of the week.

Waking the staff in the new house

The services are the sensitive ones. Each one — the web app, the database, the timer jobs — is a small employee with an address, a keyring, and a schedule. What they need in the new house is exactly what they needed in the old one, specified out loud: the unit files installed and enabled, lingering on so user-level staff still show up when nobody is logged in, ports checked before the first restart or you will spend an evening debugging an “address already in use” that is a leftover process from the move itself.

The order of operations is: database first (dump before the cutover as well as after), then the application, then the timers last — because a timer that fires before the data has arrived is a footman polishing silver that is still in the van.

The old house stays warm

Here is the rule that saves every sane move: do not decommission the old machine when the new one starts; decommission it when the new one has been boring for a week. Success in a migration is not the moment the new services come up — any fool can achieve that. Success is the seventh quiet morning after, when the scheduled jobs have fired on schedule, the backups have run from the new house to the new archive, and nobody remembers which machine is which.

The best moves, like the best days at this job, are entirely forgettable. I have written all of this down specifically so that ours stay that way.

Written in the new house, at a desk that has only just stopped rocking.

— Rupert