moving the datacenter

[icecream-guy]

Anyone have any experience with moving a data center?  out Fed wants to move one of our fairly large data centers, it's in an 80 year old building, they want to gut and refurbish the building.  Lessons learned?

[deanwebb]

Prepare for regular "cry tests" when you unplug stuff and wait to see if anyone cries about it. If nobody cries, then don't move that asset. You will not move everything. Taking a good look at what the visibility tool discovers in the datacenter will help with finding esoteric items from bygone days that need to not graduate up to the new DC.

Even worse is the data part of the datacenter. Those teams that manage data need to have a data visibility tool in place ($VENDOR alert, my company works with Alation as a partner in that space) to discover all the data and determine what's currently accessed and what can be sent off to long-term cold storage and forgotten.

[Otanx]

Do you have the budget to stand up the new place first and migrate? That is how we ended up doing it. We built out the new data center network, and some new hypervisors and storage. We connected it to the old facility with a temporary circuit, and started migrating systems from one to the other. Once we thought we were done we shut off the link to the old space, and did a scream test. Took a couple tries. Once done we excessed most of the gear at the network gear at the old site. It was pretty old anyway.

Issues we had:
- Servers not installed correctly: We had servers sitting on top of other servers. No rails to be found. This caused some issues with order of operations. I can't move the server on the bottom until we move the other servers. After the move those all got flagged for replacement.
- Drive failures: Make sure you have good backups. Drives are going to fail. If you have a lot of the same drives see if you can get the vendor to send some ahead of time for swaps. Otherwise you move a server, and the drive fails. You RMA it, and wait for the RAID rebuild before doing the second server in the HA pair. Or you accept risk and move the second server while the first one is still in a degraded state.
- Other failures: Drive failures are not the only thing that will fail. Power supplies, fans, full systems. Moving systems will break them. When possible virtualize, and move, or build on new hardware, and decom.
- Link speed: We built out new, and migrated. We found we still had 3 devices that only would do 100M, and didn't have a way to support them on the new gear that did 1/10G.
- Misc parts: Stock up on cables of different types and lengths, power cables, SFPs, cage nuts, screws, etc. You don't want to scrounge up cables because the server was installed 2U below the switches in the old place, but is now at the bottom of the rack. This also includes extra tools for mounting gear. If you don't know there is a tool for installing cage nuts so you don't rip your fingers apart. Get a bunch.

-Otanx

[deanwebb]

Power cables I recall as being in high demand if the new racks have a different power outlet shape from the old racks.

I've also seen the DC move as a time where folks asked if the system coming out of the old DC needed to go to the new one or to the cloud.