Hell story


My own personal hell story was moving database from one server to another, but we were recycling some of the hardware (disks and memory), so it was a unload-to-tape, reload-from-tape job.

The servers were in Port Hedland (a rat-infested dusty 110-degrees-plus 98% humidity hell hole... Hi to anyone in Port Hedland ... ). I was not physically present - don't you love it when IT companies think "remote login" is always the best way...Anyway, because of the dust issues, I unloaded three copies of the database to three tapes, and then did a verification read of all them. An on-site non-techie took care of unloading/reloading tapes for me.

He chucks the disk drives and memory from the old server plus the three tapes into a truck and drive 20 km's across town (where the new server is). Then I spend an hour on the phone trying to explain blind to him how to plug all this stuff in to the new server.

He turns it on... smoke starts billowing out of the box. Emergency shutdown (ie, rip out power cord). He's plugged some of the memory in wrong so its munched one of the boards. So (over the phone again) we talk through removing all the memory from just that board but leaving the other memory in.

Finally, the machine does in fact boot nicely. I log in as root and mount the new disks and start restoring from tape.

30mins in... first tape dies with a read error. About half the database restored. "No problem" I think... we load the second tape.

Another 30 mins, second tapes dies with a read error. Everything restored except SYSTEM tablespace. "Luckily we've got that third tape..." I think.

Third tape goes in... 'ufsrestore' command just hangs... Phone rings. On-site techie says "I can see tape spewing out of the Exabyte drive...."

So all tapes used, and no SYSTEM tablespace... As they say in the classics,

"Thats when you discover that adrenalin is brown and sticky".

I'm thinking about career moves etc, when the phone rings again. Its my wife.

"How's thing going at work?" she asks.

"Don't ask!..." I say and give her the run-down on what's happened..."Anyway, how's things at home?"

There's a long pause... followed by her bursting into tears....

"My pet budgerigar got out of the cage and has flown away....(sob) (sob)... he's in the tree in the back yard...(sob) (sob) can you please come home and try catch him..."

Not really what I needed at that point in time...

So I tell the techie I'm off to get a coffee... I jump in the car and scoot home to console the wife..."Where is the little bird?"

"That tree there..." says my wife as she points to our 100 year old citradora, around 150ft high. With the binoculars I can just see the little yellow budgie about 149.5ft up...

wife - "Can you climb up and get him?"

me (mentally) - "Are you out of your frickin mind ?!?!?!"
me (verbally) - "Of course dear, fetch me the ladder..."

So I start climbing this stupid tree, knowing full well that I'll never get this stupid bird, and that even if I did, I would wring its stupid little neck....30 mins later, I'm teetering at about 80ft and the branches won't support my weight anymore. So I climb back down, covered in sweat and scratches, but the effort seems to have appeased my wife...

So I spend another 10 mins crapping on about how "its better now that the the little bird is free" etc etc, knowing full well the neighbour's cat will have him for dinner within a matter of hours.

I scoot back to work...its amazing how teetering on the brink at 80ft gives you some clarity. I have a sneaking suspicion that the SYSTEM tablespace was on one of the disks that was NOT recycled from the old server. I get the techie to drive back to the old server, get all the remaining disks.

Another hour to get them all plugged in correctly (remember the good old days of SCSI terminators....) and voila! I find the SYSTEM tablespace on the old disks.

So at the end of all this drama, I naturally send my status email to management:

"Server moved, no errors encountered"