For two days MilanIN server was not fully functional because of hardware failure. It was a Maxtor IDE disk that failed us. Well, such things have happend also before (usually one or two days after the warranty is over) and from my experience such thing happen when you don't expect such an ambush.
Anyway, the story started from uploading some files to the server and noticing I/O errors. A quick look into the log cleared the situation, The filesystem was mounted read-only and a backup of all-the-important-stuff was done.
Afterwards the hosters support has changed the disks and tried to do a copy of its content using, as I asked a LiveCD, but unfortunately, the server did not boot from CD with 2 HDD connected. So we ended up with a virgin Fedora and a root partition of the old disk. This means that all the custom stuff had to be recompiled, reinstalled and reconfigured. Mostly that was related to our e-mail system based on postfix mta and modified dbmail servers (not to mention outdated software). And that is why it took so long for the server to be back. Luckily, the visible part (which is the web site and database) were backed up fully and the recovery was smooth.
The hard thing for me in such circumstances is that I always have to "design" all the interconnections from scratch because I don't remember how I did it before. Every system installation for me tends to be a peace of creativity or a kind of a challenge. At the end the art looks great but I don't remember what's inside. That is sad, and I am trying to make notes on what I am doing, but this time the notes where on the same server and I did not backup my home directory.
P.S. The strangest thing is that while writing this post I did not say a word about what I think of Fedora, yum and Chineese hardware :)
Keywords: backup, milanin.com, server, web platform, linux
RSS