Results 1 to 2 of 2

Thread: PDGS Was Down

  1. #1

    PDGS Was Down

    For approx 26hrs PDGS and related websites were down due to a catastrophic failure at the datacenter that houses our servers. This is one of the largest (if not the largest) datacenters in the US (and a very large company).

    It was basically due to the error of a low level tech at the datacenter (not an employee of mine, but of the datacenter). They were supposed to be adding a hard drive to the server and accidentally formatted/wiped the WRONG drive. It took getting the CEO of the company directly involved, calling in top recovery experts and my staying up most of the 48 surrounding hours (2.5 hrs sleep).

    We are back and with 0 data loss (keeping fingers crossed!).

    Going to sleep now.


  2. #2
    (Woke up, will probably nap again in a bit)

    FYI even if the entire server was destroyed in a fire/flood/earthquake/etc we have regular off-site backups of everything at a location hundreds of miles away. This occurs at least every 3 days (soon to change to every day). So worst case scenario ever, we could lose a few days of posts and be down for a day or two switching to another server. We actually have redundant hardware on the server so even if a drive fails the server keeps running fine. This was a freakish human error (and yes they are in trouuuubbbblllle).


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts