RE: GroupStudy Server crash

Supported

I have just donated to Paul at Paypal

Login to paypal |
Choose the register "Send Money" |
Enter the e-mail pborghese_at_groupstudy.com and the amount/currency |
click continue
confirm payment
Then you will see the reassuring "Daniel Kutchin, you have sent $XXX,00 USD
to Artisan Networking Inc."

-
Daniel

-----Original Message-----
From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf Of
Elliott Reyes
Sent: Freitag, 13. Mai 2011 00:35
To: Haroon
Cc: ccieagent; Paul Borghese; ccielab_at_groupstudy.com
Subject: Re: GroupStudy Server crash

We should take up a collection to help paul out

On Thu, May 12, 2011 at 3:32 PM, Haroon <itguy.pro_at_gmail.com> wrote:

> Paul,
>
> Thats some ordeal... sorry to hear!
>
> Your backup strategy is great.... I also use rsync to backup over
> 110GB to remote servers twice a day but my stuff (linux server, mysql
> DBs, etc.) is not on vm.
>
> ESXi 3.5 comes with very little tools for recovery.
>
> Have you tried enabling SSH on esxi??? That may give you more control
> and go under the hood as far as vmware is concerned...
>
> As far as your datacenters, what datacenter do you use in Dallas?
> Softlayer?
> Instead of copying 200GB to your house (residential cable/dsl?), I can
> recommend a vendor that I use to backup data right in dallas
> datacenter using rsync.
>
> Please keep us posted.
>
> Thanks,
>
> Haroon
>
>
> On Thu, May 12, 2011 at 2:00 PM, ccieagent <ccieagent_at_verizon.net> wrote:
>
> > Paul,
> > Good to see you got it working again. I was beginning to wonder if
> > the pressure of our joining GroupStudy and the CCIE Flyer was going
> > to
> happen!
> > LOL
> > Sorry to hear about your ordeal. Talk to you soon.
> >
> > -----Original Message-----
> > From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf
> > Of Paul Borghese
> > Sent: Thursday, May 12, 2011 4:40 PM
> > To: ccielab_at_groupstudy.com
> > Subject: GroupStudy Server crash
> >
> > On Monday one of the GroupStudy servers in Atlanta had a
> > catastrophic
> disk
> > failure. This brought down the entire site and every mailing list.
The
> > good news is this list is impacted only minimally from the failure
> > and should (may?) be back to normal shortly. I apologize for any
> > inconvenience.
> >
> > Since this is a technical list I will go into the details for those
> > that are interested. Please comment if you have any suggestions.
> > On Monday I decided it had been too long since I backed up the
> > Atlanta server, which
> > is now running as a VM on a VMWare ESXi 3.5 server. At one time I had
> an
> > elaborate backup mechanism where every evening critical database
> > file
> were
> > copied to a directory then rsync ed to a linux server at my house.
> > But over the last two years we moved a number of times and the
> > remote server is still packed away in a storage unit.
> >
> > I decided since it was a VM I could simply scp the .vmdk file to my
> house,
> > thus creating a perfect backup. The problem was the VM contained
> > multiple 200 GB files. I noticed part of the storage size was caused
> > because a snapshot had been taken on the GroupStudy server. To reduce
> > the size of the backup, I decided to delete the snapshot. This
> > action should have merged the snapshot with the primary. But
> > instead it removed all .vmdk disks and left me with a .vmdk file
> > that was simply 26k
> (instead
> > of 200+ GB)!! Oops. The GroupStudy disk was totally destroyed.
> >
> > If the VMWare server was running on another OS, I could have simply
> > gone
> in
> > with an undelete program and tried to recover the files. But VM
> > ESXi
> runs
> > a
> > propriety locked-down OS with very few tools. I hate to say this,
> > but my last backup of the server was made two years ago, just before I
> > moved (yea I know but hey it is a hobby). I called a number of
> people
> > asking for advice. I would like to thank in particular Darby Weaver
> > who found a VMWare guy who was quite knowledgeable. But even he was
stumped.
> >
> > I decided to take it slow and not do anything that may prevent a
> recovery.
> > In the evenings (I still have a day job :-) ) I started reading
> everything
> > I could find about VM ESXi. I have now read more VMWare
> > knowledge base articles then I care to admit. I am thinking about
> seeing
> > what certifications VMWare offers and simply taking the test.
> >
> > VMWare does offer support, for $300/call, which I really did not
> > want to spend. But it was obvious I was not getting anywhere trying to
figure it
> > out myself. So I reluctantly plunked down my credit card. In my
> > research I found VM did offer an undelete program for the ESX platform.
> I
> > was hopeful my support request would, at minimum, give me access to
> > the undelete program for ESXi. Or maybe some internal use only
> > magic VMWare has in their back pocket. But in all honestly, they
> > were not much help.
> > The VMWare tech support is not that great. Cisco TAC will escalate
your
> > problem until you are fixed. VMWare support seems to be for people
that
> > don t know how to read manuals. In all fairness, it could be that my
> > $300 support call is not going to the same people that handle high
paying
> > corporate outages. VMWare suggested I call a data recovery service.
> The
> > data recovery service said this happens all the time and they could
> > recover the data for $3-$5k. I simply do not want to spend that kind
of
> > money.
> >
> > The VMWare server contains two 1 GB disks, a primary and extra disk.
> > The original GroupStudy VM was running on the primary. I used the
> > two-year-old backup disks to create a new GroupStudy VM on the extra
> disk,
> > thus preserving the primary to the best of my ability. Of course
> > the backups were created before migrating to VMWare, so none of the
> > kernel drivers worked out of the box.
> >
> > After fixing the kernel and initrd boot files, the GroupStudy
> > website has been restored literally to the date of the Obama
> > Inauguration. So
> welcome
> > back to January 2009 (quick buy Apple Stock and gold!). With regards
> > to the CCIE List, this actually has less impact then you would think.
> The
> > actual mailing list is running off a Linux server in Dallas, and has
> > been unaffected. What we lost was two years of archives. But I may
> > be able
> to
> > get them back as the Dallas server has copies of the archives in a
> > MySQL DB.
> > If they are complete, I can simply write a Perl script to extract
> > the archives to a format the website can use.
> >
> > Bu there is other lists that are affected more and frankly being a
> > techie
> I
> > hate to give up. We know the data is most likely still on the disk. We
> > just need to find it. I feel with the disk, a hex editor, and some
> > voodoo I could recover the data I needed. Frankly I only need one of
the
> > backup files that was created daily, not the entire disk. The problem
> is
> > the Primary disk is where the VMWare OS is located, so I can t simply
> > remove it. And it currently resides in a data center in Atlanta, a 10
> > hour drive from my house!
> >
> > So now I am trying to MacGyver my way to the disk. The extra disk
> > contains about 400 GB of free space. It turns out VMWare does offer
> > disk dump and gzip on the ESXi platform. I am disk dumping the
> > entire primary hard drive to the extra drive, using gzip to compress
> > the data. I am praying for a much better then 2:1 compression
> > ratio! If that works I
> will
> > download the dd file and restore to another 1 TB hard drive, thus
> > creating a copy of the primary drive. Then I need to figure out the
> > VMWare partition tables and vmdk disk formats. If the 400GB of free
> > space is not enough, I am considering mounting an Amazon EC2 NFS
> > server
> on
> > the VMWare file system and trying again. I also called the VMWare
> support
> > engineer (that poor guy) and asked him to send me any documentation
> > he
> can
> > find about the VMFS and VMDK file structures. I also found an open
> source
> > VMFS driver (http://code.google.com/p/vmfs/) that may be of use.
> >
> > So if you have any suggestions, please send them to me! No matter
> > how
> bad
> > it got, I kept on thinking at least I am not Sony!
> >
> > Paul Borghese
> >
> >
> > Blogs and organic groups at http://www.ccie.net
> >
> > ____________________________________________________________________
> > ___ Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
> >
> >
> > Blogs and organic groups at http://www.ccie.net
> >
> > ____________________________________________________________________
> > ___ Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
>
>
> Blogs and organic groups at http://www.ccie.net
>
> ______________________________________________________________________
> _ Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html

Blogs and organic groups at http://www.ccie.net
Received on Fri May 13 2011 - 01:12:03 ART