Server down - Sunday, September 12th, 2004

[admin]

Good morning,

We have a server that is down currently. It has had some data corruption and we are in the process of trying to repair the hard drives. We have all of the data and there has been no loss of data since midnight, when the trouble first appeared. We have no ETA on when we will get the server back on line, but our technicians are working hard to keep the downtime to a minimum.

This effects all customers with IP addresses between:

69.44.16.73 and 69.44.16.130

If you have any further questions or concerns, please submit a trouble ticket.

https://www.hardhathosting.com/helpdesk/

The Hard Hat Guys

[admin]

Update...

We are still working on getting this server back online. We have had to get some help from our software vendor. We still have not ETA, but we are working hard to get your websites back online. Thank you for your patience.

The Hard Hat Guys

[admin]

Update...

OK, we finally have got the server back up and running. We now have to restore all of the VDS accounts on the server. We are using a backup and restore plan provided by our software vendor. If this does not work, we will have to re-create each account by hand and restore them each by hand. Either route, this will take some time. We will probably not have another update until later this evening.

We apologize for the time between updates, but we are focusing on the server and getting things back to normal for our customers.

[admin]

September 14th update...

Our hosting vendor is close to getting the server back online. However, we do not know exactly when this will happen. They have a whole team of developers working on it.

We understand that this is the 3rd day that this server has been offline. If you are unable to wait any longer, we will setup a new account for you on a new server and give you your backup files. Once we setup the new account, we will assist in anyway we can to get you back to where you were before the crash. You will need to re-install ValueApps, for example, MySQL, mod_php, etc. You will also need to setup all of your email/FTP users.

To request a new VDS account, please submit a trouble ticket via:
https://www.hardhathosting.com/helpdesk/index.php?department=Support

Thank you for your patience and understanding.

Sincerely,

The Hard Hat Guys

[admin]

Update...

The server is back online. 90% of the accounts are back to normal. The other 10% will have to have some ValueApps installed/un-installed. Others may have to be deleted and resetup with backup files. We are going thru every account 1 by 1 and checking for errors, etc. SharedSSL is also broken on the server and we are working to repair this. Their will be a periodic reboots of the server throughout the day. Please take a look at your website and report any errors, using our Helpdesk.

https://www.hardhathosting.com/helpdesk/index.php?department=Support

More updates will follow...

The Hard Hat Guys

[admin]

We are doing an update of the "locate" database on this server. It may be a bit slow for twenty minutes or so.

Thanks,
The Hard Hat Guys

[admin]

We are down to just a handfull of VDS accounts that are still having some problems. Please be sure to take a look at your account and make sure everything is working as before. For example, MySQL databases, ValueApps installed, etc.

In addition, the SharedSSL on the server is still broken. If this poses a problem for you, please submit a ticket and we will do whatever we can to help.

https://www.hardhathosting.com/helpdesk/index.php?department=Support

Thanks again, to you, our customers, for your patience and understanding during this server failure. We very much appreciate it. :) We will be sending out an email to all customers on this server, when everything is back to normal.

Sincerely,
The Hard Hat Guys