Unplanned Email Service Interruption

Unplanned Email Service Interruption, Wednesday 9/12

Posted approximately 2007-09-17, 4:00 PM

This message is directed to the approximately 250 users who may have lost mail received between Tuesday, September 11th at 11:00 AM and Wednesday, September 12th at 12:10 PM.

--------------------------------------------------------------------------------

Today we have new information available about messages you may have lost. But first, a brief history of what's happened since the mail system failure on Wednesday, September 12th at 12:10 PM:

• On the evening of September 12th, your mail was restored using our backup from Tuesday, September 11th at 11:00 AM. If you were not collecting mail using Eudora or Outlook, you may have lost messages sent to you between Tuesday at 11:00 AM and Wednesday at 12:10 PM.
• On Thursday, September 13th, we resent campus broadcasts to the most commonly used mailing lists in case you lost any of those messages.
• The entire community was notified about the loss and encouraged to resend important messages.

Today we are making a tool available to help you identify who sent you messages between Tuesday, September 11th at 11:00 AM and Wednesday the 12th at 12:10 PM. We created this tool by compiling our server log records in a format that allows you to view messages from that period. Although the log is limited, it can show you the sender address for messages which may have been lost. Hopefully, this information will better inform you and will offer you the option of contacting senders where necessary.

If you wish to view your records from this period, you must log in to the new Log Viewing Tool with your Haverford College email user name and password. If you are unable to activate this link while reading your email, enter the following into your web browser:
http://cgi.haverford.edu/acc/mail-configurator/list/list

Things to Note About Viewing Your Log:

Depending on the volume of mail you receive, viewing these records may be confusing. If you are subscribed to many mailing lists, you will likely see unusual sender addresses. Although the log may show that a message was delivered to your inbox, because of list servers and distribution lists, it is possible that the sender did not explicitly address the message to you. To shorten the length of the log, sender email addresses will be listed only once. A counter indicates if multiple messages were received from the same sender.

If you have questions or need assistance reviewing these logs, please use the Contact Us link included with the Log Viewing Tool.

Below, we have included a copy of the original notice sent to your attention on Wednesday. For a complete summary of events, see our announcements web page.

Again we apologize for the loss caused by this service interruption.

Posted approximately 2007-09-13, 3:30 PM

As a result of the mail system problem on Wednesday, September 12th, all messages sent to select distribution lists on the college mailing list server between Tuesday, September 11th at 11:00 AM and Wednesday, September 12th at 12:10 PM have been resent. These mailing lists include: HC07, HC08, HC09, HC10, HC11, hc-allstudents, faculty, and the staff distribution list. See the notice to the community about resent messages.

Posted approximately 2007-09-12,   7:30PM

The college email system was restored to normal service at 7:00PM, Wednesday, September 12, 2007. The system had crashed during the day on the 12th at approximately 12:10 PM.

All messages sent and received between September 12th at 12:10 PM and 7:00PM were queued and delivered.

The crash damaged approximately 250 mailboxes on the mail server. Users with corrupted inboxes were notified by email that their inboxes were restored using our system backup from Tuesday, September 11th at 11:00 AM. Those users whose mailboxes were not affected were also informed about the loss impacting other users.

View list of impacted mailboxes (restricted for campus users):   By Mailbox   |   By Type

Beginning around 12:10 PM, the mail system crashed and is preventing users from sending and receiving mail. Most webmail and squirrelmail users are seeing this problem as error messages when attempting to login. Users with pop mail clients like Eudora are experiencing this as an inability to collect new mail. Though it may appear that Eudora is sending mail, pop mail users are also unable to send mail; messages sent by Eudora users are being queued and will be delivered once the mail system is available.

Before we can make the mail system available again, it is required that we run a maintenance procedure on the data volumes. Because of their size, maintenance on the volumes may take some time. We will notify the community when the mail system is available. We apologize for any inconvenience this may have caused.

If you have questions about this interruption, please contact your computing support liaisons:

Students, faculty, and academic staff please contact Academic Computing:
Telephone: 610-896-1480

Administrative staff please contact Administrative Computing:
Telephone: 610-896-1044