Downtime later today


Advanced search

Message boards : News : Downtime later today

Author Message
Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4422
Credit: 2,094,806
RAC: 0
Message 10942 - Posted: 9 May 2011, 6:27:40 UTC
Last modified: 9 May 2011, 6:30:47 UTC

The usual Monday maintenance will be extended throughout Tuesday as well.
During the downtime the power wires in the webserver will be upgraded and server software updates will be run.

Connectivity and work availability will be intermittent during this time.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4422
Credit: 2,094,806
RAC: 0
Message 10948 - Posted: 10 May 2011, 18:50:31 UTC - in response to Message 10942.
Last modified: 13 May 2011, 21:07:31 UTC

Status update:
Due to full system backups taking much longer than expected the maintenance window will be extended to Friday this week. Better safe than sorry.

[10th 22:15 UTC] System/user backup completed. Starting installation of new backup storage.
[11th 06:10 UTC] Consistency check complete on new backup storage, creating 4TiB ext4 on LVM2
[11th 06:50 UTC] ext4 created, starting 1st backup pass (out of 2) of BURP data
[13th 14.00 UTC] starting 2nd backup pass (out of 2)...
[13th 14.08 UTC] backup complete, system update starts, web goes offline
[13th 18.00 UTC] System software update completed. Web back online, some services unavailable
[13th 19.00 UTC] BOINC services available temporarily to take care of queued work
[13th 20.00 UTC] System goes down for hardware upgrades
[13th 21.00 UTC] System is back up, restarting services, network is degraded. BOINC services available temporarily to take care of queued work

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4422
Credit: 2,094,806
RAC: 0
Message 10949 - Posted: 13 May 2011, 21:13:03 UTC
Last modified: 14 May 2011, 9:26:08 UTC

Status update:
Currently the server is not fully back to normal operation. The maintenance window will have to be extended until (and including) Monday.
The main remaining issues are related to networking routes, scheduled tasks and BURP statistics, session encoding and last but not least: checking that everything works.
[13th 23.58 UTC] Webserver crashes for unknown reasons
[14th 08.00 UTC] Webserver is restarted, debugging begins
[14th 09.23 UTC] Issue involving infinite loop in Apache webserver resolved (could be the cause of the crash last night). All system configuration files have been merged and system reboot tested.


Post to thread

Message boards : News : Downtime later today