Server connectivity issues 2012-05-13


Advanced search

Message boards : Server backend and mirrors : Server connectivity issues 2012-05-13

Author Message
Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11339 - Posted: 17 May 2012, 22:52:09 UTC
Last modified: 17 May 2012, 22:59:01 UTC

The primary internet connection is down (again!) since around 18:00 UTC Sunday evening. It has taken the IPv6 connection down with it so most of the stuff that relies on that is either not working or really slow (development site, the news feeds, some stats and images).

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11341 - Posted: 18 May 2012, 12:20:04 UTC
Last modified: 18 May 2012, 12:20:54 UTC

No immediate solution could be found for the issue. Currently we're looking at two different scenarios. Either it is a hardware thing with the router that the ISP has on-site or it is a problem with the physical connection. As noted in an earlier post there has been a lot of digging and comm-tech-upgrades going on here recently so it is likely that something somewhere has gone wrong.

A new router is being shipped right now (should be here Monday or Tuesday) and also a technician is being booked to stop by next week to inspect the cabling in the streets and in the distribution network.

So far all in-house cables and equipment except their router checks out fine.

If your BOINC client says that it is unable to upload you should try restarting it a few times (old clients have this issue). Eventually it should pick up the backup network connection and upload (although very very slowly).

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11344 - Posted: 19 May 2012, 11:02:33 UTC
Last modified: 22 May 2012, 15:22:00 UTC

I've just talked to the provider for the backup connection, the connection which is currently heavily overloaded due to the main connection being down. We've reached an agreement on temporarily upgrading the connection and re-routing everything through that. The upgrade should kick in in less than one hour from now.

[Edit:] Or maybe that is "less than one work-hour from now" - in which case that means Monday.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11352 - Posted: 22 May 2012, 15:18:03 UTC

The provider of the secondary link had made an error and accidentally also sent us a new router while at the same time upgrading the network speed. Unfortunately the network speed upgrade is tied to the new router hardware ID, so it doesn't apply to the hardware which is already here... *sigh*. I called them and they did schedule a speed upgrade for the existing hardware as well now, but they could provide no info about when the settings would be applied by their tech department - "probably tomorrow, maybe".

Seems like we'll have to keep going for a few more days in backup mode before either the primary link or the secondary link can be upgraded. Tough luck.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11356 - Posted: 23 May 2012, 15:57:01 UTC

The new router for the primary connection arrived this morning. I just installed it and updated the routing info to include it. Let's see if it stays online.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11359 - Posted: 26 May 2012, 11:10:40 UTC

Although the new hardware is in place and both connections are now capable of running the entire load from the site things are still not working perfectly yet. The primary connection seems to have reverted to its old good mood with the new hardware. The secondary connection, however, crashed yesterday and things are generally a bit strange with it.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11367 - Posted: 29 May 2012, 6:05:01 UTC

The primary connection is down again. Someone dug into a fibre cable during roadworks outside a nearby network distribution center.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 11374 - Posted: 30 May 2012, 5:45:30 UTC - in response to Message 11367.

Both connections are back up.


Post to thread

Message boards : Server backend and mirrors : Server connectivity issues 2012-05-13