Holes


Advanced search

Message boards : Server backend and mirrors : Holes

Author Message
Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12235 - Posted: 7 Jan 2014, 16:45:30 UTC

Good news everyone! Tomorrow people will start drilling holes in the building that houses the BURP servers.

Normally having people drill holes in your buildings is a pretty bad idea but in this case the holes will be filled with fiber optical cables that will provide us with low-latency and high throughput internet with enough bandwidth to support BURP hosting for the foreseeable future.
Currently the three ISPs that deliver network for the platform that runs BURP are based on a set of old copper and coax cables. When requesting high speeds (like during the release of Big Buck Bunny 3D Sunflower) things tend to get a bit expensive and they do not provide the low latency that fiber offers. The new setup is higher speed, better latency, lower cost - win/win!
The connection upgrade was ordered last July but it is quite a cumbersome process to roll out fiber, so it will take a while still before the new connections can be activated. For the past few months the red sticks in the ground have been moving closer and closer and right now the splicing tent is about 150 meters away. Let's hope that it arrives at the hosting facility before winter freezes everything over.

No downtime is expected because of this unless someone accidentally trips a circuit breaker.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12344 - Posted: 8 Feb 2014, 13:29:07 UTC - in response to Message 12235.
Last modified: 8 Feb 2014, 13:48:44 UTC

The splicing tent is now just 20 meters away and massive rolls of cable are waiting all over the place.
For the next month or so downloads may seem slower than usual - this is part of the transition plan and hopefully it will not be too much of a problem. We're basically trying to avoid subscribing to too many different ISPs simultaneously by juggling the connections a bit while waiting for the new route to be ready.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12373 - Posted: 20 Feb 2014, 21:11:28 UTC
Last modified: 20 Feb 2014, 21:20:44 UTC

The cables are now inside the building but still a couple of rooms away from the server room. The ISP has been pretty quiet about what is happening so far, but at least there is steady visible progress. From what I can tell from the now empty rolls of cable, they've installed almost half a kilometer of fiber optic cables in the building so far; impressive!

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12391 - Posted: 28 Feb 2014, 7:00:48 UTC

An update:
Our very own fiber cable pair is now somewhere in the ceiling a few meters above the servers. Hardware delivery and final splicing at the termination point close to the BURP server cluster is scheduled for 12th of March and if everything goes well we should be able to carry out the first tests on the new networking route shortly after that.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12417 - Posted: 11 Mar 2014, 7:11:59 UTC

We've now gracefully expired two of the three old contracts on networking connectivity. Hopefully we can attach to the new uplink tomorrow, perform some tests and then turn it on either Thursday or Friday. The rest of today and most of tomorrow downloads will be slower than usual since everything will be running on just the backup link.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12419 - Posted: 12 Mar 2014, 15:39:45 UTC
Last modified: 12 Mar 2014, 15:45:34 UTC

Some tests have started with the new connection.
Initially the connection checked out just fine so the download server was switched to use it, but after a short while the termination box became unstable.
There's going to be some disconnected downloads while we work with the ISP to optimize everything and figure out the problems.

Currently they are working on installing new firmware and backing out a set of settings that were accidentally pushed to the box shortly after initialization. Simultaneously network diagnostics is being run at the BURP servers to figure out if one of the network cables or network interfaces in the servers there could be the cause of any instability. Since this involves physically adding and removing network cards from the servers this will cause some downtime.

Next update around 16.30 UTC

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12420 - Posted: 12 Mar 2014, 16:37:23 UTC - in response to Message 12419.
Last modified: 12 Mar 2014, 17:39:39 UTC

Network diagnostics have ruled out an issue at the BURP side of things, so now we're waiting for the ISP to finish looking at their side. Meanwhile the connection will occasionally be up as more test data is sent through.

Update [17.30 UTC]: We're closing up for today without finding the source of the instability. The hunt continues tomorrow.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4483
Credit: 2,094,806
RAC: 0
Message 12444 - Posted: 19 Mar 2014, 19:49:01 UTC - in response to Message 12420.

After the ISP updated the equipment with the latest settings the connection has been very stable. Most of the work relating to the installation is now finished and only some minor issues relating to how to handle power failures remain. Hopefully downloads should no longer be dreadfully slow.


Post to thread

Message boards : Server backend and mirrors : Holes