Mirror Status


Advanced search

Message boards : Server backend and mirrors : Mirror Status

Author Message
Andrew Admire
Send message
Joined: 23 Mar 05
Posts: 108
Credit: 94,266
RAC: 0
Message 10660 - Posted: 20 Dec 2010, 2:29:42 UTC

When will the mirror status page be accessible? I wanted to know if my mirror was still being used and if there were any mirror software updates. My hosting company has changed my disk limit to unlimited.

Thanks
____________
When life is to big to tackle, kick it in the shins.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4487
Credit: 2,094,806
RAC: 0
Message 10664 - Posted: 20 Dec 2010, 9:56:12 UTC

Current status: All mirrors are inactive because of an incompatibility with BURPv.3 - this is quite unfortunate and I'm very unhappy with the fact that it has taken me so long to fix the issues in this particular area.

A software update for the mirrors will be deployed towards the end of next month (January 2011), you will be informed by email and on the front page when this happens.
Next year we will be needing the mirrors more than ever. Stay tuned

Independence
Send message
Joined: 16 Mar 05
Posts: 18
Credit: 23,433
RAC: 0
Message 10833 - Posted: 8 Mar 2011, 8:46:58 UTC

Hi Janus,

I am ready for the mirror software update.

Do you have any time information for deploying from the mirror software?

Cheers

Indy
____________

Matrix World Search Team
Matrix Forum

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4487
Credit: 2,094,806
RAC: 0
Message 10836 - Posted: 8 Mar 2011, 22:11:17 UTC
Last modified: 8 Mar 2011, 22:12:40 UTC

Thanks for your interest in helping out by providing a mirror/cache. Unfortunately it is not ready to fly yet.
The original intent was for it to be ready for Sunflower. However, it turned out not to be necessary for that at all since the new bigger internet connections at the main server could provide the required bandwidth directly. This poses the interesting question of whether it is now overkill to design a mirror network given the fact that the bandwidth available to the project is bound to increase even further over the next years.

Currently I'm spending most of my development time on fixing bugs and preparing scenes for Sunflower so I don't even have a good time estimate for this yet. Sorry

Independence
Send message
Joined: 16 Mar 05
Posts: 18
Credit: 23,433
RAC: 0
Message 10865 - Posted: 23 Mar 2011, 6:07:41 UTC - in response to Message 10836.

Hi Janus,

thanks for your update.

I hope you can solve soon all bugs :-)

Cheers Indy
____________

Matrix World Search Team
Matrix Forum

Independence
Send message
Joined: 16 Mar 05
Posts: 18
Credit: 23,433
RAC: 0
Message 11267 - Posted: 2 Apr 2012, 7:06:18 UTC

Hi Janus,

any update for us?

Thanks :-)

Indy
____________

Matrix World Search Team
Matrix Forum

Profile noderaser
Project donor
Avatar
Send message
Joined: 28 Mar 06
Posts: 512
Credit: 1,553,018
RAC: 79
Message 11268 - Posted: 2 Apr 2012, 21:16:52 UTC

I still have plenty of disk space and bandwidth to contribute.
____________

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4487
Credit: 2,094,806
RAC: 0
Message 11269 - Posted: 3 Apr 2012, 19:24:30 UTC
Last modified: 3 Apr 2012, 19:48:23 UTC

Yes, it is about time for an update on this topic.

Back when I posted on the topic in March I was slowly getting convinced that a mirror network may be rather overkill for what we were using it for.
Originally when the project was in its infancy everything was run from a very slow internet connection (think ISDN speeds) and the mirrors were essential in pushing anything bigger than a megabyte out from the central server. The mirrors kept us going. Since then, through donations and other means, the network speed on the central server has increased greatly. Additionally new techniques to handle overload situations means that when the server is overloaded it will simply just send stuff slower rather than swamping completely.
Furthermore the current connections can be scaled up on-demand. So in case we get some really demanding stuff it is possible to simply pay a bit more for a few months and get a faster throughput and then scale down again later.
The old mirror network was a real pain to maintain - new hosts joining and old hosts blacking out or leaving etc. was all handled semi-manually. Being free of this kind of daily maintenance stuff really is worth the extra cost on the central connection for me on a personal level.

So, all of the above points towards the idea of dropping mirror networks entirely. I am not sure that this is what is going to happen, though. There is one use-case where a mirror network is extremely useful: Flash crowds. A flash crowd is where suddenly (typically without any form of warning) EVERYONE wants to visit your site and download something. Although the central server has much better connectivity now than earlier it would still be swamped by a flash crowd and it would be difficult to scale up quickly enough to respond to it properly. A mirror network would be able to handle it much better because the load spreads to multiple servers all over the world. People have been extremely happy about by providing mirrors - in fact so many mirrors that there has never been much load on each individual mirror so far.

The truly golden question: How do you keep a mirror network on stand-by in a way that:
1) It shouldn't cost mirror owners anything extra compared to what they usually pay
2) It is ready to kick in when needed
3) Requires no (or very little) maintenance on the central server

Half of the solution, when talking about flash crowds at least, is Bittorrent. It costs people next to nothing to set up a Bittorrent seed on their computers/servers. It works really well when many people download the same thing. Also the only required central component is a tracker which is very low volume and almost fully automatic. Bitttorrent is the perfect solution for flash crowds.
This leaves just one group of people: The people who, for some reason or another, cannot or prefer not to use Bittorrent (maybe it is blocked/throttled by their ISP, maybe they like to watch the streaming version etc.). These people would need an HTTP download - the mirror network.

There is one problem, though. In theory the mirrors cannot be trusted. With BOINC this was not much of a problem since BOINC validates input files that are downloaded through the mirror network and re-downloads them from a different mirror if they do not match with what the central server tells it. Real people, however, do not do automatic MD5 tests on stuff they download from the web, they also don't wait and retry. They need the real deal right away.

And this leads us to the conclusion about the current state of affairs:
On a day-to-day basis things are working really well without mirrors, but sometimes for really popular content they could come in handy. Some of that load could be offloaded easily to Bittorrent, but the rest of it is really difficult to handle without creating a maintenance hell.

This also brings up the question: Should BURP be both a renderfarm and a distribution platform? Or, should we leave it up to the community to find hosting solutions for particularly popular content on a case-by-case basis?

Comments and good ideas are welcome!


Post to thread

Message boards : Server backend and mirrors : Mirror Status