Yes, it is about time for an update on this topic.
Back when I posted on the topic in March I was slowly getting convinced that a mirror network may be rather overkill for what we were using it for.
Originally when the project was in its infancy everything was run from a very slow internet connection (think ISDN speeds) and the mirrors were essential in pushing anything bigger than a megabyte out from the central server. The mirrors kept us going. Since then, through donations and other means, the network speed on the central server has increased greatly. Additionally new techniques to handle overload situations means that when the server is overloaded it will simply just send stuff slower rather than swamping completely.
Furthermore the current connections can be scaled up on-demand. So in case we get some really demanding stuff it is possible to simply pay a bit more for a few months and get a faster throughput and then scale down again later.
The old mirror network was a real pain to maintain - new hosts joining and old hosts blacking out or leaving etc. was all handled semi-manually. Being free of this kind of daily maintenance stuff really is worth the extra cost on the central connection for me on a personal level.
So, all of the above points towards the idea of dropping mirror networks entirely. I am not sure that this is what is going to happen, though. There is one use-case where a mirror network is extremely useful: Flash crowds. A flash crowd is where suddenly (typically without any form of warning) EVERYONE wants to visit your site and download something. Although the central server has much better connectivity now than earlier it would still be swamped by a flash crowd and it would be difficult to scale up quickly enough to respond to it properly. A mirror network would be able to handle it much better because the load spreads to multiple servers all over the world. People have been extremely happy about by providing mirrors - in fact so many mirrors that there has never been much load on each individual mirror so far.
The truly golden question: How do you keep a mirror network on stand-by in a way that:
1) It shouldn't cost mirror owners anything extra compared to what they usually pay
2) It is ready to kick in when needed
3) Requires no (or very little) maintenance on the central server
Half of the solution, when talking about flash crowds at least, is Bittorrent. It costs people next to nothing to set up a Bittorrent seed on their computers/servers. It works really well when many people download the same thing. Also the only required central component is a tracker which is very low volume and almost fully automatic. Bitttorrent is the perfect solution for flash crowds.
This leaves just one group of people: The people who, for some reason or another, cannot or prefer not to use Bittorrent (maybe it is blocked/throttled by their ISP, maybe they like to watch the streaming version etc.). These people would need an HTTP download - the mirror network.
There is one problem, though. In theory the mirrors cannot be trusted. With BOINC this was not much of a problem since BOINC validates input files that are downloaded through the mirror network and re-downloads them from a different mirror if they do not match with what the central server tells it. Real people, however, do not do automatic MD5 tests on stuff they download from the web, they also don't wait and retry. They need the real deal right away.
And this leads us to the conclusion about the current state of affairs:
On a day-to-day basis things are working really well without mirrors, but sometimes for really popular content they could come in handy. Some of that load could be offloaded easily to Bittorrent, but the rest of it is really difficult to handle without creating a maintenance hell.
This also brings up the question: Should BURP be both a renderfarm and a distribution platform? Or, should we leave it up to the community to find hosting solutions for particularly popular content on a case-by-case basis?
Comments and good ideas are welcome!