The mirror network and you!

Message boards : Server backend and mirrors : The mirror network and you!
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9444 - Posted: 20 Apr 2009, 18:58:05 UTC
Last modified: 20 Apr 2009, 19:00:48 UTC

One of the big showstoppers right now is that our current mirror network is not compatible with the new version of BURP. The plan is to make a new kind of network which works more smothly and can handle requests on-demand rather than having long syncing periods before it can be used.

The current mirror network is mostly based on servers and PHP. This has the advantage that most servers have great internet connections and lots of bandwidth in spare. It has the disadvantage, however, that PHP is mostly a query-reply language. A client downloading from a mirror works smoothly, but mirror-to-mirror transfers and syncing has always been a big mess, especially when some mirrors drop out of the network and rejoin later.

One of the possible future plans deals with the idea of moving the mirror network closer to the people who use BURP. It is not the only plan and multiple other plans are being weighed in at the moment.
Let me first describe the idea and then ask you a question afterwards.

The idea: Rewrite the mirror software in Java rather than PHP. This has the advantage that it is suddenly much easier to do long-lasting communications and figure out when a mirror node drops out of the network and enters again. It has the disadvantage that most servers do not support Java.
Instead of running the Java mirror node on servers it could be running on computers from the community. Seeing that symmetric 5Mbit connections are no longer such a rarity this may or may not be as powerful a solution as the server-based approach.

It would be an icon that sits in your system tray on your desktop machine (or runs in the background on your Java capable server) and uses spare outbound internet bandwidth to host scene and animation data files for the rendering nodes. It would take up a configurable amount of harddrive space to store the files and you would need to open port 80 to allow it to act as an internet server.

The question:
Would you be interested in running such a utility? Why or why not? If you are interested, how much upload bandwidth (in Mbit per second and/or gigabytes per month) would you want to donate? - And how much of the day would the utility be running on average?

What are your thoughts and comments on this?
ID: 9444 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
noderaser
Project donor
Avatar

Send message
Joined: 28 Mar 06
Posts: 516
Credit: 1,567,702
RAC: 0
Message 9446 - Posted: 21 Apr 2009, 4:07:11 UTC

My shared hosting is capable of running JRE 1.6, but it is not currently installed; would this be compatible with the Java solution you're thinking about? If so, then I would be more than happy to continue contributing with my current storage/bandwidth limits of 40GB/500GB. I am not so sure about running something on my local computer, however, because my connection is only 3Mbps down/768 Kbps up and I'm pretty sure that I would notice the upload throughput being consumed.
Click here to see My Detailed BOINC Stats
ID: 9446 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9447 - Posted: 21 Apr 2009, 13:35:38 UTC

Yes, Java 1.6.x would in fact be required.

Also, I suggest that the minimal donation is around 1Mbit for effeciency reasons. That would mean participants would have to have 1Mbit upload (if they donate all their bandwidth) or more (if they donate a percentage only, which is the recommended thing to do) to take part in the mirror network.
ID: 9447 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
00quick00

Send message
Joined: 9 Dec 07
Posts: 21
Credit: 3,169
RAC: 0
Message 9454 - Posted: 24 Apr 2009, 3:16:41 UTC - in response to Message 9447.  

Java uses up too much memory so it cannot be run on Virtual Private Servers plus it's only installed on a very small amount of Linux servers.

You'll most likely lose the majority of current mirrors and will instead be replaced by unreliable desktop machines with slow connections and high latency.


Maybe a new system incorporating PHP could be developed? Eg. Important and large files are only sent to the most reliable mirrors in terms of uptime and low connection dropouts.
ID: 9454 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9455 - Posted: 24 Apr 2009, 6:28:36 UTC - in response to Message 9454.  

Java uses up too much memory so it cannot be run on Virtual Private Servers

I think it's hard to talk about the memory consumption of a program that has yet to be written. If you are talking about Java in general then I'm not sure I entirely understand the problem - how much is too much? Do you have an example?

For comparison the entire BURPv3 server runs in about 20MB of space on average (increasing to around 64MB for performance reasons when encoding video files).

plus it's only installed on a very small amount of Linux servers.

Yes.

You'll most likely lose the majority of current mirrors and will instead be replaced by unreliable desktop machines with slow connections and high latency.
Maybe a new system incorporating PHP could be developed? Eg. Important and large files are only sent to the most reliable mirrors in terms of uptime and low connection dropouts.

Using Java is not the only plan - multiple other plans are being weighed in at the moment; including rewriting the old PHP mirror system from scratch to support this kind of "streaming".
ID: 9455 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
00quick00

Send message
Joined: 9 Dec 07
Posts: 21
Credit: 3,169
RAC: 0
Message 9456 - Posted: 24 Apr 2009, 8:16:18 UTC - in response to Message 9455.  

Java uses up too much memory so it cannot be run on Virtual Private Servers

I think it's hard to talk about the memory consumption of a program that has yet to be written. If you are talking about Java in general then I'm not sure I entirely understand the problem - how much is too much? Do you have an example?

For comparison the entire BURPv3 server runs in about 20MB of space on average (increasing to around 64MB for performance reasons when encoding video files).


Sorry, I'm sort of stereotyping Java itself there.

The majority of Java programs I have come across are riddled with memory leaks and use an excessive amount of memory compared to other applications. However like you said, you haven't programmed it yet. :)


In my opinion, a dominant programming language should be favoured over one that is installed on a very small amount of machines.
ID: 9456 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9457 - Posted: 24 Apr 2009, 18:20:08 UTC - in response to Message 9456.  
Last modified: 24 Apr 2009, 18:23:06 UTC

In my opinion, a dominant programming language should be favoured over one that is installed on a very small amount of machines.

Although it isn't something that I have direct experience with I know that Java code can compile to native executables. Apart from that there's also the option of bundling a stripped down mini-jvm with the application. In other words the choice of language and the choice of deployment method are seperate things.
The reason why I don't consider writing this in straight C/C++ is that I don't trust my skills in those languages enough to write a service that would be interfacing directly with the internet - buffer overruns etc., you know the drill.

Anyhow, the intention was not to talk about technical issues but rather to figure out if there would be community interest in running such an application.
ID: 9457 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
noderaser
Project donor
Avatar

Send message
Joined: 28 Mar 06
Posts: 516
Credit: 1,567,702
RAC: 0
Message 9458 - Posted: 25 Apr 2009, 1:48:56 UTC - in response to Message 9457.  

Anyhow, the intention was not to talk about technical issues but rather to figure out if there would be community interest in running such an application.


Anything that's beneficial to the performance of BURP!
Click here to see My Detailed BOINC Stats
ID: 9458 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile [boinc.at]Bountin

Send message
Joined: 7 Jan 07
Posts: 25
Credit: 16,742
RAC: 0
Message 9460 - Posted: 25 Apr 2009, 10:23:50 UTC

You could use a bittorrent system for the servers. There are already solutions and programs for most of the distributions and you would only have to "feed" them .. for example with a RSS feed.

The normal workflow would be like that:
- After rendering, postprogressing etc a torrent is created
- This torrent is announced on a RSS or a Atom Feed *
- After some time the mirror loads the Feed, loads the torrent and finally loads the file.

* An additional idea: The Feeds are generated for each server mirror and for each client who is participating in this system. So the feed generator can control the whole distribution system.


If you just want this bittorent system, it's fine but the files could additionally be in a public folder of the webserver and distributed via HTTP and via BT.

Here a list of Clients who can use feeds: http://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clients#Features_II

To the server - I've used BT myself for some file distributions. On my linux server I had a small shellskript running which ran a BT program. On Debian it's just the package bittorrent (with etch) and it reads all torrents in a directory. So I would only have to fill this directory with some torrent file which I get from a feed and everything is fine =)

Sorry for my English, I'm not a native speaker. I hope it is quite comprehensible ;)
ID: 9460 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9461 - Posted: 25 Apr 2009, 18:30:05 UTC

Ah, but the role of the mirror network is going to change as well. The whole "download a movie file from the mirror network"-thing is being replaced by streaming video.
The primary role of the new mirror network is to support the BOINC clients with application data and scene input files - and to do so without syncing the file to a mirror first. With Bittorrent you would have to first download the file to the mirror (sync) and then make it available over HTTP to the BOINC client. That makes it very similar (in fact extremely similar) to the current mirror network.

Instead the plan is to consider files to be cacheable streams of data. As soon as a mirror has received the first bytes of a file it can start relaying it to HTTP clients (i.e. no syncing is needed).
ID: 9461 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile PowerRenderer
Project donor

Send message
Joined: 31 Mar 05
Posts: 16
Credit: 97,751
RAC: 0
Message 9462 - Posted: 30 Apr 2009, 15:51:53 UTC - in response to Message 9461.  

Would you be interested in running such a utility?
I would like to but I wouldn't meet the bandwidth requirements.

Ah, but the role of the mirror network is going to change as well. The whole "download a movie file from the mirror network"-thing is being replaced by streaming video.
[snip]
Instead the plan is to consider files to be cacheable streams of data. As soon as a mirror has received the first bytes of a file it can start relaying it to HTTP clients (i.e. no syncing is needed).



That is an interesting concept. How would the boinc clients handle a partially completed file (mirror side) when downloading? I will admit, I haven't looked at the code; but I would guess it would retry, because of the size discrepancy and discard what is has already downloaded, being that is the simplest solution.

Although, a new server-client protocol could be created to better facilitate this type of communication. Not sure how much work would be required to send that upstream.

Or perhaps I am confused.

I think it is a great idea but HTTP is more of a limiting factor.
ID: 9462 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9463 - Posted: 2 May 2009, 11:58:03 UTC - in response to Message 9462.  

I think it is a great idea but HTTP is more of a limiting factor.

Surprisingly HTTP seems like the perfect choice of protocol for this kind of thing. Through headers and stream control it has support for placing the stream pointer at arbitrary locations in a stream and start streaming from there. If a client was disconnected before it received the entire file it may restart the stream later from the same or a different mirror. The BOINC client is based on cURL as far as I remember, and has good support for most of the signals that can be sent using HTTP.

Such a setup is typically called a reflector network rather than a mirror network. A mirror/reflector may receive a stream from the central server while at the same time retransmitting that same stream to 10 different destinations. So the central server uses the reflector to bounce data to more destinations than it could have serviced itself directly. Coordinating the input stream on a reflector with the (potentially) multiple output streams is what makes this hard to do in PHP and much easier to do in a language that has proper cross-thread communication built in.
Once a reflector has received and reflected a particular stream it may re-send it any number of times (and now it simply acts like a normal mirror).
ID: 9463 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Wizzo

Send message
Joined: 6 Sep 06
Posts: 14
Credit: 1,174,276
RAC: 0
Message 9469 - Posted: 8 May 2009, 6:55:03 UTC

First, are you locked into port 80? Someone with highspeed connections may already have a website, maybe at the actual mirror to mirror sharing level another port could be used, somthing along the lines of pick a port when you join the network.

Second, with the current network, say a mirror starts downloading the file from the main server, can it send it out to another server as it is being downloaded?
I spent 5 minutes reading about bittorrent, and it appears that it can. If 1 mirror only uploaded to 1 other mirror when the syncing is occuring, and lets say the main server uploaded the file to the mirriors with the highest outbound speed first, would that reduce the syncing time enough to not require java, and loose alot of the current mirrors? For example the project server uploads the file to mirror 1, once mirror 1 has buffered a few megs, ir starts uploading to mirror 2, while still downloading from the main server. Mirror 2, after buffering a few megs starts uploading to mirror 3. If a mirror has more upload speed then the speed the file is coming in, it can send it out to more then one mirror.

Third, have a master mirror list, with ip / port / upload speed / download speed / online status syncing between the mirrors, send a heartbeat between 3-5 random mirrors every few minutes, and if the online status changes, push a new master list. In the heartbeat send %bandwidth available & timestamp on the mirrors it has pinged, that way the network knows who is busy, who is not wothout wasting too much bandwidth.

In theory all the mirrors should have about the same % of a new file, limited by the upload speed of the project server. Once the file is complete on the first mirror, is should be close to complete on the last mirror. If a mirror is missing the file, it should attempt to stream it from as many mirrors as possible to minimise the impact on bandwidth on any 1 mirror. If the mirror is already using all of its bandwidth, it may refer to another server. A user attempting to get files would also be refered to another mirror.
ID: 9469 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
baracutio
Project donor

Send message
Joined: 29 Mar 05
Posts: 96
Credit: 174,604
RAC: 0
Message 9470 - Posted: 8 May 2009, 11:42:12 UTC
Last modified: 8 May 2009, 11:42:30 UTC

torrent is a good choice. everyone who has fast internet can help. that means you have per file hundreds of seeders and more or less unlimited bandwidth. the only problem is boinc, does it already support torrent downloads?!

but one question...
janus said:

The whole "download a movie file from the mirror network"-thing is being replaced by streaming video.

do we really want streaming video or good old file download? i think both. but how can you combine torrent and streaming?!


greets
- bara
ID: 9470 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile [boinc.at]Bountin

Send message
Joined: 7 Jan 07
Posts: 25
Credit: 16,742
RAC: 0
Message 9471 - Posted: 8 May 2009, 12:38:30 UTC - in response to Message 9470.  


do we really want streaming video or good old file download? i think both. but how can you combine torrent and streaming?!


There are some applications which do so! For example Joost and Miro use(d) torrent. Zatoo also uses a distributed model but I think it isn't torrent.
ID: 9471 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael

Send message
Joined: 7 Jan 08
Posts: 8
Credit: 45,171
RAC: 0
Message 9472 - Posted: 8 May 2009, 15:49:18 UTC

I feel the need to point out that while torrents are a good thing, and quite handy when used correctly and when not interfered with, some home Internet providers(mine included) will do nasty things to your connection if you use them. My connection, for instance will become unusable if I try to broadcast a torrent(last time I tried was a month ago for a Linux ISO). With torrents, there would need to be a "backup" method for those who can't get good results(due to ISP interference). I would be happy to contribute my bandwidth upstream if I could...
ID: 9472 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 9476 - Posted: 10 May 2009, 12:01:44 UTC

At one point I was working on a bit torrent system for BURP which as come to a stall. However there are a few things that I have to offer on that note:

If all of BURP just used bit torrent a user could run bit torrent, the java app in question here could be replaced with Vuze or another bit torrent client, and the php script could be a simple bit torrent client as well. If that were the case mirror outages are not a problem, and it could run on just about anything. It could also be faster downloads for some users.

The streaming wouldn't work with the bit torrent, however bit torrent does download the whole file for re-distribution. As an option in the php client this folder could be world readable, which could allow for streaming or non-torrent downloads of those files.

Getting more specific here is my idea: there is an xml file with the torrents for all of the files that need to be hosted. This RSS feed would be hosted on the BURP server. Any person who would like to help distribute files would point their torrent application at this feed, and would then be done. As for the concerns of only wanting to donate X amount of disk space my script I was working on for BURP allowed for that, and could possibly be used or re-written for this. The servers would have a similar setup, however they would store the files in a world readable folder, and seed the files with bit torrent. This world readable feature would be in place for those who can't use bit torrent, either personal preference, or packet shaping reasons.

Over all the result would be more load balanced file distributions on an existing system. It could also result in faster downloads if there are a lot of active peers.

~please let me know what you think, Istvan.
ID: 9476 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9477 - Posted: 10 May 2009, 14:19:11 UTC

First, are you locked into port 80?

Yes, the BOINC client does not support other protocols than HTTP and some firewalls block HTTP requests on other ports than 80.

Someone with highspeed connections may already have a website

That's correct. Since the reflector is to be HTTP-based anyways I assume it's not too much of a problem to serve it through a PHP (or similar) wrapper script to allow it to play nicely with existing server setups.
ID: 9477 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 9480 - Posted: 10 May 2009, 14:28:55 UTC

I take that to mean you did a head-palm after I came back with my bit torrent ideas ;)
ID: 9480 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9484 - Posted: 11 May 2009, 18:35:07 UTC - in response to Message 9480.  

Well, considering that BOINC does not support Bittorrent yet I think it's wiser to go with something more traditional - like HTTP.
ID: 9484 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Server backend and mirrors : The mirror network and you!