rss torrent feed

Message boards : Server backend and mirrors : rss torrent feed
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8942 - Posted: 29 Dec 2008, 21:18:41 UTC
Last modified: 29 Dec 2008, 21:41:48 UTC

I am starting a new thread on this in the dev section since that is where this now belongs. I have done some research, and the following is the format that will be used for the rss torrent:

http://istvan.s3.amazonaws.com/burp_torrent-feed_layout.txt

Please feel free to edit this, as it is not final. I will continue work, but a quick question: Janus mentioned there are some interfaces between scripts and DB queries, after some poking around the svn I saw some things, but are there any great examples I should see? The gallery page seems to do the Db queries w/o an interface...
~please check out that file, and tell me what needs to be changed, istvan.

[edit]: janus mentioned something about developing for v3, but in the svn there is no v3 folder... or is there?
ID: 8942 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8943 - Posted: 29 Dec 2008, 22:41:44 UTC

In some of the files there is a db query like the following:

$sessDB = $burpMainFactory->getDatabaseHandler()->getSessionList($offset.\",16\",\"end_time!=0\",\"end_time\");

for ($i=0;$i<sizeof($sessDB)-1;$i++){
$sessionList[]=new Session($sessDB[$i]->id);
};

_________
Is this the only way to get a list of completed sessions? In v3 will the setup be different? What I need to do is get a list of completed sessions, and then for each one get the output file names (I can steal that from mirror_download.php) and then for each file create an rss entry in the form in my previous post.

On an un-related note, I seem to have found a bug. try the link:

http://burp.boinc.dk/mirror/mirror_download.php?getid=1


There is a lot of code, and lots of includes to follow in this code, for the momment I can\'t seem to find the function getFile as used in

$file=getFile(intval($getid));

Any hints as to where I should look?
ID: 8943 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8947 - Posted: 30 Dec 2008, 0:57:26 UTC - in response to Message 8943.  

for the momment I can\'t seem to find the function getFile as used in

$file=getFile(intval($getid));

Any hints as to where I should look?


people say there is no such thing as a stupid question, but that was one. I think it is nap time if I can\'t find a function defined in the same file as it is used...
ID: 8947 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8948 - Posted: 30 Dec 2008, 1:59:05 UTC

I\'m going to bed now because I really do need sleep! you know it is time for bed when you can\'t understand simple code. The problem I am working on now is:

$file->filename

I see where $file is initialized, $file=getFile(intval($getid));, but where does filename come in? What is the arrow pointing to? I can\'t find where filename is initialized, probably because I\'m tired.

Also, there is a debugging line:

#print_r($file);exit;

what would the result of that be? (i forgot to mention that i\'m looking at mirror_download.php). I am back-tracking and trying to run some of the scripts mentally backwards to find out how they work.

~and you all know that as soon as i post this I will go off and find the answer no problem, Istvan.
ID: 8948 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8953 - Posted: 30 Dec 2008, 15:35:52 UTC

yah, after a few hours of sleep I found it right away.

So Janus, here is what I can make of the table of files: there is a column with file name, and file id. this script:

http://burp.boinc.dk/mirror/mirror_download.php?getfile=

will get the file specified by name, so he mirror_download simply converts the the file id to the file name if necessary and links to that script. Now here is my question: for my purposes, if I make a mysqql query for all files in that table would it give me anything I don\'t want? so another way of asking would be are the files you want linked to from the rss feed identical to the ones found in the table?

~if so that would make life really easy!
ID: 8953 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8954 - Posted: 30 Dec 2008, 15:38:03 UTC

also by chance can I see the torrent_cache.php file? it is not in the svn, and I would like to see what security features are in it (such as will it exit if a file with \'..\' is found in it).
ID: 8954 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8955 - Posted: 30 Dec 2008, 22:42:12 UTC

One more question (for now): I need to put in a pubdate, but from what I can scavenge the table mirrorfiles does not contain a column on the date of creation of the torrent. What should I put there?
ID: 8955 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8956 - Posted: 30 Dec 2008, 23:05:41 UTC

Here is a tentative script:

http://pastebin.com/fb0b8307

It is fairly simple, and I am a big fan of commenting. Possibly could you export part of the db table for me Janus for testing? This script also assumes that every file in the db table A. has a .torrent and B should be put in the feed. This is really an alpha version of the script, as I don\'t know if it functions yet, haven\'t tested it for errors etc. (I can\'t, I know it will give me some because I don\'t have the db). Please let me know of what needs to be changed and I will make the changes. Like I say this is a simple test which will evolve into what is required.

~this is a start, Istvan.
ID: 8956 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8958 - Posted: 31 Dec 2008, 13:28:30 UTC

it is a new day, and as I look at this thread I see all of the ramble that I have posted. I will clean up a little by telling you to ignore all of the above posts and re-state what I have to say that is still valid ;-)

1.) Here is a tentative script:

http://pastebin.com/fb0b8307

It is fairly simple, and I am a big fan of commenting. Possibly could you export part of the db table for me Janus for testing? This script also assumes that every file in the db table A. has a .torrent and B should be put in the feed. This is really an alpha version of the script, as I don\'t know if it functions yet, haven\'t tested it for errors etc. (I can\'t, I know it will give me some because I don\'t have the db). Please let me know of what needs to be changed and I will make the changes. Like I say this is a simple test which will evolve into what is required.

2.)as hinted to above I need to know table structure, and also I would need to know if there is anything in that table that shouldn\'t be put in the feed or does not have a .torrent. (or does the torrent script take any files? does it create the torrent as needed?)

3.) what I need most here is critique, I don\'t know if this script will work, or any of that so A little feedback would be nice.

4.) Janus: you said that the DB shouldn\'t be accessed directly, is that just for connecting? or should the query in that file also be thru some other interface. I tried my best to do it the same way the other files I could find do it, so I hope it is OK.

5.) Janus: you also mentioned that I should develop for v3, but I didn\'t see a folder of code in the SVN for v3, so I am assuming that all of the other functions I call (one or two) will exist in v3, and that the table setup will be what I guessed it to be (a column for name, and another for id)

6.) File names: from what I can see the file name is actually the path to the file. To a user sifting thru many torrents looking for a specific video it might be nice to have a small meaningful file name. I could explode the name and just take the string after the last slash, which would be like

s000642_movie_mpeg4_q96.avi

which I think is nicer than the whole path, and has all of the necessary info. I am also not sure if a slash is valid in a rss feed title (might be, haven\'t looked)

7.) Janus: as I am starting to develop, I am realizing that a way for users to test developments would be nice. For example debugging, testing, and exploring (ex. print_r an array to see if it contains what I think it will). Also a good documentation would be nice for the database, and what functions can be found where (what file). I would offer to write this, but I myself am not sure of the database, tho I can help my documenting the .inc files if you would like.

~I think it is time I gave back to burp, Istvan.
ID: 8958 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8963 - Posted: 3 Jan 2009, 19:35:25 UTC

After looking around I see that there are many files that have no peers for download. So for example the linked torrent on this page

http://burp.boinc.dk/mirror/mirror_download.php?getfile=/sessions/842/produced/s000842_movie_mpeg4_q96.avi

won\'t download because there are no peers. Once this script has been in use a little there should be a few more peers, but for them to get the files to seed... I was just wondering if there is even a seed for files like this.
ID: 8963 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 8975 - Posted: 7 Jan 2009, 11:34:00 UTC - in response to Message 8958.  
Last modified: 7 Jan 2009, 11:35:32 UTC

As stated elsewhere it would be a good idea to use BURPv.3 to do this since BURPv.2 is very close to being deprecated. To do this you should check out burp3 instead of just burp:
cvs -d :pserver:anonymous@burp.boinc.dk:/root checkout burp3


2.) as hinted to above I need to know table structure

Which in fact means you need to know the member list of the domain class \"Session\" and the storage class \"AttachmentFile\". These are located in src/burp/domain/Session.java and src/burp/storage/AttachmentFile.java.

or does not have a .torrent. (or does the torrent script take any files? does it create the torrent as needed?)

The torrents get created and cached on-the-fly. However, the .torrent system does not yet support BURPv.3 since an entirely new storage system is used (a virtual one where files can be all over the place).
As a consequence you must instead link in the following way:
http://bittorrent.burp.boinc.dk/torrent_cache.php?fileid=<insert file id here>

Don\'t worry about the .torrent script - I\'ll get around to that eventually.

you said that the DB shouldn\'t be accessed directly

Check the BURPv.3 page \"sessions.php\" located in html/user/.
Everything (even completed sessions) is put into queues in BURPv.3 - so you can get the most recently completed sessions using code like this:
$sessionList = $session->getSessionsInQueue($statusEnum->QUEUE_COMPLETED, 0, 16);

Also check the download section in session.php (note, no \'s\' at the ending!) for how to get a list of attached files.

File names: from what I can see the file name is actually the path to the file. To a user sifting thru many torrents looking for a specific video it might be nice to have a small meaningful file name. I could explode the name and just take the string after the last slash, which would be like
s000642_movie_mpeg4_q96.avi

You always index files by ID in the new system. A single file may be referenced from different contexts (ie. an output file can be attached as a streamable video but can also be attached as an input file for another session under a different name etc. etc.).
By default video files are named like \"s172streamable.mp4\".

as I am starting to develop, I am realizing that a way for users to test developments would be nice. For example debugging, testing, and exploring (ex. print_r an array to see if it contains what I think it will). Also a good documentation would be nice for the database, and what functions can be found where (what file). I would offer to write this, but I myself am not sure of the database, tho I can help my documenting the .inc files if you would like.

As with any other big project like this it is a good idea to create a local version for testing purposes. To do this you need to install a BOINC server, and install BURP on top of that. The process is outlined in the (currently non-existent, check the \'setup\' file instead) readme-file at the root of the BURPv.3 directory.
This is slightly complicated still, I\'m afraid, since some of the setup files are not yet ready (including the part that creates the database).

I can\'t give you print_r-access to the public project nor to the internal test project here, security reasons, I\'m sorry.

After looking around I see that there are many files that have no peers for download. So for example the linked torrent on this page

http://burp.boinc.dk/mirror/mirror_download.php?getfile=/sessions/842/produced/s000842_movie_mpeg4_q96.avi

won\'t download because there are no peers. Once this script has been in use a little there should be a few more peers, but for them to get the files to seed... I was just wondering if there is even a seed for files like this.

The .torrents are backed by webseeds. If your client supports webseeds (Azureus/Vuze, uTorrent etc.) then it will connect to a webseed after a few minutes of inactivity on the normal BitTorrent network.
ID: 8975 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8980 - Posted: 7 Jan 2009, 16:28:58 UTC - in response to Message 8975.  

As stated elsewhere it would be a good idea to use BURPv.3 to do this since BURPv.2 is very close to being deprecated. To do this you should check out burp3 instead of just burp:
cvs -d :pserver:anonymous@burp.boinc.dk:/root checkout burp3


Ok, thanks. I hadn\'t seen that, does it make sense to put that link here:
http://burp.boinc.dk/opensource_dev.php


File names: from what I can see the file name is actually the path to the file. To a user sifting thru many torrents looking for a specific video it might be nice to have a small meaningful file name. I could explode the name and just take the string after the last slash, which would be like
s000642_movie_mpeg4_q96.avi

You always index files by ID in the new system. A single file may be referenced from different contexts (ie. an output file can be attached as a streamable video but can also be attached as an input file for another session under a different name etc. etc.).
By default video files are named like \"s172streamable.mp4\".


ok, So the link to the .torrent cache script will not contain that link at all? rather it will contain a number; the file id?


as I am starting to develop, I am realizing that a way for users to test developments would be nice. For example debugging, testing, and exploring (ex. print_r an array to see if it contains what I think it will). Also a good documentation would be nice for the database, and what functions can be found where (what file). I would offer to write this, but I myself am not sure of the database, tho I can help my documenting the .inc files if you would like.

As with any other big project like this it is a good idea to create a local version for testing purposes. To do this you need to install a BOINC server, and install BURP on top of that. The process is outlined in the (currently non-existent, check the \'setup\' file instead) readme-file at the root of the BURPv.3 directory.
This is slightly complicated still, I\'m afraid, since some of the setup files are not yet ready (including the part that creates the database).

I can\'t give you print_r-access to the public project nor to the internal test project here, security reasons, I\'m sorry.


That is what I figured. Well I\'ll see what I can figure, tho that will probably result is bugs as I am not always a good guesser ;-)


After looking around I see that there are many files that have no peers for download. So for example the linked torrent on this page

http://burp.boinc.dk/mirror/mirror_download.php?getfile=/sessions/842/produced/s000842_movie_mpeg4_q96.avi

won\'t download because there are no peers. Once this script has been in use a little there should be a few more peers, but for them to get the files to seed... I was just wondering if there is even a seed for files like this.

The .torrents are backed by webseeds. If your client supports webseeds (Azureus/Vuze, uTorrent etc.) then it will connect to a webseed after a few minutes of inactivity on the normal BitTorrent network.


Ah, ok. Well I\'ll have to be more patient then =). I hope that with this script there will be enough peers that it will never have to go that far :).

After I wake up enough to think, some tea coffee, or something else I\'ll take a look at that SVN. My last question is didn\'t I see that you were trying to go to all PHP in the burp system? Two of the links to classes you gave me were to java files, and I\'v never written in java, so I\'ll see if I can take a crash course in the BURP system. Are they java because of limitations of PHP? Is the goal really to move away from java and other languages? If not I might have to go and find some nice tutorials on java... too bad winter break is over, it would have been a great time to learn a new language...

~well that first script was a good learning experience, I always enjoy reading others code. I learn some new tricks, and have to go back to good habits, fast code, lots of comments, etc. - Istvan.
ID: 8980 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8982 - Posted: 7 Jan 2009, 18:12:48 UTC

A few things: first, why is there no edit post in this section of the forum?

and second, can getSessionsInQueue be used to get ALL sessions? I havn\'t looked into it much yet, as my main computer is busy rendering and isn\'t really usable now because it is so laggy.

Based on the session.php file it looks to me like there is a Que of completed files, and in each file\'s case there is a java variable of files, which is converted to php, and if the file is larger than 0 then print a link to redirector.php (what is XML_URL_BASE?). Should my script point to that php script, or should there be a direct link to

http://bittorrent.burp.boinc.dk/torrent_cache.php?fileid=<insert file id here>

If I use that it might pose a speed problem, as for each file that exists, each time the rss feed is loaded that script would run for each file in the rss feed, which means many instances of that script running at the same time. This is also the case with any other scripts called, so any that take some time to work might really tie up your servers for a few seconds while the rss feed is being checked on the client\'s side.

~still poking around, and will really dive in when my main computer is done rendering - Istvan.
ID: 8982 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 8986 - Posted: 9 Jan 2009, 11:41:31 UTC - in response to Message 8982.  

and second, can getSessionsInQueue be used to get ALL sessions?

Yes, but please only include a subsection of those by default. For instance the 15 latest sessions.

it looks to me like there is a Que of completed files, and in each file\'s case there is a java variable of files, which is converted to php, and if the file is larger than 0 then print a link to redirector.php

Session has a getAttachmentFiles()-method which returns a List. If the size of the list is greater than 0 it will run through the list and output a link to the download redirection script.

what is XML_URL_BASE?

The base URL of the XML and non-HTML interfaces. Stuff like mirror redirects, the bittorrent .torrent generator, XML-RPC interface etc. will be located here.

should there be a direct link to
http://bittorrent.burp.boinc.dk/torrent_cache.php?fileid=

Please use XML_URL_BASE/torrent_cache.php?fileid=
just like the redirector but with the torrent_cache.php script instead.

If I use that it might pose a speed problem, as for each file that exists, each time the rss feed is loaded that script would run for each file in the rss feed, which means many instances of that script running at the same time. This is also the case with any other scripts called, so any that take some time to work might really tie up your servers for a few seconds while the rss feed is being checked on the client\'s side.

That will be taken into account in torrent_cache.php. It only spends time generating the .torrent once, subsequent requests will get a copy of the cached .torrent file
ID: 8986 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8991 - Posted: 10 Jan 2009, 0:19:22 UTC
Last modified: 10 Jan 2009, 0:37:07 UTC



and second, can getSessionsInQueue be used to get ALL sessions?


Yes, but please only include a subsection of those by default. For instance the 15 latest sessions.



If this is done then only the most recent sessions will be put into the torrents, so if a new torrent user comes along and subscribes to the feed they will not get any files older than say the last 15.

What does this mean for the BURP system? Well it means that each time a new user comes along the torrent script will have to dish out lots of torrents, but if as you say it keeps them instead of generating them each time they are asked for that shouldn\'t be such a big deal. It would be a load to generate the feed tho, but a fairly small load as it is simply generating a text feed, similar to a forum generating script on a long thread.

However if the script only uses the last XX files then only the newest (and most likely used files) would be in the script, and if an older file was popular then there would be lots of other peers anyway.

So what am I saying? well I\'m just saying the facts, and playing devils advocate as I am not in charge of the system. After reading this Janus, how many .torrents would you like in the feed at a time? assuming that there is a session completed every week, each generating 5 (eventually source files, 3 movie files, and a margin of one) there will be 5 new entries a week. If a user checks the feed every 3 weeks that would be the last 15 items in the feed, which should fairly reasonable number I think, and also would play nicely with the system.


what is XML_URL_BASE?

The base URL of the XML and non-HTML interfaces. Stuff like mirror redirects, the bittorrent .torrent generator, XML-RPC interface etc. will be located here.

should there be a direct link to
http://bittorrent.burp.boinc.dk/torrent_cache.php?fileid=<insert file id here>

Please use XML_URL_BASE/torrent_cache.php?fileid=<insert file id here>
just like the redirector but with the torrent_cache.php script instead.


So I don\'t have to set XML_URL_BASE or replace it with anything? All I need is

<a href=\\\"\".XML_URL_BASE.\"torrent_cache.php?fileid=<insert file id here>\"\\\">

~thanks for putting up with my questions, I am running low on time to browse the code and find my own answers as school has started again - Istvan.
ID: 8991 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 8992 - Posted: 10 Jan 2009, 2:21:29 UTC

I have a test version, with a TODO at the start. How should I get stuff to you? should I put them on my website, post them in this forum (it would be nasty!) or e-mail them to you?
ID: 8992 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9008 - Posted: 11 Jan 2009, 10:29:31 UTC - in response to Message 8991.  
Last modified: 11 Jan 2009, 10:29:56 UTC

assuming that there is a session completed every week, each generating 5 (eventually source files, 3 movie files, and a margin of one) there will be 5 new entries a week.

Each session now generates only 2 movie files. One for web streaming and one for download. Both files will be available through the mirrors (and hence BitTorrent too).

So I don\'t have to set XML_URL_BASE or replace it with anything? All I need is
\"\\\">

Yes, XML_URL_BASE is a constant, the php interpreter replaces it with the URL relevant for the given BURP installation (keep in mind that this site is not the only BURP site - ie. other sites have other URLs which is why you can\'t hardcode them).
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 9009 - Posted: 11 Jan 2009, 15:50:11 UTC

Ah, Ok, that makes sense. So how should I get code to you? it seems that me putting them on s3 for you works, should I just do that?
ID: 9009 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 9011 - Posted: 11 Jan 2009, 16:35:18 UTC - in response to Message 9009.  

That\'s fine
ID: 9011 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Istvan Burbank
Avatar

Send message
Joined: 3 Apr 08
Posts: 312
Credit: 58,920
RAC: 0
Message 9043 - Posted: 14 Jan 2009, 0:50:42 UTC

One more thing I need to add before I am done: to make the feed I need to put the time of the creation of the item, where can I find this?

~After that I will do a quick read-thru, and post it for you to download - Istvan.
ID: 9043 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · 3 · Next

Message boards : Server backend and mirrors : rss torrent feed