Left Input files

Message boards : Server backend and mirrors : Left Input files
Message board moderation

To post messages, you must log in.

AuthorMessage
Achim

Send message
Joined: 17 May 05
Posts: 183
Credit: 2,642,713
RAC: 0
Message 4781 - Posted: 2 Mar 2007, 21:12:44 UTC

HI,

I just saw I have all still all the old Input files in the burp project directory.

I think the server should request to delete them, as soon as the session is full rendered, right?(As done in the einstein project)

Thanks

Achim

</project>
<app>
<name>blender</name>
<user_friendly_name>Blender</user_friendly_name>
</app>
<file_info>
<name>131in0.zip</name>
<nbytes>2025878.000000</nbytes>
<max_nbytes>0.000000</max_nbytes>
<md5_cksum>1f3f7bf93130e936c3eac0a4df8f2e62</md5_cksum>
<status>1</status>
<sticky/>
<report_on_rpc/>
<url>http://burp.boinc.dk/mirror/mirror_download.php?getfile=/131in0.zip</url>
</file_info>
<file_info>
<name>184in0.zip</name>
<nbytes>3883230.000000</nbytes>
<max_nbytes>0.000000</max_nbytes>
<md5_cksum>808dd4bfe466ab803f2ca58994760845</md5_cksum>
<status>1</status>
<sticky/>
<report_on_rpc/>
<url>http://burp.boinc.dk/mirror/mirror_download.php?getfile=/184in0.zip</url>
</file_info>
ID: 4781 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 4782 - Posted: 2 Mar 2007, 22:21:59 UTC - in response to Message 4781.  
Last modified: 2 Mar 2007, 22:33:01 UTC

Yes this is part of locality scheduling which is to be experimentally enabled soon. It adds quite some load to the database, so the result table will have to be cleaned quite a bit first.

The cleaning has started today and will continue during the weekend.
ID: 4782 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Achim

Send message
Joined: 17 May 05
Posts: 183
Credit: 2,642,713
RAC: 0
Message 4783 - Posted: 2 Mar 2007, 22:39:34 UTC - in response to Message 4782.  

It adds quite some load to the database


Yes Einstein had some trouble due to this.

I\'m quite sure burp db is smaller than einsteins, but it might be worse to check there optimisations. A description is in there forum

So take your time (I assume you are quite busy currently).
ID: 4783 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 25 Apr 05
Posts: 347
Credit: 4,618
RAC: 0
Message 4836 - Posted: 5 Mar 2007, 3:46:19 UTC - in response to Message 4782.  

Yes this is part of locality scheduling which is to be experimentally enabled soon. It adds quite some load to the database, so the result table will have to be cleaned quite a bit first.

The cleaning has started today and will continue during the weekend.

mysqldump the result table. Drop it. Load the backup. There\'s your speed back.

I did it with renderfarm@home, only that it was the whole project database (not only result table), and I deleted the InnoDB files after dropping. For some really weird reason, ibdata1 kept being 6 gigabytes even after dropping DB and having no InnoDB tables anywhere else! After the dump/drop/delete/restore, ibdata1 is like 170MB.
ID: 4836 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 4846 - Posted: 5 Mar 2007, 13:12:30 UTC
Last modified: 5 Mar 2007, 13:12:58 UTC

Ah, was talking about cleaning the content of the table not the table itself. In fact the cleaning process is much similar to the BOINC db_purge utility except BURP doesn\'t use zip\'ed XML-files but a seperate database for it.
ID: 4846 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
thierry.l

Send message
Joined: 1 Sep 06
Posts: 7
Credit: 10,368
RAC: 0
Message 6974 - Posted: 1 Nov 2007, 14:32:53 UTC - in response to Message 4846.  

Hi,
What is the purpose of keeping old session files on client ?
I mean, I deleted manually 631in0.zip to 660in0.zip (about 110MB),
and boinc try to get those files from server but didn\'t find them.
So if they are not on server anymore, they should not be sticky too as it is not possible to satisfy the request.

Regards.
ID: 6974 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 6984 - Posted: 7 Nov 2007, 14:29:47 UTC - in response to Message 6974.  

They are sticky because that way you will only have to download them once for an entire session. The server will then attempt to give you work that you already have the input files for.
However, the server is also responsible for deleting the files again, and that part is rather expensive database-wise so it\'s only turned on from time to time when there\'s no other tasks around.
One of the reasons why this is currently not running all the time is because there\'s a lot of debug info in the information returned with results, so the result table in the database is a lot larger (>10GB) than what would be expected when we switch to beta/live and turn debugging off (~3GB). This, in turn, means that there\'s currently not enough room in system memory (RAM) to keep it all in there at the same time - so those scans for workunit files that are no longer in use become rather \"expensive\" because it generates a lot of I/O activity.

This was a rather technical explanation, but the essence is this: It is turned off right now due to debugging, it will be turned back on when debugging of the client is turned off.
ID: 6984 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Achim

Send message
Joined: 17 May 05
Posts: 183
Credit: 2,642,713
RAC: 0
Message 6985 - Posted: 7 Nov 2007, 14:40:11 UTC - in response to Message 6984.  

However, the server is also responsible for deleting the files again, and that part is rather expensive database-wise so it\'s only turned on from time to time when there\'s no other tasks around.

Why not start with a maybe easier way to find out.
In the server status you have all session where still some work is open.
So delete inputfile where the session is not listed on the status page any more is a very good way, or?
ID: 6985 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 6986 - Posted: 7 Nov 2007, 14:46:24 UTC - in response to Message 6985.  
Last modified: 7 Nov 2007, 14:51:19 UTC

In the server status you have all session where still some work is open.
So delete inputfile where the session is not listed on the status page any more is a very good way, or?

It is true that the BURP part of the system is perfectly aware of what is going on. The server part responsible for sending the \"delete\" message, however, is part of the BOINC scheduler. Making the scheduler aware of the BURP concept of \"sessions\" would require a rewrite of the BOINC code. Something I\'ve been thinking about doing (for other (mainly performance) reasons) but haven\'t gotten around to. The issue is that this particular component changes a lot and gets new features and bugfixes from BOINC all the time, keeping up a modified copy may be a bit too much work considering that this will only benefit us during alpha and not in later project stages.
Most other projects have taken the easy way out: Install enough RAM on the DB system to contain the entire result table.

Well, it basicly all comes down to a bad DB layout from BOINC in two of the central database tables (which was pointed out on one of the mailinglists recently), but changing it so that it would work better on small server systems like the BURP one is low priority.
ID: 6986 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Achim

Send message
Joined: 17 May 05
Posts: 183
Credit: 2,642,713
RAC: 0
Message 6987 - Posted: 7 Nov 2007, 16:57:12 UTC

How is it done with einstein?
I assume the problem is very similar, as they have as well a inputfile used by serveral WU\'s.
ID: 6987 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 6988 - Posted: 8 Nov 2007, 0:48:16 UTC - in response to Message 6987.  

How is it done with einstein?

The easy way. I haven\'t got a link, but I seem to remember that their specs were quite nice.
ID: 6988 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
M0CZY
Avatar

Send message
Joined: 31 Jul 09
Posts: 25
Credit: 279,718
RAC: 0
Message 10642 - Posted: 30 Nov 2010, 10:11:12 UTC

In order to save a bit of disk space I reset the BURP project, which deleted the Blender
application files, and I manually deleted the old session work files.
The server immediately resent me file1867409, which is 150 MB.
If that session is finished and dead, why have I been sent that file?
When a session is finished, the server should send a command to delete old work files.
The biggest threat to public safety and security is not terrorism, it is Government abuse of authority.
Bitcoin Donations: 1Le52kWoLz42fjfappoBmyg73oyvejKBR3
ID: 10642 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 10643 - Posted: 2 Dec 2010, 19:56:19 UTC
Last modified: 2 Dec 2010, 19:56:56 UTC

Not much has changed here since the thread was originally created. BOINC still does not cache files well and the solution has been to track the files server-side and delete them manually from time to time. Next time will be around new year I think.

The reason why files are left on the hosts are that they are reused for several WUs - and recently for several sessions as well.
ID: 10643 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
noderaser
Project donor
Avatar

Send message
Joined: 28 Mar 06
Posts: 517
Credit: 1,567,702
RAC: 0
Message 10649 - Posted: 6 Dec 2010, 0:38:23 UTC

It could be worse; ORE/Renderfarm.fi is currently using 1.53 GB on my machines, while BURP is only at 348.43 MB. Fortunately, storage is not a concern.
Click here to see My Detailed BOINC Stats
ID: 10649 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
M0CZY
Avatar

Send message
Joined: 31 Jul 09
Posts: 25
Credit: 279,718
RAC: 0
Message 10679 - Posted: 22 Dec 2010, 14:16:10 UTC

I also participate in Renderfarm, and their server can delete old work files.

22-Dec-2010 12:59:30 [Renderfarm.fi] Sending scheduler request: Requested by project.
22-Dec-2010 12:59:30 [Renderfarm.fi] Requesting new tasks
22-Dec-2010 12:59:34 [Renderfarm.fi] Scheduler request completed: got 0 new tasks
22-Dec-2010 12:59:34 [Renderfarm.fi] Message from server: Project has no tasks available
22-Dec-2010 12:59:34 [Renderfarm.fi] Got server request to delete file file478276
22-Dec-2010 12:59:34 [Renderfarm.fi] Got server request to delete file file478493
22-Dec-2010 12:59:34 [Renderfarm.fi] Got server request to delete file file480567

The biggest threat to public safety and security is not terrorism, it is Government abuse of authority.
Bitcoin Donations: 1Le52kWoLz42fjfappoBmyg73oyvejKBR3
ID: 10679 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 10680 - Posted: 22 Dec 2010, 16:35:30 UTC - in response to Message 10679.  

I also participate in Renderfarm, and their server can delete old work files.

22-Dec-2010 12:59:30 [Renderfarm.fi] Sending scheduler request: Requested by project.
22-Dec-2010 12:59:30 [Renderfarm.fi] Requesting new tasks
22-Dec-2010 12:59:34 [Renderfarm.fi] Scheduler request completed: got 0 new tasks
22-Dec-2010 12:59:34 [Renderfarm.fi] Message from server: Project has no tasks available
22-Dec-2010 12:59:34 [Renderfarm.fi] Got server request to delete file file478276
22-Dec-2010 12:59:34 [Renderfarm.fi] Got server request to delete file file478493
22-Dec-2010 12:59:34 [Renderfarm.fi] Got server request to delete file file480567

Yes, we are working on a solution that will allow the server to delete old files during scheduler requests based on how recently the files have been used. Olivier from Renderfarming.fi has been writing the BOINC scheduler patch and testing that part for the past few days.

My guess is that we'll start deleting files from clients attached to this project around new year as well.
ID: 10680 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Server backend and mirrors : Left Input files