3026

Message boards : Comments and discussion : 3026
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4570
Credit: 2,100,463
RAC: 8
Message 14431 - Posted: 9 May 2016, 19:32:58 UTC
Last modified: 9 May 2016, 19:34:24 UTC

Planning to deploy a fix for the slow service backend issue tomorrow afternoon and using this session to test it. Let's see how fast we can go before and after the fix.
ID: 14431 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Wilhelm

Send message
Joined: 26 Feb 16
Posts: 7
Credit: 241,699
RAC: 0
Message 14432 - Posted: 9 May 2016, 20:48:24 UTC - in response to Message 14431.  

on a side note to that, on my machine the work does take the same time with 2 threads as with 12.
there might be a possible chance that there is room for improvement in,
the perspective of rendering more that one image on the same machine,
at the same time. But that is probably something for a future feature.
ID: 14432 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Roland

Send message
Joined: 6 May 16
Posts: 6
Credit: 0
RAC: 0
Message 14433 - Posted: 10 May 2016, 1:00:03 UTC

I've lots more of virtually identical projects for you to test with :)
ID: 14433 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4570
Credit: 2,100,463
RAC: 8
Message 14434 - Posted: 10 May 2016, 20:19:46 UTC
Last modified: 10 May 2016, 20:30:05 UTC

Thanks Roland, but I think this one will be the last of this kind of very fast + very long sessions that we will accept for now - the farm really was not designed for workunits shorter than half a minute and the servers are taking a beating.

The farm shines in the 10mins-6hours time per frame area. Shorter than that and overhead gets too large (both on the client and on the server), longer than that then dropped/re-queued/restarted frames start becoming an issue due to lack of checkpointing.
ID: 14434 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
jhiswin

Send message
Joined: 3 Jan 16
Posts: 5
Credit: 832,868
RAC: 0
Message 14437 - Posted: 11 May 2016, 3:55:07 UTC
Last modified: 11 May 2016, 3:57:31 UTC

Disclaimer: I don't understand the implementation and problem domain, so there is probably good reason that I just don't understand.

More than 3/4 of the time is used by unzip. It's essentially 15 seconds to unzip and <5 seconds to render. But, unzip uses almost no CPU time and IO, and does not take advantage of multithreading.

Wouldn't it be easier/more efficient to bundle it into one big zip or fork unzip processes? Or maybe just use a better unzip (or compression alg) implementation?
Also, why make each job take up all CPU slots? Found that if you trick BOINC in to treating the jobs as less than all CPUs, you can make it run jobs concurrently with very little impact and faster.
ID: 14437 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Roland

Send message
Joined: 6 May 16
Posts: 6
Credit: 0
RAC: 0
Message 14438 - Posted: 11 May 2016, 7:45:44 UTC - in response to Message 14434.  
Last modified: 11 May 2016, 8:09:54 UTC

> I think this one will be the last of this kind of very fast + very long sessions that we will accept for now

Gosh darnit :'( Does anybody have any suggestion for a renderfarm which could work for me?

The project is not really very long (70 hours on my desktop), but the BURP overhad makes it very long.

May I suggest the option to render multiple frames into short videos, client side? That's probably the only way to get an acceptably low overhead on this kind of projects.
The serverload would go down dramatically as it could just concatenate the videos and needs not store PNGs.


PS. my client keeps telling :Sheduler request completed: got 0 new taks", while there should be tasks available now. I'd like to contribute, but something is wrong. I can contribute to other BOINC projects.
ID: 14438 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile DoctorNow
Project donor
Avatar

Send message
Joined: 11 Apr 05
Posts: 403
Credit: 2,189,214
RAC: 7
Message 14440 - Posted: 11 May 2016, 11:33:55 UTC - in response to Message 14438.  
Last modified: 11 May 2016, 11:35:35 UTC

Never mind, hit the wrong button... ;-)
Life is Science, and Science rules. To the universe and beyond
Proud member of BOINC@Heidelberg
My BOINC-Stats
ID: 14440 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile EdPha
Avatar

Send message
Joined: 28 Apr 16
Posts: 19
Credit: 164,395
RAC: 0
Message 14441 - Posted: 11 May 2016, 11:36:09 UTC - in response to Message 14438.  
Last modified: 11 May 2016, 11:52:18 UTC

hey man, give a try, ppl here are offering help, no magic. you can claim you money back, or think that it´s free you resources to still doing other jobs. i read one time : "if you think that it´s more fast on you computer, so maybe you are right" we see jobs that are better to render alone, at home, and others that only got finished because everybody are working together.

if you really need control the entire process time, so, it´s better to render on you private rig, but if you can spare some time, let the farm do it freely .

also, you can add Enigma@home and use it to take jobs while burp don´t have other jobs.
ID: 14441 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Roland

Send message
Joined: 6 May 16
Posts: 6
Credit: 0
RAC: 0
Message 14443 - Posted: 11 May 2016, 13:18:30 UTC - in response to Message 14441.  

Dear EdPha, I think we're misunderstanding each other.

I don't know what I should "give a try". Do you mean "give a break"?

I'm grateful I could use BURP, but I was just told my type of projects will no longer be accepted because of the way BURP (currently) works. There is an overhead in the way BURP works, which is at its worse when render time of a frame is low.
I don't blame anyone, I only gave my ideas on what could make my type of project (and possibly others which only care for the resulting animation) acceptable again and could improve BURP as a whole. nobody benefits form having to reject incompatible projects.
I understand and respect they currently can not allow any more of my projects.

I've got 50 or so more of these projects to do, which on a single computer is going to take many months.
After my initial projects there will be only 6 or so per year and I'd be happy to contribute the idle time, which will be much more than the render time my projects require(d).
I would like my BOINC client to contribute to BURP projects (including my own), but the server keeps telling my client there is nothing to do. It's a problem, not something I desire.
ID: 14443 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Roland

Send message
Joined: 6 May 16
Posts: 6
Credit: 0
RAC: 0
Message 14445 - Posted: 11 May 2016, 13:50:18 UTC - in response to Message 14431.  

@ Janus: I don't know if it was something you did, but I noticed my previous project took almost 69 hours to render, while this one was done in 37 hours :)

Too bad there is still 11 hours of (video?) encoding to be done by the server. I guess the huge number of PNG's to process is to blame. I'd estimate it at 55 GB. Sorry for that.
ID: 14445 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile EdPha
Avatar

Send message
Joined: 28 Apr 16
Posts: 19
Credit: 164,395
RAC: 0
Message 14446 - Posted: 11 May 2016, 13:55:25 UTC - in response to Message 14443.  

Dear EdPha, I think we're misunderstanding each other.

I don't know what I should "give a try". Do you mean "give a break"?



give a try : let´s try again, let´s try in another way, no "give up" only "let´s rethink and tray again"

in the first time, we always hit the wall instead the target, so we rethink and ask each other to archive a way to optimize the jobs.

all of us are always learning. i bet that someone have a tip to the next job.
ID: 14446 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Senilix

Send message
Joined: 15 Oct 14
Posts: 11
Credit: 823,221
RAC: 0
Message 14447 - Posted: 11 May 2016, 15:38:22 UTC - in response to Message 14443.  

I would like my BOINC client to contribute to BURP projects (including my own), but the server keeps telling my client there is nothing to do. It's a problem, not something I desire.

The reason why you don't get any work now is that the project has none atm.

The reason why you got no work for the last batch is that it was Windows only.
You'll have to wait for the next Linux batch I'm afraid...
ID: 14447 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4570
Credit: 2,100,463
RAC: 8
Message 14448 - Posted: 11 May 2016, 15:54:06 UTC - in response to Message 14437.  

@jhiswin: in fact we could do one better and only unpack Blender once per client release and reuse that - instant WU starts! Unfortunately this has turned out to be quite complicated when tested a few years ago. The test-client that actually did this turned out to be very buggy and difficult to maintain. It is still on the todo list - maybe a feature suggestion for BOINC even... other projects could benefit a lot from that too.

@Roland: yup the backend was optimized to better handle this kind of fast session, giving about 4x better performance when I ran the test (the test was only running for the second half of the session).
Returning multiple frames per WU would definitely be the way to go to properly optimize this fully. In fact we could do this even without closing Blender between each frame because the rendering is controlled by our client-side script inside Blender, so the unpacking and loading issue becomes much smaller.
The hard part is making the server "understand" that kind of WU. So far we have only had 2 kinds of WU: full-frame and split-frame. The new one would be multi-frame. I added it to the TODO-list, but will need some time to think about it. Thanks for the suggestion.
ID: 14448 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Roland

Send message
Joined: 6 May 16
Posts: 6
Credit: 0
RAC: 0
Message 14449 - Posted: 12 May 2016, 13:30:09 UTC

Did my project break the server?
My previous, almost identical project took 11 hours to 'encode':
2016-05-09 15:57.44 Encoding begins
2016-05-10 02:54.26 Encoding ends

While this project (3026) seems to be stuck somewhere:
2016-05-11 08:06.24 Encoding begins
.. and still not ended.
ID: 14449 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile EdPha
Avatar

Send message
Joined: 28 Apr 16
Posts: 19
Credit: 164,395
RAC: 0
Message 14450 - Posted: 12 May 2016, 15:56:37 UTC - in response to Message 14449.  
Last modified: 12 May 2016, 15:57:34 UTC

i think that the server are under attack of random aliens of DDos, because it´s too much time without response of the server.

i think: that this kind of work from the server can be done by the farm too ?
ID: 14450 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4570
Credit: 2,100,463
RAC: 8
Message 14451 - Posted: 13 May 2016, 16:57:21 UTC

Odd, I tried poking it with a stick - seems to be humming along now at 22641 of 88000 frames encoded
ID: 14451 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile EdPha
Avatar

Send message
Joined: 28 Apr 16
Posts: 19
Credit: 164,395
RAC: 0
Message 14452 - Posted: 13 May 2016, 17:13:26 UTC - in response to Message 14451.  
Last modified: 13 May 2016, 17:14:23 UTC

its´s an epic work, i think that it´s a story to figure in the first page alongside with big buck bunny ... at least because it´s an water divisor...

"before the overwhelming"
"after the overwhelming"
ID: 14452 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
den777

Send message
Joined: 10 Dec 13
Posts: 2
Credit: 267,555
RAC: 0
Message 14455 - Posted: 15 May 2016, 7:18:58 UTC - in response to Message 14452.  

its´s an epic work, i think that it´s a story to figure in the first page alongside with big buck bunny


Agree :D

It seems that it broke Boincstats https://boincstats.com/en/forum/5/10910,1 (this session or the prevoius one)

Also boinc client on my Windows host became unresponsive after some minutes of doing this session, i.e. boinc manager or boinccmd could not connect so I had to kill processes manually and restart boinc. However prevoius linux session was working fine.
ID: 14455 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Comments and discussion : 3026