Session 3391


Advanced search

Message boards : Problems and Help : Session 3391

1 · 2 · Next
Author Message
Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15194 - Posted: 11 Aug 2017, 15:56:05 UTC

Finally got a task from Burp, and when it started, estimated time to completion was 6:04. Progress stopped at 0.031 percent, when 6:04 elapsed was reached, progress is still at 0.031 percent and estimated time remaining started climbing. it is now at 51 days and increasing. Should the unit be aborted?

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15195 - Posted: 11 Aug 2017, 16:34:56 UTC

After running for an hour now (on 2 cpus) progress is up to 0.040 percent and time to completion is at 106 days and climbing.

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15196 - Posted: 11 Aug 2017, 17:46:13 UTC

After 2 hours, the task is now up to 0.050 percent complete and estimated finishing time is 183 days and rising.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4478
Credit: 2,094,806
RAC: 0
Message 15197 - Posted: 11 Aug 2017, 19:21:55 UTC

Don't trust the runtime estimator, it is severely broken

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15200 - Posted: 11 Aug 2017, 21:06:02 UTC

After 5 hours progress is up to 0.080% and 283 days remaining. Boinc is running the task in high priority EDF mode. The tasks does not appear to be checkpointing, so I hope the system doesn't crash. Is it normal for a task to have a virtual memory size of 4586MB

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15201 - Posted: 12 Aug 2017, 1:48:54 UTC

9 hours 43 minutes, I'm looking at the slots directory in Explorer, read the contents of the stderr.txt file when all the files disappear. Look up above the window at the Manager's window and see that the task has gone from 0.119% complete to 100% and is uploading. Estimated time remaining had made it up to 389 days when task completed. The event log never had any reports from the task that it had check-pointed, NOT GOOD. Another task has started and is repeating the same pattern. When I get back from taking the son to work, the task should be about an hour and a half into its run, I'm going to shut Boinc down and restart it to see how the task responds.

Profile noderaser
Project donor
Avatar
Send message
Joined: 28 Mar 06
Posts: 507
Credit: 1,549,461
RAC: 95
Message 15203 - Posted: 12 Aug 2017, 2:09:32 UTC

Unfortunately, the progress estimator doesn't work so both the percentage and remaining time estimations are complete rubbish. Check-pointing is also not supported at this time, due to limitations in the software.
____________

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15204 - Posted: 12 Aug 2017, 5:12:52 UTC
Last modified: 12 Aug 2017, 5:15:20 UTC

Looking at the readme.html file in the slots directory tonight, followed its links to the blender website. The blender manual states that blender has an autosave option and crash recovery and restart options using autosave. So theoretically, it would just involve changing the burp 5.09 wrapper to pass autosave timer settings to the blender process. The wrapper could then poll the autosave directory and verify that the save was made and report this back to Boinc as having checkpointed. And since the wrapper records in its stderr.txt file that Boinc requested it to quit, it should be able to read this at a restart and have blender resume from the autosave file.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4478
Credit: 2,094,806
RAC: 0
Message 15205 - Posted: 12 Aug 2017, 15:37:22 UTC - in response to Message 15204.

Could you elaborate a bit? I'm not sure that feature does what you think it does.

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15210 - Posted: 12 Aug 2017, 18:14:36 UTC - in response to Message 15205.
Last modified: 12 Aug 2017, 18:33:44 UTC

From the readme.html file found in the slot directory, I clicked on the link for the Blender user manual:

https://docs.blender.org/manual/en/dev/

Now I haven't read all of it, in fact just a very little. The manual is for the entire Blender stand-alone package, but since Burp uses the same core program, theoretically it should be a matter of having the Burp wrapper program feed the core program the correct settings, THEORETICALLY. From that page I went to the user preferences page and clicked on the Open User Preferences and on that page clicked on the File link.

https://docs.blender.org/manual/en/dev/preferences/file.html

Scrolling down the page you'll come to the Auto Save option description and a link to a deeper discussion of it.

Blender appears to written in Python and uses a lot of scripts for controlling it, as described on this File page. Blender's user interface appears to be like Boinc Manager, just a gui for taking input from the user and feeding it to the Blender client program. Take output from the Blender client and provide it to user in a useful manner.

So, again theoretically, it looks like a matter of using the correct Python scripts to have Blender perform autosaves, and graceful shutdowns, and restarts. THEORETICALLY.
____________

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4478
Credit: 2,094,806
RAC: 0
Message 15211 - Posted: 12 Aug 2017, 20:10:18 UTC
Last modified: 12 Aug 2017, 20:13:40 UTC

Still not entirely sure I follow. Please correct me if I got it wrong

The autosave file is just a copy of the scene file. Like autosave in Word, Blender saves a copy every now and then - it doesn't include the rendering data (which is typically many gigabytes of data). So restoring from the autosave file is exactly the same as restoring from the scene file: it starts over from 0%.

Blender appears to written in Python and uses a lot of scripts for controlling it

Yes, a fairly large portion of the UI and some of the underlying code is written in Python, the rest is C++. Python is actually also how the current client sets up the rendering for BURP, we inject a small python script into the scene that controls how Blender renders it. The python script connects to Glue3 which in turn talks to BOINC.
Unfortunately there are limits to how much control you have from Python in Blender - and saving and restoring the entire render state is not currently an option that I know of. Especially so if it is to be done in a way that works with both Blender Internal renderer and the Cycles renderer.

That said, you're right to some extent. It is theoretically possible to get some sessions to suspend/resume with python. Unfortunately it is the same kind of sessions that could simply be split into more pieces from the beginning, so not much effort was put into pursuing that path any further.

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15212 - Posted: 12 Aug 2017, 22:22:01 UTC - in response to Message 15211.
Last modified: 12 Aug 2017, 22:38:19 UTC

Understand that I have never used Blender, and what I am saying is just from reading a couple of pages in the manual in the last 24hrs.

From the manual File page:

Auto Save

Keep Session
Always saves the blend-file after quitting Blender and reloads it after re-starting Blender.

Auto Save Temporary Files
Enable Auto Save (create a temporary file).
Checking this box tells Blender to automatically save a backup copy of your work-in-progress to the Temp directory (refer to the File tab in the User Preferences for its location).
The Auto Saved files are named using a random number and have a blend extension.

Timer
Time to wait between automatic saves.
This specifies the number of minutes between each Auto Save. The default value of the Blender installation is 5 (5 minutes). The minimum is 1, and the Maximum is 60 (Save at every one hour).


So what is the blend-file? It appears to me that it is the finished output that Blender is creating.

What are the Temporary files? Are they the input files that are being blended?

The timer description sure sounds like what we call a checkpoint in a Boinc task.

From the Recovering Data page: https://docs.blender.org/manual/en/dev/troubleshooting/recover.html#troubleshooting-file-recovery

At the System Level (relating to Files)

There are options to save your files like Auto Save that saves your file automatically over time, and Save on Quit, which saves your blend-file automatically when you exit Blender.

Note

In addition to these functions being enabled by default, the Save on Quit functionality cannot be disabled.


So if Boinc were to ask Blender to quit properly, it should be able to save everything, and resume when restarted. The resumption should be at the point in blending that was reached when the quit command was issued. It would make no sense to have this feature if resumption of the blend has to resume from the very beginning.

Save and Auto Save

Computer crashes, power outages, or simply forgetting to save can result in the loss or corruption of your work. To reduce the chance of losing files when those events occur, Blender can use an Autosave function. The File tab of the User Preferences allows you to configure the two ways that Blender provides for you to regress to a previous version of your work.


This paragraph implies again that Blender can save a blend in progress and resume it. What is the mechanism that the Blender gui uses to send commands to the Blender client?

Recovering Auto Saves

Recover Last Session

File ‣ Recover Last Session will open the quit.blend that is saved into the Temporary Directory when you exit Blender. Note that files in your Temporary Directory may be deleted when you reboot (depending on your system configuration).


Session sure sounds like all of the input files to be blended, along with the output blend-file.

Recover Auto Save

File ‣ Recover Auto Save… allows you to open the Auto Saved file. After loading the Auto Saved version, you may save it over the current file in your working directory as a normal blend-file.

Important

When recovering an Auto Saved file, you will lose any changes made since the last Auto Save was performed. Only one Auto Saved file exists for each project (i.e. Blender does not keep older versions. Hence, you will not be able to go back more than a few minutes with this tool).


From these two pages of the manual, it sure sounds like Blender.exe is capable of checkpointing, shutting down gracefully, and resuming. It seems to be a matter of how to give it the appropriate commands.
____________

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4478
Credit: 2,094,806
RAC: 0
Message 15214 - Posted: 13 Aug 2017, 6:25:13 UTC
Last modified: 13 Aug 2017, 6:26:37 UTC

A .blend-file is a scene description - it usually contains no information from the rendered output.

Think of it like an origami instruction; it contains information on how to fold a piece of paper into a beautiful bird but if something goes wrong right now we have to restart with a new piece of paper because there is no way to keep track of progress on the old one. The instructions, on the other hand, haven't changed and can be re-used.
What we want to do is to be able to put the partially built paper bird away and continue on it later - but there is no way to do that that I know of yet.

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15215 - Posted: 14 Aug 2017, 14:51:27 UTC - in response to Message 15214.
Last modified: 14 Aug 2017, 15:01:23 UTC

Ok, so what you saying is that on my processor running 3391 on two cores takes approximately 9.6 hours. If 3391 was run in the full standalone Blender would it also take 9.6 hours? If it does, and the system suffers a crash at say 7 hours into the task, when the system is restarted and standalone Blender resumes, are the 7 hours of processing time lost and does the frame processing restart from zero?
____________

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15216 - Posted: 14 Aug 2017, 15:00:21 UTC
Last modified: 14 Aug 2017, 15:04:09 UTC

And now for something completely different,

Time to complete a task is inversely proportional to the number of cores used.

T = k(1/#cores) based on the average of three units of 3391 on two cores, k = 19.3

For three cores, estimated time is 6.4hrs to run, average of two tasks with three cores turned out to be 6.3 hrs.

For four cores, estimated time is 4.8hrs, will find out how close the predictions is in about an hour. From watching the Progress indicator, it looks like the prediction will be close.

What's strange is that the more cores you throw at the task, the less credit you get for the task. One would think that the more resources you give a task, would increase the credit given.
____________

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15218 - Posted: 14 Aug 2017, 16:38:16 UTC - in response to Message 15216.

So the four core prediction was 4.8hrs, and the actual time was 4.84hrs.

However, the credits are inconsistent as always:

3 Tasks on two cores averaged 1146.8
2 Tasks on three cores averaged 666.6
1 Task on four cores got 1820.53

Inconsistent, of course CreditNew has always been. But who crunches just for credits? I just hate the inconsistency.
____________

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4478
Credit: 2,094,806
RAC: 0
Message 15219 - Posted: 14 Aug 2017, 16:49:18 UTC - in response to Message 15215.

If 3391 was run in the full standalone Blender would it also take 9.6 hours?

Usually it would be a bit slower because standalone Blender also launches the UI. Right now it may actually be a bit faster in standalone than on the farm but that will be fixed with the next release of our client where we are once again back to being slightly faster than the standalone Blender with UI.

the system suffers a crash at say 7 hours into the task, when the system is restarted and standalone Blender resumes, are the 7 hours of processing time lost and does the frame processing restart from zero?

Correct. At least in general.
If you had made any changes to the scene without saving them before starting the render you would be able to recover those, but not the rendered data, in the autosave directory.

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15223 - Posted: 14 Aug 2017, 19:15:05 UTC - in response to Message 15219.
Last modified: 14 Aug 2017, 19:22:47 UTC

the system suffers a crash at say 7 hours into the task, when the system is restarted and standalone Blender resumes, are the 7 hours of processing time lost and does the frame processing restart from zero?

Correct. At least in general.
If you had made any changes to the scene without saving them before starting the render you would be able to recover those, but not the rendered data, in the autosave directory.


So in the 2.78 manual when they say:
Recover Auto Save

File ‣ Recover Auto Save… allows you to open the Auto Saved file. After loading the Auto Saved version, you may save it over the current file in your working directory as a normal blend-file.

Important

When recovering an Auto Saved file, you will lose any changes made since the last Auto Save was performed. Only one Auto Saved file exists for each project (i.e. Blender does not keep older versions. Hence, you will not be able to go back more than a few minutes with this tool).


The few minutes of going back only refers to changes you've made in the scene, and not to the actual rendering of the scene?

Do they have any plans to add saving of a partial rendering?
____________

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4478
Credit: 2,094,806
RAC: 0
Message 15224 - Posted: 15 Aug 2017, 15:05:14 UTC
Last modified: 15 Aug 2017, 15:05:59 UTC

The few minutes of going back only refers to changes you've made in the scene, and not to the actual rendering of the scene?

Yup

Do they have any plans to add saving of a partial rendering?

It is becoming increasingly difficult to do this for arbitrary checkpoints and arbitrary renderers (both Windows and Linux are moving away from a process system where we can freeze and restore entire processes). The amount of data involved is fairly large too - since we aren't just talking about images but also the entire state of the renderer. We may see checkpoints for layers at some point using the "Save Buffers" feature, though - this would be useful for very complex scenes with many layers.

So to answer your question: Essentially no. No-one is currently working on pause/resume-to-disk for Blender renderers. There are other renderers out there that support it, though.

Darrell
Avatar
Send message
Joined: 24 Mar 05
Posts: 16
Credit: 36,510
RAC: 22
Message 15225 - Posted: 15 Aug 2017, 16:09:25 UTC - in response to Message 15224.
Last modified: 15 Aug 2017, 16:12:41 UTC


Do they have any plans to add saving of a partial rendering?

It is becoming increasingly difficult to do this for arbitrary checkpoints and arbitrary renderers (both Windows and Linux are moving away from a process system where we can freeze and restore entire processes). The amount of data involved is fairly large too - since we aren't just talking about images but also the entire state of the renderer. We may see checkpoints for layers at some point using the "Save Buffers" feature, though - this would be useful for very complex scenes with many layers.

So to answer your question: Essentially no. No-one is currently working on pause/resume-to-disk for Blender renderers. There are other renderers out there that support it, though.


One would think that if other renderers support it, and since it would be a good feature to have on a computer, especially with the advent of Windows 10 where you have little control over when it is going to do an update that could potentially cause a problem with the program, one would think it would become a high priority feature to implement.

On another note, in the stderr.txt that each tasks returns there is a line that says the wrapper should check to see if the GPU should be used and of course it says no. The question would be if yes, what kind of processing on the GPU has Blender implemented?
____________

1 · 2 · Next
Post to thread

Message boards : Problems and Help : Session 3391