[Solved] Exception caught: Worker application apparently died prematurely


Advanced search

Message boards : Problems and Help : [Solved] Exception caught: Worker application apparently died prematurely

Author Message
111aaa
Send message
Joined: 18 May 16
Posts: 8
Credit: 1,137,690
RAC: 0
Message 14559 - Posted: 14 Jun 2016, 11:30:13 UTC

Hi all,
I have had a lot of failed units / errors. I have searched for the error and see a few people had it some years ago but no real definitive answer. Anyone know the cause or fix please?

These are the last few lines of the text :

|CUDA cuInit: Unknown error

|Error: EXCEPTION_ACCESS_VIOLATION


---------------------------
Exception caught: Worker application apparently died prematurely
Status: -9
---------------------------
Forcibly stopping worker threadCalling exit()...


Thanks



Task 9571194

Name ses0000003099frm0000000049prt00038_1
Workunit 2825457
Created 14 Jun 2016, 9:28:46 UTC
Sent 14 Jun 2016, 9:37:54 UTC
Report deadline 18 Jun 2016, 14:57:54 UTC
Received 14 Jun 2016, 9:43:15 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -9 (0xfffffffffffffff7) Unknown error number
Computer ID 70984
Run time 3 min 21 sec
CPU time 6 min 34 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 30.42 GFLOPS
Application version Blender (Windows) v5.04 (mt)
Peak working set size 5,551.58 MB
Peak swap size 6,401.36 MB
Peak disk usage 779.10 MB
Stderr output
<core_client_version>7.6.22</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code -9 (0xfffffff7)
</message>
<stderr_txt>
boinc_init_diagnostics() completed
boinc_init_options() completed
boinc_get_init_data() completed
CPU performance profile completed: 3802714444.336989 fpops, 10989575049.108023 iops reported. p_c is 1465739703.936752
Checking if GPU should be enabled...
No, using CPU
Mapping logical files to physical destinations:
in => in
out.zip => ../../projects/burp.renderfarming.net/ses0000003099frm0000000049prt00038_1_0
./windows_zip.exe => ./windows_zip.exe
./windows_unzip.exe => ./windows_unzip.exe
Project Directory Base => C:\ProgramData\BOINC/projects/burp.renderfarming.net
Unpacking archives:
blender_5.04_windows_x86_64__mt.zip => blender_5.04_windows_x86_64__mt.zip
./windows_unzip.exe -o -d "." blender_5.04_windows_x86_64__mt.zip...done
Creating worker...
Worker constructing...
Worker constructed.
$Id: glue.cpp 1827 2014-08-02 13:28:01Z jbk $
$Id: BOINCHandler.cpp 1824 2014-07-29 12:27:55Z jbk $
$Id: Controller.cpp 1824 2014-07-29 12:27:55Z jbk $
$Id: ProgressMonitor.cpp 1278 2011-01-23 09:22:45Z jbk $
Executing blender.exe -noaudio --factory-startup -y -b in -P clirender.py -- -F PNG -t 8 -f 49 0.30353022 0.0 0.34820098 0.5
po_r aft0x2cb210
po_r aft0xb4
Created pipes
Child created.
Worker thread started
Worker thread monitor up.
|('Observer constructed',)

|('Python Main',)

Application reports 'Booted'
|('Reading args',)

|('Preparing scenes',)

|('Autodetected rendering engine: CYCLES',)

|('CPU rendering',)

|('Unpacking texture files',)

|found bundled python: C:\ProgramData\BOINC\slots\9\2.77\python

|read blend: C:\ProgramData\BOINC\slots\9\in

|Dependency cycle detected:

| Near gnome Skin depends on Far gnome Skin through Field Collision.

| Far gnome Skin depends on Near gnome Skin through Field Collision.

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\1.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Lysimachia punctata 'Alexander'_List_opas.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Alpha_Farn.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Grass density.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Ground displacement.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Leaves0118_6_S.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\leavesTextureNo6557_1024x768.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Rocks map.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Sky.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Stump house bump.jpg

|Info: Total files 10 | Changed 0 | Failed 0

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\1.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Lysimachia punctata 'Alexander'_List_opas.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Alpha_Farn.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Grass density.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Ground displacement.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Leaves0118_6_S.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\leavesTextureNo6557_1024x768.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Rocks map.png

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Sky.jpg

|Info: Saved packed file to: C:\ProgramData\BOINC\slots\9\textures\Stump house bump.jpg

|Info: Total files 10 | Changed 0 | Failed 0

|('Using cycles samples:', 400)

|('Estimating render properties',)

|('Renderer: ', 'CYCLES')

|('Samples: ', 400)

|('Additional: ', 110)

|('Total work: ', 155465.46875)

|('Scene parsing done',)

|('Cleaning old files',)

|('No need to delete out',)

|('No need to delete out.png',)

|('Launching Cycles Render',)

|CUDA cuInit: Unknown error

|Error: EXCEPTION_ACCESS_VIOLATION


---------------------------
Exception caught: Worker application apparently died prematurely
Status: -9
---------------------------
Forcibly stopping worker threadCalling exit()...

</stderr_txt>
]]>

111aaa
Send message
Joined: 18 May 16
Posts: 8
Credit: 1,137,690
RAC: 0
Message 14560 - Posted: 18 Jun 2016, 12:04:00 UTC - in response to Message 14559.

No one else had this problem?

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4487
Credit: 2,094,806
RAC: 0
Message 14561 - Posted: 18 Jun 2016, 19:37:59 UTC
Last modified: 18 Jun 2016, 19:47:30 UTC

I'm a little surprised to see this line "CUDA cuInit: Unknown error" in the CPU client, especially after it has gone through the two CPU startup phases:

CPU performance profile completed: 3802714444.336989 fpops, 10989575049.108023 iops reported. p_c is 1465739703.936752 Checking if GPU should be enabled... No, using CPU

and
|('CPU rendering',)

111aaa
Send message
Joined: 18 May 16
Posts: 8
Credit: 1,137,690
RAC: 0
Message 14562 - Posted: 19 Jun 2016, 6:34:07 UTC - in response to Message 14561.
Last modified: 19 Jun 2016, 6:39:51 UTC

Any ideas though? Occasionally one or two will complete ok, but mostly they fail. Tasks - http://burp.renderfarming.net/results.php?hostid=71165

Profile DoctorNow
Project donor
Avatar
Send message
Joined: 11 Apr 05
Posts: 392
Credit: 2,168,338
RAC: 3
Message 14563 - Posted: 19 Jun 2016, 9:10:47 UTC - in response to Message 14561.
Last modified: 19 Jun 2016, 9:13:05 UTC

@jossdwyer: Just a hunch, but since you have so many results with

|Error: EXCEPTION_ACCESS_VIOLATION

and this is a session with a lot of RAM usage it could be that you have bad memory.
Maybe you should run a program to check that.


Janus wrote:
I'm a little surprised to see this line "CUDA cuInit: Unknown error" in the CPU client

It doesn't look as if this line is in all of his logs, only a few, the access violation is in the vast majority though.
Moreover, there are also tasks which end with:
|Error: EXCEPTION_STACK_OVERFLOW

Don't know if this is related to the access violation, but maybe it's also session specific.
____________
Life is Science, and Science rules. To the universe and beyond
Proud member of BOINC@Heidelberg
My BOINC-Stats

111aaa
Send message
Joined: 18 May 16
Posts: 8
Credit: 1,137,690
RAC: 0
Message 14564 - Posted: 21 Jun 2016, 17:42:57 UTC - in response to Message 14563.

Cheers. Will check the RAM and report back.
Thanks

111aaa
Send message
Joined: 18 May 16
Posts: 8
Credit: 1,137,690
RAC: 0
Message 14566 - Posted: 22 Jun 2016, 8:29:23 UTC - in response to Message 14564.

@DoctorNow - Well, I changed out a stick of RAM and the last three tasks have completed and are pending validation so fingers crossed.
Thank you for your help.
I saw the problem came up a bit with a google search but no real answer.

If the tasks are validated as OK I will edit the title (if possible) as solved to hopefully assist others.

Thanks again.

https://burp.renderfarming.net/results.php?userid=297564

111aaa
Send message
Joined: 18 May 16
Posts: 8
Credit: 1,137,690
RAC: 0
Message 14567 - Posted: 22 Jun 2016, 11:42:45 UTC - in response to Message 14566.

Well, the dodgy RAM was definitely the issue - Well done and thanks for the assistance.
Back to BURPing...

111aaa
Send message
Joined: 18 May 16
Posts: 8
Credit: 1,137,690
RAC: 0
Message 14568 - Posted: 22 Jun 2016, 11:45:01 UTC

ADMIN - If possible - please mark this as SOLVED in the title to assist others
Thanks

Profile DoctorNow
Project donor
Avatar
Send message
Joined: 11 Apr 05
Posts: 392
Credit: 2,168,338
RAC: 3
Message 14569 - Posted: 22 Jun 2016, 11:49:32 UTC - in response to Message 14566.

If the tasks are validated as OK I will edit the title (if possible) as solved to hopefully assist others.

I think editing the title isn't necessary - and not possible anyway (only mods and admins can do that permanently).
Good to see that my hunch was correct. Sometimes the most unexplainable errors are hardware problems and with a bit logic solvable. ;-)
____________
Life is Science, and Science rules. To the universe and beyond
Proud member of BOINC@Heidelberg
My BOINC-Stats

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4487
Credit: 2,094,806
RAC: 0
Message 14570 - Posted: 22 Jun 2016, 16:46:15 UTC

Had a host once that would work perfectly as long as nothing used more than 8GB of memory - above that? Instant reboot. Turned out to be a tiny spec of dust that had gotten itself trapped between the motherboard connector and one of the memory pins. Took a little while to figure that one out.
Today, as soon as something acts odd, it gets a Memtest86 run before anything else.

Wensil
Send message
Joined: 2 Jul 16
Posts: 5
Credit: 3,544
RAC: 0
Message 14724 - Posted: 16 Sep 2016, 12:33:53 UTC - in response to Message 14570.

have searched for the error and see a few people had it some years ago but no real definitive answer.
____________
Regards,
Website - Visit here


Post to thread

Message boards : Problems and Help : [Solved] Exception caught: Worker application apparently died prematurely