Linux Automatic Linking Feedback Thread


Advanced search

Message boards : Client : Linux Automatic Linking Feedback Thread

Author Message
Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4461
Credit: 2,094,806
RAC: 0
Message 14963 - Posted: 15 Dec 2016, 21:30:35 UTC
Last modified: 15 Dec 2016, 22:39:17 UTC

The latest version of the Linux client released Dec 16th contains an entirely new way to launch the Blender executable. This thread is for feedback about this feature.

For a while, now, we've had issues with Linux systems not having any/all of the dynamic link libraries required to render images. The solution so far has been to ship the libraries with the client but in some cases those libraries were incompatible with the rest of the system for some reason or another or had lower performance compared to native libraries already present on the system. One of the incompatibility issues was due to the use of a special loader for these dynamic libraries and for people who had all the right native libraries it was sometimes easier to just switch off the special loader via a file/folder called "native" in the project directory.
The new client does away with the special library loader all-together and for that reason no longer has the "native" file option. Instead what was previously "native" will be the default and only if some or all of the required native libraries are missing will the client switch to a fallback mode where it attempts to use the bundled libraries.

If you go to a workunit instance on your account and check the online log for one of the new client workunits it will have an completely new section in the log that deals with dynamic link libraries. This section should make it a lot easier to debug what the client is doing and what is going wrong when it is failing.
Here is an example where some native libraries are missing and the client switches to fallback-mode:

boinc_init_diagnostics() completed boinc_init_options() completed boinc_get_init_data() completed CPU performance profile completed: 4276435039.422476 fpops, 15252283756.845736 iops reported. p_c is 1480973420.532927 Checking if GPU should be enabled... No, using CPU Mapping logical files to physical destinations: in => in out.zip => out.zip ./linux_zip_x64_r600 => ./linux_zip_x64_r600 ./linux_unzip_x64_r600_static => ./linux_unzip_x64_r600_static Project Directory Base => Unpacking archives: blender_5.09_x86_64-pc-linux-gnu__mt.zip => blender_5.09_x86_64-pc-linux-gnu__mt.zip ./linux_unzip_x64_r600_static -o -d "." blender_5.09_x86_64-pc-linux-gnu__mt.zip...done CHMODing executable... Resolving native dynamic link libraries: linux-vdso.so.1 (0x00007ffdc1590000) librt.so.1 => /lib64/librt.so.1 (0x00007f45b23ca000) libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007f45b2122000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f45b1f07000) libGLU.so.1 => not found Missing library detected: Please install this library for optimum performance, falling back to internal libraries... libGL.so.1 => not found Missing library detected: Please install this library for optimum performance, falling back to internal libraries... libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007f45b1bc8000) libXi.so.6 => /usr/lib64/libXi.so.6 (0x00007f45b19b8000) libXxf86vm.so.1 => not found Missing library detected: Please install this library for optimum performance, falling back to internal libraries... libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007f45b17ae000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f45b15ab000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f45b13a7000) libc.so.6 => /lib64/libc.so.6 (0x00007f45b100f000) /lib64/ld-linux-x86-64.so.2 (0x00007f45b25d2000) libm.so.6 => /lib64/libm.so.6 (0x00007f45b0d0e000) libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.4/libgcc_s.so.1 (0x00007f45b0af8000) libz.so.1 => /lib64/libz.so.1 (0x00007f45b08e2000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f45b06d2000) libpng16.so.16 => /usr/lib64/libpng16.so.16 (0x00007f45b04a0000) libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007f45b0280000) libXext.so.6 => /usr/lib64/libXext.so.6 (0x00007f45b006e000) libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007f45afe6a000) libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x00007f45afc64000) Attempting to use slow fallback bundled dynamic link libraries Creating worker... Worker constructing... Worker constructed. |[-->] Info: |child <9333> to stdout |0 - ./blender |1 - -noaudio |2 - --factory-startup |3 - -y |('Observer constructed',) |('Python Main',) Application reports 'Booted' |('Reading args',) |("Preparing disk cache based on basedir",) |('Preparing scenes',) |('Autodetected rendering engine: CYCLES',) |('CPU rendering',) ...

The first few lines are identical to what the old client did: Connecting to BOINC, checking if we need to enable the GPU, mapping files and unpacking the Blender renderer.
The line about the special loader is gone and has been replaced by the new big section starting with "Resolving native dynamic link libraries...". In this section Glue3 analyzes the required dynamic link libraries and presents an overview of which ones it found and where. If any libraries are missing it will print a warning for that library and switch to fallback-mode.
Immediately following the dll analysis section is a line saying either that Glue3 will launch Blender in high-performance native mode or in slow fallback mode. The lines following are the same as in the old client and contain information from Blender and the rendering threads. The line saying "Application reports 'Booted'" signals the point where everything has been successfully loaded (either natively or with fallback libraries) and control is handed over to the next stage - the stage that performs the actual rendering.

If Blender is launched in fallback mode a set of bundled libraries will be used. Special care has been taken to select the libraries in such a way that they are compatible with as broad a range of modern 64-bit systems as possible. This, unfortunately, also means that the libraries may not contain all optimizations for the particular system that the rendering is going to be performed on - and as such they may be somewhat slower than what could be potentially be achieved by installing the corresponding libraries from the system package manager natively.

If the "Application reports 'Booted'"-line is missing there will typically be another few lines explaining an issue. It could be a problem with the native libraries or it could be a problem with the bundled ones - or something entirely different. Testing will show if this is something that actually happens.

If your Linux client used to work and stopped working please post about it - similarly if your client used to fail and started working now, give us a shout. Did you have to use the "native"-file before and now you don't have to? Let us know.
The hope is that this change should dramatically increase the number of hosts that "just work" and minimize the amount of effort and voodoo needed in order to get started rendering on Linux.

Profile Janus
Volunteer moderator
Project administrator
Avatar
Send message
Joined: 16 Jun 04
Posts: 4461
Credit: 2,094,806
RAC: 0
Message 14964 - Posted: 15 Dec 2016, 22:03:25 UTC
Last modified: 15 Dec 2016, 22:35:32 UTC

Technical Notes:

The library "linux-vdso.so.1", the very first one in the list in the example above, is treated specially since it doesn't actually reside anywhere in the file-system but is instead part of the Linux kernel. This library is always present and will always be loaded from the system kernel regardless of whether the client is in native or fallback mode.
Another special case is "/lib64/ld-linux-x86-64.so.2", which is the library that Glue3 uses to produce the list of libraries and to load them. If this library is missing, the entire list will be empty and Glue will be unable to launch Blender in any mode. This library is an essential part of 64-bit Linux userland and not having it should be an extremely rare situation (although not impossible). The library is heavily dependent on the specific version of GLIBC that the system was compiled with and is difficult to redistribute. For that reason this library will also always be loaded from the system, no bundled version is provided.

If even a single native library (apart from the exceptions above) is missing, ALL bundled libraries will be used instead of native ones. This doesn't mean that the client runs exclusively on bundled libraries, in contrast with the old client there are a set of basic libraries that are assumed to be present on most systems. Part of the evaluation of the new client will be to see if this is correct.

Senilix
Send message
Joined: 15 Oct 14
Posts: 11
Credit: 822,856
RAC: 0
Message 14974 - Posted: 28 Dec 2016, 14:52:54 UTC - in response to Message 14963.

Well, I guess we need some work units to test the new version with ...


Post to thread

Message boards : Client : Linux Automatic Linking Feedback Thread