Web frontend issues Aug 2011

Message boards : Server backend and mirrors : Web frontend issues Aug 2011
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 11014 - Posted: 9 Aug 2011, 18:27:07 UTC

Just a little heads-up that the website may be a little flaky this month.

Recently we switched backend from a standard Java installation with some custom code to a fancy Tomcat-based server with lots of nice improvements for admins. As part of this endeavour much code was moved around or changed slightly and it seems that some of the new code for handling EXR images is causing the server to leak a serious lot of memory and sometimes even crash in vivid bright flames of ones and zeros.
When this happens the site will appear to be very slow or will show errors like:
Notice: fwrite() [function.fwrite]: send of 5 bytes failed with errno=32 Broken pipe in /mnt/hosting/www/burp_production/libraries/php-java-bridge/Java.inc on line 831

Notice: Uninitialized string offset: 6 in /mnt/hosting/www/burp_production/libraries/php-java-bridge/Java.inc on line 787

Fatal error: An unchecked exception occured during script execution. Please check the server log files for details. in /mnt/hosting/www/burp_production/libraries/php-java-bridge/Java.inc on line 777

Usually this resolves itself in a minute or two, so just try again after a short while.

I'm working on isolating the memory leak but it appears to be outside of the BURP code and is related to the way that two of the external libraries that we use interact with each other when running on Tomcat. There's a possible fix but it will take a little while to implement, test and deploy here.

Another consequence is that the upload webform is offline until the issue has been properly resolved.
ID: 11014 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Janus
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 16 Jun 04
Posts: 4574
Credit: 2,100,463
RAC: 8
Message 11016 - Posted: 14 Aug 2011, 14:57:26 UTC

I've deployed a fix for this issue and things seem to be running much better now (see graph). Too much data was being logged to log files on the server which caused it to go bonkers when logging the unusually large data structures related to handling EXR images. These log messages would then queue up and eventually use all memory and I/O resources on the system, causing it to slow down and then crash.



This graph shows the system during a crash and the performance after applying the fix. The yellow line is I/O wait - the amount of time the system spends waiting for data to be read or written to disk. The green line is system performance measured in megabits per second with 100Mbps being the cap of this test (as you see on the right side when the fix was applied).

There may be more sources of the instability but this certainly was a big portion of it.
ID: 11016 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Server backend and mirrors : Web frontend issues Aug 2011