At http://burp.renderfarming.net/status.php, a new Data Compression item appears showing some finished sessions. Is this a new feature or bug?

Feature, yay! A BARF milestone subproject.

Part of the running cost of BURP is storage. If we are to open up BURP and scale it up with a factor of 10x or 100x in the future then we need to be much more conscious about server-side storage. Currently, for a number of reasons, the strategy has been to store literally everything (including multiple copies of each frame rendered from each render node, both parts and the stitched frames etc.). That way when something goes wrong, it costs very little to recreate it from some deeper storage layer.
In recent years fewer and fewer things have gone wrong where the extra storage was needed, and when the custom Cycles validator project is done there is a very large portion of that kind of "almost but not quite duplicate" storage that can be freed up for other purposes.
In the future the hope is to make the service a lot more slim, so that it scales better without unnecessary waste of storage.

The session data that will be put in long-term storage is going to consist primarily of

• One canonical copy of each rendered, final frame
  
• An encoded preview video file with all the frames
  
• The original input file(s) used for the session
  
• Some logs, checksums and performance data
  

In that list the first item is typically the one using the most space. By using lossless compression it is possible to store the exact same data using less space. How much less differs a lot from session to session but it is typically between 5%-50%.

What you are seeing is the new CruncherService going through the backlog of all the old rendered sessions and performing two tasks: validating that the data is still correct and compressing rendered frames using lossless compression.
At the moment it has been granted very few CPU resources and it is running as a background service so it is running fairly slowly - but it will pick up in speed once the new server is in place. The process is fairly I/O heavy so we're keeping it on the server cluster instead of distributing it to the farm via BOINC.

Storage problems? I'll be some kind of cloud storage, say a network of mirrors, could be part of a solution there... :P

---

Click here to see My Detailed BOINC Stats

The initial tests from the past two weeks showed a number of issues with the compression system and how jobs were being scheduled across the server cluster. Most of the performance issues have been ironed out now.

One interesting issue was that the tests themselves were bugged: When a session is compressed the system carries out a test to see that it can decompress the compressed session and get the original data back. The original and decompressed data is compared using strict validation (a variant of our normal validator) before the original data is disconnected (discarded from active storage). However, it turned out that when a frame contained no colourful pixels it could be compressed with greyscale instead of RGB (saving even more space) but when the test tried to re-open the image it would incorrectly interpret the greyscale image as being different and mark the frame as failed.
It turned out to be related to how Java's built-in image reader maps gamma for different kinds of images. By-passing the gamma correction step and reading the values directly from the raw image not only fixed the issue but also had the interesting side-effect of making our normal validator (the one targeted at Blender's internal renderer) a bit faster too.

This data compression background task has had a few additional resources assigned to it and will continue at a slightly faster pace over the next few weeks. It will be interesting to see if it hits another snag or if it will just keep going.

Got the new compute node hardware up and running and the first task assigned to it has been to help with the data compression. We're down to around 1000 sessions in the compression queue until the service has cleared the backlog.

There is a small amount of sessions that have gotten stuck somehow, still trying to figure out exactly why.

Down to less than 400 sessions still waiting for compression now, processing has been slowed down a lot due to the summer heat.

Down to 300 sessions now.
Pausing this temporarily since there is currently a heat wave going on and it is getting pretty hot in the server room

Hi Janus,

The Data Compression process is clearly a computationally expensive task.
Is it possible to distribute the work via a BOINC application taking the load off your systems when the renderfarm is idle?

I am aware that process deals with massive amounts of data that may not be economically viable to distribute but I would be interested to hear your thoughts.

---

You are right, it is certainly a balance between CPU and network bandwidth, but I definitely wouldn't rule it out at some point. However, these kinds of tasks are never really high priority since it is mostly a matter of trading storage space for CPU cycles. It is fine to pause it for long periods of time then continue from where it left at a later point in time.

Right now there are no compression tasks running on the server so all the images are simply being queued up for later at some point.

Also, we're getting considerable computational capabilities in the server as an upgrade later this year. The hope is to be able to run Blender physics simulations for sessions on it; and then also these compression tasks when it is otherwise mostly idle.