Deadline - Event Plugins
We have a number of custom Event Plugins on our render farm; these are mostly quality of life improvements to keep the farm running smoothly. Here's an overview of the current plugins and what they do.
EnergyCostCalculator
This plugin calculates the estimated energy cost of a job after it finishes.
It does this with a simple calculation: the render time of all tasks of a job × the cost per render hour.
You can set that cost in Tools > Configure Event Plugins > EnergyCostCalculator.
The total goes into Extra Info 0, the cost per frame into Extra Info 1. Turn those columns on in the Monitor to see them.
Is a job suspended, or does it fail? Then it gets the cost of the frames that were already finished. With Scripts > CalculateEnergyCosts you can also calculate it by hand for older jobs.
It's not a problem if a job is expensive, the render farm is there to be used; but as an artist it's good to be aware that renders aren't free.
Originally written by Mervin.
ErrorTriage
This plugin looks at every error of a job to figure out what's going on, so people can quickly see what's wrong with their job, and a broken job doesn't keep rendering forever.
-Do several PCs get exactly the same error (frame numbers and paths don't count)? Then it's your scene, and the job is set to failed. Fix your scene and submit again.
-Is it the PC (for example "unable to find Houdini on the machine")? Then only that PC is blocked for your job; the rest keeps rendering.
-Does a PC run out of memory? Then that PC is taken off your job. If that happens on 3 PCs, your job is too heavy and it halves the number of tasks per PC by itself (10 → 5 → 3 → 2 → 1).
You can see what it did at the start of the job comment, prefixed with [ErrorTriage].
Originally written by Robbert.
FrameValidator
This plugin checks after a job whether all your frames are really there.
A frame is "bad" when the file is missing, or when it's much smaller than the frames around it (for example a half-written EXR). Fades and the like are not flagged as bad.
Bad frames are automatically re-rendered once. For big tasks, like simulations or a whole sequence in one task, it only reports them, because re-rendering would take hours.
Is only your last frame missing, but is there a file one frame before your first frame? Then your scene numbers its output one frame too low. It reports this, but re-rendering won't fix it; check your time offset.
The bad frames are listed at the start of the job comment, prefixed with [FrameValidator]. Movies (.mov, .mp4) are skipped.
Originally written by Robbert.
HealthCheck
Every render PC checks itself 4× a day (8:00, 12:00, 16:00 and 20:00) for things that can break renders: environment variables, pipeline version, whether Nuke/Houdini/Maya/Blender are installed, the GPU, the network drives, free disk space and how long it's been since the PC was last restarted.
If everything is fine, you won't hear from it. If something is wrong, it goes at the start of the Worker comment, prefixed with [HealthCheck date time], and it disappears again once it's fixed.
Want to check right now? Select Workers, right-click > Scripts > HealthCheckNow.
This is mainly useful for us admins; as an artist you don't need to do anything with it, but if you see a [HealthCheck] comment on a PC, you know why it's acting up.
Originally written by Robbert.
LicenseLimiter
Some submitters, like our Nuke submitter, send along how many render licenses a job may use.
Some of our nodes (for example ReduceNoise from NeatVideo) only have a limited number of floating licenses; if every machine on our farm renders frames with that node, you get errors. On submission, this plugin sets the job's Machine Limit to that number, so no more machines render a job than we have licenses.
If you had already set a stricter Machine Limit yourself, that one stays. Jobs without this number are left unchanged.
Originally written by Mervin.
LocalActivityOptOut
This lets you temporarily take a PC out of the render farm, for example when you're running caches on it.
Fill in something in the Worker Extra Info "No rendering - Until morning" or "No rendering - Permanent", for example your name or a reason. As long as something is in there, that PC won't pick up any jobs.
"Until morning" is cleared automatically every day after 9:00; "Permanent" stays until someone removes it. Clear it = the PC renders along again.
Use this fairly; we want as many machines as possible to keep rendering.
Originally written by Mervin.
NightlyWorkerRestart
Every evening from 21:00, every Worker gets restarted.
A Worker that's idle restarts right away; a Worker that's rendering only restarts once its current task is done, so your render isn't lost. Workers that are offline or stuck (stalled) are started up again.
This way the PCs pick up pipeline updates and the farm renders at full power at night.
Only PCs that are switched on and running the Deadline Launcher can be reached.
Originally written by Robbert.
NukeCleanup
Nuke first writes each frame to a temporary .tmp file (something like -935512534559029538.tmp) and renames it when the frame is done. If a render is interrupted, those files are left behind in your render folder.
This plugin cleans them up after a Nuke job. It only removes files with Nuke's own naming, and only if they haven't been touched for 30 minutes. So files from a render that's still running are left alone.
This is our modified, safer version of Deadline's standard NukeCleanup plugin.
Originally written by Robbert.