automatic simultaneous jobs scaling

[oa@…]

The max number of simultaneous postprocessing jobs setting is problematic in that it either does not fully utilise available resources (at value equal to number of CPUs, a real-time commflag process will leave plenty of CPU idle) or will run too many jobs at once (at value greater to number of CPUs, transcodes compete for CPU time or (for lossless transcodes) hog I/O bandwidth and cause disk trashing). It would be better for the jobqueue processor to monitor for idle resources and start jobs when capacity is available. The attached patches will do that. This can entirely replace the maxjobs setting and thus simplify configuration.

[oa@…]

bugfix - process jobqueue stop and pause commands even when running max jobs

[oa@…]

dynamic simultaneous jobs - includes CPU monitoring for Linux only

[oa@…]

final part - eliminate the setting for number of simultaneous jobs

[oa@…]

after first job, start new ones only if idle time remains - replaces mythtv_jobqueue_2.patch

[oa@…]

dynamic jobqueue size patch, updated for 0.21-fixes

[anonymous]

Updated patch for 0.21 attached. Again, what this patch does:

• Add method for detecting system utilization, taking into account both user time CPU and I/O wait
• Allow job scheduler to start new background jobs as long as system has free resources
• Remove the "Maximum jobs" jobqueue setting as superfluous

[oa@…]

fwiw, I've rebuilt my main myth backend on a dual-core system a couple of months ago, and this patch works beautifully without any changes required. The system starts up to two transcode processes in addition to 3-5 recordings (on two DVB cards) and their respective real-time commflags, utilizing both cores to 99% capacity, 0% iowait, and no issues with simultaneous playback on the integrated ATI RS680 video.

In other words, it "just works". Anything further I could do to get this included?

[cpinkham]

Make automatic scaling optional

[cpinkham]

Can you take a look at the attached scaling_jobqueue_ctp_1.diff patch and see if it still has the desired effect on your server. I don't have time to test my changes thoroughly over the next few days, so you may have more time to run it through it's paces than I can.

I'd like to keep the max jobs setting for people who don't want the JobQueue? to consume all available CPU such as those running the JobQueue? on machines doing software encoding or on workstations doing other things. I moved some things around in the patch to accomplish this. Just to be safe, the code also falls back to a max of 1 job if the user sets the setting to 'Auto' but runs the JobQueue? on a machine that isn't supported by the getCPUStats() call.

[cpinkham]

Is anyone using the version of the patch I attached to this ticket last year?

[cpinkham]

I won't have time to fully test this prior to 0.22, so moving to 0.23.

[oa@…]

the original patch updated for 0.22

[oa@…]

Chris, your comments on this didn't reach me at the time. I haven't tested your patch, but back when I updated to 0.22, I forward-ported the original. It continues to work beautifully under all circumstances on my box.

I follow the principle of "make defaults work, don't introduce options", so personally, I think the part which removes the configuration for maxjobs is the most important bit of this patch.

[cpinkham]

Moving some things to 0.24 since we're almost at the 0.23 feature freeze.

[cpinkham]

Unsetting milestone on this. Some people (such as myself) only want to dedicate X number of cores on a multi-core box for running jobs, so the 'max' setting still needs to stay. Hopefully after 0.24 I'll have time to test this long-term, I would like to see the auto-scaling functionality added.

[stuartm]

Milestone 0.25 deleted

[cpinkham]

Closing this since the Raymond Wagner is currently working on a major rework of the JobQueue? which will include this functionality and have a true job scheduler.