Opened 17 years ago
Closed 13 years ago
#2782 closed enhancement (Won't Fix)
automatic simultaneous jobs scaling
Reported by: | Owned by: | cpinkham | |
---|---|---|---|
Priority: | minor | Milestone: | 0.25 |
Component: | MythTV - Mythjobqueue | Version: | head |
Severity: | medium | Keywords: | |
Cc: | Ticket locked: | no |
Description
The max number of simultaneous postprocessing jobs setting is problematic in that it either does not fully utilise available resources (at value equal to number of CPUs, a real-time commflag process will leave plenty of CPU idle) or will run too many jobs at once (at value greater to number of CPUs, transcodes compete for CPU time or (for lossless transcodes) hog I/O bandwidth and cause disk trashing). It would be better for the jobqueue processor to monitor for idle resources and start jobs when capacity is available. The attached patches will do that. This can entirely replace the maxjobs setting and thus simplify configuration.
Attachments (7)
Change History (21)
Changed 17 years ago by
Attachment: | mythtv_jobqueue_1.patch added |
---|
Changed 17 years ago by
Attachment: | mythtv_jobqueue_2.patch added |
---|
dynamic simultaneous jobs - includes CPU monitoring for Linux only
Changed 17 years ago by
Attachment: | mythtv_jobqueue_3.patch added |
---|
final part - eliminate the setting for number of simultaneous jobs
comment:1 Changed 17 years ago by
Owner: | changed from Isaac Richards to cpinkham |
---|
Changed 17 years ago by
Attachment: | jobqueue_dynamic.patch added |
---|
after first job, start new ones only if idle time remains - replaces mythtv_jobqueue_2.patch
comment:2 Changed 16 years ago by
Milestone: | unknown → 0.22 |
---|---|
Version: | 0.20 → head |
Changed 16 years ago by
Attachment: | dynamic_jobqueue-0.21.patch added |
---|
dynamic jobqueue size patch, updated for 0.21-fixes
comment:3 Changed 16 years ago by
Updated patch for 0.21 attached. Again, what this patch does:
- Add method for detecting system utilization, taking into account both user time CPU and I/O wait
- Allow job scheduler to start new background jobs as long as system has free resources
- Remove the "Maximum jobs" jobqueue setting as superfluous
comment:4 Changed 15 years ago by
fwiw, I've rebuilt my main myth backend on a dual-core system a couple of months ago, and this patch works beautifully without any changes required. The system starts up to two transcode processes in addition to 3-5 recordings (on two DVB cards) and their respective real-time commflags, utilizing both cores to 99% capacity, 0% iowait, and no issues with simultaneous playback on the integrated ATI RS680 video.
In other words, it "just works". Anything further I could do to get this included?
Changed 15 years ago by
Attachment: | scaling_jobqueue_ctp_1.diff added |
---|
Make automatic scaling optional
comment:5 Changed 15 years ago by
Status: | new → accepted |
---|
Can you take a look at the attached scaling_jobqueue_ctp_1.diff patch and see if it still has the desired effect on your server. I don't have time to test my changes thoroughly over the next few days, so you may have more time to run it through it's paces than I can.
I'd like to keep the max jobs setting for people who don't want the JobQueue? to consume all available CPU such as those running the JobQueue? on machines doing software encoding or on workstations doing other things. I moved some things around in the patch to accomplish this. Just to be safe, the code also falls back to a max of 1 job if the user sets the setting to 'Auto' but runs the JobQueue? on a machine that isn't supported by the getCPUStats() call.
comment:6 Changed 14 years ago by
Is anyone using the version of the patch I attached to this ticket last year?
comment:7 Changed 14 years ago by
Milestone: | 0.22 → 0.23 |
---|
I won't have time to fully test this prior to 0.22, so moving to 0.23.
comment:8 Changed 14 years ago by
Component: | mythtv → MythTV - Mythjobqueue |
---|
Changed 14 years ago by
Attachment: | dynamic_jobqueue-0.22.patch added |
---|
the original patch updated for 0.22
comment:9 Changed 14 years ago by
Chris, your comments on this didn't reach me at the time. I haven't tested your patch, but back when I updated to 0.22, I forward-ported the original. It continues to work beautifully under all circumstances on my box.
I follow the principle of "make defaults work, don't introduce options", so personally, I think the part which removes the configuration for maxjobs is the most important bit of this patch.
comment:10 Changed 14 years ago by
Milestone: | 0.23 → 0.24 |
---|
Moving some things to 0.24 since we're almost at the 0.23 feature freeze.
comment:11 Changed 13 years ago by
Milestone: | 0.24 → unknown |
---|---|
Status: | accepted → assigned |
Unsetting milestone on this. Some people (such as myself) only want to dedicate X number of cores on a multi-core box for running jobs, so the 'max' setting still needs to stay. Hopefully after 0.24 I'll have time to test this long-term, I would like to see the auto-scaling functionality added.
comment:12 Changed 13 years ago by
Milestone: | unknown → 0.25 |
---|
comment:14 Changed 13 years ago by
Resolution: | → Won't Fix |
---|---|
Status: | assigned → closed |
Closing this since the Raymond Wagner is currently working on a major rework of the JobQueue? which will include this functionality and have a true job scheduler.
bugfix - process jobqueue stop and pause commands even when running max jobs