wiki:JobQueueRewrite

Task Rewrite of JobQueue?

This is a list of goals for the rewrite of the JobQueue?.

MythSystem?

  • Expand myth_system to a full featured class, to replace existing uses of myth_system, popen, and QProcess.
  • Support both direct execution, and execution through bourne shell.
  • Support input and output pipes, as well as buffered output (for 64KB pipe limit) managed by the reaper.
  • Allow independent timeouts for wait call and automatic termination.
  • (optional) Expand reaper to pull CPU usage.
  • (optional) Allow CPU time limits on processes in addition to real time limits.

JobQueue?

  • Move user job definition out of settings table and into dedicated table, to allow an arbitrary amount of user jobs.
  • Allow per-machine, per-command time constraints for jobs to run between, with the option to forcefully terminate when the run window has closed.
  • Remove special handling of transcode and commflag jobs. Each transcode profile would be given an independent command, and any custom status updates should be performed by the external command.
  • Set master backend as a job scheduler, doling out tasks to available connected machines. The existing system is effectively a 'free-for-all'.
  • (optional) Migrate mythfilldatabase and any other external 'housekeeping' processes to the jobqueue.
  • (optional) Improve support for use of jobqueue in MythVideo?, and other non-recording related tasks.
  • (optional) Allow maximum CPU time limits, and termination of idle tasks, to prevent deadlocked tasks.
  • (optional) Allow automatic scaling of concurrent jobqueue tasks. Preferably this would run properly without configuration, and can track both CPU and IO usage.
  • (optional) Force all communication to happen over the backend protocol, removing the need for database access for child queues, to facilitate a future move to an embedded database.
  • (optional) Break the jobqueue out of the backend completely in favor of mythjobqueue. The master backend would subsequently exist only as a job scheduler.
  • (very optional) Create repository for job scripts, similar to the new theme downloader.
Last modified 8 years ago Last modified on Oct 29, 2010, 1:55:32 AM