Opened 12 years ago

Closed 11 years ago

Last modified 11 years ago

#5746 closed patch (fixed)

With multiple disk of different sizes, place new recording on partition which will have the lowest priority files deleted

Reported by: John.r.fulton@… Owned by: cpinkham
Priority: minor Milestone: unknown
Component: mythtv Version: 0.21-fixes
Severity: low Keywords: mutple disk, expire,
Cc: Rob Smith, stuartm Ticket locked: no

Description

When using multiple disks of different sizes the new recording is not always placed on the disk which has the lowest priority files to be expired.

This patch adds a new pass to the placement algorithm which simulates expiring the old files in the users required order. When a partion is found with enough space that partition is selected for the new file.

pass 1: As before - find any partition with enough space to fit the file pass 2: simulate expiration until a partition is found with enough space pass 3: (old pass 2) - find any partition that will take the file.

I've tested this patch for over a month (it takes that long) with 1 TB and 1 500 GB disk.

Attachments (2)

file_placement.patch (7.0 KB) - added by John.r.fulton@… 12 years ago.
patch to scheduler.cpp for mythbackend
file_placement.2.patch (8.0 KB) - added by John.R.Fulton@… 12 years ago.

Download all attachments as: .zip

Change History (9)

Changed 12 years ago by John.r.fulton@…

Attachment: file_placement.patch added

patch to scheduler.cpp for mythbackend

comment:1 Changed 12 years ago by Rob Smith

Cc: Rob Smith added

Changed 12 years ago by John.R.Fulton@…

Attachment: file_placement.2.patch added

comment:2 Changed 12 years ago by John.R.Fulton@…

Updated patch "file_placement.2.patch":

  • Made estimated placement size of files 1/3 bigger. This fixes a minor bug in all 3 passes where a file system may be selected as having enough free space but because the auto-expire algorithm assumes 1/3 more space over what is required and would expire a file.
  • When I file can not be found in pass 2 it is ignored instead of aborting.
  • verbosity changed to VB_FILE on some printouts and each pass indicated. Run mythbackend with "-v file" when reporting problems.
  • fixed memory leak: freed some memory allocated in pass 2 as a result of calling GetAllExpiring?

comment:3 Changed 12 years ago by ylee@…

I am pleased to report good preliminary results in the first couple of days of testing the newest version of John Fulton's patch. All recent "Expired program" entries in System Status|Log Entries are for either deleted or expire-after-x recordings, and the Deleted recording group has been whittled down to only a few entries. This is consistent with expectations for a setup, like mine, always running at maximum capacity.

comment:4 Changed 12 years ago by cpinkham

Owner: changed from Isaac Richards to cpinkham
Status: newassigned

This doesn't appear to take into account filesystems which are not accessible on the master backend, it only does a CheckFile?() looking for the file. That needs to be taken into account before it can be committed. I also would want this to be an option. I see 2 main balancing ideas behind Storage Groups. The current mode is to balance I/O load across filesystems, that takes precendence over disk space. This mode will make disk space and expirable recordings take precedence over I/O (weight) on a filesystem. I think that there should be an option to control these two modes. I think this second mode should set the starting weight of local filesystems to 0 and might want to sort the directories by free space first then by weight.

If you want to discuss further, reply on the mailing list. I'm all for adding a feature like this, but don't want it to take away from current "balance the I/O load" functionality for those that still prefer the current mode (like myself).

comment:5 Changed 11 years ago by stuartm

Cc: stuartm added

comment:6 Changed 11 years ago by cpinkham

Resolution: fixed
Status: assignedclosed

(In [21225]) Add some more intelligence to the Storage Scheduler.

When using the new "Balance Free Space" method, the Storage Scheduler will now perform an intermediate step if it can not find a directory with enough free space to hold the recording. Previously, the scheduler would go through the directory list again and put the recording on the first acceptable directory in the list and hope that something could be expired from that directory. The new middle pass will go through the directory list again looking at what can be expired from each directory to find out the best place to put the recording based on what recording(s) may have to be expired to make room for the new recording.

One warning.... The corner case of full drives and multiple recordings starting at once will still not be handled effectly by this 2nd pass. The 2nd recording will probably be put on the same filesystem as the first since "free space" is the deciding factor. It may be that the "free space"-only method may work better to use a fuzzy comparison and then take weight into account which would help balance out multiple recordings in this scenario.

This patch also does 2 things regarding the default value of the StorageScheduler? setting:

1) The default method on the setup screen is now 'Balanced Free Space'. 2) A DB upgrade inserts a setting into the database to set the method

to 'Combination' so that users upgrading from previous versions will see the same scheduling logic they have been seeing. Users can change the value in mythtv-setup anytime and restart the master backend for the new value to take effect.

Closes #5746 using modified patch originally by John R Fulton.

comment:7 Changed 11 years ago by cpinkham

(In [21227]) Missed one file in [21225].

Refs #5746.

Note: See TracTickets for help on using tickets.