RingBuffer/ThreadedFileWriter + fadvise

[danielk]

These modifications should allow these classes to interact better with Linux disk caching and disk IO schedulers. However they need to be tested, and may need to be modified a bit to work well with the ffmpeg's read patterns.

[danielk]

Initial patch

[danielk]

Synced to trunk

[danielk]

This adds ioprio support (untested)

[danielk]

The 9150-ioprio-v1.patch can be applied with or without 9150-v2.patch. The problem it is intended to solve is a stutter you can get while mythcommflag is consuming copious amounts of disk I/O and you are using the CFQ I/O scheduler. This doesn't happen if you use a more MythTV appropriate disk I/O scheduler like deadline, but since CFQ is the default this is worth addressing. With 9150-v2 applied this doesn't happen for me with local disks or with myth streaming from a locally visible disk, but it still happens with NFS shares.

The 9150-ioprio patch is untested, but I've tested the ioprio concept via ionice and it works. I just can't test it right now without affecting WAF. (I'm also setting the process priority and ionice might be setting the process group priority, I didn't look at their code. Or I may have made a typo or something like that.)

[danielk]

(In [27257]) Refs #9150. Reduces I/O priority of background video processing tasks, mythcommflag (when not run with --cpuhog) in particular. This allows MythTV to behave better with the CFQ I/O scheduler. The CFQ I/O Scheduler isn't ideal for the MythTV use case, but it is the Linux default so we should try our best to work well with it.

[danielk]

(In [27285]) Fixes #9190. This tweaks file writing and reading a bit to better interact with modern VM and disk i/o schedulers.

To explain... First, we were sometimes using the wrong syscall to sync data to disk in the TFW. The sync_file_range call only syncs allocated blocks, which is often a no-op when appending to a file. It is useful for database type workloads, but what we really want is fdatasync(). Second, we were only using posix_fadvise to tell the OS what we wanted it to cache (some of the time), but we were not using it to hint to the OS what data we won't need in the near future. I've added the hinting for the seek to end optimization and I've added the calls to tell the OS what data we don't need it to cache. The later is important for Linux in particular as it will penalize the next application that requests memory for our sins in keeping unneeded disk buffers in memory after we're done with them.

This is based on Ingo Molnar's comments in the "The State of Linux IO Scheduling For the Desktop" story on slashdot.org 2010-10-24, I've run it for several weeks without any problems. It does not have as great a positive effect as I had hoped, but it does provide some benefit. This may provide more benefit on machines with less RAM.

[danielk]

(In [27286]) Fixes #9252. Refs #9150. Fix compile where fadvise is not supported.

[anonymous]

27285 breaks playback for me would upload a sample but if the file gets shortened it plays for some reason so here is the logs from when file was shortened and fullsize using -v playback,extra,file

file shortened with dd to a 25mb sample log that works ​http://pastebin.com/M667B8KE

full file log that doesn't work ​http://pastebin.com/vD7ctp71

[danielk]

Anon, does changing "off_end <= 250000" to "off_end == 250000", fix the problem for you?

I'm assuming this only affects mkv files as well, is that correct?

[anonymous]

yes it does fix the problem

most of my collection is mkv so i dont know if its only mkv that has this issue there is atleast 1 blueray 720p mkv rip that works fine without the change(2GB) there was a fairly large(1.8GB) mp4 that worked fine too i can look through my collection for some more non mkv files to test if you need it

[anonymous]

ok i found some smaller mp4 files that suffer from it too and produce log output of ~2mb with alot of

2010-11-26 18:43:43.973 [h264 @ 0x7fe00cfbef20]AVC: nal size <number>
2010-11-26 18:43:43.973 [h264 @ 0x7fe00cfbef20]no frame!
2010-11-26 18:43:44.215 AFD Error: Unknown decoding error

changing that if statement fixes those files too

[danielk]

(In [27379]) Fixes #9150. Reverts a small part of [27285] and adds comment explaining why we only apply that optimization in one very narrow case.