Opened 10 years ago

Closed 10 years ago

#9150 closed enhancement (Fixed)

RingBuffer/ThreadedFileWriter + fadvise

Reported by: danielk Owned by: danielk
Priority: minor Milestone: 0.25
Component: MythTV - General Version: Master Head
Severity: medium Keywords:
Cc: Ticket locked: no


These modifications should allow these classes to interact better with Linux disk caching and disk IO schedulers. However they need to be tested, and may need to be modified a bit to work well with the ffmpeg's read patterns.

Attachments (3)

9150-v1.patch (7.7 KB) - added by danielk 10 years ago.
Initial patch
9150-v2.patch (8.2 KB) - added by danielk 10 years ago.
Synced to trunk
9150-ioprio-v1.patch (5.2 KB) - added by danielk 10 years ago.
This adds ioprio support (untested)

Download all attachments as: .zip

Change History (15)

Changed 10 years ago by danielk

Attachment: 9150-v1.patch added

Initial patch

comment:1 Changed 10 years ago by danielk

Status: newaccepted
Type: defectenhancement

Changed 10 years ago by danielk

Attachment: 9150-v2.patch added

Synced to trunk

Changed 10 years ago by danielk

Attachment: 9150-ioprio-v1.patch added

This adds ioprio support (untested)

comment:2 Changed 10 years ago by danielk

The 9150-ioprio-v1.patch can be applied with or without 9150-v2.patch. The problem it is intended to solve is a stutter you can get while mythcommflag is consuming copious amounts of disk I/O and you are using the CFQ I/O scheduler. This doesn't happen if you use a more MythTV appropriate disk I/O scheduler like deadline, but since CFQ is the default this is worth addressing. With 9150-v2 applied this doesn't happen for me with local disks or with myth streaming from a locally visible disk, but it still happens with NFS shares.

The 9150-ioprio patch is untested, but I've tested the ioprio concept via ionice and it works. I just can't test it right now without affecting WAF. (I'm also setting the process priority and ionice might be setting the process group priority, I didn't look at their code. Or I may have made a typo or something like that.)

comment:3 Changed 10 years ago by danielk

(In [27257]) Refs #9150. Reduces I/O priority of background video processing tasks, mythcommflag (when not run with --cpuhog) in particular. This allows MythTV to behave better with the CFQ I/O scheduler. The CFQ I/O Scheduler isn't ideal for the MythTV use case, but it is the Linux default so we should try our best to work well with it.

comment:4 Changed 10 years ago by danielk

Resolution: Fixed
Status: acceptedclosed

(In [27285]) Fixes #9190. This tweaks file writing and reading a bit to better interact with modern VM and disk i/o schedulers.

To explain... First, we were sometimes using the wrong syscall to sync data to disk in the TFW. The sync_file_range call only syncs allocated blocks, which is often a no-op when appending to a file. It is useful for database type workloads, but what we really want is fdatasync(). Second, we were only using posix_fadvise to tell the OS what we wanted it to cache (some of the time), but we were not using it to hint to the OS what data we won't need in the near future. I've added the hinting for the seek to end optimization and I've added the calls to tell the OS what data we don't need it to cache. The later is important for Linux in particular as it will penalize the next application that requests memory for our sins in keeping unneeded disk buffers in memory after we're done with them.

This is based on Ingo Molnar's comments in the "The State of Linux IO Scheduling For the Desktop" story on 2010-10-24, I've run it for several weeks without any problems. It does not have as great a positive effect as I had hoped, but it does provide some benefit. This may provide more benefit on machines with less RAM.

comment:5 Changed 10 years ago by danielk

(In [27286]) Fixes #9252. Refs #9150. Fix compile where fadvise is not supported.

comment:6 Changed 10 years ago by anonymous

27285 breaks playback for me would upload a sample but if the file gets shortened it plays for some reason so here is the logs from when file was shortened and fullsize using -v playback,extra,file

file shortened with dd to a 25mb sample log that works

full file log that doesn't work

comment:7 Changed 10 years ago by danielk

Anon, does changing "off_end <= 250000" to "off_end == 250000", fix the problem for you?

I'm assuming this only affects mkv files as well, is that correct?

comment:8 Changed 10 years ago by anonymous

yes it does fix the problem

most of my collection is mkv so i dont know if its only mkv that has this issue there is atleast 1 blueray 720p mkv rip that works fine without the change(2GB) there was a fairly large(1.8GB) mp4 that worked fine too i can look through my collection for some more non mkv files to test if you need it

comment:9 Changed 10 years ago by anonymous

ok i found some smaller mp4 files that suffer from it too and produce log output of ~2mb with alot of

2010-11-26 18:43:43.973 [h264 @ 0x7fe00cfbef20]AVC: nal size <number>
2010-11-26 18:43:43.973 [h264 @ 0x7fe00cfbef20]no frame!
2010-11-26 18:43:44.215 AFD Error: Unknown decoding error

changing that if statement fixes those files too

comment:10 Changed 10 years ago by robertm

Resolution: Fixed
Status: closednew

comment:11 Changed 10 years ago by robertm

Status: newassigned

comment:12 Changed 10 years ago by danielk

Resolution: Fixed
Status: assignedclosed

(In [27379]) Fixes #9150. Reverts a small part of [27285] and adds comment explaining why we only apply that optimization in one very narrow case.

Note: See TracTickets for help on using tickets.