Opened 12 years ago

Closed 4 years ago

#10793 closed Patch - Feature (Unverified)

CommFlagger improvements

Reported by: Mark Spieth Owned by: mspieth
Priority: minor Milestone: unknown
Component: MythTV - Mythcommflag Version: Unspecified
Severity: medium Keywords:
Cc: Ticket locked: no

Description (last modified by Mark Spieth)

This ticket is to share work on comm flagger improvements.

The comm flagger has been working worse for me over time so I decided to have a look at the algorithms.

The main components are:

  1. Command line method parsing fix. Was completely broken. See first patch
  2. Audio silence and change data fusion
  3. Scene changer param change. 0.85 was way too high to get useful scene cut info. Still not sure what the right value is. Debugging procedure to adventurers will follow.

2 and 3 are in the comm-detector-audio patch. This patch is completely independent of the myth libraries so that it is easier to test. There are some enum changes that will eventually make it into libmyth.
Only the all method is implemented.
Audio power min and max dont appear to add anything useful and will probably be removed.

It seems to work better so far for the few Ive tested.

DB Settings to play with (new and old)

Things to do:

  1. Subtitle info fusion
    This should help with segment classification
  2. Logo detector is sporadic. Need to find out why. Some programs (same channel) can find the logo but other can.

Test with

DEBUGCOMMFLAG=1 nice -19 ./mythcommflag --chanid $CHAN --starttime $STARTTIME --loglevel debug --outputmethod full --method all --outputfile output.txt -v commflag --nodblog --logpath .

Grab interesting things parsing the log e.g. for scene change values

cat x.log | perl -e 'print "frame,sc\n"; while (<>) {($x, $y) = /Frame:\s+(\d+) ->\s+\d+\s+\d+\s+\d+\s+(\d+)/; print "$x,$z\n";}' > sc.csv

Plot with your favorite app (I use R from pkg r-base)

sc = read.table("$FILE",sep=",", header=TRUE)
plot(sc)
hist(sc\$sc, breaks=100)

Also examine the log for common comm lengths and update CommDetectCommLengths? to suit.

For convenience, convert output.txt to min:seconds with attached showoutput.pl

Or just replace the installed mythcommflag with a link to the temp built one.

As always testing is appreciated.

Attachments (16)

commflag-options.patch (1003 bytes) - added by Mark Spieth 12 years ago.
comm-detector-audio.1.patch (54.0 KB) - added by Mark Spieth 12 years ago.
showoutput.pl (311 bytes) - added by Mark Spieth 12 years ago.
convert output.txt to min:sec format
run.sh (1.3 KB) - added by Mark Spieth 12 years ago.
sample script run debug mode mythcommflag in gdb
comm-detector-nextgen.2.patch (168.7 KB) - added by Mark Spieth 12 years ago.
improved comm detector patch #2
mythcommflag.patch (84.5 KB) - added by bryan@… 11 years ago.
Changes to Logo Detection, Scene Change Detection, and Heurisitic scoring (all in one)
logodetector.patch (27.6 KB) - added by bryan@… 11 years ago.
LogoDetector? changes
scenechangedetector.patch (7.7 KB) - added by bryan@… 11 years ago.
SceneChange? and Histogram changes
commdetector.patch (48.8 KB) - added by bryan@… 11 years ago.
Heuristic Scoring changes (ClassicCommDetector?)
scene-change-detector.diff (8.1 KB) - added by bryan@… 11 years ago.
Updated Scene Change Detector (uses averaging to make the detection more reliable)
logo-detector.diff (29.8 KB) - added by b 11 years ago.
20141111-26-helen-mcf-bytesperline.patch (13.2 KB) - added by Mark Spieth 9 years ago.
20141111-27-comm-detector-fixes.patch (4.8 KB) - added by Mark Spieth 9 years ago.
20141111-28-commdetector2-fixes.patch (1.1 KB) - added by Mark Spieth 9 years ago.
20141111-29-comm-detector-nextgen.patch (205.9 KB) - added by Mark Spieth 9 years ago.
20141111-30-helen-mcf.patch (7.9 KB) - added by Mark Spieth 9 years ago.

Download all attachments as: .zip

Change History (29)

Changed 12 years ago by Mark Spieth

Attachment: commflag-options.patch added

Changed 12 years ago by Mark Spieth

Attachment: comm-detector-audio.1.patch added

Changed 12 years ago by Mark Spieth

Attachment: showoutput.pl added

convert output.txt to min:sec format

Changed 12 years ago by Mark Spieth

Attachment: run.sh added

sample script run debug mode mythcommflag in gdb

comment:1 Changed 12 years ago by Mark Spieth

Description: modified (diff)

Changed 12 years ago by Mark Spieth

improved comm detector patch #2

comment:2 Changed 12 years ago by Mark Spieth

patch 2 added with support for audio and subtitles (teletext tested thoroughly, dvbsubtitles, cc608 and cc708 need verification). Back out patch 1 and apply this one.

Patch still is independent of main lib and can be build on any installed base as previously able. Eventually 2 classes for subtitle support need MTV_PUBLIC.

DB settings

this uses a simple but effective fusion of blank detect, audio silences, severe scene cuts and subtitles (no logo detection or rating) to perform commercial detection. Quite good. Got all breaks today in latest Offspring including end of Masterchef.

0x49f = 1183 uses the older style ClassicCommDetector? with silences added. Not tested much.

Currently running this replacement on my main system.

comment:3 Changed 12 years ago by warpme@…

Mark, I'm trying to improve commflaging in my system (current 0.25-fixes). Tying compiling v2 patch gives me following error:

SubtitleChangeDetector.cpp: In member function ‘virtual void SubtitleChangeDetector::processFrame(MythPlayer*, VideoFrame*)’:
SubtitleChangeDetector.cpp:131:17: error: cast from ‘TeletextSubPage*’ to ‘uint {aka unsigned int}’ loses precision [-fpermissive]
make[2]: *** [SubtitleChangeDetector.o] Error 1
make[2]: Leaving directory `/home/piotro/ArchMyth/ABS/mythtv/src/mythtv-0.25-build/mythtv/programs/mythcommflag'
make[1]: *** [sub-mythcommflag-make_default] Error 2
make[1]: *** Waiting for unfinished jobs....

comment:4 Changed 12 years ago by Mark Spieth

this is a 32/64 bit problem. change uint to ulong.

Ill try to get an update out this weekend with algo tweeks.

comment:5 Changed 11 years ago by Wayne McDougall <waynemcdougall@…>

I had to add

#include "mythmiscutil.h"

under MythTV headers to

NextgenCommDetector.cpp

to process the clamp[] macros

comment:6 Changed 11 years ago by Wayne McDougall <waynemcdougall@…>

In SubtitleChangeDetector?.cpp there is the following code:

            //subtitle.start_display_time;    /* relative to packet pts, in ms */
            //subtitle.end_display_time;  /* relative to packet pts, in ms */
            //subtitle.pts;               /* in AV_TIME_BASE, use frame number * fps */

            int start = frameNumber + subtitle.start_display_time * fps / 1000;
            int end   = frameNumber + subtitle.end_display_time * fps / 1000;

It seems to me this code assumes start_display_time and end_display_time are relative times to the frame number. Instead in my recordings they are absolutes timecodes in ms, directly related to the timecodes associated with each video frame. And the above code yields incorrect framenumbers, and subtitle detection fails.

I guess we could subtract the the initial frame's timecode, if we can assume fps is constant for recordings. I'm not sure that is a safe assumption. I suspect the time code wraps around to zero at some point, and that would need to be handled.

Instead I think it would be safer to follow the pattern of subtitlescreen.cpp which just compares the subtitle start and end timecodes with the frame timecodes.

Presumably the current code works for some people, although I find it hard to see how. It's entirely possible I'm not understanding something deeper here. Comments welcome. But I'm hoping if I do a good job, the changed code should still work correctly for them and also work for me.

Changed 11 years ago by bryan@…

Attachment: mythcommflag.patch added

Changes to Logo Detection, Scene Change Detection, and Heurisitic scoring (all in one)

Changed 11 years ago by bryan@…

Attachment: logodetector.patch added

LogoDetector? changes

Changed 11 years ago by bryan@…

Attachment: scenechangedetector.patch added

SceneChange? and Histogram changes

Changed 11 years ago by bryan@…

Attachment: commdetector.patch added

Heuristic Scoring changes (ClassicCommDetector?)

comment:7 Changed 11 years ago by bryan@…

I decided to put these here since a lot of the work is similar; my apologies if they should have gone in their own ticket. I have been these in testing now for about a month with good results. It's not perfect of course, but it works better for me (US Eastcoast mostly HiDef? from a HDHRP) that the baseline 0.25/0.26 comm flagger. I've got 4 diffs here, and I apologize for the size... I got kind of carried away and I should have probably tried to do these tasks more independently.

The largest one is all of the others combined. I split it up into 3 individual pieces for easy reviewing, but I didn't generate them separately so I'm not sure they will apply as individuals. If there's problems with that and using the monolithic one isn't a good idea, I can split them up and diff them individually against the baseline.

The "commdetector.diff" file contains my extensive changes to BuildAllMethodsCommList?(). I did a lot of reorganizing in here to help me wrap my head around what was going on. I'm not sure that any of my changes netted all that much positive here, except where related to the other two patches. I think the cleanups make it easier to follow, but that could just be me.

I mostly rewrote the LogoDetector? to use "Sobel" edge detection (there's a decent wikipedia article about it). That plus a few other changes to the thresholds and border areas makes the logo detector a lot more reliable. Because of that, I recommend making changes to BuildAllMethodsCommList?() to require less frames containing the logo before giving a score boost.

I rewrote Histogram similarity detection to allow slight changes in overall brightness by averaging over a range of intensities. This seems to work well. Bundled together, I added a bit of code to the scene change detector to avoid rapid scene changes over a very short (< 1s) period of time (reducing them to 1). I also added a bit of a "memory" of the previous scene, so that when the scene switches to something, and then switches back to the previous scene, neither is counted as a scene change. This is common in programming when there's dialog going on. These changes resulted in a much lower (about half as much) scene changes being reported, but nearly no legitimate changes are missed (except those purposely not reported because they appear to be part of programming). This necessitates changes to BuildAllMethodsCommList?() to account for the lower scene change rate. I ended up going with about half of what all of the old schene change rate requirements were.

I hope these are helpful. I'm happy to answer questions on them or try to clean them up a bit if needed. I can also change them to be against THE Nextgen detectors rather than Classic if that works better.

comment:8 in reply to:  7 ; Changed 11 years ago by Wayne McDougall <waynemcdougall@…>

Replying to bryan@…:

Hi Bryan. Please understand this is feedback - a couple of data points. I certainly appreciate you sharing your efforts.

I successfully added your mythcommflag.patch to the current development tree. I ran it on my two test recordings I'm working with, and using --method 7 (ie all) and also 6 (logo+scene) I didn't detect any commercials. The current master detects 2 of the 3 commercial breaks (and gets the timing correct on one of them). What I observe is that fewer Blanks are being detected (27 versus 35), and not consecutively. That combined with my observation that all methods are dependent on blank detection (I have found it useless to use any method without also detecting blanks), than this is a problem in my test set.

Hope you find the feedback useful. I shall go back to trying to figure out whether New Zealand TV2 is sending NULL/blank subtitles to clear subtitles off the screen... I wish there was some good debugging code to break down what is going on with DVB-T subtitle streams in a H.264 broadcast.

comment:9 in reply to:  8 Changed 11 years ago by bryan@…

Replying to Wayne McDougall <waynemcdougall@…>:

... Hope you find the feedback useful ...

I very much appreciate the feedback. I went back through and cleaned up my patches. I've reduced it to 2 (they both overlap a little in ClassicCommDetector?): a new SceneChangeDetector? and a new LogoDetector?. My other patch had made some more changes to the overall commflagging method that may have been a bad idea for non-US so I decided to abandon those changes. These 2 (scene change and logo detection) have both worked really well for me over the last 2 months.

Changed 11 years ago by bryan@…

Attachment: scene-change-detector.diff added

Updated Scene Change Detector (uses averaging to make the detection more reliable)

Changed 11 years ago by b

Attachment: logo-detector.diff added

Changed 9 years ago by Mark Spieth

Changed 9 years ago by Mark Spieth

Changed 9 years ago by Mark Spieth

Changed 9 years ago by Mark Spieth

Changed 9 years ago by Mark Spieth

Attachment: 20141111-30-helen-mcf.patch added

comment:10 Changed 9 years ago by Mark Spieth

After being prompted by helen, Ive produced a timely update to my patchset which incorporates helens findings and some others that I noticed while testing. Use all patches prefixed 20141111- in that order.

My results are as follows:

NameProgramTypeActualng all hiresng all newOrig allHelen all
The Loop1011 20141031220000SD121213610
2014 V8s - Gold Coast 6001073 20141029010000HD12111200
V8 Life1073 20140629030000HD43325
V8xtra1073 20140524020000HD55523
Party Tricks - The Campaign Trail1010 20141027093000SD99924
NASCAR Sprint Cup Qualifying - Race 34: A.A.A Texas 500 at Texas - Qualifying 100120141107173000SD1281134
Sons and Daughters1072 20141105162300SD 4:3 PB64602
Downton Abbey1007 20140427110000SD119953

The previously mentioned settings are still relevant.

  • method is 1055 or ng_all for ng all new column
  • Actual is a guess from the R plot but is at most 1 out.
    R
    > sc = read.table("loms1.csv",sep=",", header=TRUE)
    > plot(sc$Frame/25/60,sc$max)
    
  • Orig all is without any patches
  • Helen all is with helens patches and method 31 or all
  • ng all hires is my previous algo before helen's inspired updates

The updates are:

  • use original logo detector with helens mods
  • other helens mods for pillar box
  • updated to use helens bytesperline patch instead of the one I was using previously.
  • fix to coalesce segment scores which wasn't being done before helps with short program segments.
  • new log detector which is incomplete can be accessed with method ng_allx or 1055+256
  • use correct fps as players is just a wrong default. Only affects log output.

I like the results Im getting and will put this into my main backend shortly. Not perfect by better than I am currently getting. I tried to pick a varied set of material. Thanks to helen for her work that prompted me to revisit something I'd been using for ages.

As always feedback appreciated and YMMV. Note: view table in browser, mail won't do it justice.

comment:11 Changed 4 years ago by Stuart Auchterlonie

Owner: changed from cpinkham to mspieth
Status: newassigned

Mark,

Is there anything we need to do with this ticket, or should it be closed out?

comment:12 Changed 4 years ago by mspieth

I have been updating the patches locally but not submitted them.

They need a general cleanup.

8 years is a long time to keep features open!

I'll have to find some time.

comment:13 Changed 4 years ago by Stuart Auchterlonie

Resolution: Unverified
Status: assignedclosed

Closing all old tickets in trac.

If your issue still persists, please open an issue in Github https://github.com/MythTV/mythtv/issues

and reference the existing trac ticket.

Note: See TracTickets for help on using tickets.