Modify
Warning Please read the Ticket HowTo before creating or commenting on a ticket. Failure to do so may cause your ticket to be rejected or result in a slower response.

Opened 23 months ago

Last modified 16 months ago

#10793 new Patch - Feature

CommFlagger improvements

Reported by: markspieth Owned by: cpinkham
Priority: minor Milestone: unknown
Component: MythTV - Mythcommflag Version: Unspecified
Severity: medium Keywords:
Cc: Ticket locked: no

Description (last modified by markspieth)

This ticket is to share work on comm flagger improvements.

The comm flagger has been working worse for me over time so I decided to have a look at the algorithms.

The main components are:

  1. Command line method parsing fix. Was completely broken. See first patch
  2. Audio silence and change data fusion
  3. Scene changer param change. 0.85 was way too high to get useful scene cut info. Still not sure what the right value is. Debugging procedure to adventurers will follow.

2 and 3 are in the comm-detector-audio patch. This patch is completely independent of the myth libraries so that it is easier to test. There are some enum changes that will eventually make it into libmyth.
Only the all method is implemented.
Audio power min and max dont appear to add anything useful and will probably be removed.

It seems to work better so far for the few Ive tested.

DB Settings to play with (new and old)

Things to do:

  1. Subtitle info fusion
    This should help with segment classification
  2. Logo detector is sporadic. Need to find out why. Some programs (same channel) can find the logo but other can.

Test with

DEBUGCOMMFLAG=1 nice -19 ./mythcommflag --chanid $CHAN --starttime $STARTTIME --loglevel debug --outputmethod full --method all --outputfile output.txt -v commflag --nodblog --logpath .

Grab interesting things parsing the log e.g. for scene change values

cat x.log | perl -e 'print "frame,sc\n"; while (<>) {($x, $y) = /Frame:\s+(\d+) ->\s+\d+\s+\d+\s+\d+\s+(\d+)/; print "$x,$z\n";}' > sc.csv

Plot with your favorite app (I use R from pkg r-base)

sc = read.table("$FILE",sep=",", header=TRUE)
plot(sc)
hist(sc\$sc, breaks=100)

Also examine the log for common comm lengths and update CommDetectCommLengths? to suit.

For convenience, convert output.txt to min:seconds with attached showoutput.pl

Or just replace the installed mythcommflag with a link to the temp built one.

As always testing is appreciated.

Attachments (11)

commflag-options.patch (1003 bytes) - added by markspieth 23 months ago.
comm-detector-audio.1.patch (54.0 KB) - added by markspieth 23 months ago.
showoutput.pl (311 bytes) - added by markspieth 23 months ago.
convert output.txt to min:sec format
run.sh (1.3 KB) - added by markspieth 23 months ago.
sample script run debug mode mythcommflag in gdb
comm-detector-nextgen.2.patch (168.7 KB) - added by markspieth 23 months ago.
improved comm detector patch #2
mythcommflag.patch (84.5 KB) - added by bryan@… 18 months ago.
Changes to Logo Detection, Scene Change Detection, and Heurisitic scoring (all in one)
logodetector.patch (27.6 KB) - added by bryan@… 18 months ago.
LogoDetector? changes
scenechangedetector.patch (7.7 KB) - added by bryan@… 18 months ago.
SceneChange? and Histogram changes
commdetector.patch (48.8 KB) - added by bryan@… 18 months ago.
Heuristic Scoring changes (ClassicCommDetector?)
scene-change-detector.diff (8.1 KB) - added by bryan@… 16 months ago.
Updated Scene Change Detector (uses averaging to make the detection more reliable)
logo-detector.diff (29.8 KB) - added by b 16 months ago.

Download all attachments as: .zip

Change History (20)

Changed 23 months ago by markspieth

Changed 23 months ago by markspieth

Changed 23 months ago by markspieth

convert output.txt to min:sec format

Changed 23 months ago by markspieth

sample script run debug mode mythcommflag in gdb

comment:1 Changed 23 months ago by markspieth

  • Description modified (diff)

Changed 23 months ago by markspieth

improved comm detector patch #2

comment:2 Changed 23 months ago by markspieth

patch 2 added with support for audio and subtitles (teletext tested thoroughly, dvbsubtitles, cc608 and cc708 need verification).
Back out patch 1 and apply this one.

Patch still is independent of main lib and can be build on any installed base as previously able. Eventually 2 classes for subtitle support need MTV_PUBLIC.

DB settings

this uses a simple but effective fusion of blank detect, audio silences, severe scene cuts and subtitles (no logo detection or rating) to perform commercial detection. Quite good. Got all breaks today in latest Offspring including end of Masterchef.

0x49f = 1183 uses the older style ClassicCommDetector? with silences added. Not tested much.

Currently running this replacement on my main system.

comment:3 Changed 22 months ago by warpme@…

Mark, I'm trying to improve commflaging in my system (current 0.25-fixes). Tying compiling v2 patch gives me following error:

SubtitleChangeDetector.cpp: In member function ‘virtual void SubtitleChangeDetector::processFrame(MythPlayer*, VideoFrame*)’:
SubtitleChangeDetector.cpp:131:17: error: cast from ‘TeletextSubPage*’ to ‘uint {aka unsigned int}’ loses precision [-fpermissive]
make[2]: *** [SubtitleChangeDetector.o] Error 1
make[2]: Leaving directory `/home/piotro/ArchMyth/ABS/mythtv/src/mythtv-0.25-build/mythtv/programs/mythcommflag'
make[1]: *** [sub-mythcommflag-make_default] Error 2
make[1]: *** Waiting for unfinished jobs....

comment:4 Changed 22 months ago by markspieth

this is a 32/64 bit problem. change uint to ulong.

Ill try to get an update out this weekend with algo tweeks.

comment:5 Changed 19 months ago by Wayne McDougall <waynemcdougall@…>

I had to add

#include "mythmiscutil.h"

under
MythTV headers
to

NextgenCommDetector.cpp

to process the clamp[] macros

comment:6 Changed 18 months ago by Wayne McDougall <waynemcdougall@…>

In SubtitleChangeDetector?.cpp there is the following code:

            //subtitle.start_display_time;    /* relative to packet pts, in ms */
            //subtitle.end_display_time;  /* relative to packet pts, in ms */
            //subtitle.pts;               /* in AV_TIME_BASE, use frame number * fps */

            int start = frameNumber + subtitle.start_display_time * fps / 1000;
            int end   = frameNumber + subtitle.end_display_time * fps / 1000;

It seems to me this code assumes start_display_time and end_display_time are relative times to the frame number. Instead in my recordings they are absolutes timecodes in ms, directly related to the timecodes associated with each video frame. And the above code yields incorrect framenumbers, and subtitle detection fails.

I guess we could subtract the the initial frame's timecode, if we can assume fps is constant for recordings. I'm not sure that is a safe assumption. I suspect the time code wraps around to zero at some point, and that would need to be handled.

Instead I think it would be safer to follow the pattern of subtitlescreen.cpp which just compares the subtitle start and end timecodes with the frame timecodes.

Presumably the current code works for some people, although I find it hard to see how. It's entirely possible I'm not understanding something deeper here. Comments welcome. But I'm hoping if I do a good job, the changed code should still work correctly for them and also work for me.

Changed 18 months ago by bryan@…

Changes to Logo Detection, Scene Change Detection, and Heurisitic scoring (all in one)

Changed 18 months ago by bryan@…

LogoDetector? changes

Changed 18 months ago by bryan@…

SceneChange? and Histogram changes

Changed 18 months ago by bryan@…

Heuristic Scoring changes (ClassicCommDetector?)

comment:7 follow-up: Changed 18 months ago by bryan@…

I decided to put these here since a lot of the work is similar; my apologies if they should have gone in their own ticket. I have been these in testing now for about a month with good results. It's not perfect of course, but it works better for me (US Eastcoast mostly HiDef? from a HDHRP) that the baseline 0.25/0.26 comm flagger. I've got 4 diffs here, and I apologize for the size... I got kind of carried away and I should have probably tried to do these tasks more independently.

The largest one is all of the others combined. I split it up into 3 individual pieces for easy reviewing, but I didn't generate them separately so I'm not sure they will apply as individuals. If there's problems with that and using the monolithic one isn't a good idea, I can split them up and diff them individually against the baseline.

The "commdetector.diff" file contains my extensive changes to BuildAllMethodsCommList?(). I did a lot of reorganizing in here to help me wrap my head around what was going on. I'm not sure that any of my changes netted all that much positive here, except where related to the other two patches. I think the cleanups make it easier to follow, but that could just be me.

I mostly rewrote the LogoDetector? to use "Sobel" edge detection (there's a decent wikipedia article about it). That plus a few other changes to the thresholds and border areas makes the logo detector a lot more reliable. Because of that, I recommend making changes to BuildAllMethodsCommList?() to require less frames containing the logo before giving a score boost.

I rewrote Histogram similarity detection to allow slight changes in overall brightness by averaging over a range of intensities. This seems to work well. Bundled together, I added a bit of code to the scene change detector to avoid rapid scene changes over a very short (< 1s) period of time (reducing them to 1). I also added a bit of a "memory" of the previous scene, so that when the scene switches to something, and then switches back to the previous scene, neither is counted as a scene change. This is common in programming when there's dialog going on. These changes resulted in a much lower (about half as much) scene changes being reported, but nearly no legitimate changes are missed (except those purposely not reported because they appear to be part of programming). This necessitates changes to BuildAllMethodsCommList?() to account for the lower scene change rate. I ended up going with about half of what all of the old schene change rate requirements were.

I hope these are helpful. I'm happy to answer questions on them or try to clean them up a bit if needed. I can also change them to be against THE Nextgen detectors rather than Classic if that works better.

comment:8 in reply to: ↑ 7 ; follow-up: Changed 18 months ago by Wayne McDougall <waynemcdougall@…>

Replying to bryan@…:

Hi Bryan. Please understand this is feedback - a couple of data points. I certainly appreciate you sharing your efforts.

I successfully added your mythcommflag.patch to the current development tree. I ran it on my two test recordings I'm working with, and using --method 7 (ie all) and also 6 (logo+scene) I didn't detect any commercials. The current master detects 2 of the 3 commercial breaks (and gets the timing correct on one of them). What I observe is that fewer Blanks are being detected (27 versus 35), and not consecutively. That combined with my observation that all methods are dependent on blank detection (I have found it useless to use any method without also detecting blanks), than this is a problem in my test set.

Hope you find the feedback useful. I shall go back to trying to figure out whether New Zealand TV2 is sending NULL/blank subtitles to clear subtitles off the screen... I wish there was some good debugging code to break down what is going on with DVB-T subtitle streams in a H.264 broadcast.

comment:9 in reply to: ↑ 8 Changed 16 months ago by bryan@…

Replying to Wayne McDougall <waynemcdougall@…>:

... Hope you find the feedback useful ...

I very much appreciate the feedback. I went back through and cleaned up my patches. I've reduced it to 2 (they both overlap a little in ClassicCommDetector?): a new SceneChangeDetector? and a new LogoDetector?. My other patch had made some more changes to the overall commflagging method that may have been a bad idea for non-US so I decided to abandon those changes. These 2 (scene change and logo detection) have both worked really well for me over the last 2 months.

Changed 16 months ago by bryan@…

Updated Scene Change Detector (uses averaging to make the detection more reliable)

Changed 16 months ago by b

Add Comment

Modify Ticket

Action
as new .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.