stream reading eats up all memory in all myth apps

[Hans-Peter Oeri <hp@…>]

Recently, I noticed a server halting. mythbackend on preview generation took up all memory. Upon further inspection, both mythtranscode and mythcommflag do the same.

mythcommflag on a 20MB sample stream eats up 120MB of memory - for full recordings, the system runs OOM. Of course, I tried to "rebuild" seektables, but the bug seems to be "before" that.

From valgrind's/massif's output, it seems that from AVFormatDecoder::OpenFile on, av_malloc'ed and av_mallocz'ed memory is never freed.

[Hans-Peter Oeri <hp@…>]

valgrind log file

[Hans-Peter Oeri <hp@…>]

massif log file (ms_printed)

[Hans-Peter Oeri <hp@…>]

Unfortunately, I don't see through the whole allocing/freeing businness in libavformat.

I found, however, that removing a special "stream" from the mpeg ts changes the memory behaviour:

I have the following streams in the file:
0xA0 => MPEG2VIDEO
0xA1 => MP2
0xA3 => AC3
0xA5 => Teletext (TTX)
0xA9 => VPS

It seems, myth recognizes 0xA9 as "second" video stream, which it isn't (​http://en.wikipedia.org/wiki/Video_Program_System). Removing 0xa9 from the file (mpegdemux -r / project-x) is a work-around for me.

As VPS is not interpreted by mythtv anyway, and there is no information in VPS to keep, such streams may be safely discarded.

[Hans-Peter Oeri <hp@…>]

with eliminated VPS stream; massif log file (ms_printed)

[Hans-Peter Oeri <hp@…>]

If I eliminate the VPS stream but keep the TS format, everything works fine (see massif.new log)

[Hans-Peter Oeri <hp@…>]

Correction... it seems that all recordings from ORF (Austria) are somehow buggy. I don't know what, the usual tools don't display anything unusual, but a plain conversion through project-x (with all streams intact) solves the problem, too.

Still, bringing down the whole machine is bad(tm). AVFormatDecoder should probably just bail out, if something goes awry.

As a workaround:

• userjob to go over all recordings with project-x;
• configure apache to block all preview images (of recordings which might be not yet "cleansed")

[Hans-Peter Oeri <hp@…>]

Learning to cope with threaded programs ;) I use mythtranscode --mpeg2 --video --infile to really limit the surroundings.

The first ever call to ac_read_frame_internal at libs/libavformat/utils.c:2162 never returns...

[stephen.hocking@…]

Just a note to confirm that I've seen this too - on Ubuntu 8.04 64bit - it's weird to see mythbackend eating up to 21GB mem, and also Mac OS/X intel, 10.5.5 with MythFrontend. In both cases, svn 19244.

[Hans-Peter Oeri <hp@…>]

I made gdb log all calls to av_malloc and av_free (and av_realloc)... It seems, that after some startup work, some kind of loop allocating 192/64 byte sizes is entered. Those blocks are never freed (at least until the program reaches its ulimit and gets killed).

See gdb.text attached (truncated).

[Hans-Peter Oeri <hp@…>]

log of av_(malloc|free|realloc), truncated

[Hans-Peter Oeri <hp@…>]

Working on a less complex sample of 20MB, and with --verbose all, I get the following:

2008-12-07 15:48:41.384 Transcoding from /home/hpoeri/Desktop/trial20.mpg to out20.mpg
2008-12-07 15:48:41.393 Opening /home/hpoeri/Desktop/trial20.mpg
2008-12-07 15:48:41.829 ac-tex damaged at 19 2
2008-12-07 15:48:41.829 Warning MVs not available
2008-12-07 15:48:41.830 concealing 1156 DC, 1156 AC, 1156 MV errors
2008-12-07 15:48:41.857 Could not find codec parameters (Audio: mp3, s16)
2008-12-07 15:48:41.857 Could not find codec parameters (Video: 0x0000)
2008-12-07 15:48:42.233 Input #0, mpegts, from '/home/hpoeri/Desktop/trial20.mpg':
2008-12-07 15:48:42.233   Duration: N/A, bitrate: N/A
2008-12-07 15:48:42.253     Stream #0.0[0x1fa]: Video: mpeg2video, yuv420p, 544x576 [PAR 24:17 DAR 4:3], 3900 kb/s, 25.00 tb(r)
2008-12-07 15:48:42.253     Stream #0.1[0x1fb](ger): Audio: mp3, s16
2008-12-07 15:48:42.253     Stream #0.2[0x1fd](ger): Data: 0x0000
2008-12-07 15:48:42.253     Stream #0.3[0x200]: Video: 0x0000, 90000.00 tb(r)
2008-12-07 15:48:42.254 Skipping invalid audio stream: 1
2008-12-07 15:48:42.254 Skipping unsupported codec 2 on stream 2
2008-12-07 15:48:42.700 Found end of file without finding  any frames
2008-12-07 15:48:42.701 Transcoding /home/hpoeri/Desktop/trial20.mpg failed

Which would seem coherent with the no-freeing noticed above. Although the sample file is playable with all relevant players I have, myth's libavformat seems to eat up memory while searching for a frame (or frame boundaries?)

Unfortunately, I don't know anything about mpegts contents, yet.

However, myth really should give up before EOF or OOM ;)

[Dibblah]

This looks like it's closely related to #5016 - This time with an audio channel rather than video.

[janne]

(In [19279]) enhance detection of VBI streams in mpeg ts. Fixes #5971

[janne]

(In [19295]) ignore PRIVATE_DATA streams for now. Refs #5971

They cause huge memleaks with CODEC_ID_PROBE added in [19173].
ffmpeg TS demux ignores them too and we should correct the stream type
it contains useful data.

[stephen.hocking@…]

I can confirm that a memory leak matching this still occurs as of svn 19289 - both Mac OS/X & Linux 64bit.