Memory leak during playback of recent recordings (PVR-250)

[aaron <memoryguy@…>]

Recordings made in the last month or so seem to make mythfrontend leak memory during playback. Typically I use timestretch 1.3x or higher, up to 2x, not sure if it is relevant. About halfway through a one hour program I have to restart the frontend to release the memory; playback gets stuttery due to excessive paging.

All of my recordings are standard-def MPEG2, produced from a PVR-250. But only the more recent recordings show this problem.

My backend/frontend has 512 MB of RAM (+2 GB paging space). When the frontend needs to be restarted, htop reports the "VIRT" at 800+ MB.

After starting the frontend VIRT is 337 MB.

I am running master built from git. The current version I am running (and do see the problem) is:

I have tried to run valgrind:

valgrind --log-file=vglog.log --leak-check=full --error-limit=no --show-reachable=yes mythfrontend -O UIPainter=qt

When I started playback it complained about not being able to allocate/resize ALSA buffers, and it looked like it was not using the proper playback timing methods. Perhaps this is because I was using an SSH session (although displaying on the local display, not remote). Hopefully the log managed to capture something meaningful.

Playback stuttered horribly with valgrind running, so I left it going as long as I could stand.

Attached are the frontend output and valgrind log.

Please let me know if there is something else that would be helpful.

[aaron <memoryguy@…>]

Valgrind logs

[sphery]

aaron, we need a valgrind log run against a profile build of mythtv. You should be able to install debugging packages, as described at http://www.mythtv.org/wiki/Debugging , and then re-run valgrind. Thanks.

[aaron <memoryguy@…>]

Sorry that it has taken a while. I finally had a chance to make a Profile build and try to capture the valgrind output. This time I hit mute and let it run for about half an hour. The memory usage started at around 960MB (I think) and I stopped it when it went over 1 GB.

It did run for about half an hour, but with the overhead of Valgrind the audio was stuttering horribly (the video never did actually appear) so I'm not sure how much actual recording was played.

When I stopped the frontend (using Esc, trying to stop cleanly), Valgrind wrote a 1 GB logfile, which suggested to me something had probably gone horribly wrong. It looks like it got into a loop trying to dump the memory/stack details, and something was corrupted... or something. It eventually did end. I snipped out the part at the top of the file that looked potentially meaningful, and I included the end of the file (including Valgrind's message about crashing). The result is too large to attach to trac, so I put it in my Dropbox... hopefully this link works:

​http://dl.dropbox.com/u/1897138/vglogs2.tar.gz

If you want to see the whole 1 GB log, let me know and I'll try to put it up (it compresses to about 20 MB).

I know that the build I'm using (same version as in my initial report, just with --build-type=profile) is a bit old at this point. If you need and are willing to wait a bit, I would be willing to try again with a more up-to-date build.

Hopefully this is helpful... let me know if you need anything more.

Thanks! aaron

[aaron <memoryguy@…>]

I updated to version:

myth@myth:~$ mythbackend --version
Please attach all output as a file in bug reports.
MythTV Version   : v0.25pre-1788-g0e22930
MythTV Branch    : master
Network Protocol : 65
Library API      : 0.25.20110414-3
QT Version       : 4.5.1
Options compiled in:
 linux profile use_hidesyms using_alsa using_oss using_backend using_bindings_perl using_bindings_python using_bindings_php using_dvb using_frontend using_hdhomerun using_hdpvr using_iptv using_ivtv using_libxml2 using_libudf using_lirc using_mheg using_qtdbus using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_bindings_perl using_bindings_python using_bindings_php using_mythtranscode using_opengl using_ffmpeg_threads using_live using_mheg using_libxml2 using_libudf

And ran valgrind again. I think it might have worked better this time... the video was actually getting (slowly) updated --- one frame every few seconds --- and the VIRT was increasing. It went from around 960 MB at the start of playback to 1021 MB at the time I stopped. It took about half an hour to get that far, although based on the video frames, only a couple of minutes (if even that much) had played. Hopefully it shows something useful. The log is too large to attach so I put it in my Dropbox:

​http://dl.dropbox.com/u/1897138/valgrind_logs_1788.tar.gz

This time the log seems to be complete, and I don't think it says anything about memory corruption. Please let me know if there is anything more you want me to collect.

Thanks! aaron

[sphery]

Valgrind log and frontend log from comment:4

[sphery]

Attaching logs from comment:4 , so they're still available if expired from dropbox.

Thanks for getting the new logs with debug symbols, Aaron.

[beirdo]

All of the potential memory loss vectors seem to be related to the Qt painter. Is there any reason you'd be using Qt rather than OpenGL?

[aaron <memoryguy@…>]

Replying to beirdo:

All of the potential memory loss vectors seem to be related to the Qt painter. Is there any reason you'd be using Qt rather than OpenGL?

I'd love to do that, but when I enable OpenGL (or rather, when I don't override it) my X server crashes (sigsegv, if I recall correctly from the X logfile) pretty much as soon as I start mythfrontend (or mythwelcome). It used to work, but that was a long time ago and there were often drawing artifacts and/or very slow performance relative to the Qt painter. Often times it would seem fine, until I had watched a recording or two, then it would start taking a really long time to render the menus and transition the screens.

I noticed that there was recently a move to require a newer version of Qt than I have installed. When I have a bit of time I can try to upgrade Qt and see if that helps anything.

[Github]

libmythui: Fix a leak (flood) in MythPainter?.

The software cache size was never increased as the images hadn't been created when we queried their size - and hence nothing was ever removed from the software cache.

There will still be reports of un-allocated images in the MythPainter? destructor. These are the main OSD window images that are still being held by the main UI image cache. Not yet sure of the best approach with those - I don't really want to leave them hanging.

Refs #9707.

Branch: master Changeset: 612469827c34e7dc02fa026c10f646472b6fa4dd

[markk]

Aaron - can you confirm whether this is now fixed? thanks, Mark

[aaron <memoryguy@…>]

Replying to markk:

Aaron - can you confirm whether this is now fixed? thanks, Mark

I would be happy to try it out, except that somewhere along the line (I haven't updated my build in some time) something "broke" and I can't compile trunk anymore... it gets partway through and then starts complaining about redeclarations of structures/types. If I turn off V4L support (--disable-v4l2) then the compile finishes, but then it won't recognize my capture cards (PVR-250). I do realize that is expected since I told it not to compile capture support.

I'm wondering if the problem is that my kernel is too old? I'm running 2.6.29.6. I ask because I also updated my Qt from 4.5 to 4.7.0

Here's the first part of the errors I get; the rest are similar but for different structures:

g++...include -I. -o v4lchannel.o v4lchannel.cpp
In file included from /usr/include/sys/time.h:31,
                 from /usr/include/linux/videodev2.h:59,
                 from /usr/include/linux/videodev.h:17,
                 from v4lrecorder.cpp:4:
/usr/include/sys/select.h:78: error: conflicting declaration 'typedef struct fd_set fd_set'
/usr/include/linux/types.h:12: error: 'fd_set' has a previous declaration as 'typedef struct __kernel_fd_set fd_set'

[markk]

Aaron - if you're still having v4l issues in master then you need to open another ticket and try and get it resolved.

Mark

[aaron <memoryguy@…>]

Okay, I was able to update and try the patch:

myth@myth:~$ mythfrontend --version
Please attach all output as a file in bug reports.
MythTV Version : v0.25pre-2402-gbcadb9f
MythTV Branch : master
Network Protocol : 66
Library API : 0.25.20110620-2
QT Version : 4.7.0
Options compiled in:
 linux release use_hidesyms using_alsa using_oss using_backend using_bindings_perl using_bindings_python using_bindings_php using_dvb using_frontend using_hdhomerun using_hdpvr using_iptv using_ivtv using_libxml2 using_libudf using_lirc using_mheg using_opengl_video using_qtdbus using_qtwebkit using_v4l2 using_v4l1 using_x11 using_xrandr using_xv using_bindings_perl using_bindings_python using_bindings_php using_mythtranscode using_opengl using_ffmpeg_threads using_live using_mheg using_libxml2 using_libudf

The leak seems to be much improved, but I think it still leaks, as I still see the memory usage increasing. I thought it might be related to Timestretch, as I was watch *everything* at 1.3x or faster, and the higher the timestretch value, the faster it seems to leak. However, I tested today with playback left at 1x and I still saw the memory continue to increase.

I will make a profile build and try to capture a valgrind log again. Might take me a couple of days, though.

Thanks! aaron

[Raymond Wagner]

User is unable to provide new valgrind logs at this time.

[markk]

If you can verify with some valgrind logs, please reopen and I'll take another look.