Opened 13 years ago

Closed 12 years ago

#3317 closed defect (fixed)

mythbackend memory leak

Reported by: anonymous Owned by: Stuart Auchterlonie
Priority: major Milestone: 0.21
Component: mythtv Version: head
Severity: high Keywords:
Cc: Ticket locked: no

Description

I seem to have a problem with a memory leak. The backend runs fine for a while but after a day or two it starts to eat up all of the swap space on my machine. Valgrind gives me the following after running it for a couple hours:

==22603==    definitely lost: 7,483,036 bytes in 7,906 blocks.
==22603==      possibly lost: 54,448 bytes in 55 blocks.
==22603==    still reachable: 13,296,516 bytes in 287,277 blocks.
==22603==         suppressed: 0 bytes in 0 blocks.

I've never used valgrind before but it seems to imply that around 7 megs was lost.

I'm running ubuntu edgy (64 bit) on an Athlon 64 X2 4200+ Qt is version 3.3.6-3u Gcc is version 4.1.2 20060928 (ubuntu 4.1.1-13ubuntu5)

I configured using the following: ./configure --enable-xvmc --enable-dvb --enable-dts

My system has 2 gb of ram and 3.5 gb of swap. I have a pchdtv hd5500 and a wintv pvr-150 The video card is a a GeForce? 7300GS

The backend and frontend are on the same machine. I have attached the valgrind log.

Attachments (9)

valgrind.log (56.7 KB) - added by jaggroups@… 13 years ago.
valgrind log
mythbackend.log.gz (9.0 KB) - added by jaggroups@… 13 years ago.
backend log
valgrind.2.log (33.3 KB) - added by jaggroups@… 13 years ago.
new valgrind log
backend.log (2.0 KB) - added by jaggroups@… 13 years ago.
new backend log
valgrind.29752 (30.0 KB) - added by Jeremy Green <jaggroups@…> 12 years ago.
valgrind log file with backend compiled with --compile-type=debug
ticket3317-memleak.diff (491 bytes) - added by Stuart Auchterlonie 12 years ago.
Mem leak in upnp code
valgrind.4674 (29.1 KB) - added by Jeremy Green <jaggroups@…> 12 years ago.
valgrind log (after patch was applied)
valgrind.10476 (30.1 KB) - added by Jeremy Green <jaggroups@…> 12 years ago.
valgrind log while attempting to connect from upnp avr-4306
memleak.patch (361 bytes) - added by billymacdonald@… 12 years ago.
Patch to resolve Memory Leak

Download all attachments as: .zip

Change History (34)

Changed 13 years ago by jaggroups@…

Attachment: valgrind.log added

valgrind log

Changed 13 years ago by jaggroups@…

Attachment: mythbackend.log.gz added

backend log

Changed 13 years ago by jaggroups@…

Attachment: valgrind.2.log added

new valgrind log

Changed 13 years ago by jaggroups@…

Attachment: backend.log added

new backend log

comment:1 Changed 13 years ago by anonymous

Version: unknownhead

comment:2 Changed 13 years ago by Stuart Auchterlonie

Milestone: unknown0.21
Owner: changed from Isaac Richards to Stuart Auchterlonie

I'll have a look at this, although it looks like it's in the upnp code.

For better backtraces within the valgrind logs you should rebuild myth with --compile-type=debug

Stuart

comment:3 in reply to:  2 Changed 12 years ago by Jeremy Green <jaggroups@…>

I'm attaching a valgrind log file with debugging turned on. Thanks!

Changed 12 years ago by Jeremy Green <jaggroups@…>

Attachment: valgrind.29752 added

valgrind log file with backend compiled with --compile-type=debug

Changed 12 years ago by Stuart Auchterlonie

Attachment: ticket3317-memleak.diff added

Mem leak in upnp code

comment:4 Changed 12 years ago by Stuart Auchterlonie

Can you try the patch i've attached to the ticket? The patch should fix the issue.

Stuart

Changed 12 years ago by Jeremy Green <jaggroups@…>

Attachment: valgrind.4674 added

valgrind log (after patch was applied)

comment:5 in reply to:  4 Changed 12 years ago by Jeremy Green <jaggroups@…>

It looks like the patch did something but that there are still some leaks in there. Or am I reading the valgrind log wrong? I'm running it for a little bit longer test (while it records a show) to see what happens. It seems to eat up most of the systems swap after about two days.

-Jeremy

comment:6 Changed 12 years ago by Stuart Auchterlonie

(In [13317]) Refs #3317. Plugs a memory leak in the upnp code.

comment:7 Changed 12 years ago by Stuart Auchterlonie

There's nothing wrong in the latest valgrind output you've sent. Please run it for a few days and report back. Also try doing what ever else it was you were doing when you made your first report. That exercised the upnp subsytem more extensively. Perhaps there were other media clients running on the network at the time (such as games consoles etc.)

Otherwise i'll close this in a few days...

Stuart

comment:8 Changed 12 years ago by Jeremy Green <jaggroups@…>

Well, I don't know if this is abnormal or not but after 2 days of running top reports the following: 6339 mythtv 15 0 3002m 1.4g 14m S 0 69.1 8:49.08 1.6g mythbackend

So, mythbackend is using 3Gb of memory and 1.6Gb of swap. Is that normal?

Unfortunately I haven't been running valgrind because the backend becomes useless for HD content when running in valgrind.

comment:9 in reply to:  8 Changed 12 years ago by Stuart Auchterlonie

Replying to Jeremy Green <jaggroups@brownout.com>:

Well, I don't know if this is abnormal or not but after 2 days of running top reports the following: 6339 mythtv 15 0 3002m 1.4g 14m S 0 69.1 8:49.08 1.6g mythbackend

So, mythbackend is using 3Gb of memory and 1.6Gb of swap. Is that normal?

No definitely not. We'll need some more information about what devices you have on your network (like media players, that sort of thing...)

Stuart

comment:10 Changed 12 years ago by Jeremy Green <jaggroups@…>

I have a Denon AVR 4306 that is the receiver for my AV system. It supports upnp but I don't use it at all. I just played around with it a little. It sees the myth backend on the network. I'll attach another set of logs from when I was messing around with it over the network. I have an ADS Media-Link but I don't think that supports upnp. I also have a buffalo Linktheater that is currently not on the network.

Changed 12 years ago by Jeremy Green <jaggroups@…>

Attachment: valgrind.10476 added

valgrind log while attempting to connect from upnp avr-4306

comment:11 in reply to:  10 Changed 12 years ago by Jeremy Green <jaggroups@…>

I'm going to try running the backend for a while with --noupnp on the command line to see if it makes any difference to the memory usage. I'll let you know if I notice anything.

comment:12 Changed 12 years ago by Jeremy Green <jaggroups@…>

Well, the backend seems to still be using an ungodly amount of memory:

10970 mythtv 15 0 1577m 1.3g 14m S 0 64.0 4:44.61 289m mythbackend

This less than a day after I started it with --noupnp So, I don't know how much of the upnp code this disables but it disables it all then that seems to imply that the problem is elsewhere. Right?

comment:13 Changed 12 years ago by spam@…

Not sure if this is related or not, but there seems to be a leak somewhere in the code that runs the built in webserver on a master backend. I was monitoring my backend via nagios and after a re-install using monit, and memory usage climbing extremely fast.

I've updated the monit config on my master backend to not do the http check, only a PID and memory resource check about a week ago and since then it has not grown in size.

To confirm, I ran 'links -source http://localhost:6544/ > /dev/null' in a 100 loop for loop and memory jumped from 241virt/21res to 349virt/123res in less than a minute.

I am running monit (checking the http port) on a slave backend and it does not have this leak.

comment:14 Changed 12 years ago by Darryl Ross (spam@…

Sorry, forgot to mention, this is on SVN 13468.

comment:15 Changed 12 years ago by anonymous

I see the same problem running r13445. I recently installed monit to restart in the event of a crash. After monitoring the system overnight, mythbackend was using 70% of the memory. Eventually it crashes.

comment:16 Changed 12 years ago by postmaster@…

comment:17 in reply to:  16 Changed 12 years ago by Stuart Auchterlonie

Status: newassigned

Replying to postmaster@profesionalweb.com:

same problem here r13181...
probably memory leak introduced here:
http://svn.mythtv.org/trac/changeset/13022/trunk/mythtv/programs/mythbackend/httpstatus.cpp

The only way to prove that's at fault is to update back to 13021 and try that, if that doesn't leak then move to 13022 and try that and observe the leak. *Probably* isn't of much use.

Anyway, I can now reproduce this to some degree. I'm just trying to work out exactly who is the culprit. It's not leaking as such, somebody is holding onto memory.

Stuart

comment:18 Changed 12 years ago by postmaster@…

Hi stuart, if it really helps you i can do the revision (r13021 and r13022) test to be 100% sure that revision its the faulty one, that will maybe save you some time... If you are already on the track that may not be necessary.

ismael

comment:19 Changed 12 years ago by billymacdonald

I recently upgrade to SVN 13790 and have a process that queries the http server on 6544. That seems to be eating memory still.

comment:20 Changed 12 years ago by anonymous

I get a lot of these using pmap:

b20b4000    8188 rwx-- 00000000b20b4000 000:00000   [ anon ]

in fact:

pmap -d `pidof mythbackend` | grep 8188 | wc -l
84

I believe these are stack allocations from each time a new thread is created, so it implies that maybe we're not joining/terminating threads properly (this is after about 3 days with mythbackend using 774MB of virtual memory) Is there a quick way to determine which thread a block of memory belongs to, and where that thread got created?

comment:21 Changed 12 years ago by anonymous

A followup to my last post.... I checked the number of running threads, and found it to be significantly smaller than the number of stack allocations reported by pmap.

I wrote a wrapper around libpthread to keep track of stack allocations and do a backtrace of where the 'pthread_create' call came from

It turns out that all of the cases where there is allocated stack but no running thread came from dvbrecorder.cpp::StartDummyVideo?() (there is a pthread_create in that function that is initiating this memory usage)

At this point I run out of steam. I don't see anything obvious in run_dummy_video() or RunDummyVideo?() that would prevent libpthread from reclaiming the stack-space, yet it appears that is what is happening at least in my case. But maybe this will be useful to someone more familiar with this code.

comment:22 Changed 12 years ago by anonymous

Sorry for the rapid-fire posts, but here is an answer to the issue I found above. I found that simply doing:

pthread_detach(_video_thread);

right after the 'pthread_create' in dvbrecorder.cpp:StartDummyVideo() allows the stack allocation to be cleaned up properly.

comment:23 Changed 12 years ago by Stuart Auchterlonie

Priority: minormajor
Severity: mediumhigh

Changed 12 years ago by billymacdonald@…

Attachment: memleak.patch added

Patch to resolve Memory Leak

comment:24 Changed 12 years ago by billymacdonald@…

Attached is a patch to fix this leak I believe.

The ConfigurationGroup? class has a virtual function for deleteLater() which does not call it's base class's deleteLater(). Therefore, the EventLoop? never knows to delete this derived class of QOBject. I haven't done a lot of C++ coding so I don't know if what I did is the "proper" way to call the base class.

This would effect any derived class of ConfigurationGroup?, which ScheduledRecording? is derived from, which is a member of the ProgramInfo? class.

comment:25 Changed 12 years ago by Janne Grunau

Resolution: fixed
Status: assignedclosed

(In [13823]) Closes #3317. Fixes Memleak by applying patch from billymacdonald <AT> gmail <DOT> com

ConfigurationGroup::deleteLater wasn't calling QObject's deletaLater resulting in never actually deleting the object

Note: See TracTickets for help on using tickets.