0.24 Crash after Recording Finished with memory Corruption (glibc)

[flybrick@…]

I am running a dedicated mythbackend with 4 PVR-150 cards. I am seeing a repeated crashes of the backend after running for several days. It appears that the series of events leading up to the event is that at least 1 PVR-150 card becomes unavailable for a period of time, then a recording finished, then scheduling starts, then a malloc memory corruption error is generated.

I am attaching the mythbackend.log with the events leading up to the crash and the crash dump itself. The crash occurs about 1/2 way through the log at "2011-03-28 23:01:00.737"

[flybrick@…]

mythbackend.log for crash

[robertm]

In order to debug something like this, we will need a backtrace of the crash per the manual:

http://www.mythtv.org/wiki/Debugging

[flybrick@…]

I am currently rebuilding with "--compile-type=profile" so as to generate a better backtrace from a core dump. It tends to take several days to a week to crash.

Unfortunately, I had already pulled the latest fixes/0.24, not the one that crashed here.

In the meantime, I was able to extract a C++ mangled-name backtrace from the last crash, and tracing back through the code, it looks like it crashed in I am currently rebuilding with "--compile-type=profile" so as to generate a better backtrace from a core dump. It tends to take several days to a week to crash.

Unfortunately, I had already pulled the latest fixes/0.24, not the one that crashed here.

In the meantime, I was able to extract a C++ mangled-name backtrace from the last crash.

It looks like the crash is somehow entered in the method "void DeviceReadBuffer::fill_ringbuffer(void)" in DeviceReadBuffer?.cpp, at one of the calls to lock.lock() (~line 256 or 304).

From there, it just continues through libQtCore until glibc detects the memory corruption. If it really is a memory corruption issue, it is quite possible it didn't occur anywhere in the call chain, but at some other unrelated point.

2011-03-28 23:01:00.737 scheduler: Finished recording: Castle "Law & Murder": channel 1013
*** glibc detected *** /usr/bin/mythbackend: malloc(): memory corruption: 0x09fc5c39 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x22cd591]
/lib/tls/i686/cmov/libc.so.6(+0x6e395)[0x22d0395]
/lib/tls/i686/cmov/libc.so.6(__libc_malloc+0xec)[0x22d202c]
/usr/lib/libQtCore.so.4(_Z7qMallocj+0x1d)[0x7c5db8d]
/usr/lib/libQtCore.so.4(_ZN10QByteArrayC1Eic+0x36)[0x7c65b66]
/usr/lib/libQtCore.so.4(_Z15qt_error_stringi+0x59)[0x7c5a749]
/usr/lib/libQtCore.so.4(+0x655ff)[0x7c625ff]
/usr/lib/libQtCore.so.4(+0x65780)[0x7c62780]
/usr/lib/libQtCore.so.4(_ZN6QMutex4lockEv+0x132)[0x7c5e052]
/usr/lib/libmythtv-0.24.so.0(_ZN16DeviceReadBuffer15fill_ringbufferEv+0x2c)[0x12a449c]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0x232fa4e]

[Raymond Wagner]

Errors like that typically indicate some form of hardware or system instability outside of the program itself. Are you experiencing anomalous faults in any other programs? Have you tried running memtest or similar software?

[John Werner <flybrick@…>]

Replying to wagnerrp:

Errors like that typically indicate some form of hardware or system instability outside of the program itself. Are you experiencing anomalous faults in any other programs? Have you tried running memtest or similar software?

I've been wondering the same thing. The machine only runs mythbackend. I will try to find sometime when it is not scheduled to record something and boot-up into memtest and see what happens.

[After running MythTv? for 4+ years, its amazing how much you end up recording.]

[danielk]

I rewrote the DRB buffer handling in the mythtv-rec2 branch, but I'm pretty certain the existing code was not writing past the end of the buffers. So I believe memory overwriting bug is probably at some distance from where the backtrace is from and so this debugging is not useful in finding it.

A valgrind log would change my mind or some repeats..