Segv from mythshutdown in MSqlDatabase

[keemllib@…]

Caused when running:

mythshutdown --quiet --nodblog --unlock

This does not happen every time the above is run. When trying the above with --logpath /tmp --loglevel debug, no log file is produced. mythlogserver is running.

Perhaps related, but without dropping core, running the above may print:

Error: Not all threads were shut down properly: 
Thread SignalingTimer is still running

or

Real-time signal 0

This system is running: Ubuntu 12.04 LTS with a 3.2.0 kernel.

[Bill Meek <keemllib@…>]

Update:

Turns out that mythbuntu's apport was stomping on my:

kernel.core_pattern = /tmp/core.%t.%u.%p.%e

With that restored, I'm now seeing the same core dumps on mythavtest, mythfilldatabase, mythbackend --setloglevel and mythcommflag.

These SEGVs can occur on the 1st run of a program or it has taken me as many as 40 attempts to get a core file.

I'm now running on v0.26-pre-836-g42b2b24.

Please note that the "Error: Not all threads were shut down properly:" symptom above matches #10898

[danielk]

Looks like the MythSignalingTimer? for logging isn't being shutdown in or before the MythContext destructor is run.

[beirdo]

This is due to the following code in MythSignalingTimer::start():

        while (dorun && !running)
            usleep(10 * 1000);

That is where it is in the backtrace. If we are done and shutdown quickly enough, and stop the timer correctly, it never actually stops the timer as dorun is mutex protected, and the stop() is blocked waiting for it. I'm going to add some lock frobbage in here.

[Gavin Hurlbut <ghurlbut@…>]

In 1dc048f1b71c92e4bb81a9e2ae6766cbd2df1c16/mythtv:

Unlock the startstop mutex while waiting for timer startup

If we start and stop the timer too quickly, the lock can be still held by
the start() function, and the stop() sits deadlocked waiting for it. As this
delay is primarily to wait for the thread to come up before returning from
start(), it should be safe to release the mutex while waiting to allow the
stop() to take effect.

If this doesn't fix the issue, please reopen the ticket

Fixes #10867

[Bill Meek <keemllib@…>]

No joy. I ran 50 mythshutdown --logpath /tmp --loglevel debug --lock tests and got the following:

10 core files
 8 Real-time signal 0s
23 Error: Not all threads were shut down properly:

I only looked at the backtrace on the last core file and it appears the same as before v0.26-pre-840-g1dc048f. As before, no mythshutdown...log files were created.

[beirdo]

I can not recreate the problem here at all. As for no logging, it could just be that mythshutdown --lock doesn't log anything. I don't use mythshutdown at all, so I've never really looked.

I'd suggest a make uninstall, delete any remaining installed files, make distclean, make, make install. It could be you have a mix of old and new code somehow.

[beirdo]

Hmm, for logging, I think I see the issue. However, I never see a crash.

[beirdo]

Yes, add -v general to your commandline if you want any logging. mythshutdown has a default mask of -v none. And with that, I see the threads not shut down properly nonsense. Ahh, it's because of no CleanupGuard? in mythshutdown. I'll try to get you a fix momentarily.

[Gavin Hurlbut <ghurlbut@…>]

In fb9b72518b2b2b3912f3ee633f02c185fd93865d/mythtv:

Delete logThread in logStop

Seems I never actually deleted the logThread when stopping logging. This was
causing issues as then the timers don't get stopped (as that's in the dtor)
on quick runs, causing nasty messages, and maybe even crashes.

Refs #10867

[beirdo]

Please let me know if that cleans it up for you. So surprised I never noticed this before.

[Bill Meek <keemllib@…>]

Sorry for the delay, ISP went belly up.

I believe the "Error: Not all threads were shut down properly:" is fixed.

100 Tests on v0.26-pre-842-gfb9b725 adding -v general --quiet --nodblog

 1 core file
19 Real-time signal 0s
 0 Error: Not all threads were shut down properly:

100 with the original test:

 1 core file
16 Real-time signal 0s
 0 Error: Not all threads were shut down properly:

Perhaps specifying --logAnything without -v isn't a valid test, so the next 100 were just mythshutdown --lock.

15 core files
10 Real-time signal 0s
 0 Error: Not all threads were shut down properly:

All builds done with distclean and removal of everything MythTV under /usr/local. Even do a git clean -xfd.

[beirdo]

Ok. Could you post an updated coredump for me so I can have sure I'm chasing the right things? Thanks.

[Bill Meek <keemllib@…>]

mythshutdown --lock running on v0.26-pre-842-gfb9b725

[Bill Meek <keemllib@…>]

Fresh backtrace attached. With my kernel.core_pattern fixed, I see that when my backend runs: mythshutdown -c --quiet --syslog none, it too drops core at times.

[beirdo]

The source of this coredump seems to be something different now. As I can not reproduce this at all, and I can't see it being related to the code I was tweaking anymore, I'm punting this back for others to look at.