mythlogserver segfault when ending

[JYA]

Starting mythlogserver manually, no daemon. After a while it will exit due to inactivity, it will then segfault:

(gdb) bt full
#0  0x00000001036e85c8 in QObject::metaObject ()
No symbol table info available.
#1  0x00000001036eed27 in QObject::disconnect ()
No symbol table info available.
#2  0x000000010219f342 in QObject::disconnect (this=0x109e00040, receiver=0x106d25930, member=0x0) at qobject.h:252
No locals.
#3  0x000000010349a26d in nzmqt::PollingZMQContext::unregisterSocket (this=0x106d25930, socket_=0x109e00040) at nzmqt.hpp:703
	lock = {
  val = 4409416041
}
	poIt = (iterator) 0x109e007f0
	soIt = (iterator) 0x106d28588
#4  0x0000000103499848 in nzmqt::PollingZMQContext::qt_static_metacall (_o=0x106d25930, _c=QMetaObject::InvokeMetaMethod, _id=3, _a=0x106d2e0a0) at moc_nzmqt.cpp:401
	_t = ('nzmqt::PollingZMQContext' *) 0x106d25930
#5  0x00000001036e98e1 in QObject::event ()
No symbol table info available.
#6  0x00000001036d4d7f in QCoreApplicationPrivate::notify_helper ()
No symbol table info available.
#7  0x00000001036d5355 in QCoreApplication::notify ()
No symbol table info available.
#8  0x00000001036d517c in QCoreApplication::notifyInternal ()
No symbol table info available.
#9  0x00000001036d65a0 in QCoreApplicationPrivate::sendPostedEvents ()
No symbol table info available.
#10 0x000000010370a502 in QEventDispatcherUNIX::processEvents ()
No symbol table info available.
#11 0x00000001036d4094 in QEventLoop::processEvents ()
No symbol table info available.
#12 0x00000001036d4444 in QEventLoop::exec ()
No symbol table info available.
#13 0x00000001035bcf08 in QThread::exec ()
No symbol table info available.
#14 0x000000010208e355 in MThreadInternal::exec (this=0x106d23900) at mthread.cpp:85
No locals.
#15 0x000000010208dd89 in MThread::exec (this=0x106d24550) at mthread.cpp:327
No locals.
#16 0x000000010219b1a3 in LogServerThread::run (this=0x106d24540) at loggingserver.cpp:830
	locker = {
  val = 4331302216
}
	ctx = ('nzmqt::PollingZMQContext' *) 0x106d25930
	abortThread = false
	e = (ZMQException &) @0x107913800: {
  <std::exception> = {
    _vptr$exception = 0x10386f3b0
  }, 
  members of zmq::error_t: 
  errnum = 134303232
}
#17 0x000000010219b43c in non-virtual thunk to LogServerThread::run() () at loggingserver.cpp:1389
No symbol table info available.
#18 0x000000010208fb6a in MThreadInternal::run (this=0x106d23900) at mthread.cpp:79
No locals.
#19 0x00000001035c05ba in QThreadPrivate::start ()
No symbol table info available.
#20 0x00007fff86c81782 in _pthread_start ()
No symbol table info available.
#21 0x00007fff86c6e1c1 in thread_start ()
No symbol table info available.

[Jonatan Lindblad]

Log from valgrind

[Jonatan Lindblad]

Valgrind log added and reported upstream at ​https://github.com/jonnydee/nzmqt/issues/8.

[faginbagin <mythtv@…>]

I am submitting a patch (mythlogserver-segv.patch) that eliminates the segfault reported here and in other tickets. It changes the parent of the polling socket from the LoggerThread to the polling context. As a result, the polling socket is not deleted when the LoggerThread is deleted and the destroyed signal is never triggered leading to the call to disconnect() the deleted polling socket from the polling context.

This patch should fix the following tickets: #11043 #11378

[JYA]

Mark #11043 and #11378 as duplicate

[Kenni Lund [kenni a kelu dot dk]]

Mark #11395 as a duplicate.

[Craig Treleaven <ctreleaven@…>]

Re mythlogserver-segv.patch, I was doing some testing this morning (no segfaults!) but I think it may be causing another problem. I believe that the client is not answering the heartbeat which leads mythlogserver to expire the client after 5 seconds. Extract from mythlogserver log follows:

2013-02-17 14:05:53.476554 I [77036/24579] LogForward loggingserver.cpp:1310 (forwardMessage) - New Client: 00042aee859b7e4afca074dff43b352167 (#2)
2013-02-17 14:05:53.476560 I [77036/24579] LogForward loggingserver.cpp:1315 (forwardMessage) - Aborting shutdown timer
2013-02-17 14:05:53.476764 I [77036/24579] LogForward loggingserver.cpp:138 (FileLogger) - Added logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540512 I [77036/24579] LogForward loggingserver.cpp:1204 (expireClients) - Expiring client 00042aee859b7e4afca074dff43b352167 (#1)
2013-02-17 14:05:59.540551 I [77036/24579] LogForward loggingserver.cpp:148 (~FileLogger) - Removed logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540690 I [77036/24579] LogForward loggingserver.cpp:1234 (expireClients) - Starting 10 sec shutdown timer, MacPorts mod

I found the client being expired 6 seconds after it was created on a regular basis.

[faginbagin <mythtv@…>]

FWIW, I'm not seeing running clients being expired. mythbackend and mythfrontend client connections are not being expired. The only ones being expired are for mythpreviewgen, presumably being spawned and terminating as I navigate through my recordings. I'm also using syslogging, not file logging. Are you seeing this behavior for clients other than mythtv-setup?

[Craig Treleaven <ctreleaven@…>]

I don't have a full 0.26 test system so my testing has primarily been with mythtv-setup.

[hugh@…]

In my case, I just stopped the backend, in preparation to do a database restore. Before I did anything else (I'm a slow), Ubuntu told me a crash happened and asked if I wanted to report it. That's how I'm here.

I've seen this before, I think after I stopped the backend.

[jonny.dee@…]

I am the author of ​nzmqt project, which seems to be used by mythTV project. BTW, glad to here this, I had not been aware of this :)

Some time ago a ​bug has been reported that seems to be related to this bug. Unfortunately, I didn't have time to investigate this. But now I have changed implementation such that no 'destroyed' signal is used for cleanup work anymore (which als makes a call to 'disconnect' obsolete). Instead, cleanup work is done by a direct method call as soon as a socket instance gets deleted.

If it is still an issue for you, would you please try the fixed version and see if the issue is resolved? I don't know if you use 0MQ 2.2.x or 3.2.x release. In the former case you should try out the latest version in ​2.x branch. In the latter case use the latest version in master.

If the bugfix works I'd release a new version, so please let me know.

[Jonatan Lindblad <jlindblad@…>]

In 6f39379cfa8f63c84548d9a25cc33264d27b478e/mythtv:

Resync nzmqt with upstream. Fixes #10924.

[JYA]

update of nzmqt consistently crashes on startup here... so reverting that change for the time being.

[Jean-Yves Avenard <jyavenard@…>]

In ced01f695b396326a713cf99cebe3b430229779b/mythtv:

Revert "Resync nzmqt with upstream. Fixes #10924."

Consistently segfault at startup

This reverts commit 6f39379cfa8f63c84548d9a25cc33264d27b478e.

[JYA]

backtrace

[JYA]

jonny.dee@…: this probably is for you.

following the update to the latest nzmqt (not sure which version was used). The code now crashes at startup. Pretty weird crash, the crash indicates that the Sockets vector has a size of -32. Which when doing append on it will cause a crash as malloc return NULL.

[Jonatan Lindblad]

Did you try a distclean?

[jonny.dee@…]

My guess is also that the mentioned crash results from binaries not compatible to each other. So, indeed, a distclean might help here. Would you mind to give it a try?

[JYA]

there has been more than one distclean there :)

actually, everything is built in a jail, and all is wiped out before being re-compiled

[jonny.dee@…]

Ok, strange... Are the ZMQContext and ZMQSocket instances created in different threads?

[JYA]

The code is there if you want to review it...

​https://github.com/MythTV/mythtv/blob/master/mythtv/libs/libmythbase/loggingserver.cpp and here: ​https://github.com/MythTV/mythtv/blob/master/mythtv/libs/libmythbase/logging.cpp

those are the two components using zmq

[jonny.dee@…]

I looked at the sources you pointed out, but I couldn't find any ZMQ related problems. Also I reviewed my nzmqt code and couldn't find any problematic code either. However, the crash report looks like the PollingZMQContext instance is not correctly initialized in memory. In loggingserver.cpp (line 60) there is a global variable definition "LogServerThread? *logServerThread = NULL;". And in logging.cpp (line 307) this variable is accessed for obtaining a ZMQContext instance. I don't know how the whole thing is bootstrapped, but if we assume this variable has never been initialized with a correct pointer to a ZMQContext instance it might contain some non-null value until it is actually initialized with NULL according to loggingserver.cpp, line 60. As the initialization order of static/global variables is not guaranteed, could it be that for some reason logging.cpp accesses variable "logServerThread" before it is ever initialized to NULL? Because in this case the NULL-check in logging.cpp (line 315) would pass and createSocket() method would be called on an invalid ZMQContext instance pointer.

[airdrummer]

mythlogserver.crashlog

[air_drummer@…]

i just installed mythtv on my network (backend on linux, frontend on mac) but it's unusable: the mythlogserver crashes as soon as the f/e launches it, EVERY 6 SECS!-P

[air_drummer@…]

workaround: start mythlogserver b4 f/e

[jonny.dee@…]

regaring nzmqt: I've changed code executed if 'ZMQContext' and 'ZMQSocket' instances are deleted a bit. Additionally, I've add (succeeding) unit tests for PUB-SUB, REQ-REP, and PUSH-PULL protocols. Furthermore, nzmqt can now be built as static/shared lib (see ​documentation for more details).

I don't really know what causes the crash(s) but please make sure that, if several threads are sharing the same 'ZMCContext' instance, the 'ZMQSocket' instances are created in the same thread as the 'ZMQContext' instance itself (0MQ requirement). After creation, use 'QObject::moveToThread(QThread*)' method to move the socket instances to their corresponding threads and give them a reference to their socket instances (Qt requirement). You can see how this is exactly done in nzmqt's unit test source code.

HTH?

[JYA]

mythlogserver is not used on mac bundle any more by default