Opened 8 years ago

Closed 7 years ago

#10924 closed Bug Report - Crash (Won't Fix)

mythlogserver segfault when ending

Reported by: JYA Owned by: Jonatan Lindblad
Priority: minor Milestone: unknown
Component: MythTV - Mythlogserver Version: Master Head
Severity: medium Keywords:
Cc: Ticket locked: no


Starting mythlogserver manually, no daemon. After a while it will exit due to inactivity, it will then segfault:

(gdb) bt full
#0  0x00000001036e85c8 in QObject::metaObject ()
No symbol table info available.
#1  0x00000001036eed27 in QObject::disconnect ()
No symbol table info available.
#2  0x000000010219f342 in QObject::disconnect (this=0x109e00040, receiver=0x106d25930, member=0x0) at qobject.h:252
No locals.
#3  0x000000010349a26d in nzmqt::PollingZMQContext::unregisterSocket (this=0x106d25930, socket_=0x109e00040) at nzmqt.hpp:703
	lock = {
  val = 4409416041
	poIt = (iterator) 0x109e007f0
	soIt = (iterator) 0x106d28588
#4  0x0000000103499848 in nzmqt::PollingZMQContext::qt_static_metacall (_o=0x106d25930, _c=QMetaObject::InvokeMetaMethod, _id=3, _a=0x106d2e0a0) at moc_nzmqt.cpp:401
	_t = ('nzmqt::PollingZMQContext' *) 0x106d25930
#5  0x00000001036e98e1 in QObject::event ()
No symbol table info available.
#6  0x00000001036d4d7f in QCoreApplicationPrivate::notify_helper ()
No symbol table info available.
#7  0x00000001036d5355 in QCoreApplication::notify ()
No symbol table info available.
#8  0x00000001036d517c in QCoreApplication::notifyInternal ()
No symbol table info available.
#9  0x00000001036d65a0 in QCoreApplicationPrivate::sendPostedEvents ()
No symbol table info available.
#10 0x000000010370a502 in QEventDispatcherUNIX::processEvents ()
No symbol table info available.
#11 0x00000001036d4094 in QEventLoop::processEvents ()
No symbol table info available.
#12 0x00000001036d4444 in QEventLoop::exec ()
No symbol table info available.
#13 0x00000001035bcf08 in QThread::exec ()
No symbol table info available.
#14 0x000000010208e355 in MThreadInternal::exec (this=0x106d23900) at mthread.cpp:85
No locals.
#15 0x000000010208dd89 in MThread::exec (this=0x106d24550) at mthread.cpp:327
No locals.
#16 0x000000010219b1a3 in LogServerThread::run (this=0x106d24540) at loggingserver.cpp:830
	locker = {
  val = 4331302216
	ctx = ('nzmqt::PollingZMQContext' *) 0x106d25930
	abortThread = false
	e = (ZMQException &) @0x107913800: {
  <std::exception> = {
    _vptr$exception = 0x10386f3b0
  members of zmq::error_t: 
  errnum = 134303232
#17 0x000000010219b43c in non-virtual thunk to LogServerThread::run() () at loggingserver.cpp:1389
No symbol table info available.
#18 0x000000010208fb6a in MThreadInternal::run (this=0x106d23900) at mthread.cpp:79
No locals.
#19 0x00000001035c05ba in QThreadPrivate::start ()
No symbol table info available.
#20 0x00007fff86c81782 in _pthread_start ()
No symbol table info available.
#21 0x00007fff86c6e1c1 in thread_start ()
No symbol table info available.

Attachments (4)

valgrind-mythlogserver.txt (8.6 KB) - added by Jonatan Lindblad 8 years ago.
Log from valgrind
mythlogserver-segv.patch (1.2 KB) - added by faginbagin <mythtv@…> 8 years ago.
nzmqt.crash (8.4 KB) - added by JYA 7 years ago.
mythlogserver.crashlog (51.4 KB) - added by airdrummer 7 years ago.

Download all attachments as: .zip

Change History (33)

comment:1 Changed 8 years ago by beirdo

Component: MythTV - GeneralMythTV - Mythlogserver
Owner: set to beirdo
Status: newassigned
Type: Bug Report - GeneralBug Report - Crash
Version: UnspecifiedMaster Head

Changed 8 years ago by Jonatan Lindblad

Attachment: valgrind-mythlogserver.txt added

Log from valgrind

comment:2 Changed 8 years ago by Jonatan Lindblad

Valgrind log added and reported upstream at

Changed 8 years ago by faginbagin <mythtv@…>

Attachment: mythlogserver-segv.patch added

comment:3 Changed 8 years ago by faginbagin <mythtv@…>

I am submitting a patch (mythlogserver-segv.patch) that eliminates the segfault reported here and in other tickets. It changes the parent of the polling socket from the LoggerThread to the polling context. As a result, the polling socket is not deleted when the LoggerThread is deleted and the destroyed signal is never triggered leading to the call to disconnect() the deleted polling socket from the polling context.

This patch should fix the following tickets: #11043 #11378

comment:4 Changed 8 years ago by JYA

Mark #11043 and #11378 as duplicate

comment:5 Changed 8 years ago by Kenni Lund [kenni a kelu dot dk]

Mark #11395 as a duplicate.

comment:6 Changed 8 years ago by Craig Treleaven <ctreleaven@…>

Re mythlogserver-segv.patch, I was doing some testing this morning (no segfaults!) but I think it may be causing another problem. I believe that the client is not answering the heartbeat which leads mythlogserver to expire the client after 5 seconds. Extract from mythlogserver log follows:

2013-02-17 14:05:53.476554 I [77036/24579] LogForward loggingserver.cpp:1310 (forwardMessage) - New Client: 00042aee859b7e4afca074dff43b352167 (#2)
2013-02-17 14:05:53.476560 I [77036/24579] LogForward loggingserver.cpp:1315 (forwardMessage) - Aborting shutdown timer
2013-02-17 14:05:53.476764 I [77036/24579] LogForward loggingserver.cpp:138 (FileLogger) - Added logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540512 I [77036/24579] LogForward loggingserver.cpp:1204 (expireClients) - Expiring client 00042aee859b7e4afca074dff43b352167 (#1)
2013-02-17 14:05:59.540551 I [77036/24579] LogForward loggingserver.cpp:148 (~FileLogger) - Removed logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540690 I [77036/24579] LogForward loggingserver.cpp:1234 (expireClients) - Starting 10 sec shutdown timer, MacPorts mod

I found the client being expired 6 seconds after it was created on a regular basis.

comment:7 Changed 8 years ago by faginbagin <mythtv@…>

FWIW, I'm not seeing running clients being expired. mythbackend and mythfrontend client connections are not being expired. The only ones being expired are for mythpreviewgen, presumably being spawned and terminating as I navigate through my recordings. I'm also using syslogging, not file logging. Are you seeing this behavior for clients other than mythtv-setup?

comment:8 Changed 8 years ago by Craig Treleaven <ctreleaven@…>

I don't have a full 0.26 test system so my testing has primarily been with mythtv-setup.

comment:9 Changed 8 years ago by hugh@…

In my case, I just stopped the backend, in preparation to do a database restore. Before I did anything else (I'm a slow), Ubuntu told me a crash happened and asked if I wanted to report it. That's how I'm here.

I've seen this before, I think after I stopped the backend.

comment:10 Changed 8 years ago by jonny.dee@…

I am the author of nzmqt project, which seems to be used by mythTV project. BTW, glad to here this, I had not been aware of this :)

Some time ago a bug has been reported that seems to be related to this bug. Unfortunately, I didn't have time to investigate this. But now I have changed implementation such that no 'destroyed' signal is used for cleanup work anymore (which als makes a call to 'disconnect' obsolete). Instead, cleanup work is done by a direct method call as soon as a socket instance gets deleted.

If it is still an issue for you, would you please try the fixed version and see if the issue is resolved? I don't know if you use 0MQ 2.2.x or 3.2.x release. In the former case you should try out the latest version in 2.x branch. In the latter case use the latest version in master.

If the bugfix works I'd release a new version, so please let me know.

comment:11 Changed 7 years ago by Jonatan Lindblad

Owner: changed from beirdo to Jonatan Lindblad

comment:12 Changed 7 years ago by Jonatan Lindblad <jlindblad@…>

Resolution: fixed
Status: assignedclosed

In 6f39379cfa8f63c84548d9a25cc33264d27b478e/mythtv:

Resync nzmqt with upstream. Fixes #10924.

comment:13 Changed 7 years ago by Jonatan Lindblad

Milestone: unknown0.27

comment:14 Changed 7 years ago by JYA

Resolution: fixed
Status: closednew

comment:15 Changed 7 years ago by JYA

update of nzmqt consistently crashes on startup here... so reverting that change for the time being.

comment:16 Changed 7 years ago by JYA

Milestone: 0.27unknown

comment:17 Changed 7 years ago by Jean-Yves Avenard <jyavenard@…>

Resolution: fixed
Status: newclosed

In ced01f695b396326a713cf99cebe3b430229779b/mythtv:

Revert "Resync nzmqt with upstream. Fixes #10924."

Consistently segfault at startup

This reverts commit 6f39379cfa8f63c84548d9a25cc33264d27b478e.

Changed 7 years ago by JYA

Attachment: nzmqt.crash added


comment:18 Changed 7 years ago by JYA

jonny.dee@…: this probably is for you.

following the update to the latest nzmqt (not sure which version was used). The code now crashes at startup. Pretty weird crash, the crash indicates that the Sockets vector has a size of -32. Which when doing append on it will cause a crash as malloc return NULL.

comment:19 Changed 7 years ago by Jonatan Lindblad

Did you try a distclean?

comment:20 Changed 7 years ago by jonny.dee@…

My guess is also that the mentioned crash results from binaries not compatible to each other. So, indeed, a distclean might help here. Would you mind to give it a try?

comment:21 Changed 7 years ago by JYA

there has been more than one distclean there :)

actually, everything is built in a jail, and all is wiped out before being re-compiled

comment:22 Changed 7 years ago by jonny.dee@…

Ok, strange... Are the ZMQContext and ZMQSocket instances created in different threads?

comment:23 Changed 7 years ago by JYA

comment:24 Changed 7 years ago by JYA

Resolution: fixed
Status: closednew

comment:25 Changed 7 years ago by jonny.dee@…

I looked at the sources you pointed out, but I couldn't find any ZMQ related problems. Also I reviewed my nzmqt code and couldn't find any problematic code either. However, the crash report looks like the PollingZMQContext instance is not correctly initialized in memory. In loggingserver.cpp (line 60) there is a global variable definition "LogServerThread? *logServerThread = NULL;". And in logging.cpp (line 307) this variable is accessed for obtaining a ZMQContext instance. I don't know how the whole thing is bootstrapped, but if we assume this variable has never been initialized with a correct pointer to a ZMQContext instance it might contain some non-null value until it is actually initialized with NULL according to loggingserver.cpp, line 60. As the initialization order of static/global variables is not guaranteed, could it be that for some reason logging.cpp accesses variable "logServerThread" before it is ever initialized to NULL? Because in this case the NULL-check in logging.cpp (line 315) would pass and createSocket() method would be called on an invalid ZMQContext instance pointer.

Changed 7 years ago by airdrummer

Attachment: mythlogserver.crashlog added


comment:26 Changed 7 years ago by air_drummer@…

i just installed mythtv on my network (backend on linux, frontend on mac) but it's unusable: the mythlogserver crashes as soon as the f/e launches it, EVERY 6 SECS!-P

comment:27 Changed 7 years ago by air_drummer@…

workaround: start mythlogserver b4 f/e

comment:28 Changed 7 years ago by jonny.dee@…

regaring nzmqt: I've changed code executed if 'ZMQContext' and 'ZMQSocket' instances are deleted a bit. Additionally, I've add (succeeding) unit tests for PUB-SUB, REQ-REP, and PUSH-PULL protocols. Furthermore, nzmqt can now be built as static/shared lib (see documentation for more details).

I don't really know what causes the crash(s) but please make sure that, if several threads are sharing the same 'ZMCContext' instance, the 'ZMQSocket' instances are created in the same thread as the 'ZMQContext' instance itself (0MQ requirement). After creation, use 'QObject::moveToThread(QThread*)' method to move the socket instances to their corresponding threads and give them a reference to their socket instances (Qt requirement). You can see how this is exactly done in nzmqt's unit test source code.


comment:29 Changed 7 years ago by JYA

Resolution: Won't Fix
Status: newclosed

mythlogserver is not used on mac bundle any more by default

Note: See TracTickets for help on using tickets.