Modify
Warning Please read the Ticket HowTo before creating or commenting on a ticket. Failure to do so may cause your ticket to be rejected or result in a slower response.

Opened 21 months ago

Last modified 7 months ago

#10924 new Bug Report - Crash

mythlogserver segfault when ending

Reported by: jyavenard Owned by: natanojl
Priority: minor Milestone: unknown
Component: MythTV - Mythlogserver Version: Master Head
Severity: medium Keywords:
Cc: Ticket locked: no

Description

Starting mythlogserver manually, no daemon.
After a while it will exit due to inactivity, it will then segfault:

(gdb) bt full
#0  0x00000001036e85c8 in QObject::metaObject ()
No symbol table info available.
#1  0x00000001036eed27 in QObject::disconnect ()
No symbol table info available.
#2  0x000000010219f342 in QObject::disconnect (this=0x109e00040, receiver=0x106d25930, member=0x0) at qobject.h:252
No locals.
#3  0x000000010349a26d in nzmqt::PollingZMQContext::unregisterSocket (this=0x106d25930, socket_=0x109e00040) at nzmqt.hpp:703
	lock = {
  val = 4409416041
}
	poIt = (iterator) 0x109e007f0
	soIt = (iterator) 0x106d28588
#4  0x0000000103499848 in nzmqt::PollingZMQContext::qt_static_metacall (_o=0x106d25930, _c=QMetaObject::InvokeMetaMethod, _id=3, _a=0x106d2e0a0) at moc_nzmqt.cpp:401
	_t = ('nzmqt::PollingZMQContext' *) 0x106d25930
#5  0x00000001036e98e1 in QObject::event ()
No symbol table info available.
#6  0x00000001036d4d7f in QCoreApplicationPrivate::notify_helper ()
No symbol table info available.
#7  0x00000001036d5355 in QCoreApplication::notify ()
No symbol table info available.
#8  0x00000001036d517c in QCoreApplication::notifyInternal ()
No symbol table info available.
#9  0x00000001036d65a0 in QCoreApplicationPrivate::sendPostedEvents ()
No symbol table info available.
#10 0x000000010370a502 in QEventDispatcherUNIX::processEvents ()
No symbol table info available.
#11 0x00000001036d4094 in QEventLoop::processEvents ()
No symbol table info available.
#12 0x00000001036d4444 in QEventLoop::exec ()
No symbol table info available.
#13 0x00000001035bcf08 in QThread::exec ()
No symbol table info available.
#14 0x000000010208e355 in MThreadInternal::exec (this=0x106d23900) at mthread.cpp:85
No locals.
#15 0x000000010208dd89 in MThread::exec (this=0x106d24550) at mthread.cpp:327
No locals.
#16 0x000000010219b1a3 in LogServerThread::run (this=0x106d24540) at loggingserver.cpp:830
	locker = {
  val = 4331302216
}
	ctx = ('nzmqt::PollingZMQContext' *) 0x106d25930
	abortThread = false
	e = (ZMQException &) @0x107913800: {
  <std::exception> = {
    _vptr$exception = 0x10386f3b0
  }, 
  members of zmq::error_t: 
  errnum = 134303232
}
#17 0x000000010219b43c in non-virtual thunk to LogServerThread::run() () at loggingserver.cpp:1389
No symbol table info available.
#18 0x000000010208fb6a in MThreadInternal::run (this=0x106d23900) at mthread.cpp:79
No locals.
#19 0x00000001035c05ba in QThreadPrivate::start ()
No symbol table info available.
#20 0x00007fff86c81782 in _pthread_start ()
No symbol table info available.
#21 0x00007fff86c6e1c1 in thread_start ()
No symbol table info available.

Attachments (4)

valgrind-mythlogserver.txt (8.6 KB) - added by natanojl 15 months ago.
Log from valgrind
mythlogserver-segv.patch (1.2 KB) - added by faginbagin <mythtv@…> 15 months ago.
nzmqt.crash (8.4 KB) - added by jyavenard 8 months ago.
backtrace
mythlogserver.crashlog (51.4 KB) - added by airdrummer 8 months ago.
mythlogserver.crashlog

Download all attachments as: .zip

Change History (32)

comment:1 Changed 21 months ago by beirdo

  • Component changed from MythTV - General to MythTV - Mythlogserver
  • Owner set to beirdo
  • Status changed from new to assigned
  • Type changed from Bug Report - General to Bug Report - Crash
  • Version changed from Unspecified to Master Head

Changed 15 months ago by natanojl

Log from valgrind

comment:2 Changed 15 months ago by natanojl

Valgrind log added and reported upstream at https://github.com/jonnydee/nzmqt/issues/8.

Changed 15 months ago by faginbagin <mythtv@…>

comment:3 Changed 15 months ago by faginbagin <mythtv@…>

I am submitting a patch (mythlogserver-segv.patch) that eliminates the segfault reported here and in other tickets. It changes the parent of the polling socket from the LoggerThread to the polling context. As a result, the polling socket is not deleted when the LoggerThread is deleted and the destroyed signal is never triggered leading to the call to disconnect() the deleted polling socket from the polling context.

This patch should fix the following tickets:
#11043
#11378

comment:4 Changed 15 months ago by jyavenard

Mark #11043 and #11378 as duplicate

comment:5 Changed 14 months ago by kenni

Mark #11395 as a duplicate.

comment:6 Changed 14 months ago by Craig Treleaven <ctreleaven@…>

Re mythlogserver-segv.patch, I was doing some testing this morning (no segfaults!) but I think it may be causing another problem. I believe that the client is not answering the heartbeat which leads mythlogserver to expire the client after 5 seconds. Extract from mythlogserver log follows:

2013-02-17 14:05:53.476554 I [77036/24579] LogForward loggingserver.cpp:1310 (forwardMessage) - New Client: 00042aee859b7e4afca074dff43b352167 (#2)
2013-02-17 14:05:53.476560 I [77036/24579] LogForward loggingserver.cpp:1315 (forwardMessage) - Aborting shutdown timer
2013-02-17 14:05:53.476764 I [77036/24579] LogForward loggingserver.cpp:138 (FileLogger) - Added logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540512 I [77036/24579] LogForward loggingserver.cpp:1204 (expireClients) - Expiring client 00042aee859b7e4afca074dff43b352167 (#1)
2013-02-17 14:05:59.540551 I [77036/24579] LogForward loggingserver.cpp:148 (~FileLogger) - Removed logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540690 I [77036/24579] LogForward loggingserver.cpp:1234 (expireClients) - Starting 10 sec shutdown timer, MacPorts mod

I found the client being expired 6 seconds after it was created on a regular basis.

comment:7 Changed 14 months ago by faginbagin <mythtv@…>

FWIW, I'm not seeing running clients being expired. mythbackend and mythfrontend client connections are not being expired. The only ones being expired are for mythpreviewgen, presumably being spawned and terminating as I navigate through my recordings. I'm also using syslogging, not file logging. Are you seeing this behavior for clients other than mythtv-setup?

comment:8 Changed 14 months ago by Craig Treleaven <ctreleaven@…>

I don't have a full 0.26 test system so my testing has primarily been with mythtv-setup.

comment:9 Changed 13 months ago by hugh@…

In my case, I just stopped the backend, in preparation to do a database restore. Before I did anything else (I'm a slow), Ubuntu told me a crash happened and asked if I wanted to report it. That's how I'm here.

I've seen this before, I think after I stopped the backend.

comment:10 Changed 12 months ago by jonny.dee@…

I am the author of nzmqt project, which seems to be used by mythTV project. BTW, glad to here this, I had not been aware of this :)

Some time ago a bug has been reported that seems to be related to this bug. Unfortunately, I didn't have time to investigate this. But now I have changed implementation such that no 'destroyed' signal is used for cleanup work anymore (which als makes a call to 'disconnect' obsolete). Instead, cleanup work is done by a direct method call as soon as a socket instance gets deleted.

If it is still an issue for you, would you please try the fixed version and see if the issue is resolved? I don't know if you use 0MQ 2.2.x or 3.2.x release. In the former case you should try out the latest version in 2.x branch. In the latter case use the latest version in master.

If the bugfix works I'd release a new version, so please let me know.

comment:11 Changed 8 months ago by natanojl

  • Owner changed from beirdo to natanojl

comment:12 Changed 8 months ago by Jonatan Lindblad <jlindblad@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In 6f39379cfa8f63c84548d9a25cc33264d27b478e/mythtv:

Resync nzmqt with upstream. Fixes #10924.

comment:13 Changed 8 months ago by natanojl

  • Milestone changed from unknown to 0.27

comment:14 Changed 8 months ago by jyavenard

  • Resolution fixed deleted
  • Status changed from closed to new

comment:15 Changed 8 months ago by jyavenard

update of nzmqt consistently crashes on startup here... so reverting that change for the time being.

comment:16 Changed 8 months ago by jyavenard

  • Milestone changed from 0.27 to unknown

comment:17 Changed 8 months ago by Jean-Yves Avenard <jyavenard@…>

  • Resolution set to fixed
  • Status changed from new to closed

In ced01f695b396326a713cf99cebe3b430229779b/mythtv:

Revert "Resync nzmqt with upstream. Fixes #10924."

Consistently segfault at startup

This reverts commit 6f39379cfa8f63c84548d9a25cc33264d27b478e.

Changed 8 months ago by jyavenard

backtrace

comment:18 Changed 8 months ago by jyavenard

jonny.dee@…: this probably is for you.

following the update to the latest nzmqt (not sure which version was used). The code now crashes at startup.
Pretty weird crash, the crash indicates that the Sockets vector has a size of -32. Which when doing append on it will cause a crash as malloc return NULL.

comment:19 Changed 8 months ago by natanojl

Did you try a distclean?

comment:20 Changed 8 months ago by jonny.dee@…

My guess is also that the mentioned crash results from binaries not compatible to each other. So, indeed, a distclean might help here. Would you mind to give it a try?

comment:21 Changed 8 months ago by jyavenard

there has been more than one distclean there :)

actually, everything is built in a jail, and all is wiped out before being re-compiled

comment:22 Changed 8 months ago by jonny.dee@…

Ok, strange... Are the ZMQContext and ZMQSocket instances created in different threads?

comment:23 Changed 8 months ago by jyavenard

comment:24 Changed 8 months ago by jyavenard

  • Resolution fixed deleted
  • Status changed from closed to new

comment:25 Changed 8 months ago by jonny.dee@…

I looked at the sources you pointed out, but I couldn't find any ZMQ related problems. Also I reviewed my nzmqt code and couldn't find any problematic code either. However, the crash report looks like the PollingZMQContext instance is not correctly initialized in memory. In loggingserver.cpp (line 60) there is a global variable definition "LogServerThread? *logServerThread = NULL;". And in logging.cpp (line 307) this variable is accessed for obtaining a ZMQContext instance.
I don't know how the whole thing is bootstrapped, but if we assume this variable has never been initialized with a correct pointer to a ZMQContext instance it might contain some non-null value until it is actually initialized with NULL according to loggingserver.cpp, line 60. As the initialization order of static/global variables is not guaranteed, could it be that for some reason logging.cpp accesses variable "logServerThread" before it is ever initialized to NULL? Because in this case the NULL-check in logging.cpp (line 315) would pass and createSocket() method would be called on an invalid ZMQContext instance pointer.

Changed 8 months ago by airdrummer

mythlogserver.crashlog

comment:26 Changed 8 months ago by air_drummer@…

i just installed mythtv on my network (backend on linux, frontend on mac) but it's unusable: the mythlogserver crashes as soon as the f/e launches it, EVERY 6 SECS!-P

comment:27 Changed 8 months ago by air_drummer@…

workaround: start mythlogserver b4 f/e

comment:28 Changed 7 months ago by jonny.dee@…

regaring nzmqt: I've changed code executed if 'ZMQContext' and 'ZMQSocket' instances are deleted a bit. Additionally, I've add (succeeding) unit tests for PUB-SUB, REQ-REP, and PUSH-PULL protocols. Furthermore, nzmqt can now be built as static/shared lib (see documentation for more details).

I don't really know what causes the crash(s) but please make sure that, if several threads are sharing the same 'ZMCContext' instance, the 'ZMQSocket' instances are created in the same thread as the 'ZMQContext' instance itself (0MQ requirement). After creation, use 'QObject::moveToThread(QThread*)' method to move the socket instances to their corresponding threads and give them a reference to their socket instances (Qt requirement). You can see how this is exactly done in nzmqt's unit test source code.

HTH?

Add Comment

Modify Ticket

Action
as new .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.