Modify

Opened 3 years ago

Closed 15 months ago

#10924 closed Bug Report - Crash (Won't Fix)

mythlogserver segfault when ending

Reported by: jyavenard Owned by: natanojl
Priority: minor Milestone: unknown
Component: MythTV - Mythlogserver Version: Master Head
Severity: medium Keywords:
Cc: Ticket locked: no

Description

Starting mythlogserver manually, no daemon. After a while it will exit due to inactivity, it will then segfault:

(gdb) bt full
#0  0x00000001036e85c8 in QObject::metaObject ()
No symbol table info available.
#1  0x00000001036eed27 in QObject::disconnect ()
No symbol table info available.
#2  0x000000010219f342 in QObject::disconnect (this=0x109e00040, receiver=0x106d25930, member=0x0) at qobject.h:252
No locals.
#3  0x000000010349a26d in nzmqt::PollingZMQContext::unregisterSocket (this=0x106d25930, socket_=0x109e00040) at nzmqt.hpp:703
	lock = {
  val = 4409416041
}
	poIt = (iterator) 0x109e007f0
	soIt = (iterator) 0x106d28588
#4  0x0000000103499848 in nzmqt::PollingZMQContext::qt_static_metacall (_o=0x106d25930, _c=QMetaObject::InvokeMetaMethod, _id=3, _a=0x106d2e0a0) at moc_nzmqt.cpp:401
	_t = ('nzmqt::PollingZMQContext' *) 0x106d25930
#5  0x00000001036e98e1 in QObject::event ()
No symbol table info available.
#6  0x00000001036d4d7f in QCoreApplicationPrivate::notify_helper ()
No symbol table info available.
#7  0x00000001036d5355 in QCoreApplication::notify ()
No symbol table info available.
#8  0x00000001036d517c in QCoreApplication::notifyInternal ()
No symbol table info available.
#9  0x00000001036d65a0 in QCoreApplicationPrivate::sendPostedEvents ()
No symbol table info available.
#10 0x000000010370a502 in QEventDispatcherUNIX::processEvents ()
No symbol table info available.
#11 0x00000001036d4094 in QEventLoop::processEvents ()
No symbol table info available.
#12 0x00000001036d4444 in QEventLoop::exec ()
No symbol table info available.
#13 0x00000001035bcf08 in QThread::exec ()
No symbol table info available.
#14 0x000000010208e355 in MThreadInternal::exec (this=0x106d23900) at mthread.cpp:85
No locals.
#15 0x000000010208dd89 in MThread::exec (this=0x106d24550) at mthread.cpp:327
No locals.
#16 0x000000010219b1a3 in LogServerThread::run (this=0x106d24540) at loggingserver.cpp:830
	locker = {
  val = 4331302216
}
	ctx = ('nzmqt::PollingZMQContext' *) 0x106d25930
	abortThread = false
	e = (ZMQException &) @0x107913800: {
  <std::exception> = {
    _vptr$exception = 0x10386f3b0
  }, 
  members of zmq::error_t: 
  errnum = 134303232
}
#17 0x000000010219b43c in non-virtual thunk to LogServerThread::run() () at loggingserver.cpp:1389
No symbol table info available.
#18 0x000000010208fb6a in MThreadInternal::run (this=0x106d23900) at mthread.cpp:79
No locals.
#19 0x00000001035c05ba in QThreadPrivate::start ()
No symbol table info available.
#20 0x00007fff86c81782 in _pthread_start ()
No symbol table info available.
#21 0x00007fff86c6e1c1 in thread_start ()
No symbol table info available.

Attachments (4)

valgrind-mythlogserver.txt (8.6 KB) - added by natanojl 3 years ago.
Log from valgrind
mythlogserver-segv.patch (1.2 KB) - added by faginbagin <mythtv@…> 2 years ago.
nzmqt.crash (8.4 KB) - added by jyavenard 2 years ago.
backtrace
mythlogserver.crashlog (51.4 KB) - added by airdrummer 2 years ago.
mythlogserver.crashlog

Download all attachments as: .zip

Change History (33)

comment:1 Changed 3 years ago by beirdo

  • Component changed from MythTV - General to MythTV - Mythlogserver
  • Owner set to beirdo
  • Status changed from new to assigned
  • Type changed from Bug Report - General to Bug Report - Crash
  • Version changed from Unspecified to Master Head

Changed 3 years ago by natanojl

Log from valgrind

comment:2 Changed 3 years ago by natanojl

Valgrind log added and reported upstream at https://github.com/jonnydee/nzmqt/issues/8.

Changed 2 years ago by faginbagin <mythtv@…>

comment:3 Changed 2 years ago by faginbagin <mythtv@…>

I am submitting a patch (mythlogserver-segv.patch) that eliminates the segfault reported here and in other tickets. It changes the parent of the polling socket from the LoggerThread to the polling context. As a result, the polling socket is not deleted when the LoggerThread is deleted and the destroyed signal is never triggered leading to the call to disconnect() the deleted polling socket from the polling context.

This patch should fix the following tickets: #11043 #11378

comment:4 Changed 2 years ago by jyavenard

Mark #11043 and #11378 as duplicate

comment:5 Changed 2 years ago by kenni

Mark #11395 as a duplicate.

comment:6 Changed 2 years ago by Craig Treleaven <ctreleaven@…>

Re mythlogserver-segv.patch, I was doing some testing this morning (no segfaults!) but I think it may be causing another problem. I believe that the client is not answering the heartbeat which leads mythlogserver to expire the client after 5 seconds. Extract from mythlogserver log follows:

2013-02-17 14:05:53.476554 I [77036/24579] LogForward loggingserver.cpp:1310 (forwardMessage) - New Client: 00042aee859b7e4afca074dff43b352167 (#2)
2013-02-17 14:05:53.476560 I [77036/24579] LogForward loggingserver.cpp:1315 (forwardMessage) - Aborting shutdown timer
2013-02-17 14:05:53.476764 I [77036/24579] LogForward loggingserver.cpp:138 (FileLogger) - Added logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540512 I [77036/24579] LogForward loggingserver.cpp:1204 (expireClients) - Expiring client 00042aee859b7e4afca074dff43b352167 (#1)
2013-02-17 14:05:59.540551 I [77036/24579] LogForward loggingserver.cpp:148 (~FileLogger) - Removed logging to /opt/local/var/log/mythtv.26/mythtv-setup.20130217190507.77026.log
2013-02-17 14:05:59.540690 I [77036/24579] LogForward loggingserver.cpp:1234 (expireClients) - Starting 10 sec shutdown timer, MacPorts mod

I found the client being expired 6 seconds after it was created on a regular basis.

comment:7 Changed 2 years ago by faginbagin <mythtv@…>

FWIW, I'm not seeing running clients being expired. mythbackend and mythfrontend client connections are not being expired. The only ones being expired are for mythpreviewgen, presumably being spawned and terminating as I navigate through my recordings. I'm also using syslogging, not file logging. Are you seeing this behavior for clients other than mythtv-setup?

comment:8 Changed 2 years ago by Craig Treleaven <ctreleaven@…>

I don't have a full 0.26 test system so my testing has primarily been with mythtv-setup.

comment:9 Changed 2 years ago by hugh@…

In my case, I just stopped the backend, in preparation to do a database restore. Before I did anything else (I'm a slow), Ubuntu told me a crash happened and asked if I wanted to report it. That's how I'm here.

I've seen this before, I think after I stopped the backend.

comment:10 Changed 2 years ago by jonny.dee@…

I am the author of nzmqt project, which seems to be used by mythTV project. BTW, glad to here this, I had not been aware of this :)

Some time ago a bug has been reported that seems to be related to this bug. Unfortunately, I didn't have time to investigate this. But now I have changed implementation such that no 'destroyed' signal is used for cleanup work anymore (which als makes a call to 'disconnect' obsolete). Instead, cleanup work is done by a direct method call as soon as a socket instance gets deleted.

If it is still an issue for you, would you please try the fixed version and see if the issue is resolved? I don't know if you use 0MQ 2.2.x or 3.2.x release. In the former case you should try out the latest version in 2.x branch. In the latter case use the latest version in master.

If the bugfix works I'd release a new version, so please let me know.

comment:11 Changed 2 years ago by natanojl

  • Owner changed from beirdo to natanojl

comment:12 Changed 2 years ago by Jonatan Lindblad <jlindblad@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In 6f39379cfa8f63c84548d9a25cc33264d27b478e/mythtv:

Resync nzmqt with upstream. Fixes #10924.

comment:13 Changed 2 years ago by natanojl

  • Milestone changed from unknown to 0.27

comment:14 Changed 2 years ago by jyavenard

  • Resolution fixed deleted
  • Status changed from closed to new

comment:15 Changed 2 years ago by jyavenard

update of nzmqt consistently crashes on startup here... so reverting that change for the time being.

comment:16 Changed 2 years ago by jyavenard

  • Milestone changed from 0.27 to unknown

comment:17 Changed 2 years ago by Jean-Yves Avenard <jyavenard@…>

  • Resolution set to fixed
  • Status changed from new to closed

In ced01f695b396326a713cf99cebe3b430229779b/mythtv:

Revert "Resync nzmqt with upstream. Fixes #10924."

Consistently segfault at startup

This reverts commit 6f39379cfa8f63c84548d9a25cc33264d27b478e.

Changed 2 years ago by jyavenard

backtrace

comment:18 Changed 2 years ago by jyavenard

jonny.dee@…: this probably is for you.

following the update to the latest nzmqt (not sure which version was used). The code now crashes at startup. Pretty weird crash, the crash indicates that the Sockets vector has a size of -32. Which when doing append on it will cause a crash as malloc return NULL.

comment:19 Changed 2 years ago by natanojl

Did you try a distclean?

comment:20 Changed 2 years ago by jonny.dee@…

My guess is also that the mentioned crash results from binaries not compatible to each other. So, indeed, a distclean might help here. Would you mind to give it a try?

comment:21 Changed 2 years ago by jyavenard

there has been more than one distclean there :)

actually, everything is built in a jail, and all is wiped out before being re-compiled

comment:22 Changed 2 years ago by jonny.dee@…

Ok, strange... Are the ZMQContext and ZMQSocket instances created in different threads?

comment:23 Changed 2 years ago by jyavenard

comment:24 Changed 2 years ago by jyavenard

  • Resolution fixed deleted
  • Status changed from closed to new

comment:25 Changed 2 years ago by jonny.dee@…

I looked at the sources you pointed out, but I couldn't find any ZMQ related problems. Also I reviewed my nzmqt code and couldn't find any problematic code either. However, the crash report looks like the PollingZMQContext instance is not correctly initialized in memory. In loggingserver.cpp (line 60) there is a global variable definition "LogServerThread? *logServerThread = NULL;". And in logging.cpp (line 307) this variable is accessed for obtaining a ZMQContext instance. I don't know how the whole thing is bootstrapped, but if we assume this variable has never been initialized with a correct pointer to a ZMQContext instance it might contain some non-null value until it is actually initialized with NULL according to loggingserver.cpp, line 60. As the initialization order of static/global variables is not guaranteed, could it be that for some reason logging.cpp accesses variable "logServerThread" before it is ever initialized to NULL? Because in this case the NULL-check in logging.cpp (line 315) would pass and createSocket() method would be called on an invalid ZMQContext instance pointer.

Changed 2 years ago by airdrummer

mythlogserver.crashlog

comment:26 Changed 2 years ago by air_drummer@…

i just installed mythtv on my network (backend on linux, frontend on mac) but it's unusable: the mythlogserver crashes as soon as the f/e launches it, EVERY 6 SECS!-P

comment:27 Changed 2 years ago by air_drummer@…

workaround: start mythlogserver b4 f/e

comment:28 Changed 23 months ago by jonny.dee@…

regaring nzmqt: I've changed code executed if 'ZMQContext' and 'ZMQSocket' instances are deleted a bit. Additionally, I've add (succeeding) unit tests for PUB-SUB, REQ-REP, and PUSH-PULL protocols. Furthermore, nzmqt can now be built as static/shared lib (see documentation for more details).

I don't really know what causes the crash(s) but please make sure that, if several threads are sharing the same 'ZMCContext' instance, the 'ZMQSocket' instances are created in the same thread as the 'ZMQContext' instance itself (0MQ requirement). After creation, use 'QObject::moveToThread(QThread*)' method to move the socket instances to their corresponding threads and give them a reference to their socket instances (Qt requirement). You can see how this is exactly done in nzmqt's unit test source code.

HTH?

comment:29 Changed 15 months ago by jyavenard

  • Resolution set to Won't Fix
  • Status changed from new to closed

mythlogserver is not used on mac bundle any more by default

Add Comment

Modify Ticket

Action
as closed The owner will remain natanojl.
The resolution will be deleted. Next status will be 'new'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.