Opened 2 years ago

Closed 22 months ago

#13068 closed Bug Report - Crash (Invalid)

Scheduler crashes before recording starts after backend has been running for a few days

Reported by: WilliamDeRieux@… Owned by: gigem
Priority: major Milestone: 0.28.2
Component: MythTV - Scheduling Version: 0.28.1
Severity: medium Keywords:
Cc: Ticket locked: no

Description

Could be related to ticket (since that issue is only 6 days old): https://code.mythtv.org/trac/ticket/13062 Although that issue is related to the scheduler not scheduling new recordings unless the backend was restarted..

This crash happended 1 minute prior to a recording being started and the backtrace has been included. The issues seems to happen after the backend has been running for a few days -- if it is restarted then new recordings are handled correctly and it does not cause a crash.

I am curently running mythbackend now with (gdb --args mythbackend --loglevel debug -v schedule) and will post a more informative backtrace, but this is what I have for now...

$ mythbackend --version

Please attach all output as a file in bug reports.
MythTV Version : v0.28.1-37-g457a417f40-dirty
MythTV Branch : fixes/0.28
Network Protocol : 88
Library API : 0.28.20161120-1
QT Version : 5.7.1
Options compiled in:
 linux debug use_hidesyms using_alsa using_pulse using_backend using_bdjava using_bindings_perl using_bindings_python using_bindings_php using_crystalhd using_dvb using_firewire using_frontend using_hdhomerun using_vbox using_ceton using_hdpvr using_ivtv using_joystick_menu using_libcec using_libcrypto using_libdns_sd using_libfftw3 using_libxml2 using_lirc using_mheg using_opengl using_opengl_video using_opengl_themepainter using_qtwebkit using_qtscript using_qtdbus using_sdl using_taglib using_v4l2 using_x11 using_xrandr using_xv using_debugtype using_systemd_notify using_bdjava using_bindings_perl using_bindings_python using_bindings_php using_fontconfig using_freetype2 using_mythtranscode using_opengl using_vaapi using_vdpau using_ffmpeg_threads using_mheg using_libass using_libxml2

$ gdb mythbackend backtrace:

2017-07-12 21:59:00.000610 I  New DB connection, total: 6

Thread 10 "Scheduler" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fffc97fa700 (LWP 9058)]
0x00007fffea53861f in __libc_send (fd=47, buf=0x7fffac0131e8, n=5, flags=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
26	../sysdeps/unix/sysv/linux/x86_64/send.c: No such file or directory.
(gdb) bt
#0  0x00007fffea53861f in __libc_send (fd=47, buf=0x7fffac0131e8, n=5, flags=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
#1  0x00007fffd8670c2d in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#2  0x00007fffd864c8f9 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#3  0x00007fffd864cbe3 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#4  0x00007fffd864d292 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#5  0x00007fffd8644a04 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#6  0x00007fffd8642008 in mysql_select_db () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#7  0x00007fffd864204a in mysql_close () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#8  0x00007ffff7feba51 in  () at /usr/lib/x86_64-linux-gnu/qt5/plugins/sqldrivers/libqsqlmysql.so
#9  0x00007ffff3cc649b in MSqlDatabase::~MSqlDatabase() (this=0x7fffac002d90, __in_chrg=<optimized out>) at mythdbcon.cpp:102
#10 0x00007ffff3cc8c59 in MDBManager::PurgeIdleConnections(bool) (this=0x555555a38db0, leaveOne=true) at mythdbcon.cpp:427
#11 0x00007ffff3cc8071 in MDBManager::popConnection(bool) (this=0x555555a38db0, reuse=true) at mythdbcon.cpp:300
#12 0x00007ffff3cc9d13 in MSqlQuery::InitCon(MSqlQuery::ConnectionReuse) (_reuse=MSqlQuery::kNormalConnection) at mythdbcon.cpp:545
#13 0x00007ffff3cd443a in MythDB::GetSetting(QString const&, QString const&) (this=0x555555a376c0, _key=..., defaultval=...) at mythdb.cpp:392
#14 0x00007ffff3cd5836 in MythDB::GetNumSetting(QString const&, int) (this=0x555555a376c0, key=..., defaultval=300) at mythdb.cpp:560
#15 0x00007ffff3d0124c in MythCoreContext::GetNumSetting(QString const&, int) (this=0x555555a2cec0, key=..., defaultval=300) at mythcorecontext.cpp:868
#16 0x0000555555641ccd in Scheduler::run() (this=0x555555a71fc0) at scheduler.cpp:2051
#17 0x00007ffff3caaf75 in MThreadInternal::run() (this=0x555555a69410) at mthread.cpp:79
#18 0x00007fffea7f3da8 in  () at /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007fffea52f494 in start_thread (arg=0x7fffc97fa700) at pthread_create.c:333
#20 0x00007fffe99d4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Change History (20)

comment:1 Changed 2 years ago by WilliamDeRieux@…

Runnig with command...

$ gdb --args mythbackend --loglevel debug -v schedule

Produced a backtrace that was identicial to the one that I had already posted and the debug log output showed nothing different.

It's happening when the Scheduler attempts to run the FillRecordingDir? function...and it happens right before something is about to record.

It crashed again about 22 hours after I opened this ticket.

comment:2 in reply to:  1 ; Changed 2 years ago by WilliamDeRieux@…

Replying to WilliamDeRieux@…:

Runnig with command...

$ gdb --args mythbackend --loglevel debug -v schedule

Produced a backtrace that was identicial to the one that I had already posted and the debug log output showed nothing different.

It's happening when the Scheduler attempts to run the FillRecordingDir? function...and it happens right before something is about to record.

It crashed again about 22 hours after I opened this ticket.

The crash seems to originate after the scheduler::run function tries to read the WakeUpThreshold? setting. In my settings table no setting named WakeUpThreshold? exists, so it would default to 300.

scheduler.cpp:2051 (in function void Scheduler::run(void))
wakeThreshold =
    gCoreContext->GetNumSetting("WakeUpThreshold", 300);

In mythdbcon.cpp:427 (in function void MDBManager::PurgeIdleConnections?(bool leaveOne)) it deletes the idle DB connection

Subsequently after the idle connections are closed the destructor is called in mythdbcon.cpp:102 (MSqlDatabase::~MSqlDatabase())

which attempts to close the db connection -- after the connection had been deleted.


I am no expert but I believe the db connection should be closed before it is deleted....

comment:3 in reply to:  2 Changed 2 years ago by WilliamDeRieux@…

Replying to WilliamDeRieux@…:

I am no expert but I believe the db connection should be closed before it is deleted....

Looking it up -- calling delete will automatically call the destructor.....so this probably won't matter.

In the destructor, however, is where it tries to close the db connection and fails.

I'm now turning on database logging to see if it sheds any more light on the issue...

comment:4 Changed 2 years ago by WilliamDeRieux@…

I just run

$ gdb --args mythbackend --loglevel debug -v database

This is the backtrace now (still not much different)

The problem seems to be a mysql driver issue when closing the idle db connection...

I'm at as loss here trying to figure it out

2017-07-14 11:34:20.598847 D  Running HouseKeeper.
2017-07-14 11:34:20.598858 D  Checking to run DBCleanup
2017-07-14 11:34:20.598862 D  Checking to run HardwareProfiler
2017-07-14 11:34:20.598874 D  Checking to run JobQueueRecover
2017-07-14 11:34:20.598876 D  Checking to run LogClean
2017-07-14 11:34:20.598878 D  Checking to run MythFillDB
2017-07-14 11:34:20.598905 D  MythFillDatabase scheduled to run at Sat Jul 15 05:32:23 2017 GMT.
2017-07-14 11:34:20.598908 D  Checking to run RecordedArtworkUpdate
2017-07-14 11:34:20.598913 D  Checking to run ThemeUpdateNotifications
2017-07-14 11:34:20.598916 I  Queueing HouseKeeperTask 'ThemeUpdateNotifications'.
2017-07-14 11:34:20.598919 D  Checking to run UpdateRadioStreams
2017-07-14 11:34:20.599190 I  MSqlQuery::exec(DBManager1) DELETE FROM settings WHERE value = 'MusicStreamListModified' AND hostname = 'desktop1' ; <<<< Took 0ms
2017-07-14 11:34:20.599202 I  Clearing Settings Cache for 'musicstreamlistmodified'.
2017-07-14 11:34:20.599410 I  MSqlQuery::exec(DBManager1) SELECT data FROM settings WHERE value = 'musicstreamlistmodified' AND hostname = 'desktop1' <<<< Took 0ms, Returned 0 row(s)
2017-07-14 11:34:20.599577 I  MSqlQuery::exec(DBManager1) SELECT data FROM settings WHERE value = 'musicstreamlistmodified' AND hostname IS NULL <<<< Took 0ms, Returned 0 row(s)
2017-07-14 11:34:20.599594 D  Waking HouseKeepingThread.
2017-07-14 11:34:20.599635 I  Database object created: DBManager17
2017-07-14 11:34:20.599639 I  New DB connection, total: 7
2017-07-14 11:34:20.599641 I  Deleting idle DB connection...

Thread 44 "HouseKeeping" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7ffef0ff9700 (LWP 1298)]
0x00007fffea53861f in __libc_send (fd=61, buf=0x7fff0400e348, n=5, flags=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
26	../sysdeps/unix/sysv/linux/x86_64/send.c: No such file or directory.
(gdb) bt
#0  0x00007fffea53861f in __libc_send (fd=61, buf=0x7fff0400e348, n=5, flags=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
#1  0x00007fffd8670c2d in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#2  0x00007fffd864c8f9 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#3  0x00007fffd864cbe3 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#4  0x00007fffd864d292 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#5  0x00007fffd8644a04 in  () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#6  0x00007fffd8642008 in mysql_select_db () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#7  0x00007fffd864204a in mysql_close () at /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
#8  0x00007ffff7feba51 in QMYSQLDriver::close() (this=0x7fff04000d70) at ../../../sql/drivers/mysql/qsql_mysql.cpp:1455
#9  0x00007ffff3cc649b in MSqlDatabase::~MSqlDatabase() (this=0x7fff04000b80, __in_chrg=<optimized out>) at mythdbcon.cpp:102
#10 0x00007ffff3cc8c59 in MDBManager::PurgeIdleConnections(bool) (this=0x555555a3f7a0, leaveOne=true) at mythdbcon.cpp:427
#11 0x00007ffff3cc8071 in MDBManager::popConnection(bool) (this=0x555555a3f7a0, reuse=true) at mythdbcon.cpp:300
#12 0x00007ffff3cc9d13 in MSqlQuery::InitCon(MSqlQuery::ConnectionReuse) (_reuse=MSqlQuery::kNormalConnection) at mythdbcon.cpp:545
#13 0x00007ffff3dbdeae in HouseKeeperTask::UpdateLastRun(QDateTime, bool) (this=0x555555aec2d0, last=..., successful=false) at housekeeper.cpp:217
#14 0x00007ffff3dbef99 in PeriodicHouseKeeperTask::UpdateLastRun(QDateTime, bool) (this=0x555555aec2d0, last=..., successful=false) at housekeeper.cpp:338
#15 0x00007ffff3dc2a96 in HouseKeeperTask::UpdateLastRun(bool) (this=0x555555aec2d0, successful=false) at housekeeper.h:61
#16 0x00007ffff3dbfd6f in HouseKeepingThread::run() (this=0x555555a6a520) at housekeeper.cpp:517
#17 0x00007ffff3caaf75 in MThreadInternal::run() (this=0x555555b13ec0) at mthread.cpp:79
#18 0x00007fffea7f3da8 in  () at /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007fffea52f494 in start_thread (arg=0x7ffef0ff9700) at pthread_create.c:333
#20 0x00007fffe99d4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

comment:5 Changed 2 years ago by WilliamDeRieux@…

This clearly shows one backtrace for the Scheduler and one for HouseKeeping? which suggests that this is really a MySQL driver issue

comment:6 Changed 2 years ago by WilliamDeRieux@…

I just installed some debug symbols for mysql and this now shows a more complete gdb backtrace:

2017-07-15 21:59:30.000847 I  Scheduler: FillRecordingDir: Starting
2017-07-15 21:59:30.000882 I  Database object created: DBManager64
2017-07-15 21:59:30.000887 I  New DB connection, total: 7
2017-07-15 21:59:30.000889 I  Deleting idle DB connection...

Thread 10 "Scheduler" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fffc97fa700 (LWP 31594)]
0x00007fffea53861f in __libc_send (fd=fd@entry=35, buf=buf@entry=0x7fffac056398, n=n@entry=5, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
26	../sysdeps/unix/sysv/linux/x86_64/send.c: No such file or directory.
(gdb) bt
#0  0x00007fffea53861f in __libc_send (fd=fd@entry=35, buf=buf@entry=0x7fffac056398, n=n@entry=5, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
#1  0x00007fffd8670c2d in inline_mysql_socket_send (src_file=0x7fffd8757c28 "/build/mariadb-10.1-Y_aRXm/mariadb-10.1-10.1.23/vio/viosocket.c", src_line=286, flags=0, n=5, buf=0x7fffac056398, mysql_socket=...) at ./include/mysql/psi/mysql_socket.h:759
#2  0x00007fffd8670c2d in vio_write (vio=0x7fffac03d778, buf=0x7fffac056398 "\001", size=5) at ./vio/viosocket.c:285
#3  0x00007fffd864c8f9 in net_real_write(NET*, uchar const*, size_t) (net=net@entry=0x7fffac072448, packet=0x7fffac056398 "\001", len=<optimized out>) at ./sql/net_serv.cc:661
#4  0x00007fffd864cbe3 in net_flush(NET*) (net=net@entry=0x7fffac072448) at ./sql/net_serv.cc:363
#5  0x00007fffd864d292 in net_write_command(NET*, uchar, uchar const*, size_t, uchar const*, size_t) (net=net@entry=0x7fffac072448, command=command@entry=1 '\001', header=header@entry=0x0, head_len=head_len@entry=0, packet=packet@entry=0x0, len=len@entry=0) at ./sql/net_serv.cc:501
#6  0x00007fffd8644a04 in cli_advanced_command (mysql=0x7fffac072448, command=<optimized out>, header=0x0, header_length=0, arg=0x0, arg_length=0, skip_check=1 '\001', stmt=0x0)
    at ./sql-common/client.c:701
#7  0x00007fffd8642008 in mysql_close_slow_part (mysql=mysql@entry=0x7fffac072448) at ./sql-common/client.c:3949
#8  0x00007fffd864204a in mysql_close (mysql=0x7fffac072448) at ./sql-common/client.c:3961
#9  0x00007ffff7feba51 in QMYSQLDriver::close() (this=0x7fffac0348b0) at ../../../sql/drivers/mysql/qsql_mysql.cpp:1455
#10 0x00007ffff3cc649b in MSqlDatabase::~MSqlDatabase() (this=0x7fffac025ef0, __in_chrg=<optimized out>) at mythdbcon.cpp:102
#11 0x00007ffff3cc8c59 in MDBManager::PurgeIdleConnections(bool) (this=0x555555a3f330, leaveOne=true) at mythdbcon.cpp:427
#12 0x00007ffff3cc8071 in MDBManager::popConnection(bool) (this=0x555555a3f330, reuse=true) at mythdbcon.cpp:300
#13 0x00007ffff3cc9d13 in MSqlQuery::InitCon(MSqlQuery::ConnectionReuse) (_reuse=MSqlQuery::kNormalConnection) at mythdbcon.cpp:545
#14 0x000055555565eb4f in Scheduler::FillRecordingDir(QString const&, QString const&, QString const&, QDateTime const&, QDateTime const&, unsigned int, QString&, std::deque<RecordingInfo*, std::allocator<RecordingInfo*> > const&) (this=0x555555a74c10, title=..., hostname=..., storagegroup=..., recstartts=..., recendts=..., cardid=1, recording_dir=..., reclist=std::deque with 33 elements = {...}) at scheduler.cpp:4795
#15 0x0000555555648341 in Scheduler::HandleRecording(RecordingInfo&, bool&, QDateTime&, QDateTime&, int) (this=0x555555a74c10, 
    ri=..., statuschanged=@0x7fffc97f9a13: false, nextStartTime=..., nextWakeTime=..., prerollseconds=0) at scheduler.cpp:2735
#16 0x0000555555642118 in Scheduler::run() (this=0x555555a74c10) at scheduler.cpp:2111
#17 0x00007ffff3caaf75 in MThreadInternal::run() (this=0x555555a62e60) at mthread.cpp:79
#18 0x00007fffea7f3da8 in  () at /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007fffea52f494 in start_thread (arg=0x7fffc97fa700) at pthread_create.c:333
#20 0x00007fffe99d4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

comment:7 in reply to:  6 Changed 2 years ago by WilliamDeRieux@…

Replying to WilliamDeRieux@…:

I just installed some debug symbols for mysql and this now shows a more complete gdb backtrace:

2017-07-15 21:59:30.000847 I  Scheduler: FillRecordingDir: Starting
2017-07-15 21:59:30.000882 I  Database object created: DBManager64
2017-07-15 21:59:30.000887 I  New DB connection, total: 7
2017-07-15 21:59:30.000889 I  Deleting idle DB connection...

Thread 10 "Scheduler" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fffc97fa700 (LWP 31594)]
0x00007fffea53861f in __libc_send (fd=fd@entry=35, buf=buf@entry=0x7fffac056398, n=n@entry=5, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
26	../sysdeps/unix/sysv/linux/x86_64/send.c: No such file or directory.
(gdb) bt
#0  0x00007fffea53861f in __libc_send (fd=fd@entry=35, buf=buf@entry=0x7fffac056398, n=n@entry=5, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:26
#1  0x00007fffd8670c2d in inline_mysql_socket_send (src_file=0x7fffd8757c28 "/build/mariadb-10.1-Y_aRXm/mariadb-10.1-10.1.23/vio/viosocket.c", src_line=286, flags=0, n=5, buf=0x7fffac056398, mysql_socket=...) at ./include/mysql/psi/mysql_socket.h:759
#2  0x00007fffd8670c2d in vio_write (vio=0x7fffac03d778, buf=0x7fffac056398 "\001", size=5) at ./vio/viosocket.c:285
#3  0x00007fffd864c8f9 in net_real_write(NET*, uchar const*, size_t) (net=net@entry=0x7fffac072448, packet=0x7fffac056398 "\001", len=<optimized out>) at ./sql/net_serv.cc:661
#4  0x00007fffd864cbe3 in net_flush(NET*) (net=net@entry=0x7fffac072448) at ./sql/net_serv.cc:363
#5  0x00007fffd864d292 in net_write_command(NET*, uchar, uchar const*, size_t, uchar const*, size_t) (net=net@entry=0x7fffac072448, command=command@entry=1 '\001', header=header@entry=0x0, head_len=head_len@entry=0, packet=packet@entry=0x0, len=len@entry=0) at ./sql/net_serv.cc:501
#6  0x00007fffd8644a04 in cli_advanced_command (mysql=0x7fffac072448, command=<optimized out>, header=0x0, header_length=0, arg=0x0, arg_length=0, skip_check=1 '\001', stmt=0x0)
    at ./sql-common/client.c:701
#7  0x00007fffd8642008 in mysql_close_slow_part (mysql=mysql@entry=0x7fffac072448) at ./sql-common/client.c:3949
#8  0x00007fffd864204a in mysql_close (mysql=0x7fffac072448) at ./sql-common/client.c:3961
#9  0x00007ffff7feba51 in QMYSQLDriver::close() (this=0x7fffac0348b0) at ../../../sql/drivers/mysql/qsql_mysql.cpp:1455
#10 0x00007ffff3cc649b in MSqlDatabase::~MSqlDatabase() (this=0x7fffac025ef0, __in_chrg=<optimized out>) at mythdbcon.cpp:102
#11 0x00007ffff3cc8c59 in MDBManager::PurgeIdleConnections(bool) (this=0x555555a3f330, leaveOne=true) at mythdbcon.cpp:427
#12 0x00007ffff3cc8071 in MDBManager::popConnection(bool) (this=0x555555a3f330, reuse=true) at mythdbcon.cpp:300
#13 0x00007ffff3cc9d13 in MSqlQuery::InitCon(MSqlQuery::ConnectionReuse) (_reuse=MSqlQuery::kNormalConnection) at mythdbcon.cpp:545
#14 0x000055555565eb4f in Scheduler::FillRecordingDir(QString const&, QString const&, QString const&, QDateTime const&, QDateTime const&, unsigned int, QString&, std::deque<RecordingInfo*, std::allocator<RecordingInfo*> > const&) (this=0x555555a74c10, title=..., hostname=..., storagegroup=..., recstartts=..., recendts=..., cardid=1, recording_dir=..., reclist=std::deque with 33 elements = {...}) at scheduler.cpp:4795
#15 0x0000555555648341 in Scheduler::HandleRecording(RecordingInfo&, bool&, QDateTime&, QDateTime&, int) (this=0x555555a74c10, 
    ri=..., statuschanged=@0x7fffc97f9a13: false, nextStartTime=..., nextWakeTime=..., prerollseconds=0) at scheduler.cpp:2735
#16 0x0000555555642118 in Scheduler::run() (this=0x555555a74c10) at scheduler.cpp:2111
#17 0x00007ffff3caaf75 in MThreadInternal::run() (this=0x555555a62e60) at mthread.cpp:79
#18 0x00007fffea7f3da8 in  () at /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007fffea52f494 in start_thread (arg=0x7fffc97fa700) at pthread_create.c:333
#20 0x00007fffe99d4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

The mysql client is libmariadbd18 (10.1.23-9+deb9u1) https://packages.debian.org/stretch/libmariadbd18

I still think this is a driver bug -- unless the mythtv code is not properly setting some parameter, etc

comment:8 Changed 2 years ago by WilliamDeRieux@…

I have worked around this issue for now....

By disabling (in my local source) the call to PurgeIdleConnections?() and it has been working.

I'll have to run it for several days (or a week or so) to see if not calling PurgeIdleConnections?() causes anything other major issues.

I know that PurgeIdleConnections?() is there for a reason, but without a resolution for the issue this might be a valid workaround in the interim.

comment:9 Changed 2 years ago by gigem

Status: newinfoneeded_new

William, since this is a serious problem and nobody else is reporting it, it doess seem to be specific to your system. Just in case it's version specific, can you easily try MariaDB version 10.1.24-6 from Debian testing? That's what I'm using and I haven't had any issues.

comment:10 in reply to:  9 Changed 2 years ago by WilliamDeRieux@…

Replying to gigem:

William, since this is a serious problem and nobody else is reporting it, it doess seem to be specific to your system. Just in case it's version specific, can you easily try MariaDB version 10.1.24-6 from Debian testing? That's what I'm using and I haven't had any issues.

Perhaps using MariaDB would help -- basically I think it's a mysql driver issue.

The thing is, as I reported earlier, I have temporarily worked around the issue on my end by disabling the call to PurgeIdleConnections?() and it has been working for more than a week.

I know PurgeIdleConnections?() is there for a reason, but right now I have it working....(I don't know how long it would take for the idle connections to build up enough to be an issue -- it's likely that it would take months and months of constant running to do it.....so I don't see an immediate issue with this workaround ATM.)

I might try updating to MariaDB if I have any further issues with this...(I think upgrading would require exporting data from mysql to mariadb, or maybe the installer does this for you -- I don't know, but like I said things are working right now and I don't want to break it anymore ATM)

Thanks

PS: I'll update this ticket with more info if I update the mariaDB (curretnly I have 10.1.21-5 available).

comment:11 Changed 2 years ago by acstadt@…

I'm not sure it is only effecting him, though in my case the occurrence rate is so infrequent ( one every 2-3 weeks ) I've never tried nailing it down.

comment:12 Changed 2 years ago by gigem

William, ah, so you're using the MariaDB client library (via libqt5sql5) with a MySQL server, correct? That could be at least part of the problem. While that combination should be compatible, there could be some corner cases that aren't.

acstadt, do you believe you have the same problem? Even if it's only every 2-3 weeks, that's too often. It shouldn't happen at all.

comment:13 in reply to:  12 Changed 2 years ago by WilliamDeRieux@…

Replying to acstadt@…:

I'm not sure it is only effecting him, though in my case the occurrence rate is so infrequent ( one every 2-3 weeks ) I've never tried nailing it down.

Replying to gigem: William, ah, so you're using the MariaDB client library (via libqt5sql5) with a MySQL server, correct? That could be at least part of the problem. While that combination should be compatible, there could be some corner cases that aren't.

acstadt, do you believe you have the same problem? Even if it's only every 2-3 weeks, that's too often. It shouldn't happen at all.

This issue was happening every 24 to 48 hours for me -- that was before I commented out PurgeIdleConnections?.

@gigem What library client do you recommend for libqt5sql5 (5.7.1+dfsg-3+b1) / mysql server (5.6.30-1) ?

Or do you recommened upgrading to mardiadb-client / mariadb-server (10.1.23-9+deb9u1).

I'm just wondering if upgrade/install process will export all of the databases or if I will have to do that manually...

comment:14 Changed 2 years ago by acstadt@…

To be honest, I was hoping (no offence William) that this is the same issue effecting my system, due to the similarities, but not 100% sure. Typically about 12-18 days from a restart (seems to friend on load/usage) the scheduler stops responding though everything else continues to work, at least for a couple days, then I'll normally get a crash just before a recording starts. Unfortunately my job keeps me on the road for a week or two at a time, and my wife, while a good computer user, is not quite up to using gdb.

I've never filed a bug report/ticket because I can't systematically reproduce the issue myself, and don't want to waste a dev's time on something that I can do easily work around (just restart mythbackend before I head out)

Andrew.

comment:15 in reply to:  14 Changed 2 years ago by WilliamDeRieux@…

Replying to acstadt@…:

To be honest, I was hoping (no offence William) that this is the same issue effecting my system, due to the similarities, but not 100% sure. Typically about 12-18 days from a restart (seems to friend on load/usage) the scheduler stops responding though everything else continues to work, at least for a couple days, then I'll normally get a crash just before a recording starts. Unfortunately my job keeps me on the road for a week or two at a time, and my wife, while a good computer user, is not quite up to using gdb.

I've never filed a bug report/ticket because I can't systematically reproduce the issue myself, and don't want to waste a dev's time on something that I can do easily work around (just restart mythbackend before I head out)

Andrew.

These issues might well be related.....but without a gdb backtrace there is no way to know for sure.

Honestly a helper script could be used that would launch the backend using gdb and this script would be setup to log the crash (backtrace) information as well as re-starting the process.

In other words....logging the backtrace and restarting the backend could be automated via script.

comment:16 Changed 2 years ago by gigem

Andrew, a backtrace can be generated from a core dump too. Just enable core dumps for mythbackend. You can then run gdb postmortem when you get home.

William, Debian has chosen MariaDB over MySQL, so you're like to have less trouble with it going forward. It's your decision, though. If you decide to switch, I can offer the guidance below since I recently made the switch myself.

If you make the switch from MySQL 5.6 in Debian 8 to MariaDB 10.1 in Debian 9, it *should* all be handled automatically. You appear to have already upgraded to Debian 9, though, so you've likely got MySQL 5.7 installed now. For reasons that I don't know, the transition from that version isn't handled automatically. For that transition, you'll have to dump and restore the MythTV database (mythconverg by default) manually. That's what I had to do. There's probably an article in the Wiki on how to do it. If there isn't, just ask on the mailing list or IRC.

comment:17 in reply to:  16 ; Changed 2 years ago by WilliamDeRieux@…

Replying to gigem:

Andrew, a backtrace can be generated from a core dump too. Just enable core dumps for mythbackend. You can then run gdb postmortem when you get home.

William, Debian has chosen MariaDB over MySQL, so you're like to have less trouble with it going forward. It's your decision, though. If you decide to switch, I can offer the guidance below since I recently made the switch myself.

If you make the switch from MySQL 5.6 in Debian 8 to MariaDB 10.1 in Debian 9, it *should* all be handled automatically. You appear to have already upgraded to Debian 9, though, so you've likely got MySQL 5.7 installed now. For reasons that I don't know, the transition from that version isn't handled automatically. For that transition, you'll have to dump and restore the MythTV database (mythconverg by default) manually. That's what I had to do. There's probably an article in the Wiki on how to do it. If there isn't, just ask on the mailing list or IRC.

I have mysql-server 5.6.30-1 installed (this was from debian 8) and the upgrade to mariaDB 10.1 should be automatic ?

mysql-server 5.7.18-1 is from sid/unstable -- which is not installed.....I suspect that the upgrade is not handled because of the way that unstable works (they are packages that are not ready for prime-time). Most people would not install from unstable and then upgrade to testing -- as this often breaks things.

comment:18 in reply to:  17 Changed 2 years ago by WilliamDeRieux@…

Replying to WilliamDeRieux@…:

Replying to gigem:

Andrew, a backtrace can be generated from a core dump too. Just enable core dumps for mythbackend. You can then run gdb postmortem when you get home.

William, Debian has chosen MariaDB over MySQL, so you're like to have less trouble with it going forward. It's your decision, though. If you decide to switch, I can offer the guidance below since I recently made the switch myself.

If you make the switch from MySQL 5.6 in Debian 8 to MariaDB 10.1 in Debian 9, it *should* all be handled automatically. You appear to have already upgraded to Debian 9, though, so you've likely got MySQL 5.7 installed now. For reasons that I don't know, the transition from that version isn't handled automatically. For that transition, you'll have to dump and restore the MythTV database (mythconverg by default) manually. That's what I had to do. There's probably an article in the Wiki on how to do it. If there isn't, just ask on the mailing list or IRC.

I have mysql-server 5.6.30-1 installed (this was from debian 8) and the upgrade to mariaDB 10.1 should be automatic ?

mysql-server 5.7.18-1 is from sid/unstable -- which is not installed.....I suspect that the upgrade is not handled because of the way that unstable works (they are packages that are not ready for prime-time). Most people would not install from unstable and then upgrade to testing -- as this often breaks things.

I believe the term is called: FrankenDebian? (which is a mix of stable/testing/unstable/etc)

https://wiki.debian.org/DontBreakDebian#Don.27t_make_a_FrankenDebian

comment:19 Changed 2 years ago by gigem

Hmm, I didn't realize MySQL 5.7 didn't make it into Debian 9. I actually use Debian testing, which is kind of a rolling/staging distribution for the next stable version. That's how I wound up with 5.7. I do know that the MariaDB install scripts explicitly list MySQL 5.6 as the maximum upgradeable version. So, in your case, the upgrade *should* work seamlessly. I would still make backups before attempting it, though. That would include running mythconverg_backup.pl *and* saving a copy of the /var/lib/mysql directory.

comment:20 Changed 22 months ago by gigem

Resolution: Invalid
Status: infoneeded_newclosed

William, do you have any new information? Since this appears to be a database issue, I'm closing this ticket for the time being. If you get anything new that shows otherwise, please reopen it.

Note: See TracTickets for help on using tickets.