libmysql deadlock

[David Asher <asherml@…>]

Ok, so I'm not exactly sure which happened first here -- in fact I think there might be 2 bugs. I was watching a show on my remote frontend. When it was over I did my usual jump to the end to get the popup for whether to delete the show. I pressed the button to tell it to delete. At this point the frontend core dumped with an index out of range ASSERT. Now looking at the logs it looks like that was related to losing connection to the backend. Looking at the backend logs it looks like 3-4 minutes prior, the backend was starting recording of a new show. That recording never fully started up -- there was no recording file in the location the log says it was going to use. In addition the backend appeared to be non-responsive to both mythweb and an attempt to restart mythfrontend.

I have logs from both the frontend and backend, and gdb backtraces of the frontend core dump, the restarted frontend (which never fully started and had to be killed), and the non-responsive backend.

Let me know if there's anything else I can do to help.

[David Asher <asherml@…>]

backtrace of frontend core dump (ASSERT)

[David Asher <asherml@…>]

backtrace of non-responsive backend

[David Asher <asherml@…>]

backtrace of restarted frontend which never fully restarted

[David Asher <asherml@…>]

log from frontend crash (ASSERT)

[David Asher <asherml@…>]

log from non-responsive backend

[David Asher <asherml@…>]

BTW, the frontend log is "important, general, playback" and the backend log is "all, nodatabase, noupnp"

[David Asher <asherml@…>]

I forgot to mention I'm running trunk r21784.

[danielk]

(In [21939]) Refs #7046. Fixes segfault portion of this bug report.

[danielk]

Note: the hang is possibly a dupe of #7046.

[danielk]

Oops: The hang is possibly a dupe of #7078, not itself :)

[danielk]

David A, janneg started looking at this problem. At first blush it appears we died within the mysql library while we were in MSqlDatabase::prepareQuery() which is locked by a static lock. Once we die with that lock in place 99% of mythtv db queries can no longer complete which pretty much stops the application dead in it's tracks.

[danielk]

Changing summary.

While it looks like the underlying problem is with libmysql we haven't eliminated the possibility that we're somehow triggering it or could deal with it in a better way. Hence I'm leaving the ticket open for now.

[danielk]

David A, what distro are you using, and what package revision of libmysql?

[David Asher <asherml@…>]

Distro is Mythbuntu 9.04 (Jaunty)

There appears to be a libmysqlclient16 as well, but I'm using libmysqlclient15off:

$ aptitude show libmysqlclient15off Package: libmysqlclient15off State: installed Automatically installed: no Version: 5.1.30really5.0.75-0ubuntu10.2 Priority: optional Section: libs Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss@…> Uncompressed Size: 4317k Depends: mysql-common (>= 5.1.30really5.0.75-0ubuntu10.2), libc6 (>= 2.4), zlib1g (>= 1:1.1.4) Conflicts: libmysqlclient15, mysql-doc-5.0 (< 5.0.56-0ubuntu1) Replaces: libmysqlclient15, mysql-doc-5.0 (< 5.0.56-0ubuntu1) Description: MySQL database client library

MySQL is a fast, stable and true multi-user, multi-threaded SQL database server. SQL (Structured Query Language) is the most popular database query language in the world. The main goals of MySQL are speed, robustness and ease of use.

This package includes the client library.

Homepage: ​http://dev.mysql.com/

[danielk]

From all appearances this is at it's root a libmysql problem so I'm marking it invalid as a MythTV ticket. But the libmysql deadlock should be reported to Canonical.

[cpinkham]

(In [22561]) Make MSqlQuery's prepareLock non-static as discussed on IRC.

This lock was originally put in a long time ago to hande the fact that QSqlQuery::prepare() was not thread safe in Qt versions prior to v3.3.2. The Qt v4 version is thread safe. We have seen issues with deadlocks in mythbackend due to this lock, so we are removing the static so that different threads do not lock each other trying to prepare a query. If there are no issues seen after the lock is made non-static, we'll commit a larger patch which removes the prepareLock totally.

The issue causing the deadlock currently is with the recorded table and in IRC, we've been able to reproduce this with current trunk. Here is the description of the one situation we've been able to reproduce with 2 concurrent recordings.

Thread #1 calls programinfo.cpp's insert_program. insert_program gets a MySQL table write lock on the recorded table. insert_program then performs a select on the recorded table.

While Thread #1 is performing the select, Thread #2 issues a MSqlQuery::prepare() call in order to update the recorded table. Thread #2 gets our MSqlQuery::prepareLock. MSqlQuery::prepare() in turn calls QSqlQuery::prepare which calls the MySQL prepare function. MySQL sees that the recorded table is locked for write and locks Thread #2 waiting for the write lock to be released.

Meanwhile, Thread #1 finishes its select on the recorded table and goes to perform an insert to insert the information for the new recording. It tries to MSqlQuery::parepare() the query, but locks trying to get the prepareLock lock. Now we have a deadlock. Thread #1 is waiting on Thead #2 to unlock prepareLock and Thread #2 is waiting on Thread #1 to unlock the recorded table.

This does update the binary api version so make clean. It might not have been totally necessary, but you had to recompile anyway because of my previous [22560] commit.

Refs #7046