Mythbackend failed recordings, deadlocks

[Peter Townsend <elpinkus@…>]

In the last week or so, my backend has been getting into a state where all recordings fail to start. Killing mythbackend and running it again makes things record again for a day or two.

This afternoon two recordings failed to start and mythbackend stopped responding through mythweb or the internal webserver.

I followed the instructions on the wiki for attaching gdb and generating a backtrace. Let me know what other information I can provide.

Running version v0.26-alpha-79-gb379de3. Possibly related to #10870.

[Peter Townsend <elpinkus@…>]

Backend log

[Peter Townsend <elpinkus@…>]

Backtrace

[Peter Townsend <elpinkus@…>]

--version output

[danielk]

What is your "ulimit -u" output ?

Also, what is the output of "cat /proc/sys/vm/overcommit_memory" ?

[Daniel Thor Kristjansson <danielk@…>]

In 9f0be65bfcf192191d8905012f173f80a19f66d5/mythtv:

Refs #10970. Don't print EOF as error if it's expected.

This isn't the cause of any issues, it's just a bit confusing when reading the logs.
When we close a DRB we are expecting a EOF so it's not an error when we see it..

[danielk]

As far as I can tell this shouldn't be happening.

StreamHandler::Start() seems to be waiting for the thread to start. It's waiting for either _running or _error to be set. I don't see any way that DVBStreamHandler::run() can exit without setting either of those. It also seems like DVBStreamHandler::RunTS() is running a piece of code that would only be run after setting _running to true, however that could be in service of another stream. It is possible that MTHread::start() has called QThread::start(), but the thread was never started because we ran out of memory or we exceeded our process allocation. However, that shouldn't occur with any normally configured Linux system. Finally, while _running is not declared as volatile, the while loop that checks it calls a function outside the current translation unit so gcc should not optimise the check away since it can't know that _running isn't set by that call.

[Peter Townsend <elpinkus@…>]

ulimit -u 31520

cat /proc/sys/vm/overcommit_memory 0

[Daniel Kristjansson <danielk@…>]

In 7d01e691d2bb58ed7dcbbdcca93ec06dc3a7381b/mythtv:

Fixes #10970. Fix StreamHandler? restart race.

We wait for _running to be set to false, but we need the thread to be
in the finish state or QThread::start() will exit silently without
starting the thread. Since we're calling QThread::wait() we don't
really need the while (_running) loop at all in either Start() or
Stop(), so I've removed it. This race probably dates back to the
pthread->QThread porting.

[Daniel Kristjansson <danielk@…>]

In eb52783e9f7ad232cbf4afb22dbc604898514e45/mythtv:

Fixes #10970. Fix StreamHandler? restart race.

We wait for _running to be set to false, but we need the thread to be
in the finish state or QThread::start() will exit silently without
starting the thread. Since we're calling QThread::wait() we don't
really need the while (_running) loop at all in either Start() or
Stop(), so I've removed it. This race probably dates back to the
pthread->QThread porting.
(cherry picked from commit 7d01e691d2bb58ed7dcbbdcca93ec06dc3a7381b)