Deadlock on slave backend disconnect

[Ian Dall <ian@…>]

I have a setup with a master BE, 2 slave BEs and 1 - 3 frontends. I am running code compiled from git (v0.25pre-2145-gf199a84-dirty).

The behaviour is that nothing works and one slave is dead and the master backend is deadlocked. FE's don't work and accessing the status port with a browser times-out.

The slave death is accompanied by kernel syslog messages like: kernel: [371375.689820] mythbackend: page allocation failure. order:0, mode:0x20 Maybe the slave death is due to a kernel bug, BUT the master should not deadlock!

The attached backtrace shows that master Thread 16 is trying, in SlaveDisconnected? to get a shedlock, when shedlock is already held by Scheduler::run further up the stack.

I saw exactly the same problem with an older version: 0.24-7.fc14 (464fa28373) but went to git head in the hope that this had been fixed :-(

I notice other deadlock tickets (esp #9745), but none seem to have quite the same description, and the version I am running has the #9745 fix included.

[Ian Dall <ian@…>]

Backtrace all threads

[Ian Dall <ian@…>]

Master backend log.

[Jonatan <mythtv@…>]

I have also seen a similar deadlock a few times on 0.24. I have been running the backend with the attached patch for a while now without any problems.

[Ian Dall <ian@…>]

I haven't tried Jonatan's patch yet but will soon (I missed it until now).

Without the patch, the issue is still there as of version: v0.25pre-2563-ga41e965

See thread 21 in the attached backtrace.

[Ian Dall <ian@…>]

Backtrace all threads as of commit a41e965

[Ian Dall <ian@…>]

Master backend log as of commit a41e965

[Ian Dall <ian@…>]

I have been running with Jonatan's patch for a week now and it seems to fix the problem. I deliberately tried to provoke it by killing and restarting the backend many times and never saw the deadlock.

Can this patch be applied?

[danielk]

Ian, Janathan's patch is more of a debugging patch, it just disables a bit of code. And can't be applied as is. But if it fixed the problem for you, it does show that you are both experiencing the same deadlock and the same fix will help both of you.

[Github]

Refs #9885. Fixes deadlock when a slave backend disconnect is first seen from within the Scheduler thread. Patch by Ian Dall.

Keeping ticket open since this should be backported to 0.24-fixes.

Branch: master Changeset: 1fae22a8bc56a5474375332f7799b0ee91bb6244

[danielk]

Fixed in [3a5f78862a5be39e02ee549551d59e9c40fa575d]

Fixes #9885. Fixes deadlock when a slave backend disconnect is first seen from within the Scheduler thread. Patch by Ian Dall.

This has been running in master for 4 weeks without reports of regression.