Threads not exiting, causing backend to hang

[dave@…]

With revision 27076 (and earlier releases of 0.24-fixes) running on Ubuntu 10.10, I'm seeing MythBackend? lock up after recording a program and running commercial detection. For the attached log, I scheduled one recording for 4PM, and two more for 4:30PM. MythBackend? recorded the first program, but failed to record the two following programs and refused all MythWeb or MythFrontend connections.

[beirdo]

Please try updating to [27084] and trying again. Additionally, you may want to try with mythbackend -v general,extra.

[dave@…]

Recompiled backend with -v general,extra

[anonymous]

I rebuilt mythtv revision 27085 with --compile-type=debug, rebooted, and started mythbackend -v general,extra, then waited until the backend locked up again and uploaded the resulting log as mythbackend.2.log

[beirdo]

OK, that's easily solved. From a mysql login:

SELECT * FROM settings WHERE value = 'Thread/HTTP/Max';

If there are none:

INSERT INTO settings (value, data, hostname) VALUES ('Thread/HTTP/Max', 20, NULL);

If there is one:

UPDATE settings SET data = 20 WHERE value = 'Thread/HTTP/Max';

Tweaking this setting will hopefully be possible from mythtv-setup (or replacement) for 0.25, but for now has to be set manually, either from mysql or from mythweb if you can get into the settings page there. The value of 20 works well for me, but you may be OK with a lower or higher value.

Please be careful :)

Oh, and after setting that, you will need to restart the backend for it to take effect. What is happening is that there are only 5 worker threads in the internal webserver by default, and you've run out.

[anonymous]

Thanks for the quick responses! I've done as you suggested and queued up a bunch of programs to record tonight. I'll let you know how it goes tomorrow.

[dave@…]

The backend locked up again, but in looking at the log this morning, it appears that the webserver still has only 5 threads:

2010-11-03 12:21:11.493 Current MythTV Schema Version (DBSchemaVer): 1264
2010-11-03 12:21:11.554 ThreadPool:HTTP: Initial 1, Max 5, Timeout 60000

but the recommended change is definitely in the database:

mysql> SELECT * FROM settings WHERE value = 'Thread/HTTP/Max';
+-----------------+------+----------+
| value           | data | hostname |
+-----------------+------+----------+
| Thread/HTTP/Max | 20   | NULL     |
+-----------------+------+----------+
1 row in set (0.00 sec)

[robertm]

Sounds like you made the change without restarting the backend...

[anonymous]

Nope, I made the change last night and restarted today just to make sure that wasn't the case (thus the 12:21 time on the log entry)

[beirdo]

OK, seems my brain sucks.

ThreadPool/HTTP/Max

I need more caffeine. So sorry.

[beirdo]

mysql> SELECT * FROM settings WHERE value like 'Thread%';
+---------------------+------+----------+
| value               | data | hostname |
+---------------------+------+----------+
| ThreadPool/HTTP/Max | 20   | NULL     |
+---------------------+------+----------+
1 row in set (0.00 sec)

[dave@…]

Yep, that did the trick, thanks!

I'll load up another group of programs and see if the problem goes away.

[dave@…]

As you can see in the attached mythbackend.3.log, the HTTP ThreadPool? was definitely set to 20, but the backend still locked up. I'm pretty sure that this is the first sign that the backend has locked up:

2010-11-03 14:00:14.154 MythCoreContext: Connecting to backend server: 192.168.1.20:6543 (try 1 of 1)
2010-11-03 14:00:21.188 MythSocket(885030:11): readStringList: Error, timed out after 7000 ms.
2010-11-03 14:00:21.233 Protocol version check failure.
                        The response to MYTH_PROTO_VERSION was empty.
                        This happens when the backend is too busy to respond,
                        or has deadlocked in due to bugs or hardware failure.

[beirdo]

(In [27090]) Increase the default maximum threadpool size to 25. Seems quite a few users are starting to be bumping against the old max, and we are likely to be reworking this a bit in 0.25.

Refs #9171

[beirdo]

(In [27091]) Merged [27090] from trunk

Increase the default maximum threadpool size to 25. Seems quite a few users are starting to be bumping against the old max, and we are likely to be reworking this a bit in 0.25.

Refs #9171

[sphery]

Originally-reported issue is fixed in [27090] and [27091]

Dave, please open another ticket for the socket errors if one of the currently existing socket-related tickets doesn't apply (at least #7608, #8979, and #9138 are open, socket-related tickets; there may be more).

[dave@…]

OK, thanks for all the help!