Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#9171 closed defect (Fixed)

Threads not exiting, causing backend to hang

Reported by: dave@… Owned by: beirdo
Priority: minor Milestone: 0.24
Component: MythTV - General Version: 0.24-fixes
Severity: medium Keywords:
Cc: Ticket locked: no

Description

With revision 27076 (and earlier releases of 0.24-fixes) running on Ubuntu 10.10, I'm seeing MythBackend? lock up after recording a program and running commercial detection. For the attached log, I scheduled one recording for 4PM, and two more for 4:30PM. MythBackend? recorded the first program, but failed to record the two following programs and refused all MythWeb or MythFrontend connections.

Attachments (3)

mythbackend.log (38.0 KB) - added by dave@… 9 years ago.
mythbackend.2.log (37.4 KB) - added by dave@… 9 years ago.
Recompiled backend with -v general,extra
mythbackend.3.log (39.3 KB) - added by dave@… 9 years ago.

Download all attachments as: .zip

Change History (20)

Changed 9 years ago by dave@…

Attachment: mythbackend.log added

comment:1 Changed 9 years ago by beirdo

Owner: set to beirdo
Status: newassigned

Please try updating to [27084] and trying again. Additionally, you may want to try with mythbackend -v general,extra.

comment:2 Changed 9 years ago by beirdo

Status: assignedinfoneeded

Changed 9 years ago by dave@…

Attachment: mythbackend.2.log added

Recompiled backend with -v general,extra

comment:3 Changed 9 years ago by anonymous

I rebuilt mythtv revision 27085 with --compile-type=debug, rebooted, and started mythbackend -v general,extra, then waited until the backend locked up again and uploaded the resulting log as mythbackend.2.log

comment:4 Changed 9 years ago by beirdo

OK, that's easily solved. From a mysql login:

SELECT * FROM settings WHERE value = 'Thread/HTTP/Max';

If there are none:

INSERT INTO settings (value, data, hostname) VALUES ('Thread/HTTP/Max', 20, NULL);

If there is one:

UPDATE settings SET data = 20 WHERE value = 'Thread/HTTP/Max';

Tweaking this setting will hopefully be possible from mythtv-setup (or replacement) for 0.25, but for now has to be set manually, either from mysql or from mythweb if you can get into the settings page there. The value of 20 works well for me, but you may be OK with a lower or higher value.

Please be careful :)

Oh, and after setting that, you will need to restart the backend for it to take effect. What is happening is that there are only 5 worker threads in the internal webserver by default, and you've run out.

comment:5 Changed 9 years ago by anonymous

Thanks for the quick responses! I've done as you suggested and queued up a bunch of programs to record tonight. I'll let you know how it goes tomorrow.

comment:6 Changed 9 years ago by dave@…

The backend locked up again, but in looking at the log this morning, it appears that the webserver still has only 5 threads:

2010-11-03 12:21:11.493 Current MythTV Schema Version (DBSchemaVer): 1264
2010-11-03 12:21:11.554 ThreadPool:HTTP: Initial 1, Max 5, Timeout 60000

but the recommended change is definitely in the database:

mysql> SELECT * FROM settings WHERE value = 'Thread/HTTP/Max';
+-----------------+------+----------+
| value           | data | hostname |
+-----------------+------+----------+
| Thread/HTTP/Max | 20   | NULL     |
+-----------------+------+----------+
1 row in set (0.00 sec)

comment:7 Changed 9 years ago by robertm

Sounds like you made the change without restarting the backend...

comment:8 Changed 9 years ago by anonymous

Nope, I made the change last night and restarted today just to make sure that wasn't the case (thus the 12:21 time on the log entry)

comment:9 Changed 9 years ago by beirdo

OK, seems my brain sucks.

ThreadPool/HTTP/Max

I need more caffeine. So sorry.

Last edited 9 years ago by beirdo (previous) (diff)

comment:10 Changed 9 years ago by beirdo

mysql> SELECT * FROM settings WHERE value like 'Thread%';
+---------------------+------+----------+
| value               | data | hostname |
+---------------------+------+----------+
| ThreadPool/HTTP/Max | 20   | NULL     |
+---------------------+------+----------+
1 row in set (0.00 sec)

comment:11 Changed 9 years ago by dave@…

Yep, that did the trick, thanks!

I'll load up another group of programs and see if the problem goes away.

comment:12 Changed 9 years ago by dave@…

As you can see in the attached mythbackend.3.log, the HTTP ThreadPool? was definitely set to 20, but the backend still locked up. I'm pretty sure that this is the first sign that the backend has locked up:

2010-11-03 14:00:14.154 MythCoreContext: Connecting to backend server: 192.168.1.20:6543 (try 1 of 1)
2010-11-03 14:00:21.188 MythSocket(885030:11): readStringList: Error, timed out after 7000 ms.
2010-11-03 14:00:21.233 Protocol version check failure.
                        The response to MYTH_PROTO_VERSION was empty.
                        This happens when the backend is too busy to respond,
                        or has deadlocked in due to bugs or hardware failure.

Changed 9 years ago by dave@…

Attachment: mythbackend.3.log added

comment:13 Changed 9 years ago by beirdo

(In [27090]) Increase the default maximum threadpool size to 25. Seems quite a few users are starting to be bumping against the old max, and we are likely to be reworking this a bit in 0.25.

Refs #9171

comment:14 Changed 9 years ago by beirdo

(In [27091]) Merged [27090] from trunk

Increase the default maximum threadpool size to 25. Seems quite a few users are starting to be bumping against the old max, and we are likely to be reworking this a bit in 0.25.

Refs #9171

comment:15 Changed 9 years ago by sphery

Resolution: Fixed
Status: infoneededclosed

Originally-reported issue is fixed in [27090] and [27091]

Dave, please open another ticket for the socket errors if one of the currently existing socket-related tickets doesn't apply (at least #7608, #8979, and #9138 are open, socket-related tickets; there may be more).

comment:16 Changed 9 years ago by sphery

Milestone: unknown0.24

comment:17 Changed 9 years ago by dave@…

OK, thanks for all the help!

Note: See TracTickets for help on using tickets.