Opened 11 years ago

Closed 10 years ago

#7195 closed task (fixed)

Fix watch recordings preview image loading speed regression

Reported by: danielk Owned by: danielk
Priority: minor Milestone: unknown
Component: MythTV - General Version: head
Severity: medium Keywords:
Cc: stuartm Ticket locked: no

Description

Due to the way preview images are loaded now extra connections are made and extra work performed as compared to 0.21. The attached patch should speed up remote fronends a bit by skipping these steps for all previews less than 200 kb in size.

Note: To test reliably run "rm -f ~/.mythtv/remotecache" prior to starting mythfrontend. Otherwise caching can effectively hide this regression until you browse into a category with seldom browsed recordings.

Attachments (11)

7195-v1.patch (20.8 KB) - added by danielk 11 years ago.
Not extensively tested yet. In particular this has not been tested with sbe's
7195-v2.patch (22.3 KB) - added by danielk 11 years ago.
updated patch
7191-gen2-v1.patch (63.8 KB) - added by danielk 10 years ago.
new set of changes to preview generation.
7191-gen2-v2.patch (77.8 KB) - added by danielk 10 years ago.
updated patch
7191-gen2-v3.patch (80.5 KB) - added by danielk 10 years ago.
7191-gen2-v4.patch (80.1 KB) - added by danielk 10 years ago.
Updated to trunk.
7191-gen2-v5.patch (85.1 KB) - added by danielk 10 years ago.
7191-gen2-v6.patch (102.0 KB) - added by danielk 10 years ago.
Updated patch
7195-gen2-v7.patch (104.4 KB) - added by danielk 10 years ago.
Updated patch
7195-gen2-v8.patch (104.7 KB) - added by danielk 10 years ago.
Updated patch (sans mythversion.cpp)
7195-gen2-v9.patch (110.5 KB) - added by danielk 10 years ago.

Download all attachments as: .zip

Change History (23)

Changed 11 years ago by danielk

Attachment: 7195-v1.patch added

Not extensively tested yet. In particular this has not been tested with sbe's

comment:1 Changed 11 years ago by danielk

(In [22108]) Refs #7195. Fixes a spelling mistake and checks for a failed mkdir. Noticed both problems when looking at the watch recordings preview image loading code.

Changed 11 years ago by danielk

Attachment: 7195-v2.patch added

updated patch

comment:2 Changed 11 years ago by danielk

(In [22164]) Refs #7195. Speed up Watch Recordings preview pixmap downloads from remote backends.

Adds a new myth proto command, so all frontends and backends need to be updated.

comment:3 Changed 10 years ago by danielk

Below is a log showing some of the inefficiencies of preview generation right now. First we try multiple times to locate the remote file on the locally accessible recording directories, in this case there is a local /jessdell-video directory, but this doesn't currently mount the remote directory.

Eventually we realize we need one of the backends to generate the preview, but then we create a local path in PreviewGenerator::CreateAccessibleFilename?() and try to download it with RemoteFile?.. which of course does not work... I've left out the multiple iterations this goes through before finally locating the preview...

2009-12-03 22:05:07.554 SG(Default): FindRecordingFile: Searching for '1139_20091203180000.mpg'           
2009-12-03 22:05:07.554 SG(Default): FindRecordingDir: Checking '/jessdell-video' for '/jessdell-video/1139_20091203180000.mpg'                                                                                     
2009-12-03 22:05:07.554 SG(Default): FindRecordingDir: Checking '/living-video' for '/living-video/1139_20091203180000.mpg'                                                                                         
2009-12-03 22:05:07.554 SG(Default): FindRecordingDir: Checking '/bedroom-storage1' for '/bedroom-storage1/1139_20091203180000.mpg'                                                                                 
2009-12-03 22:05:07.554 SG(Default): FindRecordingDir: Checking '/bedroom-storage2' for '/bedroom-storage2/1139_20091203180000.mpg'                                                                                 
2009-12-03 22:05:07.554 SG(Default): FindRecordingDir: Checking '/tmp' for '/tmp/1139_20091203180000.mpg' 
2009-12-03 22:05:07.554 SG(Default): FindRecordingDir: Checking '/video' for '/video/1139_20091203180000.mpg'                                                                                                       
2009-12-03 22:05:07.556 SG(): FindRecordingDir: Checking '/tmp' for '/tmp/1139_20091203180000.mpg'        
2009-12-03 22:05:07.556 SG(): FindRecordingDir: Checking '/jessdell-video' for '/jessdell-video/1139_20091203180000.mpg'                                                                                            
2009-12-03 22:05:07.556 SG(): FindRecordingDir: Checking '/living-video' for '/living-video/1139_20091203180000.mpg'                                                                                                
2009-12-03 22:05:07.556 SG(): FindRecordingDir: Checking '/bedroom-storage1' for '/bedroom-storage1/1139_20091203180000.mpg'                                                                                        
2009-12-03 22:05:07.556 SG(): FindRecordingDir: Checking '/bedroom-storage2' for '/bedroom-storage2/1139_20091203180000.mpg'                                                                                        
2009-12-03 22:05:07.556 SG(): FindRecordingDir: Checking '/video' for '/video/1139_20091203180000.mpg'    
2009-12-03 22:05:07.556 SG(): FindRecordingDir: Checking '/video/livetv' for '/video/livetv/1139_20091203180000.mpg'                                                                                                
2009-12-03 22:05:07.556 SG(Default) Error: FindRecordingFile: Unable to find '1139_20091203180000.mpg'!   
2009-12-03 22:05:07.556 ProgramInfo: GetPlaybackURL: Using default of: 'myth://192.168.1.160:6543/1139_20091203180000.mpg'                                                                                          
2009-12-03 22:05:07.558 MythSocket(23cab10:36): write -> 36 645     QUERY_PIXMAP_GET_IF_MODIFIED[]:[]1259895718[]:[]204800[]:[]Seinfe...                                                                            
2009-12-03 22:05:07.577 MythSocket(23cab10:36): read  <- 36 93467   1259895719[]:[]70073[]:[]57538[]:[]iVBORw0KGgoAAAANSUhEUgAAAUAAAA...                                                                            
2009-12-03 22:05:07.584 SG(Default): FindRecordingFile: Searching for '1139_20091203180000.mpg'           
2009-12-03 22:05:07.584 SG(Default): FindRecordingDir: Checking '/jessdell-video' for '/jessdell-video/1139_20091203180000.mpg'                                                                                     
2009-12-03 22:05:07.584 SG(Default): FindRecordingDir: Checking '/living-video' for '/living-video/1139_20091203180000.mpg'                                                                                         
2009-12-03 22:05:07.584 SG(Default): FindRecordingDir: Checking '/bedroom-storage1' for '/bedroom-storage1/1139_20091203180000.mpg'                                                                                 
2009-12-03 22:05:07.584 SG(Default): FindRecordingDir: Checking '/bedroom-storage2' for '/bedroom-storage2/1139_20091203180000.mpg'                                                                                 
2009-12-03 22:05:07.584 SG(Default): FindRecordingDir: Checking '/tmp' for '/tmp/1139_20091203180000.mpg'
2009-12-03 22:05:07.584 SG(Default): FindRecordingDir: Checking '/video' for '/video/1139_20091203180000.mpg'
2009-12-03 22:05:07.585 SG(): FindRecordingDir: Checking '/tmp' for '/tmp/1139_20091203180000.mpg'
2009-12-03 22:05:07.585 SG(): FindRecordingDir: Checking '/jessdell-video' for '/jessdell-video/1139_20091203180000.mpg'
2009-12-03 22:05:07.585 SG(): FindRecordingDir: Checking '/living-video' for '/living-video/1139_20091203180000.mpg'
2009-12-03 22:05:07.585 SG(): FindRecordingDir: Checking '/bedroom-storage1' for '/bedroom-storage1/1139_20091203180000.mpg'
2009-12-03 22:05:07.585 SG(): FindRecordingDir: Checking '/bedroom-storage2' for '/bedroom-storage2/1139_20091203180000.mpg'
2009-12-03 22:05:07.585 SG(): FindRecordingDir: Checking '/video' for '/video/1139_20091203180000.mpg'
2009-12-03 22:05:07.586 SG(): FindRecordingDir: Checking '/video/livetv' for '/video/livetv/1139_20091203180000.mpg'
2009-12-03 22:05:07.586 SG(Default) Error: FindRecordingFile: Unable to find '1139_20091203180000.mpg'!
2009-12-03 22:05:07.586 ProgramInfo: GetPlaybackURL: Using default of: 'myth://192.168.1.160:6543/1139_20091203180000.mpg'
[New Thread 0x7fcfbfdb7950 (LWP 5276)]
2009-12-03 22:05:07.597 MythContext: Connecting to backend server: 192.168.1.160:6543 (try 1 of 1)
2009-12-03 22:05:07.597 MythSocket(2e72000:43): write -> 43 21      MYTH_PROTO_VERSION 52
2009-12-03 22:05:07.598 MythSocket(2e72000:43): read  <- 43 13      ACCEPT[]:[]52
2009-12-03 22:05:07.598 Using protocol version 52
2009-12-03 22:05:07.598 MythSocket(2e72000:43): write -> 43 17      ANN Monitor t61 0
2009-12-03 22:05:07.613 MythSocket(2e72000:43): read  <- 43 2       OK
2009-12-03 22:05:07.613 MythSocket(2e72000:43): write -> 43 650     QUERY_GENPIXMAP[]:[]Seinfeld[]:[]The Foundation[]:[]George is rea...
2009-12-03 22:05:12.392 MythSocket(2e72000:43): read  <- 43 2       OK
2009-12-03 22:05:12.392 Preview: outfile '1139_20091203180000.mpg.png' -> '/jessdell-video/1139_20091203180000.mpg.png'
2009-12-03 22:05:12.394 MythSocket(2f0a790:43): Unable to lookup:
2009-12-03 22:05:12.394 RemoteFile::openSocket(control socket), Error:
                        Could not connect to server :6543
2009-12-03 22:05:12.394 RemoteFile::openSocket(control socket), Error: url: /jessdell-video/1139_20091203180000.mpg.png

comment:4 Changed 10 years ago by danielk

(In [23599]) Fixes #8006. Refs #7195. Fixes some regressions in preview generation. Most importantly this addresses the segfault. But it also removes some of the cruft in the PreviewGenerator? constructor that had built up to deal with places that called the preview generator without setting pathname as per the PreviewGenerator? documentation. This means we aren't doing the very expensive StorageGroup? search quite as many times.

This also saves the pathname after doing a StorageGroup? search in the PlaybackBoxHelper?.

There are still a number of ways left to speed up the PreviewGenerator?: For instance, using fork instead of running an external program to get fault tolerance, which will also address #7136. We also do not need to open a new command socket if we convert the backend preview generation to use a non-blocking call and just send a MythEvent? when we're done. But I wanted to limit this commit to really only fix regressions since we're in feature freeze (the pathname saving is pbbh is a bit of a stretch, but we actually did do this before in a different way).

comment:5 Changed 10 years ago by Dibblah

Status: newassigned

Changed 10 years ago by danielk

Attachment: 7191-gen2-v1.patch added

new set of changes to preview generation.

comment:6 Changed 10 years ago by danielk

There is no guarantee that 7191-gen2-v1.patch won't do terrible things to your computer. This is an updated to trunk version of a patch I last tested two weeks ago and lightly at that.

What it does is get rid of the extra connections to the backend to initiate preview generation and uses an asynchronous protocol instead.

There are known problems with this patch: 1st, it breaks all use of the preview generator other than standard preview generation. 2nd, if two frontends request the same preview, these requests are not combined. 3rd, new previews are broadcast to all MythEvent? listeners, not just those who have requested the preview.

Changed 10 years ago by danielk

Attachment: 7191-gen2-v2.patch added

updated patch

comment:7 Changed 10 years ago by stuartm

Cc: stuartm added

Changed 10 years ago by danielk

Attachment: 7191-gen2-v3.patch added

comment:8 Changed 10 years ago by danielk

Updated patch. With some additional protocol changes to make it more efficient.

Still a problem:

  • PreviewGen? dev docs out of date.
  • After bookmarking the preview in pbb is not updated until you reload the group.
  • PREVIEW_GENERATED messages sent to all frontends, not just the ones that need it.
  • Starting a mythpreviewgen process for every preview is inefficient & slow, these should probably be long lived processes that only get relaunched after they die...

Untested:

  • Slave backends

Changed 10 years ago by danielk

Attachment: 7191-gen2-v4.patch added

Updated to trunk.

comment:9 Changed 10 years ago by danielk

Not much different in the latest patch, but it has now been tested with slave backends & the proto and ABI versions are incremented.

Changed 10 years ago by danielk

Attachment: 7191-gen2-v5.patch added

comment:10 Changed 10 years ago by danielk

Updated patch. Fixes the network spaming previous versions did.

Remaining problems:

  • preview generator developer docs out of date.
  • After bookmarking the preview in pbb is not updated until you reload the group.
  • Starting a mythpreviewgen process for every preview is inefficient & slow, these should probably be long lived processes that only get relaunched after they die... Not planning to fix in this pass.

Changed 10 years ago by danielk

Attachment: 7191-gen2-v6.patch added

Updated patch

comment:11 Changed 10 years ago by danielk

Updated patch. This fixes the problem with the preview image not getting updated, by forcing MythUIImage to stat the preview images to make sure the cached version is not out of date.

AFAIK The only remaining problem for this patch is that the developer docs are now out of date with respect to the implementation.

Changed 10 years ago by danielk

Attachment: 7195-gen2-v7.patch added

Updated patch

Changed 10 years ago by danielk

Attachment: 7195-gen2-v8.patch added

Updated patch (sans mythversion.cpp)

Changed 10 years ago by danielk

Attachment: 7195-gen2-v9.patch added

comment:12 Changed 10 years ago by danielk

Resolution: fixed
Status: assignedclosed

(In [26106]) Fixes #7195. This speeds up preview generation and transport.

It's been some time since preview generation had any real love, it's been updated to deal with storage groups and it's been moved from the frontend to the backend and from an internal thread to an external process and generally a lot of cruft has built up. This dumps a lot of that. It abstracts the preview generation queue that was being used on the frontend so it can run on both the front and back ends and consequently limit the thread count on the backend to the number of cores *2 instead of the number of cores on all frontends * 2 as was effectively the case after it was moved out of the frontend.

There is still at least one major optimization that can be made, documented in the ticket. But this gets the performance up to an acceptable level and is all I have ready before the feature freeze anyway.

Thanks go out to Stuart Morgan who very quickly came up with a fix for a wexing problem in MythUIButtonList which allows preview generation to run well for those using the Arclight and Terra themes (I've had a patch that worked well for the MythCenter? themes for some time now :)

Note: distclean is recommended and all clients need to be updated due to a new mythproto version.

Note: See TracTickets for help on using tickets.