mythutil --scanvideos does not update database

[yiannividalis@…]

I moved a folder with movies within the storage group hierarchy and tried to use mythutil --scanvideos to update the database, but it wasn't updated. The movement was on the SB, and I ran mythutil both on the MB and on the SB without success. The xterm log output doesn't show anything like "removing x" and "Adding x" like in the frontend log. In one test it updated the location of only the first episode of the moved folder.

MythTV Version : v0.25-rc-38-gd6f038c-dirty
MythTV Branch : master
Network Protocol : 72
Library API : 0.25.20120315-2
QT Version : 4.7.4
Options compiled in:
 linux profile use_hidesyms using_alsa using_oss using_pulse using_pulseoutput using_backend using_bindings_perl using_bindings_python using_bindings_php using_dvb using_frontend using_hdpvr using_iptv using_libcrypto using_libfftw3 using_libxml2 using_libudf using_lirc using_mheg using_opengl_video using_qtwebkit using_qtscript using_qtdbus using_v4l2 using_x11 using_xrandr using_xv using_bindings_perl using_bindings_python using_bindings_php using_mythtranscode using_opengl using_vdpau using_ffmpeg_threads using_live using_mheg using_libxml2 using_libudf

[Raymond Wagner]

This should have been fixed by 0a57c4662b8. It makes the master backend open a new connection to the slave backend, rather than trying to connect back in through itself to perform the query. Could you check your slave backend logs to see if the master backend is trying to connect to perform the scan?

[yiannividalis@…]

Today I performed the following test on the SB:

The folders I tested belong to the SB. I moved a Series folder (Shaka Zulu) outside the videos storage group and ran "mythutil --scanvideos". The series was removed from the frontend.

When I moved it back where it was before and re-ran mythutil, in mythfrontend I could see the Series Title correctly, but it had created only one episode, from another series. The correct episode was S01E10 and it had the title and description of another file of another series (the first alphabetically series in the folder - BlackAdder?).

I removed the series around 11:13 I added it back around 11:15. I have edited the log to replace the Greek messages back to English, but I've left alone the Greek folder/file names. Unknown characters are not Greek, the program tries to translate to Greek and fails.

[yiannividalis@…]

Slave Backend log - I tried to attach it ucompressed but it was considered spam.

[yiannividalis@…]

Seeing ​http://www.gossamer-threads.com/lists/mythtv/users/561019, I decided to re-test the mythutil --scanvideos program to check if the problem still occurs, and I'm afraid it does.

On the SB, I moved a folder with 4 videos outside the videos scope, ran mythutil --scanvideos, and verified they were removed from the DB.

I then moved the folder back, and re-ran mythutil. Of the four videos, only the first alphabetically title was inserted in the db, but trying to play it I discoverd its file is actually the one of the last alphabetically title (4th).

[yiannividalis@…]

I ran some more tests to ascertain the exact cirumstances of the error and I came across something I think is strange.

When mythutil --scanvideos is ran on ANY backend, the hash each added video takes is "QUERY_FILE_HASH", and not some actual hash. That's why only the last video is inserted in the DB and the description is of the first video.

If I run mythutil for a second time without removing the videos, the MB log will complain:

Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner mythcorecontext.cpp:327 (ConnectToMasterServer) ERROR: Master backend tried to connect back to itself!
Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner videoscan.cpp:326 (updateDB) Hash QUERY_FILE_HASH already exists in the database, updating record 504654 with new filename ÎÏίλλεÏ/Aliens Vs Predator - Requiem.avi
Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner mythcorecontext.cpp:327 (ConnectToMasterServer) ERROR: Master backend tried to connect back to itself!
Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner videoscan.cpp:326 (updateDB) Hash QUERY_FILE_HASH already exists in the database, updating record 504654 with new filename ÎÏίλλεÏ/Pan's Labyrinth.avi

and the video played will be a different one. Usually the last two alternate.

The "QUERY_FILE_HASH" is not present in the DB if I scan for changes from either frontend.

MythTV Version : v0.28-pre-677-g864bbe0
MythTV Branch : master
Network Protocol : 79
Library API : 0.28.20140104-1
QT Version : 4.8.1
Options compiled in:
 linux profile use_hidesyms using_alsa using_oss using_backend using_bindings_perl using_bindings_python using_bindings_php using_dvb using_frontend using_ceton using_hdpvr using_ivtv using_libcrypto using_libfftw3 using_libudf using_lirc using_mheg using_opengl using_opengl_video using_qtwebkit using_qtscript using_qtdbus using_taglib using_v4l2 using_x11 using_xrandr using_profiletype using_bindings_perl using_bindings_python using_bindings_php using_mythtranscode using_opengl using_vdpau using_ffmpeg_threads using_mheg using_libass using_libudf

[Jonatan Lindblad]

Replying to yiannividalis@…:

When mythutil --scanvideos is ran on ANY backend, the hash each added video takes is "QUERY_FILE_HASH", and not some actual hash. That's why only the last video is inserted in the DB and the description is of the first video.

If I run mythutil for a second time without removing the videos, the MB log will complain:

Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner mythcorecontext.cpp:327 (ConnectToMasterServer) ERROR: Master backend tried to connect back to itself!
Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner videoscan.cpp:326 (updateDB) Hash QUERY_FILE_HASH already exists in the database, updating record 504654 with new filename ÎÏίλλεÏ/Aliens Vs Predator - Requiem.avi
Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner mythcorecontext.cpp:327 (ConnectToMasterServer) ERROR: Master backend tried to connect back to itself!
Jan 10 18:07:32 storage  mythbackend[23355]: E VideoScanner videoscan.cpp:326 (updateDB) Hash QUERY_FILE_HASH already exists in the database, updating record 504654 with new filename ÎÏίλλεÏ/Pan's Labyrinth.avi

and the video played will be a different one. Usually the last two alternate.

The "QUERY_FILE_HASH" is not present in the DB if I scan for changes from either frontend.

The reason for this seems to be that when the RemoteFile::GetFileHash? method is called from the master BE it tries to connect to itself in SendReceiveStringList? which fails. This leaves the strlist variable unmodified and its first element, "QUERY_FILE_HASH", will be used as the hash.

​http://code.mythtv.org/cgit/mythtv/tree/mythtv/libs/libmythbase/remotefile.cpp#n512

[paulh]

I don't seem to be able to reproduce this. Can you please verify it is still a problem.

[Yianni Vidalis <yiannividalis@…>]

Hi, Paul, thanks for looking into this. Yes, it still happens, I rechecked when I saw your post. Maybe it's triggered when there are two backends involved. I ran the command on the SBE, after having removed and re-inserted the last two episodes of Shaka (9 and 10). What I saw in the videos was the description of episode 9 but the actual video was of the 10th. I tried twice and both times the SBE disconnected and I had to restart the mythbackend program. The errors in the journal are:

Μάρ 02 20:33:45 storage mythbackend[13544]: 2016-03-02 20:33:45.898401 E  MythCoreContext::ConnectToMasterServer(): ERROR: Master backend tried to connect back to itself! 
Μάρ 02 20:33:46 storage mythbackend[13544]: 2016-03-02 20:33:45.911248 E  MythCoreContext::ConnectToMasterServer(): ERROR: Master backend tried to connect back to itself! 
Μάρ 02 20:33:46 storage mythbackend[13544]: 2016-03-02 20:33:45.925396 E  Hash QUERY_FILE_HASH already exists in the database, updating record 509794 with new filename Shaka Zulu S01E10.mkv 
Μάρ 02 20:33:46 storage mythbackend[13544]: 2016-03-02 20:33:46.438298 E  MythCoreContext::ConnectToMasterServer(): ERROR: Master backend tried to connect back to itself! 
Μάρ 02 20:33:46 storage mythbackend[13544]: 2016-03-02 20:33:46.438372 E  MythCoreContext::ConnectToMasterServer(): ERROR: Master backend tried to connect back to itself! 
Μάρ 02 20:33:47 storage mythbackend[13544]: 2016-03-02 20:33:47.268293 E  MythCoreContext::ConnectToMasterServer(): ERROR: Master backend tried to connect back to itself! 
Μάρ 02 20:33:47 storage mythbackend[13544]: 2016-03-02 20:33:47.268423 E  MythCoreContext::ConnectToMasterServer(): ERROR: Master backend tried to connect back to itself! 
Μάρ 02 20:33:47 storage mythbackend[13544]: 2016-03-02 20:33:47.545852 E  MythCoreContext::ConnectToMasterServer(): ERROR: Master backend tried to connect back to itself! 
Μάρ 02 20:34:18 storage mythbackend[13544]: 2016-03-02 20:34:18.176492 E  MythSocket(7f1e6000f210:126): ReadStringList: Error, timed out after 30000 ms. 
Μάρ 02 20:34:18 storage mythbackend[13544]: 2016-03-02 20:34:18.176629 E  MythSocket(7f1e6000f210:-1): No response. 
Μάρ 02 20:34:18 storage mythbackend[13544]: 2016-03-02 20:34:18.176641 E  RemoteFile::openSocket(file data socket): Did not get proper response from al2kpro:6543 
Μάρ 02 20:34:18 storage mythbackend[13544]: 2016-03-02 20:34:18.176827 E  RemoteFile::openSocket(file data socket): Failed to open socket, error was invalid response 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978047 E  MythSocket(1acb8e0:88): ReadStringList: Error, timed out after 30000 ms. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978135 E  MainServer: Slave backend: al2kpro no longer connected 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978295 E  MythSocket(1acb8e0:-1): No response. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978315 E  PlaybackSock::SendReceiveStringList(): No response. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978326 E  PlaybackSock: GetEncoderState: QUERY_REMOTEENCODER 2 gave us no response. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978389 E  MythSocket(1acb8e0:-1): WriteStringList: Error, called with unconnected socket. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978452 E  MythSocket(1acb8e0:-1): Failed to send command. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978476 E  PlaybackSock::SendReceiveStringList(): No response. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978487 E  PlaybackSock: GetEncoderState: QUERY_REMOTEENCODER 2 gave us no response. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978522 E  MythSocket(1acb8e0:-1): WriteStringList: Error, called with unconnected socket. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978553 E  MythSocket(1acb8e0:-1): Failed to send command. 
Μάρ 02 20:34:19 storage mythbackend[13544]: 2016-03-02 20:34:18.978566 E  PlaybackSock::SendReceiveStringList(): No response. 
Μάρ 02 20:34:25 storage mythbackend[13544]: 2016-03-02 20:34:25.189477 E  MythSocket(7f1e6001ba40:124): ReadStringList: Error, timed out after 7000 ms. 
Μάρ 02 20:34:25 storage mythbackend[13544]: 2016-03-02 20:34:25.189596 E  MythSocket(7f1e6001ba40:-1): No response. 
Μάρ 02 20:34:25 storage mythbackend[13544]: 2016-03-02 20:34:25.189604 E  Remote file timeout. 
Μάρ 02 20:34:25 storage mythbackend[13544]: 2016-03-02 20:34:25.189790 E  Image Download: Failed to open remote file (myth://Coverart@al2kpro/Shaka Zulu Season 1_coverart.jpg) for write.  Does Storage Group Exist?

After these errors the SBE shows disconnected in mythweb and I have to restart it.

Comment 7 still applies, the hash column of videometadata has the value QUERY_FILE_HASH, not an actual hash.

Version is

> mythfrontend --version
Please attach all output as a file in bug reports.
MythTV Version : v0.28-pre-3632-g6985cac-dirty
MythTV Branch : master
Network Protocol : 88
Library API : 0.28.20160223-1
QT Version : 5.4.1
Options compiled in:
 linux release use_hidesyms using_alsa using_oss using_pulse using_pulseoutput using_backend using_bindings_perl using_bindings_python using_bindings_php using_dvb using_frontend using_vbox using_ceton using_hdpvr using_ivtv using_libcrypto using_libfftw3 using_libxml2 using_lirc using_mheg using_opengl using_opengl_video using_qtwebkit using_qtscript using_qtdbus using_taglib using_v4l2 using_x11 using_xrandr using_bindings_perl using_bindings_python using_bindings_php using_freetype2 using_mythtranscode using_opengl using_vdpau using_ffmpeg_threads using_mheg using_libass using_libxml2

[Jonatan Lindblad]

Can you try the attached patch? I got no "Master backend tried to connect back to itself!" with it applied. I still have some NULL hashes though, but none with "QUERY_FILE_HASH".

[Yianni Vidalis <yiannividalis@…>]

Replying to natanojl:

Can you try the attached patch? I got no "Master backend tried to connect back to itself!" with it applied. I still have some NULL hashes though, but none with "QUERY_FILE_HASH".

Thank you very much for this patch. I applied it and made two tests, both on the MBE and the SBE by first removing the same two videos from the SG, then running mythutil --scanvideos, then re-add them and scanning again. No problems so far, I can see the entries in videometadata and they both have normal hashes. No disconnected SBE, either. Unless your patch affects the system in some unwanted way, which I doubt, as far as my problem is concerned, you fixed it!

Please let me know if you want me to run any more tests.

Thanks,

Yianni.

[c_franklin@…]

I having been experiencing the same problem for the last year. I have MBE, SBE, and one FE.

I applied the videoscan.patch against 0.27.6+fixes.20160213.332aa34-0ubuntu0mythbuntu4.

The problem is gone and everything else appears to be working normally.

Thank you for fixing this.

[Yianni Vidalis <yiannividalis@…>]

If a developer could please take a look at this ticket again, it seems from our point of view that it's fixed, and if you can verify this, please commit the patch and close it as fixed?

I'm bumping this as it's some days before the 0.28 release and it's a great opportunity.

Thanks,

Yianni

[Jonatan Lindblad <jlindblad@…>]

In 1d4d102d18b1266dc1a30431eabf1fbefa265378/mythtv:

Make mythutil --scanvideos work with slave backends

This uses the same workaround as in 0a57c46 where the master backend opens
temporary connections to slave backends to issue commands.

Fixes #10500

[Jonatan Lindblad <jlindblad@…>]

In 9fc0d18233dde0398f22a0b2be1db511db24fade/mythtv:

Make mythutil --scanvideos work with slave backends

This uses the same workaround as in 0a57c46 where the master backend opens
temporary connections to slave backends to issue commands.

Fixes #10500

(cherry picked from commit 1d4d102d18b1266dc1a30431eabf1fbefa265378)

[Jonatan Lindblad <jlindblad@…>]

In d8dc0e730d93cd9aad597b61374f9a90f675884f/mythtv:

Make mythutil --scanvideos work with slave backends

This uses the same workaround as in 0a57c46 where the master backend opens
temporary connections to slave backends to issue commands.

Fixes #10500

(cherry picked from commit 1d4d102d18b1266dc1a30431eabf1fbefa265378)