Auto-expire doesn't delete then loops and retries every time (endless log loop)

[stevenable@…]

When mythbackend fails to remove an autoexpired recording, it logs the condition but then tries again every time (approx 15min). Either it should remove the entry from the database or flag it as attempted/failed.

I took a look at my mythbackend log and found many copies containing the following (only showing a few lines):

Feb  5 12:49:00 host mythbackend: mythbackend[3302]: N Expire autoexpire.cpp:641 (SendDeleteMessages) 
Expiring 2606 MB for 1021 at 2016-07-22T22:33:18Z => "NBC Nightly News With Lester Holt"
Feb  5 12:49:00 host mythbackend: mythbackend[3302]: N Expire autoexpire.cpp:641 (SendDeleteMessages) 
Expiring 2811 MB for 1061 at 2016-09-23T21:59:45Z => "News 6 at 6:00p"
Feb  5 12:49:00 host mythbackend: mythbackend[3302]: N Expire autoexpire.cpp:641 (SendDeleteMessages) 
Expiring 2808 MB for 1061 at 2016-09-23T22:30:00Z => "CBS Evening News With Scott Pelley"
Feb  5 12:49:00 host mythbackend: mythbackend[3302]: E CoreContext programinfo.cpp:2358 (GetPlaybackURL) ProgramInfo(1061_20160722223034.mpg): GetPlaybackURL: '1061_20160722223034.mpg' should be local, but it can not be found.
Feb  5 12:49:00 host mythbackend: mythbackend[3302]: E CoreContext mainserver.cpp:2698 (DoHandleDeleteRecording) ERROR when trying to delete file: GetPlaybackURL/UNABLE/TO/FIND/LOCAL/FILE/ON/host/1061_20160722223034.mpg. File doesn't exist.  Database metadata will not be removed.
Feb  5 12:49:00 host mythbackend: mythbackend[3302]: E CoreContext programinfo.cpp:2358 (GetPlaybackURL) ProgramInfo(1021_20160722223318.mpg): GetPlaybackURL: '1021_20160722223318.mpg' should be local, but it can not be found.

From the dates you can see these recordings go back to 2016 and as best I can tell, this has been logging and failing since that time. The issue appears to be that by failing to remove database metadata, it shows up for autoexpire to consider again every 15 minutes. I don't know how the original shows got deleted without backend knowing; that isn't my concern here. Rather just seems that if the file does not exist, then removing metadata is desired. (I don't know the conditions under which retaining it is desired so care should be taken.)

This is certainly a minor problem, not causing issues other than slowly filling up my disk; and with a long enough list it may affect overall processing (not sure if it has throttle to skip if busy). I also apologize if this is duplicate; i searched but didn't find any other tickets that referenced this. I might try and investigate it myself if i have time but figured I'd submit it.

[sphery]

There was a conscious design decision made to ensure that no recording metadata is ever automatically deleted from the database when a recording file is missing because there are far too many reasons why a file may be temporarily missing (hard drive is unmounted, NFS/CIFS is down/unreachable, ...). Therefore, MythTV logs a message so that the user can, if desired, manually delete the recording (with appropriate "are you certain" prompts) using mythfrontend to say that it really is gone (or, using find_orphans.py at ​https://www.mythtv.org/wiki/Find_orphans.py ).

The reason it doesn't make sense to "flag it as attempted/failed" is because that file may appear at any time before the next autoexpire run, so it always makes sense to check if that file--the highest-priority-for-autoexpire--actually exists in every run. The only thing that flagging it could possibly be used to do (since we should always check to see if we can delete it) is to silence the logging, but IMHO, it's not a bad thing to keep "shouting" that something is wrong until the user notices and fixes the problem (mounts the file system, boots the file server, ...--or if nothing else, notices that a file is really missing and deletes the recording metadata using mythfrontend and ensures that he never uses a tool that removes MythTV's recording files without asking MythTV to do the deletion).

[Steve <stevenable@…>]

Thank you for your explanation. Since most users wouldn't normally see the logs I'm not sure I agree with you but okay. Yet then what is the correct procedure to identify and remove these?

I can find the offending recordings in the "auto expire list" and they are marked as 'deleted' but I find no way to remove them from this list. Requesting 'delete now' does nothing (other than generate error in the log that I don't see from the frontend [e.g. no error message given to user]). I can find the recordings in the "previously recorded" but find nothing identifying these as different from other recordings although I can request to delete the episode - and doing so removes it from this 'previously recorded' list but not the 'auto expire list'.

It does appear the 'Find_orphans' script would work but I was expected something in Myth itself. Since the auto-expire list shows they are deleted, why would you not allow me to remove it from there as one of the menu items? Or is this done somewhere else I've missed?

[Peter Bennett]

Normally you can go to the "Deleted Recordings" group, or whatever group they are in, in the frontend, them select and request Delete from the menu. It should prompt for a message saying the file cannot be found, are you sure you want to delete, then allow deletion.

The frontend does not normally display the Deleted group. To find the "Deleted" group in the frontend you have to press Menu and select 'Group filter" then select "Deleted".

[Steve <stevenable@…>]

Thank you, that worked just as you stated. And I'm afraid to admit I had never seen the menu from the recorded shows previously so I didn't even know of the various lists you could view. I only knew of what I see is called the 'default' group. Selecting the deleted list showed the expired shows that weren't going away and after a few YES selections, I was able to remove each one. I'm surprised to find a menu I didn't know existed; I'll have to try the menu button on a few other screens to see what appears. Thank you.