Transition to python3 breaks MythArchive

[rcrdnalor]

I missed the dependency from mytharchive to the MythTV's python package:

• mytharchive calls on current master mythburn.py with hard-coded executable python, which is not available on a modern linux setup, because 'python' aka 'python2' it is deprecated.

• currently, the file mythburn.py is only capabable to work with python2, but needs the MythTV python package which may be installed only for python3. The solution is to make mythburn.py compatible to python3 as well and resolve the hard-coded call to python by the configured version of 'python'.

• Fortunately, the changes are quite simple, but can't be verified by me, soon.

The enclosed patch uses the configured python version and updates mythburn.py to python2 and python3 as well.

Please note: This patch is tested at compile time only, and 'python3' is shown now in the compiled mythburn.o

I kindly ask the developer using a current setup with python3 only to test this patch.

I really appreciate your help!

I would need a week to setup an environment to test this and Ubuntu 20.04 released with MythTV v31 is knocking on the door....

Please help me on testing this patch.

Enclosed a patch for compatibility to python3 and python2 of mytharchive, using the configured python version.

[rcrdnalor]

Patch using the configured python version and updates compatibility of mythburn.py to python2 and python3 as well.

[jpilk]

I'm not the owner, but I used to create a lot of DVDs. I last did it about a year ago. Current master in Fedora 30 hung just now for me when trying to invoke psutil to set ionice -c3. I didn't follow that up.

Creation succeeded when I applied my old tweaks in mythburn.py and some of the other files. Extending your patch to those could be interactive and probably not too difficult, but does the .cpp part need to be committed?

I only ever used the Project-X option, with pre-cut mpeg2 video input and a single audio track.

[rcrdnalor]

@jpilk:

but does the .cpp part need to be committed?

It looks like that you still have the python bindings installed for python2:

Please post the output of the following commands:

python --version
python3 --version
mythpython --version
ls -la /usr/local/lib/python3.<your installed python3 minor version>/dist-packages

On a fresh install of Ubuntu 19.04, there is only python3 available. Nothing else. Therefore, the cpp part needs to be changed as well.

[jpilk]

[john@HPFed ~]$ python --version
Python 2.7.17
[john@HPFed ~]$ python3 --version
Python 3.7.5
[john@HPFed ~]$ mythpython --version 
MythTV Python Bindings
  local versions
    bindings version:         30.0.-1.0
    ttvdb version:            2.0-dev
    tmdb version:             v0.7.0
  external versions
    lxml version:             4.2.5
    MySQLdb version:          1.3.13.final.0
  protocol versions
    backend:                  91
    schema:                   1360
    music schema:             1024
    netvision schema:         1007

The last query failed, but 'rpm -qa | grep python3' found 148 packages

[rcrdnalor]

If you do in a terminal

python3
>>> import MythTV
>>> MythTV.static.PROTO_VERSION
'91'
>>>

Does this work ?

[jpilk]

[john@HPFed ~]$ python3
Python 3.7.5 (default, Oct 17 2019, 12:09:47) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import MythTV
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'MythTV'

[rcrdnalor]

That's bad. Does

mythbackend --version

show

Options compiled in:
using_bindings_python

?

If you build locally, then in your mythtv source folder

grep -i python ./config.h

should show a

#define PYTHON_EXE "python3"
using_bindings_python

otherwise you have not installed the dependencies for python3

python3-lxml
python3-MySQLdb
python3-requests
python3-simplejson
python3-future

These dependencies are checked during ./configure

[jpilk]

The 'options compiled in' include using_bindings_python (twice).  I'm building with

https://github.com/garybuhrmaster/packaging/blob/master/MythTV/rpm/SPECS/mythtv.spec
  and 
https://github.com/garybuhrmaster/packaging/blob/master/MythTV/rpm/SPECS/mythtv-plugins.spec

which IIUC will use python for el7, python2 for Fedora 30 and python3 for Fedora >30.  

I'm on 30 here, or el7 without vpx.  Not sure what disasters lurk if I were to do more editing.

[Bill Meek]

Mike,

Did you do (does the .spec do) a make distclean (I've done that plus the ./configure)? The missing

#define PYTHON_EXE "python3"

won't work correctly.

[jpilk]

The builds are done using 'mock' which IIUC always starts in a clean environment and downloads only the BuildRequires? packages and dependencies. py_prefix is set to python3 for Fedora 31, but python2 for my Fedora 30 at present. I don't know how the .cpp patch would affect things.

[Bill Meek]

The shebang in the .py file isn't in play, the python version is determined by the .cpp file. And that's set at ./configure time:

-    commandline = "python " + GetShareDir() + "mytharchive/scripts/mythburn.py";
+    commandline = PYTHON_EXE;
+    commandline += " " + GetShareDir() + "mytharchive/scripts/mythburn.py";

[rcrdnalor]

Sorry John for the confusion.

Please excuse.

I was looking for a python3 test:

I kindly ask the developer using a current setup with python3 only to test this patch.

But, on the other side, it would be great if you could test this for python2 as well.

Please apply the patch and build your setup for fedora 30 as usual, rum mytharchive and report back.

[Bill Meek]

Roland, I'm testing. Ubuntu 19.04 & 20.04, python3-cddb isn't available. I think that's where this lives:

Traceback (most recent call last):
  File "/usr/local/share/mythtv/mytharchive/scripts/mythburn.py", line 111, in <module>
    import CDROM
ModuleNotFoundError: No module named 'CDROM'

[jpilk]

I thought I might be able to create a patched build by inserting 'git apply /full/path/to.patch' after the git checkout in the build script. Initially it looked promising, and I have a full set of 31 rpms with a 31.Pre.1785.g83d8bbc825-100.fc30 label, but it looks as if the patch didn't take in the builddir in the plugins-build phase. Sorry.

[rcrdnalor]

Bill, yes I got the same error and faked the missing CDROM import. I is from /usr/lib/python2.7/plat-x86_64-linux-gnu/CDROM.py, which is not available on python3 but only needed, if you write the iso file to disk.

I did an ugly import of an old recording on my test system, already processed by projectX.

And I had to add additional changes not discovered by the 2to3 tool, but dvdauthor now hangs for more than an hour to produce an iso file. I would like to know if this recently worked with python2, I doubt that it did.

Definitely, this couldn't be solved in short time.

I propose to exclude mytharchive for the upcoming v31 release on Ubuntu. We could easily exclude this in packaging/deb/debian.

Please excuse, that I missed that dependency and many thanks to you and John spending a lot of time on working on this!

[jpilk]

My standard mode of operation was to create an iso file with embedded checksum, and burn and verify using K3B. I'll attach a diff from my most recent version of mythburn.py for python2 in the hope that it might be useful. It uses mp2 sound and non-standard versions of the encoding and intro files, but it worked two days ago.

[rcrdnalor]

I managed to get an 'iso' file from a recording via mytharchive.

See attached patch mytharchive_python3_campatibility-V2.patch

Hint: One needs to get all dependencies for mythburn.py installed! e.g.: missing mjpegtools causes dvbauthor to hang forever, mea culpa.

Note: Not tested:

• Create archive of recordings
• Write to DVD
• re-transcode with ffmpeg
• projectX clean-up

[rcrdnalor]

Patch using the configured python version and updates compatibility of mythburn.py to python2 and python3 as well.

[rcrdnalor]

Further improvements, see attached patch v3

I managed to create iso files with the

MythCenter?? animated theme MythCenter?? theme Simple - Autoplay them

I can also write to dvd ( tested with MythCenter?? animated theme). Luckily I found a spare dvd-rw medium :-).

Creating an Archive medium works as well. I could not test the import of this archive , because the channel setup on my test environment did not fit, but the created xml file looks good.

The attached patch needs #13565 and implements #13306, otherwise it will not work.

Remark: The python source file (mythburn.py) was far to old to be converted by the 2to3 tool....

Please give that patch a try.

[rcrdnalor]

Version 3 of a patch using the configured python version and updates compatibility of mythburn.py to python2 and python3 as well, needs #13565 and implements #13306.

[jpilk]

I'm not sure that I can yet build-with-patches, but will try replacing current mythburn.py and 'other.py' in my current F30 build of master-with-py2.

/home/john/packaging/MythTV/rpm/mytharchive_python3_compatibility-V3.patch:246: trailing whitespace.
        text=text.replace("%chaptertime","%s" % chapterlist[chapternumber - 1] ) 
warning: 1 line adds whitespace errors.

[jpilk]

mythburn.py under F30/py2 still needs the psutil change noted in #13306.

I have not yet found the 'external command' that my system now needs to run Project-X...

[Bill Meek]

Roland & John, I've added the psutil fix to the existing set of fixes in these 3 tickets. Plan is to release all in a single commit.

[jpilk]

It looks as if I may have an 'escape' problem in the file path; unless it's a formatting artefact in the log or this display box. The progress.log showed a gap where I have inserted

Alternatively, and perhaps more likely, it could be the double quotes round the "ionice -c3 ..." bit, which give trouble when used from the command line. The quotes aren't shown in the box that defines the 'external command'

Processing recording 1: '/home/john/SGs/RecsSG2/10091_20200112152800.mpg'

Failed while running Project-X to cut commercials and/or demux an mpeg2 file.
Result: 127, Command was "ionice -c3 java -jar projectx_link" "/home/john/SGs/RecsSG2~~~~~~/10091_20200112152800.mpg" -id '0x1e0,0x1c0' -set ExternPanel.appendPidToFileName=1 -out "/home/john/mArchive/work/1" -name stream

[rcrdnalor]

@Bill,

please check the sql query output of

select * from settings where value = 'MythArchiveProjectXCmd';

Mine is set to projectx.

What I did in the past (mythtv fixes/0.25) was to create an executable shell script in '/usr/local/bin' with name projectx:

#!/bin/sh

exec java -jar /path/to/ProjectX.jar "$@"

If java complains about not able to start in headless mode, use

#!/bin/sh

exec java -Djava.awt.headless=true -jar /path/to/ProjectX.jar/ProjectX.jar "$@"

Change the path to projectX.jar according your needs.

Alternately, you can try instead of "$@" : $@ or $* (without quotes).

Do not forget to make this script executable.

Unfortunately, I do not have this setup now on my test server, and I haven't recorded mpeg2 SD content for years.

[rcrdnalor]

Sorry, John was meant in #comment22, not Bill.

[jpilk]

OK: It worked with what is no doubt the default "projectx", without quotes, in the box. The "ionice -c3" bit is equivalent to psutil and "java -jar" was needed by my local build of PX but not by the version from the repos. Now it's looking for menumusic.mp2, which was dropped from the official build years ago and is another local tweak. The untweaked version is probably ok now. Thanks for your work and suggestions.

[Roland Ernst <rcrernst@…>]

In 20827d1bc/mythtv:

MythArchive?: Python v2/v3 changes.

Solution for 13562 - Transition to python3 breaks MythArchive?
Includes 13306 - MythArchive? fails during DVD Menu creation from: jreiser
Includes 13565 - Fix recorded markup handling in python3 bindings

Plus two of typos in: mytharchivehelper/main.cpp

Closes: #13562
Closes: #13306
Closes: #13565

Signed-off-by: Bill Meek <billmeek@…>