Opened 6 months ago

Closed 6 months ago

Last modified 6 months ago

#13614 closed Patch - Bug Fix (fixed)

Python Bindings fails on Recorded.update() during silence detection

Reported by: rcrdnalor Owned by: rcrdnalor
Priority: minor Milestone: 31.1
Component: Bindings - Python Version: v31-fixes
Severity: medium Keywords: silence detection
Cc: Ticket locked: no

Description

A user on the forum reported a traceback in the python bindings (1) when running the detection for silence according the wiki page "Commercial detection with silences" (2).

(1) https://forum.mythtv.org/viewtopic.php?f=36&t=3763

(2) https://www.mythtv.org/wiki/Commercial_detection_with_silences

This affects Ubuntu 20.04 for

  • Python 3.8
  • MySQLdb 1.4.4
  • MariaDB 10.3

I had not any errors on Ubuntu 19.10 for

  • Python 3.7
  • MySQLdb 1.3.10
  • MariaDB 10.3

The following traceback occurs when running silence.py with chanid and starttime as paramters:

Traceback (most recent call last):
  File "/home/local-admin/MythTV/silence/MythTV/_conn_mysqldb.py", line 98, in executemany
    return super(LoggedCursor, self).executemany(query, args)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 235, in executemany
    return self._do_execute_many(q_prefix, q_values, q_postfix, args,
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 264, in _do_execute_many
    rows += self.execute(sql + postfix)
  File "/home/local-admin/MythTV/silence/MythTV/_conn_mysqldb.py", line 68, in execute
    query = self._sanitize(query)
  File "/home/local-admin/MythTV/silence/MythTV/_conn_mysqldb.py", line 44, in _sanitize
    def _sanitize(self, query): return query.replace('?', '%s')
TypeError: a bytes-like object is required, not 'str'

The offended call to 'MySQLdb.curser.execute' was

bytearray(b"INSERT INTO recordedmarkup (chanid,starttime,mark,type,data) VALUES(10111,\'2020-02-09 16:10:00\',74367,4,NULL),(10111,\'2020-02-09 16:10:00\',81984,5,NULL)")

Note: This is a callback from MySQLdb as described in a similar issue #13123. MySQLdb resolves a call to 'executemany' to multiple calls of 'execute', but as bytearray and not as string.

To reproduce this issue, one needs to patch the silence.py from the wiki (2) with the attached patch for python3 ('silence_py_patch_01.patch') and install it according the wiki page.

Then run 'silence.py -- chanid YYYY --starttime YYY' with correct parameters.

To solve this issue, apply the path '0001_sanitize_mysql_queries.patch' and, of course, the patch for 'silence.py'.

Attachments (3)

silence_py_patch_01.patch (1.7 KB) - added by rcrdnalor 6 months ago.
Patch for a python3 version of silence.py on the wiki
0001_sanitize_mysql_queries.patch (778 bytes) - added by rcrdnalor 6 months ago.
Patch to fix Python Bindings for MySQLDB calls of executemany
silence_py_patch_02.patch (1.7 KB) - added by rcrdnalor 6 months ago.
Patch for silece.py tested with python3.6 and python3.8

Download all attachments as: .zip

Change History (7)

Changed 6 months ago by rcrdnalor

Attachment: silence_py_patch_01.patch added

Patch for a python3 version of silence.py on the wiki

Changed 6 months ago by rcrdnalor

Patch to fix Python Bindings for MySQLDB calls of executemany

Changed 6 months ago by rcrdnalor

Attachment: silence_py_patch_02.patch added

Patch for silece.py tested with python3.6 and python3.8

comment:1 Changed 6 months ago by rcrdnalor

Patches tested on

Ubuntu 18.04 python2 master
python: 2.7.17
python-MySQLdb: 1.3.10

Ubuntu 18.04 python3 master
python3: 3.6.9
python3-MySQLdb: 1.3.10

Ubuntu 20.04 python3 fixes/31
python3: 3.8.2
python3-MySQLdb: 1.4.4

comment:2 Changed 6 months ago by rcrdnalor

Milestone: needs_triage31.1

comment:3 Changed 6 months ago by Roland Ernst <rcrernst@…>

Resolution: fixed
Status: assignedclosed

In b2e9c6a442/mythtv:

Fix mysql cursor class to handle bytearrays

Newer Python MySQLdb modules call 'cursor.execute()' multiple times
from 'cursor.executemany()'.
With python3 and python3-MySQLdb > 1.4.0 these call-backs containing
a query are bytearrays, resulting in a traceback in the '_sanitize' method.

Note: This '_sanitize' method is only needed when creating a query within
the Python Bindings, but not necessary when python-mysqldb itself calls the
cursor.execute() method.

Fixes #13614

comment:4 Changed 6 months ago by Roland Ernst <rcrernst@…>

In ce23a0225/mythtv:

Fix mysql cursor class to handle bytearrays

Newer Python MySQLdb modules call 'cursor.execute()' multiple times
from 'cursor.executemany()'.
With python3 and python3-MySQLdb > 1.4.0 these call-backs containing
a query are bytearrays, resulting in a traceback in the '_sanitize' method.

Note: This '_sanitize' method is only needed when creating a query within
the Python Bindings, but not necessary when python-mysqldb itself calls the
cursor.execute() method.

Fixes #13614

(cherry picked from commit b2e9c6a44233570704554894bf45e01bfa8e26a7)

Note: See TracTickets for help on using tickets.