Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#13123 closed Patch - Bug Fix (fixed)

Exception using python bindings

Reported by: tom@… Owned by: Roger Siddons
Priority: minor Milestone: 29.1
Component: Bindings - Python Version: 0.28.1
Severity: medium Keywords:
Cc: Ticket locked: no

Description

With recent versions of the python MySQLdb library you can get an exception when the MythTV python bindings do executemany:

Traceback (most recent call last):
  File "./mythcommflag-silence", line 250, in <module>
    main()
  File "./mythcommflag-silence", line 217, in main
    rec.update()
  File "/usr/lib/python2.7/site-packages/MythTV/database.py", line 429, in update
    self._push()
  File "/usr/lib/python2.7/site-packages/MythTV/dataheap.py", line 354, in _push
    self.markup.commit()
  File "/usr/lib/python2.7/site-packages/MythTV/database.py", line 624, in commit
    ','.join(['?' for a in fields])), data)
  File "/usr/lib/python2.7/site-packages/MythTV/_conn_mysqldb.py", line 96, in executemany
    raise MythDBError(MythDBError.DB_RAW, e.args)
MythTV.exceptions.MythDBError: MySQL error: sequence item 0: expected string, bytearray found

The problem is that executemany calls back to the logged cursor with the query expressed as a bytearray that has been encoded using the database encoding. That then throws an exception because you can't use join on a bytearray.

I have opened https://github.com/MythTV/mythtv/pull/153 with a patch to decode back to a string in this case.

Change History (9)

comment:1 Changed 2 years ago by Stuart Auchterlonie

Milestone: needs_triage30.0

comment:2 Changed 2 years ago by Roger Siddons

Thanks for the patch.

Is this solely due to an upstream update in mysqlclient-python ? If so, are version numbers of the problematic/last-known-good available ?

comment:3 Changed 2 years ago by tom@…

Yes this was triggered by upgrading from Fedora 25 to 26 which broke the commercial flagger I was using (based on https://www.mythtv.org/wiki/Commercial_detection_with_silences#silence.py) as it was failing to update the recording markup and bombing out.

The actual change in mysqlclient-python was from 1.3.7 to 1.3.12 by the looks of it.

comment:4 Changed 2 years ago by tom@…

Looks like https://github.com/PyMySQL/mysqlclient-python/commit/49e401b3bc2c123b36949baeb409b91dcbdf85a6 is the trigger as that first appears in 1.3.8 and is a major rewrite of executemany.

It doesn't actually change the conversion of the query to a bytearray but it does change executemany to call back to self.execute (and hence get caught by Myth's LoggedCursor? derived class) rather than the self._query internal method than underlies execute.

comment:5 Changed 2 years ago by Roger Siddons

Milestone: 30.029.1
Owner: changed from Raymond Wagner to Roger Siddons
Status: newaccepted

Thanks for the details.

Ubuntu 16.04 & 17.04 use python-mysqldb 1.3.7.1; 17.10 uses 1.3.10 so will likely be affected also.

I've verified this patch has no adverse affects on an (old) 0.29 with 16.04. So I see no reason why it shouldn't be committed/backported immediately.

I'm a long way off master, up to my neck in broken code atm and not comfortable with pull requests so if some kindly soul feels inclined to do the honours, please do so.

comment:6 Changed 2 years ago by Peter Bennett

I will commit this around September 26th, giving folks a week to review it first.

comment:7 Changed 2 years ago by Mark Spieth

reviewed. LGTM

comment:8 Changed 2 years ago by Tom Hughes <tom@…>

Resolution: fixed
Status: acceptedclosed

In b5201fd40bb4259361f5e689606ff5a1cf66dba2/mythtv:

Fixes #13123 Exception using python bindings

Update log_query to handle bytearray queries

Merge remote-tracking branch 'tomhughes/bytearray'

Signed-off-by: Peter Bennett <pbennett@…>

comment:9 Changed 2 years ago by Tom Hughes <tom@…>

In b2a3174c0b66de83f8c931f157d4e5e709cb517e/mythtv:

Fixes #13123 Exception using python bindings

Update log_query to handle bytearray queries

Merge remote-tracking branch 'tomhughes/bytearray'

Signed-off-by: Peter Bennett <pbennett@…>
(cherry picked from commit b5201fd40bb4259361f5e689606ff5a1cf66dba2)

Note: See TracTickets for help on using tickets.