search returns data in ISO-8859-1 but header calls it UTF-8

[Karl Egly]

wget -S ​https://services.mythtv.org/channel-icon/search?s=WDR

Content-Type: text/plain; charset=utf-8

wget -O - ​https://services.mythtv.org/channel-icon/search?s=WDR | grep -P -e "(?:sseldorf|ln|nster)" | hd

00000000  22 33 38 35 37 35 22 2c  22 57 44 52 20 44 fc 73  |"38575","WDR D.s|
00000010  73 65 6c 64 6f 72 66 22  2c 22 68 74 74 70 3a 2f  |seldorf","http:/|
00000020  2f 6c 79 6e 67 73 61 74  2d 6c 6f 67 6f 2e 63 6f  |/lyngsat-logo.co|
00000030  6d 2f 6c 6f 67 6f 2f 74  76 2f 77 77 2f 77 64 72  |m/logo/tv/ww/wdr|
00000040  5f 64 75 73 73 65 6c 64  6f 72 66 2e 70 6e 67 22  |_dusseldorf.png"|
00000050  0a 22 33 38 35 37 36 22  2c 22 57 44 52 20 48 44  |."38576","WDR HD|
00000060  20 44 fc 73 73 65 6c 64  6f 72 66 22 2c 22 68 74  | D.sseldorf","ht|
00000070  74 70 3a 2f 2f 6c 79 6e  67 73 61 74 2d 6c 6f 67  |tp://lyngsat-log|
00000080  6f 2e 63 6f 6d 2f 6c 6f  67 6f 2f 74 76 2f 77 77  |o.com/logo/tv/ww|
00000090  2f 77 64 72 5f 64 75 73  73 65 6c 64 6f 72 66 5f  |/wdr_dusseldorf_|
000000a0  68 64 2e 70 6e 67 22 0a  22 33 38 36 36 35 22 2c  |hd.png"."38665",|
000000b0  22 57 44 52 20 48 44 20  44 fc 73 73 65 6c 64 6f  |"WDR HD D.sseldo|
000000c0  72 66 22 2c 22 68 74 74  70 3a 2f 2f 6c 79 6e 67  |rf","http://lyng|
000000d0  73 61 74 2d 6c 6f 67 6f  2e 63 6f 6d 2f 68 69 72  |sat-logo.com/hir|
000000e0  65 73 2f 77 77 2f 77 64  72 5f 64 75 73 73 65 6c  |es/ww/wdr_dussel|
000000f0  64 6f 72 66 5f 68 64 2e  70 6e 67 22 0a 22 33 38  |dorf_hd.png"."38|
00000100  35 38 32 22 2c 22 57 44  52 20 48 44 20 4b f6 6c  |582","WDR HD K.l|
00000110  6e 22 2c 22 68 74 74 70  3a 2f 2f 6c 79 6e 67 73  |n","http://lyngs|
00000120  61 74 2d 6c 6f 67 6f 2e  63 6f 6d 2f 6c 6f 67 6f  |at-logo.com/logo|
00000130  2f 74 76 2f 77 77 2f 77  64 72 5f 6b 6f 6c 6e 5f  |/tv/ww/wdr_koln_|
00000140  68 64 2e 70 6e 67 22 0a  22 33 38 36 36 38 22 2c  |hd.png"."38668",|
00000150  22 57 44 52 20 48 44 20  4b f6 6c 6e 22 2c 22 68  |"WDR HD K.ln","h|
00000160  74 74 70 3a 2f 2f 6c 79  6e 67 73 61 74 2d 6c 6f  |ttp://lyngsat-lo|
00000170  67 6f 2e 63 6f 6d 2f 68  69 72 65 73 2f 77 77 2f  |go.com/hires/ww/|
00000180  77 64 72 5f 6b 6f 6c 6e  5f 68 64 2e 70 6e 67 22  |wdr_koln_hd.png"|
00000190  0a 22 33 38 35 38 34 22  2c 22 57 44 52 20 48 44  |."38584","WDR HD|
000001a0  20 4d fc 6e 73 74 65 72  22 2c 22 68 74 74 70 3a  | M.nster","http:|
000001b0  2f 2f 6c 79 6e 67 73 61  74 2d 6c 6f 67 6f 2e 63  |//lyngsat-logo.c|
000001c0  6f 6d 2f 6c 6f 67 6f 2f  74 76 2f 77 77 2f 77 64  |om/logo/tv/ww/wd|
000001d0  72 5f 6d 75 6e 73 74 65  72 5f 68 64 2e 70 6e 67  |r_munster_hd.png|
000001e0  22 0a 22 33 38 36 36 39  22 2c 22 57 44 52 20 48  |"."38669","WDR H|
000001f0  44 20 4d fc 6e 73 74 65  72 22 2c 22 68 74 74 70  |D M.nster","http|
00000200  3a 2f 2f 6c 79 6e 67 73  61 74 2d 6c 6f 67 6f 2e  |://lyngsat-logo.|
00000210  63 6f 6d 2f 68 69 72 65  73 2f 77 77 2f 77 64 72  |com/hires/ww/wdr|
00000220  5f 6d 75 6e 73 74 65 72  5f 68 64 2e 70 6e 67 22  |_munster_hd.png"|
00000230  0a 22 33 38 35 38 31 22  2c 22 57 44 52 20 4b f6  |."38581","WDR K.|
00000240  6c 6e 22 2c 22 68 74 74  70 3a 2f 2f 6c 79 6e 67  |ln","http://lyng|
00000250  73 61 74 2d 6c 6f 67 6f  2e 63 6f 6d 2f 6c 6f 67  |sat-logo.com/log|
00000260  6f 2f 74 76 2f 77 77 2f  77 64 72 5f 6b 6f 6c 6e  |o/tv/ww/wdr_koln|
00000270  2e 70 6e 67 22 0a 22 33  38 35 38 33 22 2c 22 57  |.png"."38583","W|
00000280  44 52 20 4d fc 6e 73 74  65 72 22 2c 22 68 74 74  |DR M.nster","htt|
00000290  70 3a 2f 2f 6c 79 6e 67  73 61 74 2d 6c 6f 67 6f  |p://lyngsat-logo|
000002a0  2e 63 6f 6d 2f 6c 6f 67  6f 2f 74 76 2f 77 77 2f  |.com/logo/tv/ww/|
000002b0  77 64 72 5f 6d 75 6e 73  74 65 72 2e 70 6e 67 22  |wdr_munster.png"|
000002c0  0a                                                |.|

[Stuart Auchterlonie]

The backend database is storing data in UTF-8, so we need to translate some strings which are invalid in UTF-8 to UTF-8 before doing the lookup.

Example log showing this

I, [2016-06-01T13:30:02.543869 #18053]  INFO -- : Started POST "/channel-icon//search" for XX.XX.XX.XX at 2016-06-01 13:30:02 +0000
I, [2016-06-01T13:30:02.545446 #18053]  INFO -- : Processing by ChannelIconController#search as HTML
I, [2016-06-01T13:30:02.545605 #18053]  INFO -- :   Parameters: {"s"=>"RMC Découverte", "csv"=>"\"RMC D\xE9couverte\",\"C4135.telerama.fr\",\"RMC D\xE9couverte\",\"10\",\"0\",\"0\",\"8442\",\"2564\"\n"}
I, [2016-06-01T13:30:02.587081 #18053]  INFO -- : Completed 500 Internal Server Error in 41ms (ActiveRecord: 37.9ms)
F, [2016-06-01T13:30:02.588021 #18053] FATAL -- : 
ArgumentError (invalid byte sequence in UTF-8):
  app/models/channel_icon/icon_finder.rb:89:in `is_blocked?'
  app/models/channel_icon/icon_finder.rb:109:in `is_blocked'
  app/models/channel_icon/icon_finder.rb:147:in `block (2 levels) in search'
  app/models/channel_icon/icon_finder.rb:146:in `block in search'
  app/models/channel_icon/icon_finder.rb:143:in `each'
  app/models/channel_icon/icon_finder.rb:143:in `search'
  app/controllers/channel_icon_controller.rb:88:in `search'

[Stuart Auchterlonie]

Moving remaining open 0.28.1 tickets to 0.28.2

[Stuart Auchterlonie]

Closing any remaining tickets for 0.28, if the issue persists, feel free to reopen and align to v29 or master