Opened 12 years ago

Closed 12 years ago

#4519 closed defect (fixed)

MythWeather Environment Canada grabber broken

Reported by: Joe Ripley <vitaminjoe@…> Owned by: stuartm
Priority: minor Milestone: 0.21
Component: mythweather Version: head
Severity: low Keywords: MythWeather, Environment Canada, grabber
Cc: stuartm Ticket locked: no

Description

It looks like Environment Canada (http://www.weatheroffice.gc.ca) has changed their page layout. This has broken the ENVCAN grabber for MythWeather?.

However, it appears that the weather information is offered in an RSS feed now (i.e. http://www.weatheroffice.gc.ca/rss/city/ab-50_e.xml). I will update the grabber script and offer it as a patch shortly.

-- Joe Ripley vitaminjoe@…

Attachments (6)

4519_diff.patch (17.0 KB) - added by Joe Ripley <vitaminjoe@…> 12 years ago.
Modified to handle changes in Environment Canada HTML layout, and fixes some whitespace issues
4519_diff.2.patch (17.0 KB) - added by Joe Ripley <vitaminjoe@…> 12 years ago.
Second version of patch to get EnvCan? grabber working. Fixes a few small details from the previous patch
4519_diff.3.patch (238.2 KB) - added by Joe Ripley <vitaminjoe@…> 12 years ago.
Makes use of RSS feed from Environment Canada for weather information
4519_diff.4.patch (25.3 KB) - added by Joe Ripley <vitaminjoe@…> 12 years ago.
Update ENVCAN grabber to use RSS feed from Environment Canada, also correct some whitespace issues in other modules
4519_diff.5.patch (25.4 KB) - added by Joe Ripley <vitaminjoe@…> 12 years ago.
Fixed a couple very small details from the previous version of this patch
4519_diff.6.patch (25.8 KB) - added by Joe Ripley <vitaminjoe@…> 12 years ago.
Found another weather condition that wasn't being handled (overcast). Fixed in this version.

Download all attachments as: .zip

Change History (18)

Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Attachment: 4519_diff.patch added

Modified to handle changes in Environment Canada HTML layout, and fixes some whitespace issues

comment:1 Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Attached patch should get things working again for the Environment Canada grabber. I haven't been able to test all possible renderings of the page (it'll take a couple days). I wouldn't commit this until I'm sure the bugs are worked out. I thought I'd post it now anyway, if anyone wants to test it.

I wanted to be able to use the RSS feed for parsing, however, the data on it is more abstract than the HTML data. Some of the RSS data is written in plain language, which could change, while the HTML data is numerical and probably won't. So, I had to stick with the HTML parser.

I'll keep an eye out for anything I missed and post patches if necessary.

-- Joe Ripley vitaminjoe@…

comment:2 Changed 12 years ago by stuartm

Owner: changed from ldunning to stuartm
Status: newaccepted

Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Attachment: 4519_diff.2.patch added

Second version of patch to get EnvCan? grabber working. Fixes a few small details from the previous patch

comment:3 Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Revised patch attached. This corrects a few small details missed by the first patch. Due to some inconsistency on Environment Canada's part, the 5 day forecast uses some kind of rotating id name. This causes some missing icons (sometimes!). This patch fixes that.

Still need to verify that wind gust speed is parsed correctly.

-- Joe Ripley vitaminjoe@…

comment:4 Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Environment Canada sometimes doesn't report the high or low for a given day (not sure why). But this screws up the ordering of the data for MythWeather?. Patch is forthcoming...

-- Joe Ripley vitaminjoe@…

Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Attachment: 4519_diff.3.patch added

Makes use of RSS feed from Environment Canada for weather information

comment:5 Changed 12 years ago by Joe Ripley <vitaminjoe@…>

I've decided to use the RSS feed after all. It's layout has proven to be more consistent than the HTML layout. Also, each XML file is only about 4.5kB, which lessens the load on Environment Canada's servers as compared to the HTML files.

This patch adds the XML urls to ENVCAN-Stations.xml as well as a (nearly) completely re-written ENVCANParser.pm module. It also removes the need for the 'ENVCAN_icons' file.

Still need to do more testing, but this should work for most people.

-- Joe Ripley vitaminjoe@…

comment:6 Changed 12 years ago by stuartm

Status: acceptedstarted

comment:7 Changed 12 years ago by anonymous

Oops. I just applied an earlier patch to ENVCANParser.pm. Sorry. Stuart, I will clean up the mess if you haven't started locally applying yet. Otherwise, fire this off to me?

comment:8 Changed 12 years ago by anonymous

Where "me" = "nigel". *&$*@(% Trac auto-logouts

Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Attachment: 4519_diff.4.patch added

Update ENVCAN grabber to use RSS feed from Environment Canada, also correct some whitespace issues in other modules

comment:9 Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Just added the latest revision of the patch. I'm pretty confident that this one can be applied. I've reverted the earlier (larger) patch. The XML URLs aren't really necessary in ENVCAN-Stations.xml (neither are the HTML ones actually).

This patch does the following:

  • Cleans up whitespace issues in all ENVCAN script components
  • Modifies ENVCANParser.pm to parse XML RSS feed from Environment Canada
  • Modifies envcan.pl to work with some minor interface changes in ENVCANParser.pm

Anything I've missed is probably trivial. This patch also takes care of issues that I brought up in ticket #4416. That one can be closed as invalid.

-- Joe Ripley vitaminjoe@…

comment:10 in reply to:  7 Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Replying to anonymous:

Oops. I just applied an earlier patch to ENVCANParser.pm. Sorry. Stuart, I will clean up the mess if you haven't started locally applying yet. Otherwise, fire this off to me?

nigel: The patch I just submitted was diff'd against 15650, so it should apply cleanly to the changes that you applied in [15648].

-- Joe Ripley vitaminjoe@…

Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Attachment: 4519_diff.5.patch added

Fixed a couple very small details from the previous version of this patch

comment:11 Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Sorry folks. I just noticed that the grabber was outputting 'mph' for wind speed when it wasn't necessary (i.e. the plugin does this already). Also, I had to correct for the instance where Environment Canada doesn't supply the 'high-2' data point.

Should be good to go now.

-- Joe Ripley vitaminjoe@…

Changed 12 years ago by Joe Ripley <vitaminjoe@…>

Attachment: 4519_diff.6.patch added

Found another weather condition that wasn't being handled (overcast). Fixed in this version.

comment:12 Changed 12 years ago by Nigel

Resolution: fixed
Status: startedclosed

(In [15682]) Canadian grabber upgrades by Joe Ripley. Closes #4519

Note: See TracTickets for help on using tickets.