Mythweb Weather throws error when MSNBC unavailable

[anonymous]

Just checking weather in the latest svn r13224 and have got the following errors:

Error at /var/www/localhost/htdocs/modules/weather/includes/objects/WeatherSite.php, line 56:
file(): HTTP request failed! HTTP/1.1 404 NotFound

Error at /var/www/localhost/htdocs/modules/weather/includes/objects/WeatherSite.php, line 56:
file(http://www.msnbc.com/m/chnk/d/weather_d_src.asp?acid=ASXX0301): failed to open stream: Success

Error at /var/www/localhost/htdocs/modules/weather/includes/objects/WeatherSite.php, line 58:
Invalid argument supplied for foreach()

Error at /var/www/localhost/htdocs/modules/_shared/tmpl/default/header.php, line 16:
Cannot modify header information - headers already sent by (output started at /var/www/localhost/htdocs/includes/errors.php:117)

I think maybe a catch/try or whatever the construct is could be used for the data fetch from the msnbc website.

This *does not* cause an unrecoverable error, just a bit ugly and doesn't let the user know what has really happened.

[anonymous]

Some further info:

This problem also affects the mythweather plugin it seems.

I have tried the link reported above and just get a 404 error. I think that the url for the weather data may have changed.

I can get weather using the following url:

http://weather.msn.com/local.aspx?wealocations=wc:ASXX0111

the original doesn't work.

Unfortunately I hadn't ever visited this weather site before to know if the actual page has changed or not.

[maverik044]

That web site won't work. The code is looking for data within the makeWeatherObj function on web site. Seems that may have changed or move.

Notice that other projects use xoap with ​http://xoap.weather.com. This code may be a good starting place: ​http://sourceforge.net/projects/xoapweather/

[maverik044]

One can get weather info like this: ​http://xoap.weather.com/weather/local/CAXX0301?cc=*&unit=C&dayf=4&prod=xoap&par=xxx&key=xxx in soap format. Not sure where to get partnar id and key, don't think appropriate to post so I put xxx, but I used from the following file for test: ​https://svn.sourceforge.net/svnroot/mediaportal/trunk/mediaportal/WindowPlugins/GUIWeather/GUIWindowWeather.cs

[jeff@…]

I can confirm that MSNBC has changed their web structure. The broken URL is hard coded in /usr/share/mythtv/mythweb/modules/weather/handler.php

[anonymous]

Maybe another website such as weather.com, wunderground.org, or weather.gov would work better. Most Linux OS'es have the builtin weather script which essentially grabs the weather info from wunderground.org. Not sure if this info is helpful, but thought it might be easier than trying to grab data from a website that could change anytime.

[reallyasi9 [at] g[oogle]mail.com]

I don't regularly use MythTV's weather plugin, but I do use mythweb's, and I was able to kludge together a working update that uses Yahoo Weather's XML feed. It's easy to parse, but only has forecast data for two days in the future (I think MSNBC used to have 5 days). The relevant URL for me is ​http://weather.yahooapis.com/forecastrss?p=USOH0212. The XML format they use is well documented at ​http://developer.yahoo.com/weather/, so you won't be surprised by any changes they might make, and this fiasco won't happen in the future.

[reallyasi9 [at] g[oogle]mail.com]

By the way, the following is the kludge I added. In the handler.php file in modules/weather/, I had to change how the data were parsed, and some of the forecast parsing routine:

<?php
/**
 * Handler for the Weather module.
 *
 * @url         $URL: svn+ssh://ijr@cvs.mythtv.org/var/lib/svn/trunk/mythplugins/mythweb/modules/weather/handler.php $
 * @date        $Date: 2006-04-02 03:40:11 -0400 (Sun, 02 Apr 2006) $
 * @version     $Revision: 9601 (PaSTE Kludge 01)$
 * @author      $Author: xris $
 * @license     GPL
 *
 * @package     MythWeb
 * @subpackage  Weather
 *
/**/

// Unit preference
    if (empty($_SESSION['siunits'])) {
        $_SESSION['siunits'] = setting('SIUnits');
    }

/**
 * @global  array   $GLOBALS['Weather_Types']
 * @name    $Weather_Types
/**/
    global $Weather_Types;
    $Weather_Types = array();

// Load the weather data
    foreach (file(modules_path.'/'.module.'/weathertypes.dat') as $line) {
        list($id, $name, $img) = explode(',', $line);
        $Weather_Types[$id] = array($img, $name);
    }

// Build a list of the known weather sites
    $WeatherSites = array();

    $sh = $db->query('SELECT data, hostname FROM settings WHERE value="locale"');
    while (list($data, $host) = $sh->fetch_row()) {
    // New data site
        if (empty($WeatherSites[$data])) {
            $WeatherSites[$data] = new WeatherSite($data, $hostname, $_SESSION['siunits']);
        }
    // Add the hostname to sites we've already seen
        else {
            $WeatherSites[$data]->host .= ', '.$host;
        }

    }
    $sh->finish();

// Print the weather page template
    require_once tmpl_dir.'weather.php';

// Exit
    exit;

class WeatherSite {

    var $acid;
    var $host;

    var $city;
    var $subdiv;
    var $country;
    var $region;

    var $use_metric;
    var $Temperature;
    var $CIcon;
    var $ConditionImage;
    var $ConditionText;
    var $WindSpeed;
    var $WindDirection;
    var $BarometricPressure;
    var $PressureRising;
    var $Humidity;
    var $Real;
    var $UV;
    var $Visibility;
    var $ConText;
    var $Forecast;

    var $RadarImage;

    var $LastUpdated;

    function WeatherSite($data, $hostname, $use_metric) {
        $this->acid       = $data;
        $this->host       = $hostname;
        $this->use_metric = $use_metric;

        $this->getData();
        $this->RadarImage = $this->getRadarURL();
    }

    function getData() {
    $data = file("http://weather.yahooapis.com/forecastrss?p=" . $this->acid);

    $forecastday = -1;
    foreach($data as $line) {
        if(strpos($line, "yweather:") === false)
        continue;

        $line = trim($line);
	preg_match_all('/ ([a-z]+)="(.*?)"/',$line,$matches,PREG_SET_ORDER);
	foreach($matches as $n => $match) {
		$name = $match[1];
		$value = $match[2];

	        switch ($name) {
        	case "city":
	            $this->city = $value;
        	    break;
/*
	        case "SubDiv":
        	    $this->subdiv = $value;
	            break;
*/
	        case "country":
        	    $this->country = $value;
	            break;

	        case "region":
        	    $this->region = $value;
            	break;

	        case "temp":
        	    $this->Temperature = $value;
	            break;

        	case "code":
	            $this->CIcon = $value;
        	    break;

	        case "speed":
        	    $this->WindSpeed = $value;
	            break;

        	case "direction":
	            $this->WindDirection = $value;
	            break;

	        case "pressure":
        	    $this->BarometricPressure = $value;
	            break;

		case "rising":
		    $this->PressureRising = $value;
		    break;

        	case "humidity":
	            $this->Humidity = $value;
        	    break;

	        case "chill":
        	    $this->Real = $value;
	            break;
/*
	        case "UV":
        	    $this->UV = $value;
	            break;
*/
        	case "visibility":
	            $this->Visibility = $value;
        	    break;

	        case "date":
        	    if (!$this->LastUpdated) $this->LastUpdated = $value;
	            break;

	        case "text":
        	    if (!$this->ConText) $this->ConText = $value;
	            break;
/*
	        case "Fore":
        	    $this->Forecast = $this->readForecast($value);
	            break;
*/
	        default:
        	   // Weird, unknown type
	           break;
		}
        }
	// Forecast
	if (stripos($line,'yweather:forecast') === false) continue;
	$forecastday++;
	preg_match_all('/ ([a-z]+)="(.*?)"/',$line,$matches,PREG_SET_ORDER);
	foreach($matches as $n => $match) {
		$name = $match[1];
		$value = $match[2];

		switch ($name) {

		case 'day':
			$wonk[$forecastday][0] = $value;
			break;

		case 'date':
			$wonk[$forecastday][1] = $value;
			break;

		case 'low':
			$wonk[$forecastday][5] = $value;
			break;

		case 'high':
			$wonk[$forecastday][4] = $value;
			break;

		case 'text':
			$wonk[$forecastday][3] = $value;
			break;

		case 'code':
			$wonk[$forecastday][2] = $value;
			break;

		default:
			break;
		}
	}
    }
    ksort($wonk);
    foreach($wonk as $datarray) {
        ksort($datarray);
    	foreach ($datarray as $datum) {
    	    if ($datastring) $datastring .= "|";
    	    $datastring .= $datum;
	}
    }
    $this->Forecast = $this->readForecast($datastring);
    

    //Are we using metric or imperial system
    if($this->use_metric == "YES") {
        $this->Temperature = round((5/9) * ($this->Temperature - 32));
        $this->Real = round((5/9) * ($this->Real - 32));
        $this->BarometricPressure = round($this->BarometricPressure * 2.54);
        $this->Visibility = round($this->Visibility * 1.609344);
        $this->WindSpeed = round($this->WindSpeed * 1.609344);
    }

    if(strlen($this->ConText) > 0) {
        $this->ConditionText = $this->ConText;
        $this->ConditionImage = getImageFromName($this->ConditionText);
        if(strlen($this->ConditionImage) == 0)
        list($this->ConditionImage, $blah) = getImageAndDescFromId($this->CIcon);
    } else {
        list($this->ConditionImage, $this->ConditionText) = getImageAndDescFromId($this->CIcon);
    }
    $this->ConditionImage = (strlen($this->ConditionImage) > 0) ? $this->ConditionImage : "unknown.png";
    }

    function getRadarURL() {
    $data = file("http://w3.weather.com/weather/map/" . $this->acid . "?from=LAPmaps&setcookie=1");
    foreach($data as $line) {
        if(substr(trim($line), 0, 29) != "if (isMinNS4) var mapNURL = \"") continue;

        $url1 = substr(trim($line), 30);
        $url1 = "http://w3.weather.com/" . substr($url1, 0, strlen($url1) - 2);

        break;
    }

    $data = file($url1);
    foreach($data as $line) {
        if(substr(trim($line), 0, 48) != "<IMG NAME=\"mapImg\" SRC=\"http://image.weather.com") continue;

        $url2 = substr(trim($line), 24);
        $url2 = substr($url2, 0, strpos($url2, '"'));
        break;
    }
    return $url2;
    }

    function readForecast($data) {
    $ret = array();

    $data = explode("|", $data);
    # XXX
    #format: dow|date|code|text|high|low|etc...
    for($i = 0;$i<2;$i++) {
	# mktime uses 0-6;  msnbc gives us 1-7;  adjust msnbc to match mktime
	switch ($data[6*$i]) {
	case 'Sun':
		$dayofweek = 0;
		break;
	case 'Mon':
		$dayofweek = 1;
		break;
	case 'Tue':
		$dayofweek = 2;
		break;
	case 'Wed':
		$dayofweek = 3;
		break;
	case 'Thu':
		$dayofweek = 4;
		break;
	case 'Fri':
		$dayofweek = 5;
		break;
	case 'Sat':
		$dayofweek = 6;
		break;
	}
	$datestring = strtotime($data[6*$i + 1]);
	$datestring = date("mdY",$datestring);
        $forecast = new Forecast($datestring,$dayofweek);
        $forecast->dayofweek = $dayofweek;
        list($forecast->DescImage,$forecast->DescText) = getImageAndDescFromId($data[6*$i + 2]);
        $forecast->DescImage = (strlen($forecast->DescImage) > 0) ? $forecast->DescImage : "unknown.png";
        $forecast->DescText = (strlen($forecast->DescText) > 0) ? $forecast->DescText : t('Unknown') . " (" . $data[6*$i + 3] . ")";
        if($this->use_metric == "YES") {
            $forecast->HighTemperature = round((5/9) * ($data[6*$i + 4] - 32));
            $forecast->LowTemperature = round((5/9) * ($data[6*$i + 5] -32 ));
        } else {
        $forecast->HighTemperature = $data[6*$i + 4];
        $forecast->LowTemperature = $data[6*$i + 5];
        }

        $ret[$i] = $forecast;
    }

    return $ret;
    }
}

function getImageAndDescFromId($my_id) {
    global $Weather_Types;
    return $Weather_Types[$my_id];
}

function getImageFromName($my_name) {
    global $Weather_Types;
    foreach ($Weather_Types as $pair) {
        if ($pair[1] == $my_name)
            return $pair[0];
    }
}

class Forecast {
    var $date;
    var $dayofweek;

    var $DescImage;
    var $DescText;

    var $HighTemperature;
    var $LowTemperature;

    function Forecast($date,$real_dayofweek) {

    $month = substr($date,0,2);
    $day = substr($date,3,2);
    $year = substr($date,6,4);

    $temp_date = mktime(0,0,0,$month,$day,$year);
    $date_time_array = getdate( $temp_date );
    $claimed_dayofweek = $date_time_array['wday'];

    if($real_dayofweek != $claimed_dayofweek)

        $this->date = date("m/d/Y", mktime(0, 0, 0, $month  , $day+1, $year));
    else
        $this->date = $date;
    }
}

[simonflood@…]

Replying to reallyasi9 [at g[oogle]mail.com ]:

By the way, the following is the kludge I added. In the handler.php file in modules/weather/, I had to change how the data were parsed, and some of the forecast parsing routine:

I get the following error when I access /mythweb/weather having replaced handler.php with this kludged version.

Fatal error: Call to undefined function: stripos() in /usr/share/mythtv/mythweb/modules/weather/handler.php on line 154

The stripos function is not defined within the old handler.php either.

HTH

Simon

[briand]

A while back, somebody asked Isaac if/when the "mythweather-revamp" branch would be merged into SVN trunk... and, if I recall correctly, he indicated that if scripts were written for the -revamp branch that allowed it to retrieve information from the same source that the current mythweather plugin uses (so that it could be the 'default' setup), he'd strongly consider merging the -revamp branch into SVN...

If, indeed, the provider has changed the page sufficiently that a re-write of code is necessary to recover functionality... then perhaps that effort is better spent within the -revamp branch.

[maverik044]

There are two packages in the ubuntu packages database (​http://packages.ubuntu.com/feisty/web/php-services-weather and ​http://packages.ubuntu.com/feisty/perl/libweather-com-perl) which look to me like they could be used to fix mythweather and mythweb-weather.

[anonymous]

I think one of the major hurdles in this problem is to find a reliable data source that could be easily parsed for using weather information. Perhaps a source that provides the information from a telnet or other text-based service would be better than trying to keep up with parsing a website that could change at anytime and break everything.

[anonymous]

Look at NOAA for the US they give all there data in RSS if you would like

​http://www.rss.noaa.gov/

[maverik044]

The "reliable source" (I believe) is ​http://xoap.weather.com/weather/local/CAXX0301?cc=*&unit=F&dayf=4&. I'm working on a patch. I hope to have something out soon. Most other software uses this as the source. The mythtv owners may need to sign up for a partner id though.

Another common source of data is the US METAR system (aviation) NOAA which does appear to offer international data. Problem with this as a source is that I don't believe that the existing city identifiers coded into myth (ie. CAXX0301) can be used. They use the airport identifiers instead. So more of the myth code may have to be changed. With the weather.com source, we don't need to change this.

[maverik044]

Preliminary patch see comment below; forecast does not work yet...

[maverik044]

Preliminary patch posted above. Forecast does not work. The patch was generated at the directory level: mythweb/modules. This patch uses xoap.weather.com. Technically I think we need to add the weather.com logo on the page according the the usage terms, could use some help with that if someone is willing. Will fix the forecast for this weekend.

Only tested for one location, please test for others.

In the future may need a partner id and key for this to work.

[simonflood]

Replying to simonflood:

I get the following error when I access /mythweb/weather having replaced handler.php with this kludged version.

Fatal error: Call to undefined function: stripos() in /usr/share/mythtv/mythweb/modules/weather/handler.php on line 154

The stripos function is not defined within the old handler.php either.

To answer my own report changing stripos to strpos fixes this error.

However be aware that Yahoo Weather uses different numbering for the weather types so weathertypes.dat needs re-doing. See ​http://developer.yahoo.com/weather/#codes for Yahoo's codes. Of course the conditions don't match those for MSNBC!

Also with this kludged script, visibility is already supplied in km (metric) in the XML feed so it doesn't need converting to imperial (and in fact needs converting to imperial if not metric).

[maverik044]

Weather codes for yahoo are the same as the ones for weather.com. So I'll fix that in the next version of the patch that includes the forecast.

[anonymous]

The info that mythweb used to get at "​http://www.msnbc.com/m/chnk/d/weather_d_src.asp?acid=" + locale" is now available at "​http://weather.msn.com/local.aspx?wealocations=wc:" + locale" using the same location codes. I think the weather descriptions and icons are the same as msnbc was using, because it turning out that it's not to dificult to write a new parsing routine in the weather.cpp file(-I LOVE OPEN SCOURCE :)-). I currently have a working libmythweather.so file that is only missing the extended forecast info and a little bit of tomorrows forecast. I tested it with a couple of different location settings and everything seems to work good, even worked with a Great Britton location, although my one hardcoded peice of info(contry code variable) shows up when its not supposed to on non USA locations but doesn't affect the workings. As soon as I have more time I will post a link to a new libmythweather.so file.

P.S. This will have NO affect on the mythweb weather module as far as I know. But it looks like some people have already started that fix. :)

[maverik044]

are you parsing the web page? The original link was a function created specifc to the locale. If you are, I'd be worried that if the web page changes, your patch may become broken. The advantage with weather.com is that the information provided is in xml format that is defined in their SDK. It is very, very unlikely that it will change to the point to break the code. I had planned on working on a patch for mythweather to use weather.com after I completed the mythweb weather page.

[maverik044]

Updated patch for mythweb weather to use xoap.weather.com

[maverik044]

icons need to latest patch - part 1

[maverik044]

icons need to latest patch - part 3

[maverik044]

icons need to latest patch - part 2

[maverik044]

icons need to latest patch - part 4

[maverik044]

icons need to latest patch - part 4

[maverik044]

Ok I've updated the patch. Everything seems to be working including the forecast.

Need to put the icon files from the four zip files in mythweb\skins\default\img\weather and I guess also mythweb\skins\grey\img\weather

Need help with mythweb\modules\weather\tmpl\default\weather.php. Put crude patch to add weather.com logo, but don't know how to format it properly.

If someone can test another location, I think it'll be ready for svn incorporation.

[anonymous]

Replying to maverik044:

are you parsing the web page? The original link was a function created specifc to the locale. If you are, I'd be worried that if the web page changes, your patch may become broken. The advantage with weather.com is that the information provided is in xml format that is defined in their SDK. It is very, very unlikely that it will change to the point to break the code. I had planned on working on a patch for mythweather to use weather.com after I completed the mythweb weather page.

Actully yes I am parsing the web page(pretty much like it used to do) and you have a very legitimate concern. However I think the old msnbc site was using the same resources as the new msn site for a while now and they just consolidated the 2. That being said, what I am doing was really meant for a quick fix and now that I opened my mouth, it's turning out to be more difficult than I first thought. Sure, the first page(current conditions) is easy but it gets a little buggy after that. I just got the icon to display in the tomorrow's outlook page. Although I will continue to work on this for my own sake(-because it's fun :)-), I would deffitnetly be interested in your update to use weather.com and I will wait patiently. :)

P.S. If you want, I can post my weather.cpp somewhere that you can get to it, BUT be forewarned it is very very sloppy right now(lots of testing).

[simonflood]

Replying to maverik044:

One can get weather info like this: ​http://xoap.weather.com/weather/local/CAXX0301?cc=*&unit=C&dayf=4&prod=xoap&par=xxx&key=xxx in soap format. Not sure where to get partnar id and key, don't think appropriate to post so I put xxx

Having registered with weather.com to get access to their SDK I see the following in a PDF document accompanying it that relates to types of applications, etc.

"If you have an application that does not fall into the categories of Web-Based or Desktop, such as applications for PDAs, cell phones, kiosks, print, fax, broadcast television, or other platforms that cannot immediately or directly link back to weather.com or, if your usage expectations exceed those outlined above, you will need to determine what other licensing options may be available to you.

Generally, if your site or application (a) has 500,000 unique visitors or greater on a monthly basis, or (b) has over 100,000 unique visitors on a monthly basis and it can generate 25,000 referrals per month to weather.com through the links provided to you, and you are interested in additional weather content, please contact our Business Development Group via e-mail at busdevmanager2@… to discuss a potential relationship."

I wonder whether it's worth someone from the mythTV dev team contacting them to see whether (no pun intended!) it's possible to get a _free_ Partner ID/License Key pair for all mythTV users (as opposed to each user having to sign up individually to get their own)?

[maverik044]

Right now my patch works without using the partner id and key. There is a comment in the code to change it if it is needed in the future.

I think that the mythtv dev team should sign up for the id and key that goes into the final code.

Media Portal uses weather.com also and uses a fixed id and key.

[simonflood]

Replying to maverik044:

Ok I've updated the patch. Everything seems to be working including the forecast.

There's a typo - you reference $this-Visibility when it should be $this->Visibility (ie. missing >)

[maverik044]

Patch updated to correct typo

[maverik044]

Oops, thanks for catching that. I've updated the patch.

[maverik044]

I've created ticket 3327 for mythweather. A patch is attached to that ticket.

[pitabread55@…]

can you do a really quick "how to" for applying these patches?

[maverik044]

Sure, sorry about that

Get mythplugins from SVN (tested on 13132, but should work on others if there are problems let me know I'll fix it)

from mythplugins/mythweb/modules dir: patch -p1 < weather.3.patch unzip the four icon zip files to mythplugins\mythweb\skins\default\img\weather and mythweb\skins\grey\img\weather. Also need to unzip na.zip also. Think I forgot one icon

install mythweb as usual.

[simonflood]

OK weather.com have now done something to their site that breaks the radar image (w3.weather.com appears to be no more). Two URLs need changing in WeatherSite?.php - replace ​http://w3.weather.com/weather/map/ with ​http://www.weather.com/outlook/travel/businesstraveler/map/ and ​http://w3.weather.com/ with ​http://www.weather.com/

[simonflood]

Replying to simonflood:

Two URLs need changing in WeatherSite?.php

It's simpler than what I'd posted as just the two references to ​http://w3.weather.com need changing to ​http://www.weather.com as rest of the URL still works (/weather/map/ redirects to /outlook/travel/businesstraveler/map/)

[KuruOujou]

Replying to maverik044:

Sure, sorry about that

Get mythplugins from SVN (tested on 13132, but should work on others if there are problems let me know I'll fix it)

from mythplugins/mythweb/modules dir: patch -p1 < weather.3.patch unzip the four icon zip files to mythplugins\mythweb\skins\default\img\weather and mythweb\skins\grey\img\weather. Also need to unzip na.zip also. Think I forgot one icon

install mythweb as usual.

I tried that, however I got this message:

can't find file to patch at input line 4 perhaps you used the wrong -p or --strip option? The text leading up to this was:

---

|dff -rup /home/mythtv/mythtv/myth_src/13238/mythplugins/mythweb/modules/weathe r/includes/objects/Forecast.php ./weather/includes/objects/Forecast.php

/home/mythtv/mythtv/myth_src/13238/mythplugins/mythweb/modules/weather/incl

udes/objects/Forecast.php 2007-04-14 01:34:39.000000000 -0400 |+++ ./weather/includes/objects/Forecast.php 2007-04-19 22:40:24.0000000000 -0 400

---

File to patch:

I don't know what is wrong and I can't seem to fix it. Can anyone help out?

[Rob Smith]

(In [13527]) Re #3303, this removes the error messages from being thrown when unable to connect to the msnbc site

[stuartm]

(In [13528]) Backports [13527] to -fixes. Refs #3303

Suppress errors messages when unable to connect to MSNBC and weather.com.

[maverik044]

updated for svn; fix radar map url; and minor clean-ups

[maverik044]

Added updated patch. This is fixed to work with current SVN Also Radar maps stopped working, so URL is fixed.

[maverik044]

same as previous, but fixing path to file to patch (local instead of /var/www)

[xris]

Can't accept this. Pretty sure that TWC logo use conflicts with GPL anti-copyright, and mythweb/mythtv don't have the rights to use weather.com xmloap services. Plus, the patches are full of ugly 8-space tab characters.

MythWeb will be updated as soon as MythWeather? itself is.