Saving an https web page to disk

[rdgerken]

I have a "smart" thermostat from Carrier/Bryant that links out to the internet. Unfortunately, I don't think I can interface with the stat directly via any kind of API, so I was trying to do some screen scraping initially to pull the temperatures from the OEM web page that the stat links up to on the internet.

Does anyone know how I can retrieve an HTTPS page to a file, that I can then later parse using EG?

I think if I could use the urlretrieve function in urrlib, that would work, except this is for an HTTPS page, so I used urllib2 instead, but it doesn't have the retrieve function. I was thinking of using this, and then adding Pako's write file plugin to drive the results out to a file, but I had trouble getting the output of the script mapped into the file.

here is what i have for a script so far... but I don't know how to get the results out to a file (or even see if it's working for that matter).

Code: Select all

import urllib2, base64

request = urllib2.Request("https://www.myevolutionconnex.bryant.com/MyLocations")
base64string = base64.encodestring('%s:%s' % ('username', 'password')).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)   
result = urllib2.urlopen(request)

I think in this script that I found, "result" is some sort of page handle... but I don't know how to link that over to the write file plugin.

Thanks for any help you can provide.

UPDATE 1:

So I was able to get the page saved off to a file, but unfortunately, the authentication is not working - so I think I have to go a different route.

[rdgerken]

I think I'd like to try something like this:

Code: Select all

import mechanize
br=mechanize.Browser()
br.open('https://www.blahblah.com/Login')
br.select_form(nr=0) 
br['user[login]']= your_login
br['user[password]']= your_password
br.submit()
br.retrieve('http://www.blahblah.com/dataiminterestedin','saveittothisfile.html')

problem is... mechanize is not included with EventGhost. I'm a total python noob. Can anyone explain to me how to load additional libraries so that I can use them in my EG scripts? I don't want to muck up my install by doing something wrong (like loading a wrong version of python, or other)

Thanks

[topix]

Try copying mechanize to c:\Program Files (x86)\EventGhost\lib26\site-packages\.

[rdgerken]

That worked perfectly! Thank you!

The snippet I posted worked fine too, except I needed to enclose my user/pass in quotes which I forgot to do for the posting.

All that's left is to write some script to scrape the information out of that file, log to MySQL, and then discard the file.