Web Scraping with AutoHotKey 101-Super simple ways to Get data from a page, handles & pointers

Web Scraping with AutoHotKey

Web Scraping with AutoHotkey: Intro

Being able to, programatically, navigate to an Internet page and scrape the contents in a reliable fashion is best things invented since sliced bread!   I spent years manually going through pages and copying/pasting contents from IE to Excel then spent even more time trying to clean it up.  Done properly you can get the data very, very close to how it is on the web with little effort.

The below video walks through using AutoHotKey to obtain basic values from a Web page.  It also demonstrates a script I wrote that helps write the syntax (yes I’m that lazy!)  The AutoHotKey script I wrote is further down this page and can also be found on the AHK forum here.

In this beginning tutorial I how to:
1) get a pointer to IE
2) navigate to a page
3) get text from a page

Web Scraping Intro with AutoHotkey

Here is the script writer to use during your web scraping intro with AutoHotkey.
Web Scraping with AutoHotKey

How to use SciTE messages to control SciTE with AutoHotkey | 63 Extremely powerful messages to control SciTE

How to use SciTE messages

How to use SciTE messages

SciTE is a great IDE that I use with AutoHotKey, SPSS, SQL, Python, XML, HTML, etc.   I love being able to use regular expressions in it to manipulate text and it has some very cool capabilities.  This video is one of my favorite demonstrations how powerful SciTE can be at manipulating text.

Here is a short tutorial and demonstration on how to manipulate SciTE editor via COM objects and Windows commands with AutoHotKey.

Tutorial How to use SciTE messages with AutoHotkey

How to customize SciTE

See the list of SciTE commands here

Take a deep-dive into Scintilla documentation

Look at the Director Interface options

To send messages in AutoHotkey review MSDN

FYI- 0x111 is the WM_COMMAND

SciTE Messages for use in AutoHotkey with a COM object

You can use spy to find wm_command

A specific version of the SciTE editor for AutoHotKey can be downloaded here and more generic documentation can be found here.

Loading contacts to SilverPop via API call & SFTP file transfer

API call & SFTP file transfer

API call & SFTP file transfer

API call & SFTP file transfer

Loading our contact lists into Engage via their Web interface (browser) takes around 7-9 minutes of focused attention per list.  Doing the same thing with an API call and Secure File Transfer (SFTP) takes seconds!  The below video demonstrates the process from using web scraping to grab the name /ID on our SharePoint server to uploading the file to Engage, creating the name in Engage and tying the new name to the contact list.  The video is longer because I also have it wait for the data job to complete and then write an email confirming the process is complete.

The API call & SFTP file transfer is free with Engage/SilverPop but it isn’t well promoted (on their website nor by their representatives.  Spending a bit of time to get it working is well worth the investment. For the SFTP file transfer I’m using WinSCP but any SFTP program will do.

 

 

API call & SFTP file transfer