Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping
The term Web Scraping is well known but I’ve yet to hear much talk about “Web Pasting”. In this Web Scraping Example with AutoHotkey video I demonstrate how I helped my Real Estate agent automate checking the status of a house on a website.
Here is a second version where, depending on the status of the home, I write a follow-up email either with an email from Outlook or within the website via COM.
I had over 100 URLs I wanted to submit to StumbleUpon.com and wanted to tag them with around 12 interests. This would have taken me quite some to do manually so I wrote an AutoHotKey script to web scrape and cranked it out in no time!
Below is the video demonstrating the Automating StumbleUpon submissions.
And here is the first draft of my script (I only included 3 urls in this example)
Interests:="Self Improvement|Gadgets|Technology|Internet|Computers|Internet Tools|Hacking|Software|Computer Science|Programming|Open Source|Computer Security|Web Development|IT|Windows" urls= (Using 6 built-in SciTE Find and Replace features- Great time savers!https://the-automator.com/gui-menu-for-api-web-serviceExcellent SciTE editor video documenting powerful features) ;***********Loop over StumbleUpon Submissions******************* Loop, parse, URLs, `n, `r ;loop over Var line by line { IfEqual, A_LoopField,,continue ;Skip loop if blank ;automating StumbleUpon submissions pwb := WBGet() pwb.Navigate("http://www.stumbleupon.com/submit") ;Navigate to URL while pwb.busy or pwb.ReadyState != 4 ;Wait for page to load Sleep, 100 ;~ pwb.document.all.url.Value :=A_LoopField ;Unique ID -no dashes Sleep, 400 Clipboard:=A_LoopField pwb.document.all.url.focus() pwb.document.ExecCommand("paste") ; paste pwb.document.all.nsfw[0].focus() ;~ pwb.document.all.nsfw[0].checked :=1 ;Set Tagname and Array value pwb.document.all.nsfw[0].Click() pwb.document.all.nsfw[0].fireEvent("onchange") Loop, parse, Interests, | ;loop over interests { DDL:=pwb.document.getElementByID("submit-search-interests-dropview").getElementsByClassName("droplist dropdown-menu")[0].getElementsByClassName("animated-item-view rendered animated-item-view-end") IfEqual, A_LoopField,,continue ;Skip loop if blank pwb.document.getElementByid("submit-search-interests").focus() ;~ pwb.document.getElementByid("submit-search-interests").value:=A_LoopField Clipboard:=A_Loopfield pwb.document.getElementByid("submit-search-interests").focus() pwb.document.ExecCommand("paste") ; paste - triggers DOM/javascript events WinActivate, StumbleUpon loop, % (ddl.length) { if (trim(DDL[A_index-1].innerTEXT) =A_LoopField){ ;~ MsgBox % trim(DDL[A_index-1].innerTEXT) Sleep, 500 pwb.document.getElementByid("submit-search-interests").focus() Sleep, 500 SendPlay, {Down} Sleep, 500 SendPlay, {Enter} } } } ;***********submit StumbleUpon submissions******************* sleep, 1000 pwb.document.getElementByID("submit-view-save").click() ;***********Cancel navigate away from page and just submit******************* WinWaitActive,Windows Internet Explorer WinActivate, Windows Internet Explorer ;~ https://autohotkey.com/board/topic/38243-fill-form-in-the-background-logging-into-account/ ControlClick,Button2,ahk_class #32770,The webpage you are viewing is trying to close the tab. Sleep, 1000 } ;End loop over pages MsgBox complete automating StumbleUpon submissions
Being able to, programatically, navigate to an Internet page and scrape the contents in a reliable fashion is best things invented since sliced bread! I spent years manually going through pages and copying/pasting contents from IE to Excel then spent even more time trying to clean it up. Done properly you can get the data very, very close to how it is on the web with little effort.
The below video walks through using AutoHotKey to obtain basic values from a Web page. It also demonstrates a script I wrote that helps write the syntax (yes I’m that lazy!) The AutoHotKey script I wrote is further down this page and can also be found on the AHK forum here.
In this beginning tutorial I how to:
1) get a pointer to IE
2) navigate to a page
3) get text from a page
