Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping
In this tutorial I walk through how to “walk the DOM” (DOM= Document Object Model) with IE and AutoHotkey.
I used to grab all the text and then try and use a regular expression to parse the data. Taking advantage of the DOM is critical to learning how to scrape pages efficiently. In this case I’m simply relying on built in tags to an HTML table.
Here is the first bit of code I mention to scrape data from a table
found here.
Someone asked me if I had a tool that would look at the clipboard and return only the URL or Rip the URL from the Clipboard. (The clipboard has various formats like: Files, Text, Rich Text Format, HTML, Images, etc.).
I used the cool WinClip library which can be grabbed here: https://autohotkey.com/board/topic/74670-class-winclip-direct-clipboard-manipulations/ and, with the help of maestrith, wrote a script which returns the various items on the clipboard in HTML format as an object.
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
#SingleInstance,Force
SetBatchLines,-1
global Com:=ComObjCreate(“HTMLFile”)
;*******************************************************
Browser_Back:: ;Change this hotkey to what you want
CB:=ClipboardAll
Clipboard:=””
while(Clipboard){
Sleep,10
}
Send,^c
while(!Clipboard){
Sleep,10
}
Clip:=Get_Clipboard_Data() ;Call the function and return the HTML data in an object
MsgBox % clip.1.HREF
DebugWindow(Obj2String(Clip),1,1,200,0)
Clipboard:=CB ;restore Clipboard
return
;********************Grab data from Clipboard***********************************
Get_Clipboard_Data(){
wc := new WinClip
wc.iSnap()
Obj:=[]
try:=wc.iGetHTML()
HTML:=SubStr(Try,InStr(Try,”<HTML>”))
Com.Close()
Com.Write(HTML)
All:=Com.GetElementsByTagName(“a”)
Url:=[]
RegExMatch(Try,”OUi)SourceURL:(.*)\R”,Found)
while(aa:=All.Item[A_Index-1]){
URL.Push({Page:Found.1,Text:aa.InnerText,HREF:aa.HREF})
}
return URL
}
;~ https://autohotkey.com/board/topic/74670-class-winclip-direct-clipboard-manipulations/
#Include B:\Progs\AutoHotkey_L\AHK Work\WinClip\WinClipAPI.ahk
#Include B:\Progs\AutoHotkey_L\AHK Work\WinClip\WinClip.ahk
I use ClassName a lot when web scraping. So much so I updated the iWB2 Learner tool to extract it from the outerHTML and have it’s own edit field (like Name and ID).
I also changed the ListView that shows Frames to be RED. I can’t tell you how much time I wasted early-on when dealing with Frames. Hopefully this will help people realize they need to take a different approach when Frames are present! Continue reading
