Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping
When I’m web scraping info off the Internet and run into Frames I first cry and then say a prayer… The below video helps review the issue and offers some solutions on how to, reliably, grab data.
Just keep in mind that each frame is, in essence, it’s own page and you’ll have to add additional values to your handle to navigate to it.
A copy of the syntax writer for Web Scraping with AutoHotkey can be found here.
In order to speed-up our page load times I like to archive items to an old folder. Unfortunately there is no easy way to select some, but not all, items. What I really wanted to do is automate select multiple items from a list.
Web scraping with AutoHotKey to the rescue!!! This simple looper is fast, flexible and a life saver. It simply sends tabs and the space key to my browser window saving me a lot of time and mundane work. Below is the AutoHotKey code which is set to work in Chrome
WinActivate, ahk_class Chrome_WidgetWin_1 ;activate chrome window- lightning fast compared to IE
Loop, 25
{
Send, {Tab 2} ;sends two tabs
sleep, 5
Send, {Space} ;selects / deselects item
}
returnAnd here is a video in case you’re not getting the value.
URL VerificationWorking in Internet Marketing I’m frequently involved with reviewing new and active web pages. When I started the process was manual and quite time consuming so I wrote a script to automate much of it. This video demonstrates it in action. It requires you to navigate in IE (yes, I know…) and uses Microsoft COM to rip all the URLs by web scraping with AutoHotkey from the active page and dumps them into a spreadsheet for URL verification. It also pulls things like the title of the page, # of words on the page, most popular words, Values for name and ID elements as well as the actual URL and outerHTML.
The script also allows you to choose your overall area which is great because there is probably a template that your company uses (and you’ll probably want to exclude items that are part of site navigation, not unique to that particular page.
