Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping
Web Scraping multiple pagesExample of how to Web Scraping multiple pages with some simple URL manipulation. This is where being able to make sure you’ve loaded a page fully, scrape it, then navigate to the next is critical. Looking for patterns in the URL will help you understand how you’ll be able to navigate to the next page.
A copy of the AutoHotKey syntax writer can be found here. Remember patience is a virtue and Happy Scraping!
Web Scraping your own site
You would think it would be easy to determine what criteria to use for targeting customers that have visited specific pages on our website. Unfortunately, it isn’t easy to get the values I need for the query so I wrote a script that automated grabbing what I need from the Web Scraping your own site.
Obviously you will have to adapt the overall concept to how your site is configured however my script has saved me, and my clients, thousands of hours over the past couple of years!
You can review some of my scripts that help automate web scraping here.
Video on Web Scraping your own site
A few years back Lexis Nexis had access to Jigsaw (a sales list provider). They had built in restrictions which allowed for exporting up to 1,000 contacts at a time.
Using AutoHotKey I was able to automate the exporting of contacts from Lexis Nexis into a great format. All-told I exported over 4 million business contacts including their Name, job title, job function, email address, phone number, company name, company revenue, company NAIC code and SIC codes and much more! It was quite a score and I was in love with Web Scraping from then on!
Example of Web Scraping with AutoHotkey
