Home Automation Archive by category "Web Scraping" (Page 11)

Category: Web Scraping

Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping

Web Scraping with AutoHotKey 108- use querySelctorAll for added flexibility

Web Scraping with AutoHotkeyWeb Scraping with AutoHotkeyWeb Scraping with AutoHotkey is fun and easy however some web pages don’t have Names, IDs, or classNames set up.  Other times the pages are pretty sophisticated and you want to grab something very specific (w/out writing a lot of code).   I recently learned about QuerySelctor which uses the CSS Selector.

This W3schools page has a great, dynamic tool, that is very helpful to understand what is being grabbed.  Here is a screenshot of clicking around on the above tool which will help you get the concept.
Web Scraping with AutoHotkey QuerySelectorAll

Below is a video documenting various method calls.Web Scraping with AutoHotkey

Web Scraping with AutoHotkey Tutorial 8- QuerySelectorAll

And lastly here is the script I used walking through the video.

Web Scraping with AutoHotKey 107- using ClassName

web scraping with AutoHotkeyTwo women in an office smiling at a computer screen, one pointing at it while the other sits beside a green bottle on a desk.When I did my original posts on web scraping with AutoHotkey, I was unaware of ClassName  (getElementsByClassName).

About a year ago I was speaking with Jackie Sztuck from http://jszapp.com (BlackHolyman on the AHK forum) and he introduced me to ClassName.  In the past year I’ve really come to rely on it as my “go-to” for grabbing specific values from a page.   If Name or ID are present I’ll still use them however more and more sites don’t take the time to provide them.

This video demonstrates a few examples where I grab the ClassName.

Web Scraping with AutoHotkey– ClassName

Close-up of a laptop keyboard and touchpad with a USB drive and Intel and Windows stickers, showing a hardware setup for scripting

Automate Unicode character encoding for HTML

Unicode character encoding

Group of coworkers seated around a table in a bright office, talking together with a green reusable water bottle in front of themOur CMS does not support Unicode text thus when we need to search-replace for characters like the ™ and © symbols. This takes a fair amount of time and is easy to miss so I wrote a script in AutoHotKey to automatically handle the Unicode character encoding.

Now I can highlight the word and click a button and  Whamo!  Instant replacement with HTML equivalents!  No more trying to scan text and find illegal characters.

A black AutoHotkey branded water bottle on a desk beside a laptop, with a person reading a newspaper in the backgroundUnicode character encoding

Here is the AutoHotKey code I use.  My code first grabs the highlighted text and copies it to the clipboard, where it manipulates it, then sends it back to the active program I was working in. No more need for Unicode character encoding!