Home Automation Archive by category "Web Scraping"

Category: Web Scraping

Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping

How to use AutoHotkey to Automate IE from within Edge | Windows 10 & Windows 11

Automate IE from within EdgeA while back someone wrote me saying how IE is dead and wondering how we can automate other browsers.  While IE is definitely on it’s death-bed, I do still automate IE for sites that will let IE load.  Someone mentioned that Windows 11 completely removes Internet Explorer thus, if you’re running Windows 11, automating IE is not an option.    I looked into this and had some very interesting discoveries:

  1. The IWB2 Learner tool works within Edge when in “IE Mode”
  2. In Windows 11, you can add IE back and still use it in IE mode.  (I’ll document how I did this in a later video)

When I realized the above, I played with Edge (in IE Mode) in Windows 10 & Windows 11 and was able to connect to the DOM!  Granted my approach sucked but I asked Tank (Charlie Simmons) to take a look at it and he borrowed on the concept and re-wrote what i did into something that is decentily reliable.  You can get the download here

Here’s a video showing how I use AutoHotkey to Automate IE from within Edge!

Discover the unbeatable benefits of using xmlHTTPRequest vs WinHTTPRequest – Don’t miss out on these game-changing advantages!

xmlHTTPRequest vs. WinHTTPRequestIn this video I walk through both the xmlHTTPRequest and the WinHTTPRequest and I compare the xmlHTTPRequest vs. WinHTTPRequest.  As I mention in the video, Jackie Sztuk and I have a great  AutoHotkey webinar on Intro to API calls.  I also have several examples on my API page.

I also mention using Fiddler to monitor the browser traffic and grab your cookies / headers.

xmlHTTPRequest vs WinHTTPRequest tutorial

The MSXML2.XMLHTTP and WinHttpRequest COM objects are both used to send HTTP requests from an AutoHotkey script. However, there are some differences between the two:

  • Compatibility: The MSXML2.XMLHTTP object is available on all versions of Windows, while the WinHttpRequest object is only available on Windows XP and later.
  • Performance: In general, the WinHttpRequest object is faster and more efficient than the MSXML2.XMLHTTP object, because it uses the Windows HTTP Services (WinHTTP) library to send requests, which is optimized for HTTP communications.
  • Features: The WinHttpRequest object supports a wider range of features than the MSXML2.XMLHTTP object, including the ability to send HTTPS requests, specify proxy settings, and authenticate with a server using different authentication methods.

In general, if you are running AutoHotkey on a newer version of Windows and you need the additional features and performance of the WinHttpRequest object, you should use that object instead of the MSXML2.XMLHTTP object. However, if you need to support older versions of Windows or you do not need the advanced features of the WinHttpRequest object, you can use the MSXML2.XMLHTTP object instead.

Here’s the syntax I used for the WinHTTPRequest example

WinHTTPRequest syntax

And here’s the corresponding two XML API calls I demonstrated in the video

xmlHTTPRequest example using Msxml2.XMLHTTP COM object

Here’s the example connecting to the IE page and sending the API request with the xmlHTTPRequest()

08: Automating Chrome with AutoHotkey: How to handle EventListeners & custom edit fields

Automating Chrome with AutoHotkeyAutomating Chrome with AutoHotkey: handle EventListeners & custom edit fields


donate to GeekDudeIf you’re loving this, please consider donating to GeekDude!

Automating Chrome with AutoHotkey Script to Handle EventListeners & custom edit fields

Notes on handle EventListeners & custom edit fields

00:09     When web scraping, often there will be buttons I can’t press, text that doesn’t get “acknowledged” that it has been entered, etc.
Continue reading