We use AI to build automation that gives you your time back. 🤖 Tools, training and custom work.
New here❓
Start Here
Discover Our
Courses
Schedule
a meeting
Downloads
Browse by Task
Testimonials
Using AI❓
AI Tools

Automation Archive by category "Web Scraping" (Page 9)

Category: Web Scraping

Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping

Automating the login to Reddit with AutoHotkey

This is the 7th video in my series of automating logging into websites with AutoHotkey. In this tutorial I show how to login to Reddit.

In this video we had a small hiccup where I used the classname and had assumed it was unique. Turns out the there was another button on the page with the exact same classnames! It was not a big issue, but definitely a good example of how you need to troubleshoot/test your code!

Below is the video showing how I automated the login to Reddit with AutoHotkey

AutoHotkey Script to Login to Reddit

#SingleInstance, Force
Browser_Forward::Reload ;RControl::
Browser_Back::
pwb := WBGet()
If (pwb.locationURL!="https://www.reddit.com/login"){
 pwb.Navigate("https://www.reddit.com/login") ;Navigate to URL
while pwb.busy or pwb.ReadyState != 4 ;Wait for page to load
	Sleep, 100
}
pwb.document.all.user_login.Value :="joetazz" ;Unique ID -no dashes
pwb.document.all.passwd_login.Value :="The_Stupidist_Password!" ;Unique ID -no dashes
pwb.document.all.rem_login.checked :=1 ;Set Tagname and Array value
pwb.document.getElementsByClassName("c-btn c-btn-primary c-pull-right")[1].click()
;************Pointer to Open IE Window******************
WBGet(WinTitle="ahk_class IEFrame", Svr#=1) {               ;// based on ComObjQuery docs
   static msg := DllCall("RegisterWindowMessage", "str", "WM_HTML_GETOBJECT")
        , IID := "{0002DF05-0000-0000-C000-000000000046}"   ;// IID_IWebBrowserApp
;//     , IID := "{332C4427-26CB-11D0-B483-00C04FD90119}"   ;// IID_IHTMLWindow2
   SendMessage msg, 0, 0, Internet Explorer_Server%Svr#%, %WinTitle%
   if (ErrorLevel != "FAIL") {
      lResult:=ErrorLevel, VarSetCapacity(GUID,16,0)
      if DllCall("ole32\CLSIDFromString", "wstr","{332C4425-26CB-11D0-B483-00C04FD90119}", "ptr",&GUID) >= 0 {
         DllCall("oleacc\ObjectFromLresult", "ptr",lResult, "ptr",&GUID, "ptr",0, "ptr*",pdoc)
         return ComObj(9,ComObjQuery(pdoc,IID,IID),1), ObjRelease(pdoc)
      }
   }
}

Automate the Login to Twitter with AutoHotkey

In the sixth post of this series I demonstrate how to Login to Twitter with AutoHotkey.

In this web scraping example we saw how the Google login looks like it is loading a separate page but, in reality, is not. Web scraping is always fun (and can be very frustrating) because HTML developers can think of so many different ways to accomplish their objectives. Check this page for the other scripts I used in the video.

Video demonstrating Login to Twitter

AutoHotkey Script I wrote to Login to Twitter with AutoHotkey

#SingleInstance, Force
Browser_Forward::Reload ;RControl::
Browser_Back::
;************************************************************
pwb := WBGet()
If (pwb.locationURL !="https://twitter.com/"){
 pwb.Navigate("https://twitter.com/") ;Navigate to URL
while pwb.busy or pwb.ReadyState != 4 ;Wait for page to load
	Sleep, 100
}
pwb.document.GetElementsByName("session[username_or_email]")[0].Value :="[email protected]"
pwb.document.GetElementsByName("session[password]")[0].Value :="The_Stupidist_Password!"
pwb.document.GetElementsByName("remember_me")[0].checked :=1 ;Set Tagname and Array value
pwb.document.getElementsByClassName("submit btn primary-btn js-submit")[0].click()
;************Pointer to Open IE Window******************
WBGet(WinTitle="ahk_class IEFrame", Svr#=1) {               ;// based on ComObjQuery docs
   static msg := DllCall("RegisterWindowMessage", "str", "WM_HTML_GETOBJECT")
        , IID := "{0002DF05-0000-0000-C000-000000000046}"   ;// IID_IWebBrowserApp
;//     , IID := "{332C4427-26CB-11D0-B483-00C04FD90119}"   ;// IID_IHTMLWindow2
   SendMessage msg, 0, 0, Internet Explorer_Server%Svr#%, %WinTitle%
   if (ErrorLevel != "FAIL") {
      lResult:=ErrorLevel, VarSetCapacity(GUID,16,0)
      if DllCall("ole32\CLSIDFromString", "wstr","{332C4425-26CB-11D0-B483-00C04FD90119}", "ptr",&GUID) >= 0 {
         DllCall("oleacc\ObjectFromLresult", "ptr",lResult, "ptr",&GUID, "ptr",0, "ptr*",pdoc)
         return ComObj(9,ComObjQuery(pdoc,IID,IID),1), ObjRelease(pdoc)
      }
   }
}

Automate the Login to Pinterest with AutoHotkey

In this 5th video of my automating website logins with AutoHotkey I demonstrate how to login to Pinterest.

In this video I move quickly covering the topics (as much of process is covered in-depth in earlier videos.

There was one difference in this video. The submit button on Pinterest does not have a name or ID so I used classname.

Check this page for the other scripts I used in the video.

Login to Pinterest with AutoHotkey

The AutoHotkey script to login to Pinterest is here:

#SingleInstance, Force
Browser_Forward::Reload ;RControl::
Browser_Back::
;************************************************************
pwb := WBGet()
;~  MsgBox % pwb.locationURL
If (pwb.locationURL !="https://www.pinterest.com/login/?referrer=home_page"){
pwb.Navigate("https://www.pinterest.com/login/?referrer=home_page") ;Navigate to URL
while pwb.busy or pwb.ReadyState != 4 ;Wait for page to load
	Sleep, 100
}
pwb.document.GetElementsByName("id")[0].Value :="[email protected]" ;Object Name- Set array value
Sleep,50
pwb.document.GetElementsByName("password")[0].Value :="The_Stupidist_Password!" ;Object Name- Set array value
Sleep,50
pwb.document.getElementsByClassName("red SignupButton active")[0].click()
;************Pointer to Open IE Window******************
WBGet(WinTitle="ahk_class IEFrame", Svr#=1) {               ;// based on ComObjQuery docs
   static msg := DllCall("RegisterWindowMessage", "str", "WM_HTML_GETOBJECT")
        , IID := "{0002DF05-0000-0000-C000-000000000046}"   ;// IID_IWebBrowserApp
;//     , IID := "{332C4427-26CB-11D0-B483-00C04FD90119}"   ;// IID_IHTMLWindow2
   SendMessage msg, 0, 0, Internet Explorer_Server%Svr#%, %WinTitle%
   if (ErrorLevel != "FAIL") {
      lResult:=ErrorLevel, VarSetCapacity(GUID,16,0)
      if DllCall("ole32\CLSIDFromString", "wstr","{332C4425-26CB-11D0-B483-00C04FD90119}", "ptr",&GUID) >= 0 {
         DllCall("oleacc\ObjectFromLresult", "ptr",lResult, "ptr",&GUID, "ptr",0, "ptr*",pdoc)
         return ComObj(9,ComObjQuery(pdoc,IID,IID),1), ObjRelease(pdoc)
      }
   }
}

Category: Web Scraping

Account

It looks like you are new to the-Automator!

In the top right corner of the page, you can access the welcome guide.