Sometimes it is not convenient to grab the specific element you want while you’re scraping a web page. In this tutorial I walk through how to use parentNode, previousSibling,nextSibling,firstChild and lastChild. Continue reading
Category: Web Scraping
Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Mozilla Firefox.
Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.
https://en.wikipedia.org/wiki/Web_scraping
Detecting Carrier Type, Company and Location from Phone Number
Detecting Carrier Type, Company and Location from Phone Number
I was talking to a company today that does “voicemail drops” (so they don’t actually call you, they just drop a message into your voicemail.
We were discussing how nice it would be if you could know who the carrier was and/or where the respondent was located.
I mentioned that I’d learned some years ago that the prefix / Exchange was tied to a given carrier and that we should be able to reverse-engineer the information pretty easily.
In this video I walk through how to automate the process and, quickly ,extract the information from the web page.
Here is the article I mentioned during the video and below is the video and script I wrote (I realized after recording the video I could have done w/o the loop. Take a look at the script below…
Video Walk Through: Detecting Carrier Type, Company and Location from Phone Number
Detecting Carrier Type, Company and Location from Phone Number
Use the following for a webpage post:
Use Fiddler to monitor network traffic
Being able to easily monitor your network traffic is very helpful when working with APIs as well as Web Scraping.
In these two videos I demonstrate how to use Fiddler to monitor network traffic. Fiddler is a great, free, tool that can help you see what traffic is going on behind the scenes from browsers, programs, apps, and AutoHotkey. FYI- If you don’t care to use Fiddler / an external tool, a lot of this can be done with the developer tools inside a browser however they are somewhat limited in functionality. I’ve tried other network monitoring tools but they are ridiculously complex.
Use Fiddler to monitor network traffic
Intro to using Fiddler to peek at traffic
Use AutoHotkey to easily clear network traffic in Fiddler
;*********************Clear traffic in Fiddler Need to uncomment********************************** ;~ #IfWinActive ahk_exe Fiddler.exe ;Uncomment this if you want to only do it when Fiddler is Active ^+c:: ;Hotkey to clear traffic ControlFocus,WindowsForms10.SysListView32.app.0.2bf8098_r6_ad12 ;Control where Traffic is (must be active) Send ^x ;Clear Traffic return ;~ #IfWinActive ;Uncomment this if you uncommend the above #IfWinactive
Using Filters in Fiddler to clean-up what Fiddler is tracking
