Supercharged Web Scraping with Asyncio and Python

Supercharged Web Scraping with Asyncio and Python

English | MP4 | AVC 1280×720 | AAC 48KHz 2ch | 3h 13m | 2.24 GB

Learn the fundamentals of asynchronous web scraping & data mining in Python to drastically improve extraction speeds.

Web scraping is simply automatically opening up any website and grabbing the data you find important on that website. It’s fundamental to the internet, search engines, Data Science, automation, machine learning, and much more.

Opening websites and extracting data are only part of what makes web scraping great. It’s the parsing of the data that’s where the value is.

This project will cover:

Basic web scraping with Python

Web scraping with Selenium

Sync vs Async

Asynchronous Web scraping with Asyncio

But why asynchronous code? What is it? How does it benefit us?

Asynchrounous code is a way to execute multiple functions basically at once. It’s not actually at the exact same time but it’s close. (They actually run concurrently). This means that we can do more things in less time and, when it comes to mining or scraping data, this time saving is absolutely significant.

Imagine for a moment you’re recreating google’s search engine. You’d have to scrape trillions (if not more) web pages on a regular interval to help with the search results. Of course you’re not going to be scraping all of the trillions of pages at once but the idea is that scraping event 1,000 pages would take a very long time doing it synchronously (like using Python requests and/or just selenium).

If you’ve done a lot of web scraping before but never used Python’s aysncio, this course will help you better understand the fundamentals and bring your scraping game to another level.

What you’ll learn

  • Basic Web scraping with Python
  • Web scraping with Selenium & Python
  • JavaScript-Heavy Website Scraping
  • Asynchronous Web scraping with Asyncio
Table of Contents

1 Welcome
2 Project Demo
3 Requirements

4 Sync vs Async
5 Blocking & Timeouts
6 Scraping with Selenium
7 Async Web Scraping with chrome driver and arsenic
8 Hide Arsenic logs

Extraction & Formatting
9 Async Data with Python Pandas
10 Prepare to Scrape Multiple URLs
11 Extract Product Data
12 Async Product Data Extraction

Prepare for Re-usability
13 Modules & Submodules
14 Service Specific Submodule
15 Decouple Logging & Scraper

Storing Data
16 Synchronous SQL Storage with Pandas
17 Store Scrapped Data to SQL Tables
18 Inspect Stored Data in Jupyter
19 Scraping URLS from Stored Links Table
20 Scrape Paginated List View
21 Results & Timing

Thank you and next steps
22 Thank you & next steps