Scrapy: Powerful Web Scraping & Crawling with Python

Scrapy: Powerful Web Scraping & Crawling with Python
Scrapy: Powerful Web Scraping & Crawling with Python
English | MP4 | AVC 1280×720 | AAC 48KHz 2ch | 7.5 Hours | 1.35 GB

Python Scrapy Tutorial - Learn how to scrape websites and build a powerful web crawler using Scrapy and Python

In this Scrapy tutorial, you will learn how to install Scrapy. You will also build a basic and advanced spider, and finally learn more about Scrapy architecture. Then you are going to learn about deploying spiders, logging into the websites with Scrapy. We will build a generic web crawler with Scrapy, and we will also integrate Selenium to work with Scrapy to iterate our pages. We will build an advanced spider with option to iterate our pages with Scrapy, and we will close it out using Close function with Scrapy, and then discuss Scrapy arguments. Finally, in this course, you will learn how to save the output to databases, MySQL and MongoDB. There is a dedicated section for diverse web scraping solved exercises... and updating.

One of the main advantages of Scrapy is that it is built on top of Twisted, an asynchronous networking framework. "Asynchronous" means that you do not have to wait for a request to finish before making another one; you can even achieve that with a high level of performance. Being implemented using a non-blocking (aka asynchronous) code for concurrency, Scrapy is really efficient.

It is worth noting that Scrapy tries not only to solve the content extraction (called scraping), but also the navigation to the relevant pages for the extraction (called crawling). To achieve that, a core concept in the framework is the Spider -- in practice, a Python object with a few special features, for which you write the code and the framework is responsible for triggering it.

Scrapy provides many of the functions required for downloading websites and other content on the internet, making the development process quicker and less programming-intensive. This Python Scrapy tutorial will teach you how to use Scrapy to build web crawlers and web spiders.

Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

Scrapy is the most popular tool for web scraping and crawling written in Python. It is simple and powerful, with lots of features and possible extensions.

Python Scrapy Tutorial Topics:

This Scrapy course starts by covering the fundamentals of using Scrapy, and then concentrate on Scrapy advanced features of creating and automating web crawlers. The main topics of this Python Scrapy tutorial are as follows:

  • What Scrapy is, the differences between Scrapy and other Python-based web scraping libraries such as BeautifulSoup, LXML, Requests, and Selenium, and when it is better to use Scrapy.
  • This tutorial starts by how to create a Scrapy project and and then build a basic Spider to scrape data from a website.
  • Exploring XPath commands and how to use it with Scrapy to extract data.
  • Building a more advanced Scrapy spider to iterate multiple pages of a website and scrape data from each page.
  • Scrapy Architecture: the overall layout of a Scrapy project; what each field represents and how you can use them in your spider code.
  • Web Scraping best practices to avoid getting banned by the websites you are scraping.
  • In this Scrapy tutorial, you will also learn how to deploy a Scrapy web crawler to the Scrapy Cloud platform easily. Scrapy Cloud is a platform from Scrapinghub to run, automate, and manage your web crawlers in the cloud, without the need to set up your own servers.
  • This Scrapy tutorial also covers how to use Scrapy for web scraping authenticated (logged in) user sessions, i.e. on websites that require a username and password before displaying data.
  • This course concentrates mainly on how to create an advanced web crawler with Scrapy. We will cover using Scrapy CrawlSpider which is the most commonly used spider for crawling regular websites, as it provides a convenient mechanism for following links by defining a set of rules. We will also use Link Extractor object which defines how links will be extracted from each crawled page; it allows us to grab all the links on a page, no matter how many of them there are.
  • Furthermore there is a complete section in this Scrapy tutorial to show you how to combine Selenium with Scrapy to create web crawlers of dynamic web pages. When you cannot fetch data directly from the source, but you need to load the page, fill in a form, click somewhere, scroll down and so on, namely if you are trying to scrape data from a website that has a lot of AJAX calls and JavaScript execution to render webpages, it is good to use Selenium along with Scrapy.
  • We will also discuss more functions that Scrapy offers after the spider is done with web scraping, and how to edit and use Scrapy parameters.
  • As the main purpose of web scraping is to extract data, you will learn how to write the output to CSV, JSON, and XML files.
  • Finally, you will learn how to store the data extracted by Scrapy into MySQL and MongoDB databases.
Table of Contents

Scrapy vs_ Other Python Web Scraping Frameworks
1 Scrapy vs_ Beautiful Soup vs_ Selenium
2 Course Tips (Must Read)

Scrapy Installation
3 Linux Scrapy Installation
4 Mac Scrapy Installation
5 Windows Scrapy Installation
6 Scrapy Installation Instructions

Building Basic Spider with Scrapy
7 Scrapy Simple Spider - Part 1
8 Scrapy Simple Spider - Part 2
9 Scrapy Simple Spider - Part 3

XPath Syntax
10 Using XPath with Scrapy
11 Tools to Easily Get XPath

Q&A
12 Do you have questions so far_

Building More Advanced Spider with Scrapy
13 Scrapy Advanced Spider - Part 1
14 Scrapy Advanced Spider - Part 2
15 Scrapy Advanced Spider - Part 3
16 Scrapy Advanced Spider - Part 4
17 Scrapy Architecture

Web Scraping Best Practices
18 Avoid Getting Banned!

Do you want to scrape a specific website_
19 Do you want to scrape a specific website_ Let's do it together!

Deploying Scrapy Spider to ScrapingHub
20 Deploying Scrapy Spider to ScrapingHub

Logging into Websites Using Scrapy
21 Logging into Websites Using Scrapy

Scrapy as a Standalone Script
22 Scrapy as a Standalone Script

Building Web Crawler with Scrapy
23 Building Web Crawler with Scrapy

Scrapy with Selenium
24 Why_When We Should Use Selenium
25 Selenium WebDriver + Scrapy Selector to Extract URLs
26 Selenium Loading Next for Data Extraction (usable even with JavaScript pages)
27 Getting Data

Scrapy Spider - Books Store
28 Grabbing URLs
29 Data Extraction

More about Scrapy
30 Scrapy Arguments
31 Scrapy Close Function

Export Output to Files
32 Scrapy Feed Exports to CSV_ JSON_ or XML
33 Export Output to Excel (NEW)
34 Downloading Images (NEW)

Scrapy Project #1_ Scraping Craigslist Eng Jobs in NY
35 Creating Scrapy Craigslist Spider
36 Craigslist Scrapy Spider #1 8O Titles
37 Craigslist Scrapy Spider #2 8O One Page
38 Craigslist Scrapy Spider #3 8O Multiple Pages
39 Craigslist Scrapy Spider #4 8O Job Descriptions
40 Editing Scrapy settings_py
41 Final Scrapy Tutorial_ Craigslist Spider Code

Extracting Data to Databases - MySQL & MongoDB
42 Installing MySQL
43 MySQL Installation and Usage
44 Writing Data to MySQL
45 Installing MongoDB
46 MongoDB Installation and Usage
47 Writing Data to MongoDB

Scrapy Project #2_ Web Scraping Course-Central_com
48 Scraping Course-Central - Part 1_ Subjects
49 Scraping Course-Central - Part 2_ Courses

Scrapy Advanced Topics (NEW)
50 Scrapy User Agent (NEW)
51 Scraping Tables (NEW)
52 Scraping JSON Pages (NEW)
53 Scrapy FormRequest (NEW)

Scrapy Project #3_ Web Scraping eplanning_ie (NEW)
54 ePlanning Scraping Project Overview (NEW)
55 ePlanning_ Extracting Initial URLs (NEW)
56 ePlanning_ Crawling Internal Pages (NEW)
57 ePlanning_ Scrapy Form Requests (NEW)
58 ePlanning_ Scraping Data (NEW)
59 ePlanning_ Checking Data Existence (NEW)
60 ePlanning_ Scraping Data from Table (NEW)

Project #4_ Web Scraping LinkedIn_com (NEW)
61 LinkedIn Scraping Project_ Overview & Requirements (NEW)
62 LinkedIn Logging in (NEW)
63 Finding LinkedIn Profiles_ Part 1 (NEW)
64 Finding LinkedIn Profiles_ Part 2 (NEW)
65 Scraping Data Points from LinkedIn Profiles_ Part 1 (NEW)
66 Scraping Data Points from LinkedIn Profiles_ Part 2 (NEW)
67 Connecting to LinkedIn Profiles (NEW)

Solved Web Scraping Exercises
68 Yield Data Items from 2 Functions
69 How to Order Exported Data
70 Xpath contains() and starts-with() functions
71 Extracting Hidden Phone Number with Selenium
72 Extracting Hidden Phone Number without Selenium

Bonus_ Web Scraping with Beautiful Soup_ Requests & Selenium Course
73 Coupon for Web Scraping with Beautiful Soup_ Requests & Selenium Course