Steps Involved in Web Crawling

To perform this tutorial step-by-step with me, you’ll need Python3 already configured on your local development machine. You can set up everything you need before-hand and then come back to continue ahead.


Creating a Basic Web Scraper

Web Scraping is a two-step process:

You send HTTP request and get source code web pages.
You take that source code and extract information from it.
Both these steps can be implemented in numerous ways in various languages. But we will be using request and bs4 packages of python to perform them.

pip install beautifulsoup4
If you want to install BeautifulSoup4 without using pip or you face any issues during installation you can always refer to the official documentation.

Create a new folder