scrapy

> Web-crawling framework.

Create a project:

scrapy startproject {project_name}

Create a spider (in project directory):

scrapy genspider {spider_name} {website_domain}

Edit spider (in project directory):

scrapy edit {spider_name}

Run spider (in project directory):

scrapy crawl {spider_name}

Fetch a webpage as Scrapy sees it and print the source to stdout:

scrapy fetch {url}

Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity):

scrapy view {url}

Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available):

scrapy shell {url}

> This work is licensed under the Creative Commons Attribution 4.0 International License (CC-BY).

-- Response ended

-- Page fetched on Tue May 21 02:01:14 2024