Scrapy for Beginners - A Complete How To Example Web Scraping Project
Why Stack Overflow recommends it
Stack Overflow users cite it as the step-by-step Scrapy tutorial they were following when they hit errors like 'Spider not found' or problems with scrapy fetch.
Recommended by 1 people across 1 different months, 2021 to 2023.
What it is
A 23-minute beginner tutorial by Python developer John Watson Rooney that walks through a complete Scrapy web scraping project, from creating a project and spider to running it and extracting product data from an online whiskey shop.
What people said
Recommends lnky · 2021-12 · Stack Overflow · on “KeyError: 'Spider not found:”
I am following the youtube video [this video] and have reached 13:45, when the creator runs his spider. I have followed the tutorial precisely, yet my code refuses to run. This is my actual code. I imported scrapy too as well. Can anyone help me figure out why scrap refuses to acknowledge my spider? The file is in the correct 'spider' file. I am so confused rn. import scrapy from scrapy.spiders import Spider class WhiskeySpider(scrapy.Spider): spider_name = 'whiskey' start_urls = [' [LINK] '] def parse(self, response): for products in response.css('div.product-item-info'): yield { 'name' : products.css('a.product-item-link::text').get(), 'price' : products.css('span.price::text').get().replace('£
Cites Bastion · 2022-10 · Stack Overflow · on “Scrapy: KeyError: 'Spider not found:”
I'm trying to write my first Scrapy spider and have been following this tutorial [this video] I am running the command exactly as shown: scrapy crawl charnley, have a name for my spider and my spider is not called "scrapy" or anything similar so I am really confused what is wrong. This is my error messages During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\shark\VisualStudioCodeProjects\env\Scripts\scrapy.exe\__main__.py", line 7, in <module> F
Cites How FunnyTv · 2023-01 · Stack Overflow · on “Problems using scrapy fetch”
EDIT: Hi, I learn some scrapy from a YouTube tutorial. I used this commands (like in this tutorial [this video] ): > scrapy startproject whiskyscraper > cd whiskyscraper > tree . ├── scrapy.cfg └── whiskyscraper ├── __init__.py ├── items.py ├── middlewares.py ├── pipelines.py ├── __pycache__ │ ├── __init__.cpython-310.pyc │ └── settings.cpython-310.pyc ├── settings.py └── spiders ├── __init__.py └── __pycache__ └── __init__.cpython-310.pyc > scrapy shell > fetch(' [LINK] ') ...and after fetch() i got this problem: >>> fetch(' [LINK] ') 2023-01-22 11:07:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET [LINK] > (referer: None) 2023-01-22 11:07:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET [LIN