Scraping Web Pages with Scrapy
Why Stack Overflow recommends it
Stack Overflow users mostly cite it as the tutorial they were following when their Scrapy spider broke, since the 2012 code relies on Python 2 and deprecated APIs like BaseSpider and HtmlXPathSelector.
Recommended by 1 people across 1 different months, 2014 to 2017.
What it is
A short screencast by Michael Herman showing how to build a web crawler with Scrapy's BaseSpider. It scrapes Craigslist San Francisco nonprofit job listings with XPath selectors and writes the results to a CSV file. A written blog post accompanies it.
What people said
Recommends Skeer · 2017-08 · Stack Overflow · on “Python web-crawl script, following tutorial and having issues”
Following a tutorial on youtube: Scraping Web Pages with Scrapy [this video] It's old, for Python 2.x and I'm learning version 3.x. So far I've ran into a few issues that I've been able to figure out via Google. However currently I'm getting an error: File "/usr/lib64/python3.5/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/skeer/PycharmProjects/scrape_craigslists/scrape_cl/scrape_cl/spiders/scrape.py", line 11, in parse xpath = scrapy.selector(response) TypeError: 'module' object is not callable Googling earlier I found references to other having this was due to a non-capitalized character, as if the 's' in
Cites gdogg371 · 2014-07 · Stack Overflow · on “Scrapy Spider not scraping correctly”
I am using Python.org 2.7 64 bit shell on Windows Vista. I have Scrapy installed and it seems to be stable and working. However, I have copied the following simple piece of code: from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector class MySpider(BaseSpider): name = "craig" allowed_domains = ["craigslist.org"] start_urls = [" [LINK] "] def parse(self, response): hxs = HtmlXPathSelector(response) titles = hxs.select("//p") for titles in titles: title = titles.select("a/text()").xpath() link = titles.select("a/@href").xpath() print title, link Contained in this Youtube video: [this video] When I run this code I get the warning: hxs = HtmlXPathSelector(response) C:\Pyth
Cites alphamonkey · 2015-06 · Stack Overflow · on “Scrapy not getting titles from website”
I'm just starting to learn webcrawling and I am having an issue about getting the correct titles from a website. I am currently watching this guide: [this video] and it tells you to make this class: class MySpider(BaseSpider): name = "craig" allowed_domains = ["craigslist.org"] start_urls = [" [LINK] "] def parse(self, response): hxs = HtmlXPathSelector(response) titles = hxs.select("//p") for titles in titles: title = titles.xpath("a/text()").extract() link= titles.xpath("a/@href").extract() print title, link The issue is that the link prints only a bunch of /for/###numbers###.html and the title doesn't print anything at all. I'm not sure why this is occuring. I've read through previous threads
Cites Kirti Chichmalkar · 2016-08 · Stack Overflow · on “Got error while using scrapy”
I'm trying to extract data using scrapy, I followed the video [this video] but i'm getting error test.py from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from craigslist_sample.items import CraigslistSampleItem from scrapy.http import Request class MySpider(BaseSpider): name = "craig" allowed_domain = ["craigslist.org"] start_urls = [" [LINK] "] def parse(self,response): hxs = HtmlXPathSelector(response) titles = hxs.select("//p") items = [] for titles in titles: item = CraigslistSampleItem() item["title"] = titles.select("a/text()").extract() item["link"] = titles.select("a/@href").extract() item.append(item) return items items.py from scrapy.item import Item, F