Python 2.7 Tutorial Pt 13 Website Scraping
Why Stack Overflow recommends it
Stack Overflow users link it as a starting point for Python web scraping, and many askers cite it as the tutorial whose code they adapted and then got stuck on.
Recommended by 3 people across 3 different months, 2011 to 2013.
What it is
A 10-minute Python 2.7 tutorial by Derek Banas on scraping websites, part of his Python 2.7 series. It introduces the urllib and Beautiful Soup modules and walks through pulling titles and links from an RSS feed.
What people said
Recommends MSey · 2011-05 · Stack Overflow · on “Mining Wikipedia for mapping relations for text mining”
Jvc, there are existing python modules that can do everything you mentioned above. For pulling information from webpages, I like to use Selenium, [LINK] . Basically, you can localize and retrieve information on any webpage using a number of identifiers (id, Xpath, etc). However, like winwaed said, it can be inflexible if you are simply "pattern matching", especially since some websites use dynamic code- meaning the identifiers can change with each subsequent reload of the page. But, this problem can be solved by adding regular expressions, i.e. (.*), to your code. Check out this youtube video, [this video] . Even though he is using BeautifulSoup to scrape the website- you can see how he uses regu
Recommends Ardy Dedase · 2013-06 · Stack Overflow · on “How can I get data from other websites?”
Python has great web scraping features: urllib, BeautifulSoup, XPath, etc. This video will get you started quickly with python web scraping: [this video] - He's using urllib and BeautifulSoup to scrape huffingtonposts [LINK] ' feed in his sample script. If you want a scraping system (scraper with web front-end and admin to publish your scraped content), this might be a good option for you - [LINK] - I would highly suggest this if you're already familiar with Django.
Recommends kylerthecreator · 2013-10 · Stack Overflow · on “Python Blog RSS Feed Scraping BeautifulSoup Output to .txt Files”
Apologies in advance for the long block of code following. I'm new to BeautifulSoup, but found there were some useful tutorials using it to scrape RSS feeds for blogs. Full disclosure: this is code adapted from this video tutorial which has been immensely helpful in getting this off the ground: [this video] . Here's my problem: the video does a great job of showing how to print the relevant content to the console. I need to write out each article's text to a separate .txt file and save it to some directory (right now I'm just trying to save to my Desktop). I know the problem lies i the scope of the two for-loops near the end of the code (I've tried to comment this for people to see quickly--it's
Cites user1205632 · 2012-02 · Stack Overflow · on “RSS feed scraping with Python”
I'm a newbie to Python and programming in general so please excuse me if the question is very dumb. I've been following this [this video] tutorial on RSS scraping step by step but I am getting a "list index out of range" error from Python when trying to gather the corresponding links to the titles of the articles being gathered. Here is my code: from urllib import urlopen from BeautifulSoup import BeautifulSoup import re source = urlopen(' [LINK] ').read() title = re.compile('<title>(.*)</title>') link = re.compile('<link>(.*)</link>') find_title = re.findall(title, source) find_link = re.findall(link, source) literate = [] literate[:] = range(1, 16) for i in literate: print find_title[i] print f
Cites user945967 · 2012-04 · Stack Overflow · on “Rss Feed scraping with BeautifulSoup”
Your Code strongly reminds me of: [this video] Why do you actually use BeautifulSoup for XML parsing? Its built for HTML-Sites and python itself has very good XML-Parsers. Example: [LINK]
Cites Robert Birch · 2013-10 · Stack Overflow · on “Python Re Beautiful Soup Unexpected String”
I am using the following code from this tutorial [this video] . import urllib2 from urllib import urlopen from BeautifulSoup import BeautifulSoup import re webpage = urlopen(' [LINK] ').read # open webpage and read it web page is variable patFinderTitle = re.compile('<title>(.*)</title>') # get characters between titles patFinderLink = re.compile('<link rel.*href="(.*)" />') findPatTitle = re.findall(patFinderTitle, webpage) # variable is declared, uses re module to find all find two variables using the following args findPatLink = re.findall(patFinderLink, webpage) listIterator - [] listIterator[:] = range(2,16) soup2 = BeautifulSoup(webpage) print soup2.findAll('title') However, I get this erro