Scrape Websites with Python + Beautiful Soup 4 + Requests -- Coding with Python
Why Stack Overflow recommends it
Stack Overflow users link it as the beginner scraping tutorial they followed, usually when asking why their Beautiful Soup code from the video throws an error or returns nothing.
Recommended by 2 people across 2 different months, 2014 to 2018.
What it is
A 2014 hands-on tutorial from CodingEntrepreneurs that walks through scraping a website in Python using the Beautiful Soup 4 and Requests libraries. It assumes basic Python knowledge and builds up from fetching a page to pulling links and text out of the HTML.
What people said
Recommends benawad · 2014-11 · Stack Overflow · on “Python: Beautiful Soup list is empy”
There is and I am following this video tutorial: [this video] it works for him to. Could it be that I am using Python 2.7?
Recommends Brendan Cott · 2015-09 · Stack Overflow · on “Python and BeautifulSoup Opening pages”
I am wondering how would I open another page in my list with BeautifulSoup? I have followed this tutorial [this video] , but it does not tell us how to open another page on the list. Also how would I open a "a href" that is nested inside of a class? Here is my code: # coding: utf-8 import requests from bs4 import BeautifulSoup r = requests.get("") soup = BeautifulSoup(r.content) soup.find_all("a") for link in soup.find_all("a"): print link.get("href") for link in soup.find_all("a"): print link.text for link in soup.find_all("a"): print link.text, link.get("href") g_data = soup.find_all("div", {"class":"listing__left-column"}) for item in g_data: print item.contents for item in g_data: print item.
Cites HassenPy · 2015-03 · Stack Overflow · on “Scrape alexa and show results in table in django”
If you want an API there is one for Alexa here [LINK] If you want too scrape, i'd suggest BeautifulSoup (scrapy is to heavy for this since the only thing you'll be doing is reading from one URL.) Doing this is simple: Make a python module that deals with pulling data from the Alexa link using BeautifulSoup, in the module make it so that it runs the task every 5 minutes or any time span your application will be efficient with, then save it to your database. To display the data you would retrieve it from the database then pass it to the template in a template variable, and the html should look something like this (don't use tables): <table> {% for site in top_20_sites %} <tr> <td>{{site.rank}}
Cites user6326823 · 2016-05 · Stack Overflow · on “BeautifulSoup: TypeError: __init__() takes at least 5 arguments”
I'm working on this example of how to webscrape that I found on youtube and just going step by step through it. [this video] Here is my code and the error I keep getting. import requests from bs4 import BeautifulSoup url = (' [LINK] ') r = requests.get(url) soup = BeautifulSoup(r.content) links = soup.find_all("a") for link in links: print "<a href='%s'>%s</a>" %(link.get("href"), link.text) g_data = soup.find_all("div", {"class": "info"}) print g_data ERROR Traceback (most recent call last): line 8, in <module> soup = BeautifulSoup(r.content) TypeError: __init__() takes at least 5 arguments (2 given) I have tried to use the soup module on other files and it still won't work. Any ideas? Thanks in
Cites rohit nair · 2016-05 · Stack Overflow · on “Python: Print text elements does not work”
@Jivan - Please can you explain how this last two lines of your for loop works? I is a tremendous work you have done. Also I dont understand the reason that in so many tutorials like `print item.contents[0].text` works but not when I do. How is it working for them?
Cites Keretto · 2016-12 · Stack Overflow · on “BeautifulSoup + Requests >> AttributeError: 'NavigableString' object has no attribute 'find_all'”
Looked everywhere and I still don't know why it's giving an error. This was a YouTube tutorial and my code is identical so I don't know why it throws this 'NavigableString' error on the mentioned line below (I tried it in both Python 2.7 and 3.5). import requests from bs4 import BeautifulSoup url= " [LINK] " r = requests.get(url) soup = BeautifulSoup(r.content) data = soup.find_all("div", {"class": "example"}) for item in data: print item.contents[0].find_all("a", {"class": "ex"}) # Error line EDIT: other usefull information tutorial: [this video] example code in tutorial (time: 29:16): [this video] url in tutorial: [LINK]
Cites K. Willem · 2018-02 · Stack Overflow · on “How to web-scrape the creator-bio on Kickstarter projects with Python & BS4”
I am a finance master student and currently writing my thesis. I am analyzing factors that have an influence on the probability on successful funding on Kickstarter. I have obtained a dataset from webrobots.io, however I am missing some variables (did not get links for their full, public databases). In fact, I need information on the founder (# of projects founded before, # of projects backed, # of fb friends). This information only pops up in a window when clicking on the founders' names. Thus, I have the URL to each of my 5000 projects in the sample and want to scrape the creator-bio for each of these links. I have started to watch tutorials (f.e." [this video] ") and also tried to come up with
Cites Nick_Doe · 2018-02 · Stack Overflow · on “Webscraping using Python BeautifulSoup Contents.Text”
I am using Python 3.6.2, Requests and BeautifulSoup4. But it won't allow me to use a certain property/attribute. Here is the code. Following this youtube video: [this video] 21:11 import requests from bs4 import BeautifulSoup url=" [LINK] " page=requests.get(url) soup=BeautifulSoup(page.content,"html.parser") g_data=soup.find_all("h3",{"itemprop":"name"}) #print(g_data) for x in g_data: print(x.contents[1].text)<---offending line. Says that there is no such attribute/property and can't choose when I hit Tab after contents[1]. I have tried this in the Python Idle as well as in the command prompt. What am I missing? Thanks in advance.