Automating & Scraping with PhantomJS and CasperJS
Why Stack Overflow recommends it
Stack Overflow users link it as a hands-on starter tutorial when asked how to scrape Ajax-heavy pages or get CasperJS scripts working.
Recommended by 3 people across 4 different months, 2013 to 2014.
What it is
A 2013 screencast by Chris Hawkes introducing PhantomJS and CasperJS, headless browser tools for automating web pages and scraping data, including pages that load content via Ajax. He walks through writing CasperJS scripts that navigate sites, fill in forms and pull out data.
What people said
Recommends Chris Hawkes · 2013-10 · Stack Overflow · on “CasperJS looking for 404 error on links site”
If you're trying to learn how to use CasperJS you need to start with a smaller example than that. That script is a mess which goes after a site named yoursite.foo (maybe you put that name in there?) I would take small steps. I have a video which may help explain how to use CasperJS. [this video]
Recommends Chris Hawkes · 2013-11 · Stack Overflow · on “HTML/Javascript login page using CasperJS”
I'm not sure anybody is going to write the code you need. Casperjs has pretty good documentation. You need to request the page, find the id's of the login and sendKeys. I haven't run into any issues of not being able to log into a web based application using casper. IF you're completely at a loss I have a video you can watch which enters data into an Ajax application, which shouldn't be too much different than what you're trying to do. Filling out Ajax forms [this video]
Recommends Chris Hawkes · 2013-11 · Stack Overflow · on “How to make casperjs work with an ajax progress form properly?”
Most likely you're not allowing the browser enough time to make the Ajax calls. You need to make sure any attempt to click or update etc.. that you pause after that to allow execution. I created a tutorial video which shows how I deal with Ajax sites by pausing after execution. [this video]
Recommends mattspain · 2014-02 · Stack Overflow · on “How do you scrape AJAX pages?”
In my opinion the simpliest solution is to use Casperjs [LINK] , a framework based on the WebKit headless browser phantomjs. The whole page is loaded, and it's very easy to scrape any ajax-related data. You can check this basic tutorial to learn Automating & Scraping with PhantomJS and CasperJS [this video] You can also give a look at this example code, on how to scrape google suggests keywords : /*global casper:true*/ var casper = require('casper').create(); var suggestions = []; var word = casper.cli.get(0); if (!word) { casper.echo('please provide a word').exit(1); } casper.start(' [LINK] ', function() { this.sendKeys('input[name=q]', word); }); casper.waitFor(function() { return this.fetchTex
Recommends szab.kel · 2014-08 · Stack Overflow · on “Greasemonkey: Send full page content forward”
Okay, so it is not a solution to the problem above, more like a better way to do it. I tried to do automation with greasemonkey, that is why it tries to send the data to another place. There is a tool for doing this, called casperJS (built onto PhantomJS, it is also required to run it). Basically, you can write JavaScript, to navigate in the site, select the data you want and post it with the casperjs's ajax util. All of this is documented well. casperjs.org [LINK] casper ajax [LINK] Tutorial: Automating & Scraping with PhantomJS and CasperJS by Chris Hawkes [this video]