A Guide to Web Scraping with Node.js
Why Stack Overflow recommends it
Stack Overflow users link it when someone needs to scrape JavaScript-rendered pages such as Instagram, since it shows a working Puppeteer headless-browser example.
Recommended by 2 people across 2 different months, 2019 to 2021.
What it is
A short Fireship tutorial on web scraping with Node.js using two approaches. The first builds a link-preview generator with Cheerio that reads page metatags, and the second builds an interactive Instagram scraper with Puppeteer and headless Chrome.
What people said
Recommends Laczkó Örs · 2021-05 · Stack Overflow · on “How to fix "please wait a few minutes before you try again" error instagram?”
I'm using puppeteer for the first time running it on locally hosted firebase cloud functions . I've been trying with different accounts, and I waited hours so that the error may resolves, but no luck. The error I'm getting: [LINK] I can't interact with the site, and even if I switch routs this is the only thing popping up. What I did/tried: I followed this tutorial and coded the exact same app: [this video] Searched for hours on google if there is anything like my problem, still no solution that worked for me. Edit: The code I'm using is basically from fireship.io [LINK] : const puppeteer = require('puppeteer'); const scrapeImages = async (username) => { const browser = await puppeteer.launch( {
Recommends Daggie Blanqx - Douglas Mwangi · 2021-10 · Stack Overflow · on “Is there a node.js package for web scraping that will capture content outside of page source-code?”
Yes, basically you are looking for a way to access content that shows up when a page loads using JavaScript. The solution you want is a headless browser - puppeteer does this very well. Checkout: [LINK] Or even a quick video: [this video]
Cites AcruxCode · 2019-08 · Stack Overflow · on “How to get information from instagram with cheerio and express”
Unfortunately, Instagram pages are rendered on the client side, so it is not possible to get information like the number of followers through this way. A solution to this problem would be to use puppeteer [this video] . With puppeteer you can start a headless version of chrome, which also interprets javascript, so that Instagram pages get rendered completely. Example code: const puppeteer = require('puppeteer') class InstagramClient { async start() { this.browser = await puppeteer.launch({ headless: true //When set to true, a new browser window will ge opened }) } async stop() { await this.browser.close() } async getFollowers(username) { if (!this.browser) throw new Error('Browser not started') c