Always Check for the Hidden API when Web Scraping
Why Stack Overflow recommends it
Stack Overflow users link it when asking about scraping dynamic or JavaScript-heavy pages, to show that calling the site's hidden API is simpler and more reliable than Selenium.
Recommended by 4 people across 4 different months, 2022 to 2023.
What it is
A short Python web scraping tutorial by John Watson Rooney showing how to use the browser's network tab to find the JSON API a site loads its data from, then call it directly with requests instead of parsing HTML or driving a browser.
What people said
Recommends econbernardo · 2022-06 · Stack Overflow · on “Download Covid patient metadata from GISAID website in R using RSelenium”
@TomWenseleers I don't know if we are talking about the same thing, but it may be worthwhile to give this video [this video] a look to understand what I mean by "hidden API".
Recommends Agricultural Research · 2022-12 · Stack Overflow · on “How to Scrape Data from a Map Using Python (Fail to Click a Layer)?”
The problem was using the wrong library. Instead selenium one should use request. The YouTube video helped me is [Always Check for the Hidden API when Web Scraping]: [this video] My final code looks: import requests from urllib3.exceptions import InsecureRequestWarning from urllib3 import disable_warnings disable_warnings(InsecureRequestWarning) url_regions = " [LINK] " payload_regions = "sLang=" headers_regions = { "Accept": "*/*", "Accept-Language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,de;q=0.6", "Connection": "keep-alive", "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8", "Origin": " [LINK] ", "Referer": " [LINK] ", "Sec-Fetch-Dest": "empty", "Sec-Fetch-Mode": "cors", "Sec-Fe
Recommends domigmr · 2023-03 · Stack Overflow · on “Webscraping sale prices from a grocery store- Am I on the right track or is there a simpler way?”
I would suggest that you avoid using Selenium in this case. You could, as Antoine suggested, use the inspect element feature to examine if there is an API exposed. I think what happens when you scroll to the bottom is that the web page fires off a request to the back end for more data. As Antoine suggested, you can mimic this request. Use Inspect Element on the webpage, navigate to Network , and then to response . Scroll to the bottom of the page and let it load, and you'll see some new requests. I'd suggest John Watson Rooney's video from here on out.https: [this video]
Recommends Frosting · 2023-05 · Stack Overflow · on “selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element”
After hours of research I realized that site fetches only 10 values on loading and fetches more as you scroll, The 'Load more' button was not clickable so I had to change my approach. I decided to fetch the data directly from the API endpoint and change the limit i.e. (10) to any value I wanted. Here is the new code import http.client import json conn = http.client.HTTPSConnection("api.betika.com") payload = "" headers = {'cookie': "__cf_bm=IUdIW5DuTUklj090LYfwfKTblBGzb6VtPvukBIEx9Mw-1684241456-0-AcfDe74HLZ85Cs2OyRKH9OSNlUm9heyhUg3A41s%2Fl%2Fu%2FHWt0w7rZS1a4wVVnL73cP8kwcj8vxOHr3CMQTwamyZk%3D" } conn.request("GET", "/v1/uo/matches?page=1&limit=200&tab=&sub_type_id=1%2C186%2C340&sport_id=14&ta
Cites Jonathan · 2022-07 · Stack Overflow · on “scraping data from ITU download links with rvest”
If you open the developer tools of your browser (`Ctrl + Shift + I`), you find a tab called "network" which lists all kinds of requests the site does to any kind of server. Most of the times data is returned in JSON format, so you can look out for that. Then click around for the correct Response, and then use this url (right-click onto the response). Sometimes you can find the request already in your Browser console. There are actually many tutorials out there that can teach you this, just search for "webscraping hidden api" e.g. [this video] )
Cites M.O. · 2022-07 · Stack Overflow · on “Error H14 heroku with selenium and fastapi using python”
You've partly answered the question yourself: the reason you are getting the heroku error is that you are not defining a web process, which you need to use to expose a web API [LINK] . You've also given the reason you're getting the other error when you do use the web process: the first line ( dict = Scraping().get_books() ) takes 2-5 minutes to run, meaning we are "stuck" waiting on the first line, so the actual FastAPI app doesn't start until after those 2-5 minutes, and heroku has a 1 minute timeout for starting the API. Also, sidenote: dict is a reserved keyword in Python, so you really shouldn't use it for variable names. Try do find a more descriptive name, e.g. book_dict . So what can