Scraping Infinite Scrolling Pages
Why Stack Overflow recommends it
Stack Overflow users link it when they need to scrape sites that load more content on scroll or via a "Load More" button, as a starting point for inspecting the underlying AJAX requests.
Recommended by 2 people across 2 different months, 2018 to 2020.
What it is
A five-minute episode from Zyte's (formerly Scrapinghub) "Learn Scrapy" series on scraping pages that use AJAX-based infinite scrolling. It shows how to find the background requests the page makes and replicate them in a Scrapy spider.
What people said
Recommends Keenan Burke-Pitts · 2018-02 · Stack Overflow · on “Scraping Website With Infinite Scroll Using Scrapy”
I am trying to scrape used items in my area from [LINK] for a personal project. I found this video helpful [this video] . However, there are some subtle differences that the video doesn't help with. The info I need is loaded asyncrhonously via json. The website loads 15 items per each scroll (except for the initial load which contains 30 items). The json object looks like this: [LINK] and the next 15 items to load look like this: [LINK] When I load the first response data = json.loads(response.text) it returns a list of the 30 items. The first item looks like this: {'attributes': None, 'category_id': 5, 'created_at': '2018-02-12T15:40:56+00:00', 'currency': 'USD', 'description': None, 'featured':
Recommends Hans · 2020-04 · Stack Overflow · on “Scrapy crawler response url vs request url”
Thank you very much guys! While doing a little reading on your suggestions I found this: [this video] which ultimately led me to this [LINK] , what helped me to find a solution to my problem.
Cites Dan · 2019-02 · Stack Overflow · on “Scrapy | Follow links that are loaded by a dynamic "Load More"-Button”
I know this question was asked a few times with similar approaches, but I can't wrap my head around it. That is why I want to open a specific case/question. The situation I crawl a website which displays articles, let's say 20 (but the website has 200). I follow the article links and there I will extract the data I need. So far, so good. Some pages appear to have a Load More button instead of "normal" pagination to show more than the 20 articles. And here is where the struggle begins. I inspected the page (like described in this video [this video] with that example [LINK] for example). However, the structure of the request URL when clicking the button differs from the one described in the video.
Cites Abdulaziz Al Jumaia · 2019-07 · Stack Overflow · on “Clicking "Load More" on webpages by using python”
I am trying to load the whole content of this link [LINK] (clicking on المزيد) I tried so many tutorials like this one here [this video] , which teaches how to work with a similar issue (Infinite Scrolling Pages). My issue is that I couldn't manage to specify the load more class on this page to click it. But, if I am not mistaken, it exists in this part of the webpage source code: <ng-include src="'loader.html'" class="ng-scope"> <div class="loading-div ng-scope ng-hide" ng-show="!loadingMoreData"> <div class="spinner"> <div class="bounce1"></div> <div class="bounce2"></div> <div class="bounce3"></div> </div> </div> </ng-include> <div class="block-hed block-link clearfix" data-ng-show="isMoreDisp