How to Get to the End of a Page with Infinite Scrolling 🖱️

You've probably been on a web page with infinite scrolling. But, have you ever needed to load all the results? 🤔

Recently, I had to scrape data of a webpage with this feature and I had to choose: either scroll down and wait for the new results to load a few times OR I could try to find a way to do it by programming.

I'll let you guess what choice I made! 😝

Fortunately, the JavaScript code to go to the bottom of the infinite scrolling page is pretty straightforward if you know the language. Or if you are on this article! 😉

Just open the DevTools (with F12) and paste the following script in the console.

// Declare some constants
const MAXIMUM_NUMBER_OF_TRIALS = 5;
const MINIMUM_SLEEPING_TIME_IN_MS = 500;
const MAXIMUM_SLEEPING_TIME_IN_MS = 2000;

// Utility functions
const sleep = (time) => new Promise((resolve) => setTimeout(resolve, time));
const randomNumber = (minimum, maximum) => Math.floor(Math.random() * maximum) + minimum;
const randomSleep = () => sleep(randomNumber(MINIMUM_SLEEPING_TIME_IN_MS, MAXIMUM_SLEEPING_TIME_IN_MS));

// How to get at the bottom of an infinity scroll
let currentScrollHeight = 0;
let manualStop = false;
let numberOfScrolls = 0;
let numberOfTrials = 0;

while (numberOfTrials < MAXIMUM_NUMBER_OF_TRIALS && !manualStop) {
  // Keep the current scroll height
  currentScrollHeight = document.body.scrollHeight;

  // Scroll at the bottom of the page
  window.scrollTo(0, currentScrollHeight);

  // Wait some seconds to load more results
  await randomSleep();

  // If the height hasn't changed, there may be no more results to load
  if (currentScrollHeight === document.body.scrollHeight) {
    // Try another time
    numberOfTrials++;

    console.log(
      `Is it already the end of the infinite scroll? ${MAXIMUM_NUMBER_OF_TRIALS - numberOfTrials} trials left.`,
    );
  } else {
    // Restart the number of consecutive trials
    numberOfTrials = 0;

    // Increment the number of successful scroll
    numberOfScrolls++;

    console.log(`The scroll #${numberOfScrolls} was successful!`);
  }
}

console.log('We should be at the bottom of the infinity scroll! Congratulation!');
console.log(`${numberOfScrolls} scrolls were needed to load all results!`);

I added a lot of comments, so you should be able to figure it out easily. You may need to adjust the constants at the top of the script if the server is taking too long to respond.