Very Simple NodeJS Web Scraper (Quick Example)

Welcome to a tutorial on how to create a simple web scraper in NodeJS. So you need to get data from a website automatically? That calls for a web scraper, and it is actually easy to create one in NodeJS.

Yes, but just how does that work? Read on for an example!

TABLE OF CONTENTS

Download & Notes

NodeJS Web Scraper

Extras

The End

EXAMPLE CODE DOWNLOAD

Source code on GitHub Gist

Just click on “download zip” or do a git clone. I have released it under the MIT license, so feel free to build on top of it or use it in your own project.

SORRY FOR THE ADS...

But someone has to pay the bills, and sponsors are paying for it. I insist on not turning Code Boxx into a "paid scripts" business, and I don't "block people with Adblock". Every little bit of support helps.

Buy Me A Coffee Code Boxx eBooks

QUICK SETUP

Run npm i jsdom to install the required modules.

PART 1) DUMMY PRODUCT PAGE

<div id="product">
  <img src="basketball.png" id="pImg">
  <div id="pName">Basketball</div>
  <div id="pPrice">$12.34</div>
  <div id="pDesc">It's a ball. Horray?</div>
  <input type="button" value="Add To Cart" id="pAdd">
</div>

First, we need a dummy page to work with, and here is a simple product page – We will be extracting the product information from this page in the NodeJS web scraper below.

PART 2) NODEJS WEB SCRAPER

// (A) LOAD JSDOM
const jsdom = require("jsdom");
const { JSDOM } = jsdom;
 
// (B) FETCH
fetch("http://localhost/1-dummy.html")
.then(res => res.text())
.then(txt => {
  // (B1) PARSE HTML
  const dom = new JSDOM(txt);
  const doc = dom.window.document;
 
  // (B2) EXTRACT INFORMATION
  console.log(doc.getElementById("pName").innerHTML);
  console.log(doc.getElementById("pPrice").innerHTML);
  console.log(doc.getElementById("pDesc").innerHTML);
  console.log(doc.getElementById("pImg").src);
})
.catch(err => console.log(err));

D:\http> node .\2-scrape.js
Empty Box
$12.34
It's an... empty box. Horray?
box.png

Yep, that’s all for the “very difficult” web scraper. As in the introduction:

• We use fetch(URL).then(res => res.text()) to get the web page as a string.
• Technically, we can use substring() and match() to extract data from the “raw HTML string”… But it’s smarter to use an HTML parser here – const dom = new JSDOM(txt).
• Finally, you should already know all the “get HTML element” functions – getElementById() querySelector() querySelectorAll()… If you don’t it’s time to catch up with the basics. Links below.

P.S. Take note that fetch() is only available in Node 17.5 and later. If you are still using an older version, it’s time to update or consider using other modules such as CURL.

LINKS & REFERENCES

• JSDOM – NPM
• http-server – NPM
• CURL – NPM
• DOM Navigation – Javascript.info