Web Scraping with NodeJS and Osmosis

What Is Web Scraping?

Web Scraping is a technique for extracting information from a web page. It allows you to obtain vast amounts of information, known as Big Data.

To use it, you need a basic to intermediate understanding of HTML and the most common tags, as well as JS and CSS, and client-side web technologies in general.

To obtain data from a website, you also need knowledge of server-side web technologies, such as PHP or NodeJS, for example. Web Scraping must be performed on the server side, which will send requests to websites and process the data. This guide will use basic commands in NodeJS.

Before you begin, you need to know that not all Web Scraping is legal, at least in Spain. According to the Spanish justice system, these are the cases in which this technique is illegal:

• When it infringes the intellectual property rights of the author of the web page being scraped.
• When there is unfair competition.
• When it violates the website's legal terms and conditions of use.
• If it fails to comply with personal data protection regulations, specifically Organic Law 15/1999 of December 13 on the Protection of Personal Data.

This post is for informational purposes, so readers are responsible for how they use these technologies.

Installing NodeJS and Osmosis

If You Have Linux

Install the nodejs and npm packages using the package manager installed on your computer; for Debian and derivatives: `

# apt install nodejs npm

Once NodeJS is installed, run:

# npm install http osmosis

If You Have Windows and MacOS

Download and run the installers from the NodeJS website: https://nodejs.org/es/download/

Creating the Server

Create a new file named index.js and add the base code for the server:

var http = require('http');
const osmosis = require('osmosis');
var server = http.createServer(function(request, response) {
response.writeHead(200, {"Content-Type": "text/plain"});
});
var port = process.env.PORT || 1337;
server.listen(port);
console.log("Server running at http://localhost:%d", port)

Create a new file (in the same folder as the previous one) named package.json and add the following code:

{
	"name": "Hola Mundo",
  "description": "holamundo",
  "version": "0.0.1",
  "private": true,
  "license": "MIT",
  "author": "My name",
  "scripts": {
  "start": "node index.js"
  },
  "dependencies": {
  "osmosis": "^1.1.4"
  }
}

where the fields can be edited as desired, except for the scripts and dependencies sections.

Basic Osmosis Usage

After the line response.writeHead(200, {“Content-Type”: “text/plain”}); write the following code:

osmosis
.get('http://www.ejemplo.es') // Specify the URL to obtain data from

.find('#ctPMain') // Choose the element from which to extract information; it is written in JQuery format, # means we are looking for an id named "ctPMain"

.set({'elements': ['td']}) // Create a JSON array with the "td" elements extracted from the previous tag (also in JQuery format)

.log(console.log)
.data(function(data){
datos = JSON.stringify(data); // Actions
})
.error(console.log);
response.end(datos); // Display the created JSON on screen

To run it on your machine, you can type the following in a terminal:

node index.js

Documentation

This application has extensive documentation available on GitHub and allows you to extract large amounts of data by nesting the options correctly.

This post was written by David Población Criado, a Computer Engineering and Statistics student at the University of Valladolid.