The ability to log data is an important capability in IoT applications. In this tutorial, we learn how to use Node.js to log data by utilizing built-in Node modules. There are certainly excellent logging modules available including pino and Winston (see my Winston tutorial here); however, our goal today is to deepen our knowledge of Node by implementing some simple logging code ourselves in order to become better Node developers.
Getting Data That is Out of This World
In our last tutorial on Tracking the ISS through the Sky, we developed a data acquisition system for retrieving data from a “sensor in the sky”. Here is the code we ultimately developed:
const got = require('got');
const geolib = require('geolib');
const delaySeconds = 3;
const url = 'http://api.open-notify.org/iss-now.json';
const myPosition = { latitude: 32.715738, longitude: -117.161084 };
function loop() {
got(url, { json: true })
.then(iss => {
const position = iss.body.iss_position;
const distanceFromIss = geolib.getDistance(myPosition, position);
const distanceFromIssMiles = geolib.convertUnit('mi', distanceFromIss, 2);
console.log(`${distanceFromIssMiles} miles`);
})
.catch(error => {
console.log(error.response.body);
});
setTimeout(loop, delaySeconds * 1000);
}
loop();
With this code, we provided our current location and created a loop to calculate our distance in miles from the International Space Station every 3 seconds. Try this out for yourself. You will need to npm install got and geolib in order for this to work.
With this foundation in place, we are ready to start logging the data that we are receiving and processing.
Logging Raw Data to a CSV File
As a first step, let’s log data to a CSV (Comma Separated Values) file. This will enable us to view and process the data further in a program such as Microsoft Excel or LibreOffice Calc.
We’ll first create a function that we can invoke to log data when given a log file name and the data to write:
const logDir = path.join(__dirname, 'logs');
function logToFile(logFileName, dataToWrite) {
const logFilePath = path.join(logDir, logFileName);
const timestamp = new Date().toLocaleString();
const data = `${timestamp}, ${dataToWrite}${eol}`;
fs.appendFile(logFilePath, data, (error) => {
if (error) {
console.error(`Write error to ${logFileName}: ${error.message}`);
}
});
}
We first define logDir to specify where our log files will be written. We define a separate directory so we don’t clutter our Node code directory with log files. We place this variable outside of our function near the top of our code so that we can easily change it later.
Next, we create the function including a timestamp variable so we automatically log the timestamp with each entry in the log.
Finally, we leverage the fs.appendFile method to write data to a file. If the file does not exist, it will automatically be created. If it does exist, data will be appended at the end of the file.
We are almost ready to go; however, we have one problem. If we don’t create the log directory in advance, our program will fail. (I learned this from firsthand experience. :)) After creating an init function to handle that scenario, we arrive at the following code that we can take for a spin:
const got = require('got');
const geolib = require('geolib');
const fs = require('fs');
const path = require('path');
const eol = require('os').EOL;
const loopSeconds = 10;
const url = 'http://api.open-notify.org/iss-now.json';
const myPosition = { latitude: 32.715738, longitude: -117.161084 };
const logDir = path.join(__dirname, 'logs');
const dataLogFile = 'issResults.csv';
function directoryExists(filePath) {
try {
return fs.statSync(filePath).isDirectory();
} catch (err) {
return false;
}
}
function init() {
if (!directoryExists(logDir)) {
fs.mkdirSync(logDir);
}
}
function logToFile(logFileName, dataToWrite) {
const logFilePath = path.join(logDir, logFileName);
const timestamp = new Date().toLocaleString();
const data = `${timestamp}, ${dataToWrite}${eol}`;
fs.appendFile(logFilePath, data, (error) => {
if (error) {
console.error(`Write error to ${logFileName}: ${error.message}`);
}
});
}
function loop() {
got(url, { json: true })
.then(iss => {
const position = iss.body.iss_position;
const distanceFromIss = geolib.getDistance(myPosition, position);
const distanceFromIssMiles = geolib.convertUnit('mi', distanceFromIss, 2);
console.log(`${distanceFromIssMiles} miles`);
logToFile(dataLogFile, distanceFromIssMiles);
})
.catch(error => {
console.log(error.response.body);
});
setTimeout(loop, loopSeconds * 1000);
}
init();
loop();
Notice that we also create a directoryExists function so we can test first to see if the log directory exists before creating it. Also, we run our loop every 10 seconds for testing purposes. If we were deploying this code in production, a value of 60 seconds would probably be more reasonable.
Conditional Logging – Log if ISS is Closer or Further Than Previous Readings
Let’s now implement some conditional logging so we can log if the ISS is closer or further than our previous observations. Here is the modified source code:
const got = require('got');
const geolib = require('geolib');
const fs = require('fs');
const path = require('path');
const eol = require('os').EOL;
const moment = require('moment');
const loopSeconds = 10;
const url = 'http://api.open-notify.org/iss-now.json';
const myPosition = { latitude: 32.715738, longitude: -117.161084 };
const logDir = path.join(__dirname, 'logs');
const dataLogFile = 'issResults.csv';
const currentDateTime = moment().format('YYMMDD[T]HHmmss');
const closestDistanceFile = `issClosest-${currentDateTime}.csv`;
const furthestDistanceFile = `issFurthest-${currentDateTime}.csv`;
let closestDistance = Number.MAX_VALUE;
let furthestDistance = Number.MIN_VALUE;
function directoryExists(filePath) {
try {
return fs.statSync(filePath).isDirectory();
} catch (err) {
return false;
}
}
function init() {
if (!directoryExists(logDir)) {
fs.mkdirSync(logDir);
}
}
function logToFile(logFileName, dataToWrite, append = true) {
const logFilePath = path.join(logDir, logFileName);
const timestamp = new Date().toLocaleString();
const data = `${timestamp}, ${dataToWrite}${eol}`;
const flags = append ? { flag: 'a' } : {};
fs.writeFile(logFilePath, data, flags, (error) => {
if (error) {
console.error(`Write error to ${logFileName}: ${error.message}`);
}
});
}
function loop() {
got(url, { json: true })
.then(iss => {
const position = iss.body.iss_position;
const distanceFromIss = geolib.getDistance(myPosition, position);
const distanceFromIssMiles = geolib.convertUnit('mi', distanceFromIss, 2);
console.log(`${distanceFromIssMiles} miles`);
logToFile(dataLogFile, distanceFromIssMiles);
if (distanceFromIssMiles < closestDistance) { closestDistance = distanceFromIssMiles; console.log(`Closer than ever: ${closestDistance}`); logToFile(closestDistanceFile, closestDistance, false); } if (distanceFromIssMiles > furthestDistance) {
furthestDistance = distanceFromIssMiles;
console.log(`Further than ever: ${furthestDistance}`);
logToFile(furthestDistanceFile, furthestDistance, false);
}
})
.catch(error => {
console.log(error.response.body);
});
setTimeout(loop, loopSeconds * 1000);
}
init();
loop();
We’ve made several modifications. Here are a few of the highlights:
- We added the
momentnpm module. (You will need tonpm install momentfor this code to work.) Moment is an amazing module that enables us to format and parse dates. In my opinion, it should ship with the core JavaScript system. We use moment to format dates in a human readable format for creating meaningful CSV file names. - We defined two new files to log our closest encounter of the ISS kind (not to be confused with this movie :)) as well as the furthest distance of the ISS from us.
- We modified our
logToFilefunction to include an optional parameter that specifies whether we should append to the log file or replace the log file. In the case of the log files for storing the closest and farthest distances, we opt to replace the log files completely rather than append. - Finally, in the main loop, we compare the current distances with previous records set for closest and furthest distance, and write to the log files if a new record is set.
Conclusion
We successfully implemented some basic logging capabilities by leveraging built-in Node modules, with the exception of moment to help us format the current date to create CSV file names. The ability to log data will serve us well in the future—whether our sensor is out of this world or right in front of us on our desk!
Follow @thisDaveJ on Twitter to stay up to date on the latest tutorials and tech articles.
Additional Articles
Node.js IoT: Tracking the ISS through the Sky
Beginner’s Guide to Installing Node.js on a Raspberry Pi
Using Winston, a versatile logging library for Node.js
Using Visual Studio Code with a Raspberry Pi (Raspbian)
Visual Studio Code Jumpstart for Node.js Developers
Just a heads up. There is some formatting issues in the loop function of the last source listing. Missing some newlines.