In our last article, we learned how to create Node.js microservices using Polybit’s stdlib platform. We created a fabulous (IMHO 🙂 ) GPS service that enabled us to retrieve the name of a city based on its GPS coordinates. Today, we’re going to learn how to consume data returned from this stdlib GPS microservice using several methods. While the information presented here is specific to consuming Polybit stdlib microservices, many aspects of this article will be generally applicable for consuming Web API JSON data from any http endpoint. Strap on your seatbelts as we embark on a whirlwind tour to learn about consuming JSON data from a variety of contexts…and I’m talking about some serious variety!
Table of Contents
Review of stdlib GPS microservice API
Let’s first review how the stdlib GPS service we created last time works so you will be equipped to write code to consume the data it returns. If you created your own GPS service in our last post, go ahead and use it. If not, feel free to use mine since it is built and ready to go!
The GPS service (documented here) provides us with the name of the city when GPS coordinates are supplied. Once again, my apologies in advance to all of you, my friends, in other countries, since we are only able to find cities in the United States. Perhaps you can employ these concepts, to create a service to find cities in other parts of the world as well.
The GPS service accepts the following keyword arguments:
- lat: latitude (the angular distance of a place north or south of the earth’s equator)
- lon: longitude (the angular distance of a place east or west of the meridian at Greenwich, England)
- fmt: format of the data to return. If fmt = simple, just the city, state, and zip code are returned rather than a full JSON object.
Let’s first try to use this microservice without writing any code. For starters, get out your pen and paper (real or virtual), and go to this GPS Coordinates website, select a place in the USA to use for testing your (yet to be written) microservice, and jot down the GPS coordinates. San Diego is near and dear to my heart, so I will be using the following GPS coordinates for my microservice:
- latitude: 32.72
- longitude: -117.16
Please observe that I have rounded off the GPS coordinates to two decimal digits. This precision will more than suffice for locating a city given the GPS coordinates.
The microservice can be invoked in a variety of ways including from a URL (HTTP GET). Here is a sample URL for the San Diego coordinates:
https://thisdavej.api.stdlib.com/gps/findcity?lat=32.72&lon=-117.16
Go ahead and launch the microservice URL (https://thisdavej.api.stdlib.com/gps/findcity?lat=32.72&lon=-117.16) in your browser. You should see the following results:
{
"zipcode": "92101",
"state_abbr": "CA",
"latitude": "32.719601",
"longitude": "-117.16246",
"city": "San Diego",
"state": "California",
"distance": 0.2343729936809432
}
Feel free to tweak the URL parameters (lat and lon) and see how the results change.
Let’s review the various components of this stdlib URL:
https://f.stdlib.com/<username>/gps/findcity?lat=32.72&lon=-117.16
- f.stdlib.com – this is the host name of the stdlib function-as-a-service platform, where all the magic takes place.
- <username> – the stdlib username which is
thisdavejin my case, and will be your username after you build your microservice. - gps – the name of the microservice. You can think of this as a namespace for various functions.
- findcity – the name of a function created within the microservice
- lat/lon – URL parameters that are supplied to the function.
Next, let’s invoke the service by setting the fmt parameter to “simple” (fmt=simple) so just the city, state, and zip code are returned rather than a full JSON object. Append the URL in your browser with the fmt URL parameter:
https://thisdavej.api.stdlib.com/gps?lat=32.72&lon=-117.16&fmt=simple
When invoked, you should see the following results showing the “simple” format:
"San Diego, CA 92101"
Excellent! Our GPS service is working well and we are equipped with the knowledge needed to move on and invoke it many different ways.
Consuming service using the lib command
A cool feature of Polybit’s stdlib is that we can invoke microservices from the terminal using a command called lib. The lib command is included when you install the lib.cli npm package globally.
As a prerequisite, you will need to install Node.js from here, if you do not already have it installed. If you wish to run Node on a Raspberry Pi, follow the steps in my Beginner’s Guide to Installing Node.js on a Raspberry Pi.
Next, enter the following from the terminal to install the lib package globally on your system:
$ npm install -g lib.cli
Perfect! We can now use lib to invoke the microservice from the terminal as follows:
$ lib thisdavej.gps.findcity --lat 32.72 --lon -117.16
Sure enough, you should see a JSON object that looks like this:
{
"zipcode": "92101",
"state_abbr": "CA",
"latitude": "32.719601",
"longitude": "-117.16246",
"city": "San Diego",
"state": "California",
"distance": 0.2343729936809432
}
Next, let’s invoke the GPS service using the simple format option:
$ lib thisdavej.gps.findcity --lat 32.72 --lon -117.16 --fmt simple
Your results should look like this:
"San Diego, CA 92101"
The lib command is very powerful and useful for testing or for invoking microservices in real world scenarios!
Consuming service from Node.js
It probably comes as no surprise to you that stdlib microservices (based on Node.js) can be invoked from Node.js itself. Let’s create a Node console app to see this in action.
First, create an empty directory and navigate into it.
$ mkdir -p stdlib-client/find-city $ cd stdlib-client/find-city
Next, create an standard package.json file, accepting all of the defaults:
$ npm init -y
We will need to install the f npm package locally and save it in package.json as a dependency.
$ npm install --save f
Here is the general syntax used by f in a Node.js program:
const f = require('f');
f('user_name/service@version/function')(arg1, arg2, {kwarg1: 'value'}, (err, result) => {
if (err) {
// handle error
}
// do something with `result` here
});
Note: For our context, we will only be using the keyword arguments kwargs rather than args. My astute readers who read my previous introductory stdlib post will recall that we did provide an option to supply latitude and longitude as args parameters too. I will leave it as an exercise for you to supply args parameters as an option in microservices. I’m focusing this tutorial on using the kwargs parameters only since the keyword parameters are simple and effective and work in all contexts including HTTP GET. (The args parameters are not available through HTTP GET.)
Ok, back to our regularly scheduled program. 🙂 Let’s create a file called index.js and add the following contents:
const f = require('f');
if (process.argv.length !== 4) {
const programName = require('./package.json').name;
console.log(`Usage: ${programName} latitude longitude`);
process.exit(1);
}
const args = process.argv.slice(2);
const lat = args[0];
const lon = args[1];
f('thisdavej/gps/findcity')({ lat, lon }, (err, result) => {
if (err) {
console.error(err);
process.exit(1);
}
console.log(`The city for those GPS coordinates is ${result.city}, ${result.state}.`)
});
In line 1, we load the f module and make it available for use in our application.
In line 3, we check and confirm that a latitude and longitude are supplied on the command line. If not, we print usage text to guide the user to successfully invoke the program.
In line 13, we invoke f with our keyword arguments by passing a keyword JSON object of { lat, lon }. Please note that we could have also passed an object of { lat: lat, lon: lon }, but opted to use the shorthand notation instead since the JSON object attribute names and object values are the same.
On line 18, we display various attributes of the JSON object returned by the GPS service such as result.city.
Let’s give our program a try from the command line:
$ node index.js 32.72 -117.16
You should see the following results:
The city for those GPS coordinates is San Diego, California.
Looking good! Let’s move on and invoke our service from HTML.
Consuming service from a web browser
We have conquered the command line so let’s move on and invoke the GPS service in an HTML file through a Web browser.
First, create an empty directory and navigate into it.
$ mkdir -p stdlib-client/web $ cd stdlib-client/web
Go ahead and install bower globally if you don’t already have it installed:
$ npm install -g bower
Next, install the f library using bower:
$ bower install poly/f
As another option, you could also download the f.js file directly from the Polybit GitHub repo.
Create an index.html page with the following contents:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Find City using GPS Coordinates</title>
</head>
<body>
<h2>Result</h2>
<span id="message">Loading...</span>
<script type="text/javascript" src="./bower_components/f/web/f.js"></script>
<script type="text/javascript">
f('thisdavej/gps/findcity')({ lat: 32.72, lon:-117.16} , function(err, obj) {
var msg;
if (err) {
msg = err;
} else {
msg = obj.city + ", " + obj.state_abbr + " " + obj.zipcode;
}
var message = document.getElementById("message");
message.innerHTML = msg;
});
</script>
</body>
</html>
Before we can test our HTML page, we need to install a local web server to host our page since we will be initiating a cross origin request via an XMLHttpRequest to the microservice and this requires a protocol such as http or https from our originating HTML page making the request.
Let’s install the excellent http-server npm package. (I’ve written an article called Create a Web Server in Node without any Code which goes into more details about the http-server package.)
We’ll install the http-server package globally:
$ npm install -g http-server
Finally, we can invoke our web server and use -o to automatically open then index.html file in our local web browser at http://127.0.0.1:8080:
$ http-server -o
Huzzah! You should see a web page rendered that includes “San Diego, CA 92101”.
Let’s step it up a notch and use the W3C Geolocation API to obtain our current GPS coordinates and invoke our microservice to determine our current city. Here we go!
Create a file called index2.html file and add the following contents:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Find City using GPS Coordinates</title>
</head>
<body>
<h2>Result</h2>
<span id="message">Loading...</span>
<script type="text/javascript" src="./bower_components/f/web/f.js"></script>
<script type="text/javascript">
var message = document.getElementById("message");
if ("geolocation" in navigator) {
navigator.geolocation.getCurrentPosition(function(p) {
var msg;
f('thisdavej/gps/findcity')(p.coords.latitude, p.coords.longitude, function(err, obj) {
var msg;
if (err) {
msg = err;
} else {
msg = obj.city + ", " + obj.state_abbr + " " + obj.zipcode;
}
message.innerHTML = msg;
});
});
} else {
message.innerHTML = 'Geolocation is NOT available';
}
</script>
</body>
</html>
This time around, we leverage the geolocation API, if it is available. Pretty amazing! Let’s see it in action:
Confirm the http-server program is still running. Navigate to http://localhost:8080/index2.html. Your web browser may prompt you to confirm you want to disclose your current location. You will, of course, want to respond in the affirmative. You should now see your current city!
Very powerful! We’re supplying live GPS coordinates to our GPS service to get our current city.
Consuming service from bash
Let’s expand our horizons and invoke our GPS service from bash. This is accomplished with the help of the venerable curl command for transferring data using http and other protocols.
Log into your Linux system and create a script called get-city.sh and add the following contents:
#!/bin/bash LAT="$1" LON="$2" content=$(curl -sS "https://thisdavej.api.stdlib.com/gps/findcity?lat=$LAT&lon=$LON&fmt=simple") # Remove double quotes on each side sed -e 's/^"//' -e 's/"$//' <<<"$content"
This script accepts two parameters from the command line for longitude and latitude. We use curl (see documentation here) to invoke an HTTP GET against our web service and return data using the “simple” format. We also use sed to remove the double quotes from both sides of our results. Otherwise, our results will look like “San Diego, CA 92101” rather than San Diego, CA 92101 since stdlib microservices return JSON objects by default.
As a side note, if you are authoring an stdlib microservice and want to return results without double quotes (to avoid using sed), you could return the results using Buffer. Here is a simple example:
module.exports = (params, callback) => {
result = 'hello, world';
//callback(null, result);
callback(null, Buffer.from(result));
};
I have commented out the typical callback method in line 5 and demonstrated the use of Buffer to return the results in line 6.
If you want to further control how content is returned through a Web browser, you can modify the Content-Type returned from the stdlib micrsoservice response header. (The default Content-Type is application/octet-stream when results are returned as a Buffer.) This is accomplished by editing the headers attribute in function.json. For example, to set the Content-Type to text/plain with UTF-8 encoding, the function.json file would be modified like this:
....
"http": {
"headers": { "Content-Type": "text/plain; charset=utf-8" }
}
I generally don’t recommend using the Buffer method since it causes unintended side effects. For example, the lib command will return an expected result of “hello, world” as “<Buffer 68 65 6c 6c 6f 2c 20 77 6f 72 6c 64>” in the terminal. When the microservice is launched from the browser, it will download a file with the results rather than displaying the results. It will look great, however, when invoked through bash. 🙂 The use of Buffer is helpful in certain contexts, but the standard JSON results returned are typically what we want.
Now back to our bash script…
Before we can use our command, we need to set the execute bit:
$ chmod u+x find-city.sh
We are then positioned to launch our newly created command:
$ ./find-city.sh 32.72 -117.16
You should see a result of: San Diego, CA 92101
Next, let’s modify our script to return the full JSON object:
#!/bin/bash LAT="$1" LON="$2" content=$(curl -sS "https://thisdavej.api.stdlib.com/gps/findcity?lat=$LAT&lon=$LON") echo $content
And…invoke it:
$ ./find-city.sh 32.72 -117.16
You should see the following results:
{
"zipcode": "92101",
"state_abbr": "CA",
"latitude": "32.719601",
"longitude": "-117.16246",
"city": "San Diego",
"state": "California",
"distance": 0.2343729936809432
}
What if we want to parse the JSON object returned from our service and customize the results displayed? We may, for example, want something different than the “simple” format—and also want something different than the full JSON object. It’s time for the jq command to come to our rescue to parse and process the JSON text returned!
Let’s install jq first:
$ sudo apt install jq
Next, modify the script to use jq for parsing:
#!/bin/bash LAT="$1" LON="$2" content=$(curl -sS "https://thisdavej.api.stdlib.com/gps/findcity?lat=$LAT&lon=$LON") echo $content | jq '. | "\(.city), \(.state_abbr)"'
We retrieve the content from our GPS service and then pipe the JSON result into jq for parsing. We ultimately display the “city” and “state_abbr” attributes of the JSON object.
The jq command is very powerful and we have just provided a high-level introduction here. I plan to release a future tutorial covering many of jq’s capabilities for parsing data from JSON Web APIs.
As a final step, you could add the bash script you created to a directory contained in the system path such as usr/local/bin so it can be invoked without placing a “./” in front of the command. You could also create a custom scripts directory and add the custom scripts directory to the PATH variable.
Consuming service from Google Sheets
Google Sheets provides yet another great way to consume our stdlib microservice and process the JSON results. In this first example, we use the built-in IMPORTDATA function to retrieve data from our GPS service and display the results. Go ahead and create a Google Sheet as shown here:
| A | B | C | |
| 1 | Latitude | Longitude | |
| 2 | =IMPORTDATA(“https://thisdavej.api.stdlib.com/gps/findcity?lat=” & B2 & “&lon=” & C2 &”&fmt=simple”) | 32.72 | -117.16 |
After creating the sheet, you should see the following results:
| A | B | C | |
| 1 | Latitude | Longitude | |
| 2 | San Diego, CA 92101 | 32.72 | -117.16 |
We call our GPS service with fmt=simple so the results will show up on one line as “San Diego, CA 92101”. If we had not included fmt=simple, our results would include the full JSON object and spill over into multiple rows in our sheet.
Bonus information: If our service returned
San Diego, CA 92101(without being wrapped in double quotes ), theIMPORTDATAfunction would have placed “San Diego” in one column and “CA 92101” in a column to the right since the data imported is processed as CSV data. The double quotes protects the returned results from being split into two columns.
Go ahead and change the latitude and longitude in the cells. Sheets will re-invoke our GPS microservice and show us the results. How cool is that?
Next, let’s create our own custom function in Sheets so we can abstract away the URL and make it easier to invoke our GPS service. To accomplish this, click on Tools from the menu and select Script editor.
Google Sheets supports JavaScript in functions so let’s go ahead and create one now!
/**
* Find the city given a latitude and longitude.
*
* @param {32.72} latitude
* latitude coordinate
* @param {-117.16} longitude
* longitude coordinate
* @customfunction
*/
function FINDCITY(lat, lon) {
var response = UrlFetchApp.fetch("https://thisdavej.api.stdlib.com/gps/findcity?lat=" + lat +
"&lon=" + lon + "&fmt=simple");
return JSON.parse(response.getContentText());
}
This function uses the built-in UrlFetchApp.fetch method to fetch the data from our GPS service. We declare our function name in upper case which is the standard convention for Google Sheets functions. Finally, you will notice that we include a JSDOC style comment above our function. Sheets uses this information to provide a tooltip on function usage when editing a cell and using the function.
Go ahead and press Ctrl+S to save your function. Let’s start using it! Modify your Sheet as follows:
| A | B | C | |
| 1 | Latitude | Longitude | |
| 2 | =FINDCITY(B2,C2) | 32.72 | -117.16 |
You should see the following results:
| A | B | C | |
| 1 | Latitude | Longitude | |
| 2 | San Diego, CA 92101 | 32.72 | -117.16 |
As a last step in our Google Sheets tour, let’s kick it into high gear and create a custom function that enables us to use Handlebars-style templating syntax with the JSON object returned from our GPS service. Once again, click on Tools from the menu and select Script editor. Add the following custom function:
/**
* Parse JSON and render the results in a string template.
*
* @param {"url"} url
* JSON API data URL
* @param {"templateText"} template
* string template for rendering results
* @customfunction
*/
function JSONTEMPLATE(url, template) {
url = encodeURI(url);
var response = UrlFetchApp.fetch(url);
var obj = JSON.parse(response.getContentText());
var result = template.replace(/\{\{\s*(.*?)\s*\}\}/g, function (match, varName) {
return obj[varName];
});
return result;
}
As before, we use the built-in UrlFetchApp.fetch method to fetch the data from our GPS service. We parse the JSON object and use regular expressions to substitute in the various attributes of our JSON object using a template.
Let’s give this a test run to see this awesomeness in action. Modify your Sheet as follows:
| A | B | C | |
| 1 | Latitude | Longitude | |
| 2 | The city is {{city}} and the state is {{state}}. | 32.72 | -117.16 |
| 3 | =JSONTEMPLATE(“http://thisdavej.api.stdlib.com/gps/findcity?lat=” & B2 & “&lon=” & C2,A2) |
Please note our template in cell A2. The JSON attributes of interest are enclosed in doubly curly braces. You should see the following results:
| A | B | C | |
| 1 | Latitude | Longitude | |
| 2 | The city is {{city}} and the state is {{state}}. | 32.72 | -117.16 |
| 3 | The city is San Diego and the state is California. |
Google Sheets provides some amazing versatility for consuming JSON data from our stdlib GPS service, and the capability to use JavaScript to create custom functions results in a huge win!
If you are using Microsoft Excel instead of Google Sheets, Excel (2013 and higher) ships with the WEBSERVICE function which can be used to invoke URLs and receive JSON data from microservices.
Consuming service from C#
To maximize our variety, let’s go ahead and develop a C# console application to invoke our GPS service. We’ll cover the steps at a very high level.
To develop C# applications in Windows, you can use free Visual Studio Community Edition IDE.
Create a C# console application called WebApiFetcher, and use the NuGet package manager to install the Newtonsoft.Json package as follows:
PM> Install-Package Newtonsoft.Json
Next, add the following code to the Program.cs file:
using Newtonsoft.Json.Linq;
using System;
using System.IO;
using System.Net;
using System.Net.Http;
namespace JsonApiFetcher
{
class Program
{
private static HttpClient Client = new HttpClient();
static void Main(string[] args)
{
if (args.Length != 2)
{
Console.WriteLine("Usage: FindCity latitude longitude");
Environment.Exit(1);
}
string lat = args[0];
string lon = args[1];
string url = $"https://thisdavej.api.stdlib.com/gps/findcity?lat={lat}&lon={lon}";
// Set result as error message and replace with data if web request succeeds
string result = "Error retrieving data";
try
{
var json = Client.GetStringAsync(url).Result;
dynamic o = JObject.Parse(json);
result = $"{o.city}, {o.state_abbr}";
}
catch (WebException ex)
{
var responseStream = ex.Response?.GetResponseStream();
if (responseStream != null)
{
using (var reader = new StreamReader(responseStream))
{
string error = reader.ReadToEnd();
result = $"Error: {error}";
}
}
}
Console.WriteLine(result);
}
}
}
Build your executable and the run your newly created program from the command line:
C:/> WebApiFetcher 32.72 -116.22
You should see the following results: San Diego, CA
Consuming service from Python
Why not invoke the GPS service from Python? We don’t want any languages to be left behind. 🙂
Download the latest version of Python 3 from the Python downloads page. Create a file called find_city.py and add the following contents:
""" Usage: find_city.py (Prompts user for GPS Coordinates and provides city name) """
import json
from urllib.request import urlopen
LAT = input('Enter latitude: ')
LON = input('Enter longitude: ')
URL = 'https://thisdavej.api.stdlib.com/gps/findcity?lat={}&lon={}'.format(LAT, LON)
RESPONSE = urlopen(URL)
CONTENT = RESPONSE.read().decode('utf-8')
OBJ = json.loads(CONTENT)
print('{}, {} {}'.format(OBJ['city'], OBJ['state_abbr'], OBJ['zipcode']))
Launch your newly created Python program:
$ python find_city.py
You will be prompted for the latitude and longitude and Python will communicate with the stdlib GPS service and return the results!
Consuming service using HTTP POST commands
We have been focusing on invoking our GPS service using HTTP GET commands. It is also possible to invoke stdlib microservices using HTTP POST commands. Microservices developed using stdlib expect to receive POST data in the following JSON format:
{
"args": [],
"kwargs": {}
}
Let’s invoke the GPS service usign an HTTP POST with the help of Postman, an awesome Google Chrome extension for developing and testing APIs. Postman is also available as a Mac App.
First, go to the Postman home page and install the Postman app.
Next, launch Postman and create an API request with the following information:
- HTTP Verb: POST
- URL: https://thisdavej.api.stdlib.com/gps/findcity
- Headers: key =
Content-Type, value =application/json(This is probably already there by default since we specified an HTTP Verb ofPOST) - For
Body, chooserawand include the following contents:
{
"args": [],
"kwargs": {
"lat": 32.72,
"lon": -117.16
}
}
Here’s a screenshot to provide some context:
Click the Send button to invoke the HTTP POST against the stdlib GPS service and you should see a JSON object returned!
Conclusion
We learned how to consume microservices created by Polybit’s stdlib platform using a wide variety of methods! JSON has clearly become the lingua franca for exchanging data on the web and we are now well positioned to take advantage of this ubiquity with the help of stdlib and the many different ways the world has provided to consume the JSON data returned from microservices.
Follow @thisDaveJ (Dave Johnson) on Twitter to stay up to date with the latest tutorials and tech articles.
Additional articles
Creating Node.js Microservices with Ease Using Stdlib
Node.js: Sending Email Notifications Using Nodemailer and Gmail
Getting Started with YAML in Node.js using js-yaml
Making Interactive Node.js Console Apps That Listen for Keypress Events
Awesome article! This is sooo helpful!