Consuming Web API JSON Data Using curl and jq

Hey everyone!  I decided to put a few extra batteries in the background color of the article image above. 🙂  I’m actually pretty charged up about our topic today, particularly about jq, which is a lightweight and flexible command-line JSON processor with “batteries included”.

JSON has become the lingua franca for exchanging data on the web, and we (as developers) need to know how to process JSON data so we can be positioned for the present and for the future. Today, we’re zooming in and learning about consuming Web API JSON data from bash using curl and jq.

Article contents

Prepare environment

We’ll be working in bash today so you will need to find a Linux machine to give this a go. I’m personally running on a Raspberry Pi (Raspbian), but this tutorial will work equally well in any Linux-based environment. (My instructions will, however, be geared toward Debian-based Linux distros since we will be using the apt command.) To set up a Raspberry Pi, you can follow the instructions in my popular Beginner’s Guide to Installing Node.js on a Raspberry Pi. We won’t need Node.js for this tutorial; however, my beginner’s guide will help you get an awesome development environment set up that optionally includes Node.js too. If you don’t have a Linux environment already, other options might include running Ubuntu on a virtual machine.

We will consume Web API JSON data with the help of the venerable curl command for transferring data using http and other protocols. As a first step, log into your Linux system and launch a terminal session to verify that curl is installed (if you are running Raspbian, curl ships with the base image):

$ which curl
/usr/bin/curl

We see that usr/bin/curl is returned so we know curl is installed. If nothing is returned, you will need to install curl:

$ sudo apt install curl

Weather web API overview – current weather conditions

As Mark Twain (or was it Charles Dudley Warner?) once said, “Everybody talks about the weather, but nobody does anything about it.” Today, we’ll be doing something about it! (Maybe not :), but we’ll at least gain some situational awareness so we know what’s heading our way.)

Wouldn’t it be cool if we could slice and dice the JSON data returned from a Weather Web API and get the current and forecast weather conditions? Have no fear, everyone… I’ve got you covered. 🙂 I created a Weather microservice using Azure Functions in Python for our tutorial today. Let’s focus first on consuming JSON data to get current weather conditions:

The current function accepts two parameters:

• loc: location (example: San Diego, CA)
• deg: degree type (F or C) Default is F.

Example: Return a JSON object with the current weather info (deg C)

https://thisdavej.azurewebsites.net/api/weather/current?loc=San Diego, CA&deg=C

This will return the following JSON object:

{
  "temperature": "18",
  "skycode": "32",
  "skytext": "Sunny",
  "date": "2016-11-23",
  "observationtime": "10:15:00",
  "observationpoint": "San Diego, CA",
  "feelslike": "18",
  "humidity": "68",
  "winddisplay": "2 km/h East",
  "day": "Wednesday",
  "shortday": "Wed",
  "windspeed": "2 km/h",
  "imageUrl": "http://blob.weather.microsoft.com/static/weather4/en-us/law/32.gif",
  "degType": "C"
}

There you have it. Perfect, we are ready to create our first script to retrieve the current weather from our cloud-based microservice!

Get raw JSON data for current weather using curl

First, create a bash script called weather.sh and add the following contents:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/current?loc=$LOCATION&deg=$DEG")

echo $content

This command accepts two parameters from the command line for the location (line 3) and degree type (F or C) on line 4.

In line 7, we use sed to HTML encode blank spaces in the LOC parameter passed from the command line so that “San Diego, CA” becomes “San%20Diego,%20CA” when invoked through a URL.

In line 9, we use curl to initiate an HTTP GET against our web service and return the current weather data. We invoke curl using curl -sS. These parameters accomplish the following:

• -s: Launches curl in silent mode so it doesn’t show errors or a progress meter as the Web API results are being downloaded.
• -S: When used with -s it shows an error message if curl fails.

See the curl man page for additional options available.

Finally, in line 10, we echo the downloaded content to the terminal.

Before we can use our command, we may need to set the execute bit:

$ chmod u+x weather.sh

We are then positioned to launch our newly created command:

$ ./weather.sh "San Diego, CA"

You should see a result that looks something like this:

{ "temperature": "66", "skycode": "32", "skytext": "Sunny", "date": "2016-11-23", "observationtime": "10:45:00", "observationpoint": "San Diego, CA", "feelslike": "66", "humidity": "68", "winddisplay": "1 mph East", "day": "Wednesday", "shortday": "Wed", "windspeed": "1 mph", "imageUrl": "http://blob.weather.microsoft.com/static/weather4/en-us/law/32.gif", "degType": "F" }

Our current weather forecast has been returned, but it is quite messy and difficult to interpret. Let’s clean it up.

Process current weather JSON data using jq

Pretty print current data

Let’s improve our ability to work with Web API JSON data with the help of jq, a lightweight and flexible command-line JSON processor. As a first step, let’s install jq:

$ sudo apt install jq

Excellent! We’re now ready to step up our JSON game and move beyond the blob of text we are currently seeing returned in our first script.

Modify your existing weather.sh script or create a new one and add the following contents:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/current?loc=$LOCATION&deg=$DEG")

echo $content | jq .

This script is identical to our last script, but we now pipe our results through jq. The jq . command processes the JSON returned and pretty prints it.

Let’s try it out:

$ ./weather.sh "San Diego, CA"

Here are some sample results:

{
  "temperature": "67",
  "skycode": "32",
  "skytext": "Sunny",
  "date": "2016-11-23",
  "observationtime": "11:10:00",
  "observationpoint": "San Diego, CA",
  "feelslike": "67",
  "humidity": "66",
  "winddisplay": "4 mph Northwest",
  "day": "Wednesday",
  "shortday": "Wed",
  "windspeed": "4 mph",
  "imageUrl": "http://blob.weather.microsoft.com/static/weather4/en-us/law/32.gif",
  "degType": "F"
}

Ah, yes. Our results are looking much more readable! The jq command is capable of a lot more as we soon will see.

Return one JSON attribute

Let’s modify our script to return just the temperature key of the weather JSON object:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/current?loc=$LOCATION&deg=$DEG")

echo $content | jq .temperature

Notice that we just changed the last line of our script to return just the temperature attribute of our JSON object. You should see the following results:

"67"

Very nice!

Return multiple JSON attributes

What if we want to return more than one attribute from our JSON current weather object? Here’s how we do it:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/current?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '. | "\(.temperature)°\(.degType) \(.skytext)"'

Here is our result:

67°F Sunny

We made a couple of changes to the last line of our script. We invoke jq with -r (raw output) so the result is no longer formatted as a JSON string with quotes. This enables the script to return 67°F Sunny instead of "67°F Sunny".

Secondly, we use a pipe (|) in jq and get the degType and skytext JSON attributes in addition to temperature. Note also the use of the backslash and parentheses syntax \(.temperature) to retrieve a given JSON attribute such as temperature.

Weather web API overview – forecast weather conditions

I believe we’ve covered the processing of the current weather JSON object sufficiently. Let’s move on and process some JSON data that provides the weather forecast. This will also give us a chance to learn how to process arrays of JSON objects.

The forecast function also accepts two parameters:

• loc: location (example: San Diego, CA)
• deg: degree type (F or C) Default is F.

Example: Return a JSON object with the forecast weather info (deg C)

https://thisdavej.azurewebsites.net/api/weather/forecast?loc=San Diego, CA&deg=C

This will return a JSON object shaped like this:

[
 {
 "low": "12",
 "high": "20",
 "skycodeday": "31",
 "skytextday": "Clear",
 "date": "2016-11-23",
 "day": "Tuesday",
 "shortday": "Tue",
 "precip": "0",
 "degType": "C"
 },
 {
 "low": "12",
 "high": "19",
 "skycodeday": "28",
 "skytextday": "Mostly Cloudy",
 "date": "2016-11-23",
 "day": "Wednesday",
 "shortday": "Wed",
 "precip": "0",
 "degType": "C"
 },
 {
 "low": "11",
 "high": "24",
 "skycodeday": "32",
 "skytextday": "Sunny",
 "date": "2016-11-24",
 "day": "Thursday",
 "shortday": "Thu",
 "precip": "0",
 "degType": "C"
 },
 {
 "low": "12",
 "high": "21",
 "skycodeday": "32",
 "skytextday": "Sunny",
 "date": "2016-11-25",
 "day": "Friday",
 "shortday": "Fri",
 "precip": "0",
 "degType": "C"
 },
 {
 "low": "14",
 "high": "18",
 "skycodeday": "30",
 "skytextday": "Partly Sunny",
 "date": "2016-11-26",
 "day": "Saturday",
 "shortday": "Sat",
 "precip": "80",
 "degType": "C"
 }]

As you can see, the forecast weather function returns a JSON object containing an array of five objects providing the forecast weather data.

Get raw JSON data for forecast weather using curl

Let’s start by returning raw data:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content

We see the following returned:

[ { "low": "53", "high": "67", "skycodeday": "31", "skytextday": "Clear", "date": "2016-11-22", "day": "Tuesday", "shortday": "Tue", "precip": "0", "degType": "F" }, { "low": "54", "high": "67", "skycodeday": "28", "skytextday": "Mostly Cloudy", "date": "2016-11-23", "day": "Wednesday", "shortday": "Wed", "precip": "0", "degType": "F" }, { "low": "52", "high": "75", "skycodeday": "32", "skytextday": "Sunny", "date": "2016-11-24", "day": "Thursday", "shortday": "Thu", "precip": "0", "degType": "F" }, { "low": "53", "high": "70", "skycodeday": "32", "skytextday": "Sunny", "date": "2016-11-25", "day": "Friday", "shortday": "Fri", "precip": "0", "degType": "F" }, { "low": "58", "high": "65", "skycodeday": "30", "skytextday": "Partly Sunny", "date": "2016-11-26", "day": "Saturday", "shortday": "Sat", "precip": "80", "degType": "F" } ]

I think it’s fair to say that this is quite messy. 🙂

Process forecast weather JSON data using jq

Pretty print forecast data

Let’s start using jq and do a pretty print of the JSON returned:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq .

As you might guess, the results returned look very similar to our JSON results presented above in the API description above. I won’t reprint them here to avoid redundancy.

Return one attribute from each JSON object in an array

What if we want to just print one attribute from each of the JSON objects in the array? Let’s do that now:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '.[] .shortday'

The .[] syntax returns all elements from the array and we print the shortday JSON attribute from each object. Once again, we use jq -r to remove the double quotes from each JSON object returned. Here are the results:

Tue
Wed
Thu
Fri
Sat

Very nice! A weather forecast the only returns the day of the week is not very insightful. 🙂

Return multiple attributes from each JSON object in an array

Let’s get multiple attributes from each JSON object in the array:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '.[] | "\(.shortday) \(.low)-\(.high)°\(.degType) \(.skytextday) ~ Rain: \(.precip)%"'

We see now that jq is using a pipe (|) through so the attributes of each JSON object can be displayed.
You will now see results that look something like this:

Tue 53-67°F Clear ~ Rain: 0%
Wed 54-67°F Mostly Cloudy ~ Rain: 0%
Thu 52-75°F Sunny ~ Rain: 0%
Fri 53-70°F Sunny ~ Rain: 0%
Sat 58-65°F Partly Sunny ~ Rain: 80%

Return a subset of JSON objects in an array based on indexes

What if we only want to see the first two elements of the JSON array? We can do that too:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '.[0,1] | "\(.shortday) \(.low)-\(.high)°\(.degType) \(.skytextday) ~ Rain: \(.precip)%"'

As you can see on the last line, we only return the first two elements [0,1] in the array of JSON objects.

Here are the results:

Tue 53-67°F Clear ~ Rain: 0%
Wed 54-67°F Mostly Cloudy ~ Rain: 0%

Return a subset of JSON objects based on a selection criteria

We can also select only certain objects in the JSON array to be returned based on a criteria. For example, let’s only return the forecast objects for days that it might rain:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '.[] | select(.precip | tonumber > 0) | "\(.shortday) \(.low)-\(.high)°\(.degType) \(.skytextday) ~ Rain: \(.precip)%"'

This jq syntax selects objects in the JSON array where the precipitation is forecasted to be greater than zero.  Please note that

This returns the following results:

Sat 58-65°F Partly Sunny ~ Rain: 80%

How about that? It rarely rains in San Diego, but it looks like it might actually rain on Saturday. I’m glad I have this script to help me. 😉

Return a sorted array of JSON objects

Can we sort the array of objects returned from the JSON Web API call? You bet! Let’s sort by the low temperature:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '. | sort_by(.low) | .[] | "\(.shortday) \(.low)-\(.high)°\(.degType) \(.skytextday) ~ Rain: \(.precip)%"'

This returns the following results:

Thu 52-75°F Sunny ~ Rain: 0%
Tue 53-67°F Clear ~ Rain: 0%
Fri 53-70°F Sunny ~ Rain: 0%
Wed 54-67°F Mostly Cloudy ~ Rain: 0%
Sat 58-65°F Partly Sunny ~ Rain: 80%

How about sorting in descending order? Yes, indeed with the help of the jq reverse function…

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '. | sort_by(.low) | reverse | .[] | "\(.shortday) \(.low)-\(.high)°\(.degType) \(.skytextday) ~ Rain: \(.precip)%"'

The results, sorted in descending orde by low temperature, look like this:

Sat 58-65°F Partly Sunny ~ Rain: 80%
Wed 54-67°F Mostly Cloudy ~ Rain: 0%
Fri 53-70°F Sunny ~ Rain: 0%
Tue 53-67°F Clear ~ Rain: 0%
Thu 52-75°F Sunny ~ Rain: 0%

Return the maximum value of a given attribute in an array of JSON objects

We can also return just the maximum value using jq’s max_by function. Let’s display the maximum temperature predicted in the forecast:

#!/bin/bash

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '. | max_by(.high) | "\(.shortday) \(.low)-\(.high)°\(.degType) \(.skytextday) ~ Rain: \(.precip)%"'

The maximum forecasted temperature is displayed:

Thu 52-75°F Sunny ~ Rain: 0%

Not surprisingly, jq also provides a min_by function too.

Use conditional logic to control the results displayed

Finally, let’s utilize some conditional logic to control what information is displayed. Let’s provide some guidance to bring an umbrella on the days where the chance of rain (precipitation) is forecasted to be greater than 20%:

#!/bin/bash

if (( $# < 1 ))
then
  echo "Usage: $(basename "$0") location [degreeType]"
  echo "Example: $(basename "$0") \"San Diego, CA\" F"
  exit 1
fi

LOC="$1"
DEG="$2"

# HTML encode string as %20
LOCATION=$(sed -e "s/ /%20/g" <<<"$LOC")

content=$(curl -sS "https://thisdavej.azurewebsites.net/api/weather/forecast?loc=$LOCATION&deg=$DEG")

echo $content | jq -r '.[] | "\(.shortday) \(.low)-\(.high)°\(.degType) \(.skytextday) ~ Rain: \(.precip)% \(if (.precip | tonumber > 20) then "(Bring umbrella!)" else "" end)"'

In this final script, we use the jq if-then-else conditional to display (Bring umbrella!) if the chance of rain is greater than 20 percent. Here are the results:

Tue 53-67°F Clear ~ Rain: 0%
Wed 54-67°F Mostly Cloudy ~ Rain: 0%
Thu 52-75°F Sunny ~ Rain: 0%
Fri 53-70°F Sunny ~ Rain: 0%
Sat 58-65°F Partly Sunny ~ Rain: 80% (Bring umbrella!)

The command line capability of the jq command is downright amazing!

You may also notice that we added a final finishing touch to our script and included some logic at the top to ensure that at least one parameter is supplied. If not, command usage text is provided. Let’s see this in action by invoking our command with a parameter of “San Diego, CA” and without the parameter.

$ ./weather.sh "San Diego, CA"
Tue 53-67°F Clear ~ Rain: 0%
Wed 54-67°F Mostly Cloudy ~ Rain: 0%
Thu 52-75°F Sunny ~ Rain: 0%
Fri 53-70°F Sunny ~ Rain: 0%
Sat 58-65°F Partly Sunny ~ Rain: 80% (Bring umbrella!)
$
$ ./weather.sh
Usage: weather11.sh location [degreeType]
Example: weather11.sh "San Diego, CA" F
$

Our script is now complete and ready for production use. 🙂

You could, for example, add the following line to your .profile file to run your weather script every time you log into your system from ssh:

~/scripts/weather.sh "San Diego, CA"

Please note that this will provide you with a weather forecast every time you log in via ssh, but it would not appear when launching terminal sessions through the Linux desktop environment.

Conclusion

Who would have thought we had so much power to consume JSON Web API data from the bash command line with the help of both curl and jq? There are many more features of jq that we have not even had time to cover. You can check out a follow-up article I wrote called Using jq and curl to Wrangle JSON Arrays from the Terminal and also the jq Manual to learn about many of jq’s other features. We’ll be back again next time for more fun consuming JSON Web API data!

Follow @thisDaveJ (Dave Johnson) on Twitter to stay up to date with the latest tutorials and tech articles.

Additional articles

Creating Node.js Microservices with Ease Using Stdlib
Consuming Node.js Microservices Created with Stdlib
Node.js: Sending Email Notifications Using Nodemailer and Gmail
Getting Started with YAML in Node.js using js-yaml