Easy methods to Parse JSON Information on the Linux Command Line with jq

0
4


A terminal prompt on a Linux PC.
Fatmawati Achmad Zaenuri/Shutterstock

JSON is without doubt one of the hottest codecs for transferring text-based knowledge across the internet. It’s in every single place, and also you’re certain to come back throughout it. We’ll present you the best way to deal with it from the Linux command line utilizing the jq command.

JSON and jq

JSON stands for JavaScript Object Notation. It’s a scheme that enables knowledge to be encoded into plain textual content recordsdata, in a self-describing approach. There are not any feedback in a JSON file—the contents ought to be self-explanatory. Every knowledge worth has a textual content string referred to as a “title” or “key.” This tells you what the information worth is. Collectively, they’re often called title:worth pairs, or key:worth pairs. A colon (:) separates a key from its worth.

An “object” is a set of key:worth pairs. In a JSON file, an object begins with an open curly brace ({) and ends with a closing brace (}). JSON additionally helps “arrays,” that are ordered lists of values. An array begins with a gap bracket ([) and ends with a closing one (]).

From these easy definitions, in fact, arbitrary complexity can come up. For instance, objects may be nested inside objects. Objects can include arrays, and arrays can even include objects. All of which might have open-ended ranges of nesting.

In follow, although, if the structure of JSON knowledge is convoluted, the design of the information structure ought to in all probability use a rethink. After all, for those who’re not producing the JSON knowledge, simply attempting to make use of it, you don’t have any say in its structure. In these circumstances, sadly, you simply should take care of it.

Most programming languages have libraries or modules that permit them to parse JSON knowledge. Sadly, the Bash shell has no such functionality.

Necessity being the mom of invention, although, the jq utility was born! With jq, we will easily parse JSON within the Bash shell. And it doesn’t matter whether or not you must work with well-engineered, elegant JSON, or the stuff nightmares are fabricated from.

Easy methods to Set up jq

We needed to set up jq on all of the Linux distributions we used to analysis this text.

To put in jq on Ubuntu sort this command:

sudo apt-get set up jq

The "sudo apt-get install jq" command in a terminal window.

To put in jq on Fedora, sort this command:

sudo dnf set up jq

The "sudo dnf install jq" command in a terminal window.

To put in jq on Manjaro, sort this command:

sudo pacman -Sy jq

The "sudo pacman -Sy jq" command in a terminal window.

Easy methods to Make JSON Readable

JSON doesn’t care about white area, and structure doesn’t have an effect on it. So long as it follows the rules of JSON grammar, programs that course of JSON can learn and understood it. Due to this, JSON is commonly transmitted as a easy, lengthy string, as a right of structure. This protects a little bit of area as a result of tabs, areas, and new-line characters don’t should be included within the JSON. After all, the draw back to all that is when a human tries to learn it.

Let’s pull a brief JSON object from the NASA web site that tells us the position of the International Space Station. We’ll use curl, which might download files to retrieve the JSON object for us.

We don’t care about any of the standing messages curl normally generates, so we’ll sort the next, utilizing the -s (silent) choice:

curl -s http://api.open-notify.org/iss-now.json

The "curl -s http://api.open-notify.org/iss-now.json" command in a terminal window.

Now, with a little bit of effort, you’ll be able to learn this. It’s a must to pick the information values, but it surely isn’t straightforward or handy. Let’s repeat this, however this time we’ll pipe it by jq.

jq makes use of filters to parse JSON, and the only of those filters is a interval (.), which implies “print your entire object.” By default, jq pretty-prints the output.

We put all of it collectively and sort the next:

curl -s http://api.open-notify.org/iss-now.json | jq .

The "curl -s http://api.open-notify.org/iss-now.json | jq ." command in a terminal window.

That’s a lot better! Now, we will see precisely what’s occurring.

The whole object is wrapped in curly braces. It incorporates two key:title pairs: message and timestamp. It additionally incorporates an object referred to as iss_position, which incorporates two key:worth pairs: longitude and latitude.

We’ll do this as soon as extra. This time we’ll sort the next, and redirect the output right into a file referred to as “iss.json”:

curl -s http://api.open-notify.org/iss-now.json | jq . > iss.json
cat iss.json

The "curl -s http://api.open-notify.org/iss-now.json | jq . > iss.json" and "cat iss.json" commands in a terminal window.

This provides us a properly laid out copy of the JSON object on our laborious drive.

RELATED: How to Use curl to Download Files From the Linux Command Line

Accessing Information Values

As we noticed above, jq can extract knowledge values being piped by from JSON. It will probably additionally work with JSON saved in a file. We’re going to work with native recordsdata so the command line isn’t cluttered with curl instructions. This could make it a bit simpler to comply with.

The best option to extract knowledge from a JSON file is to offer a key title to acquire its knowledge worth. Sort a interval and the important thing title and not using a area between them. This creates a filter from the important thing title. We additionally want to inform jq which JSON file to make use of.

We sort the next to retrieve the message worth:

jq .message iss.json

The "jq .message iss.json" command in a terminal window.

jq prints the textual content of the message worth within the terminal window.

When you’ve got a key title that features areas or punctuation, you must wrap its filter in citation marks. Care is normally taken to make use of characters, numbers, and underscores solely so the JSON key names usually are not problematic.

First, we sort the next to retrieve the timestamp worth:

jq .timestamp iss.json

The "jq .timestamp iss.json" command in a terminal window.

The timestamp worth is retrieved and printed within the terminal window.

However how can we entry the values contained in the iss_position object? We are able to use the JSON dot notation. We’ll embody the iss_position object title within the “path” to the important thing worth. To do that, the title of the item the hot button is inside will precede the title of the important thing itself.

We sort the next, together with the latitude key title (be aware there are not any areas between “.iss_position” and “.latitude”):

jq .iss_position.latitude iss.json

The "jq .iss_position.latitude iss.json" command in a terminal window.

To extract a number of values, you must do the next:

  • Listing the important thing names on the command line.
  • Separate them with commas (,).
  • Enclose them in citation marks (") or apostrophes (').

With that in thoughts, we sort the next:

jq ".iss_position.latitude, .timestamp" iss.json

The "jq ".iss_position.latitude, .timestamp" iss.json" command in a terminal window.

The 2 values print to the terminal window.

Working with Arrays

Let’s seize a unique JSON object from NASA.

This time, we’ll use a list of the astronauts who are in space right now:

curl -s http://api.open-notify.org/astros.json

The "curl -s http://api.open-notify.org/astros.json" command in a terminal window.

Okay, that labored, so let’s do it once more.

We’ll sort the next to pipe it by jq and redirect it to a file referred to as “astro.json”:

curl -s http://api.open-notify.org/astros.json | jq . > astro.json

The "curl -s http://api.open-notify.org/astros.json | jq . > astros.json" command in a terminal window.

Now let’s sort the next to verify our file:

much less astro.json

The "less astros.json" command in a terminal window.

As proven under, we now see the checklist of astronauts in area, in addition to their spacecrafts.

Output from "less astros.json" in a terminal window.

This JSON object incorporates an array referred to as folks. We all know it’s an array due to the opening bracket ([) (highlighted within the screenshot above). It’s an array of objects that every include two key:worth pairs:  title and craft.

Like we did earlier, we will use the JSON dot notation to entry the values. We should additionally embody the brackets ([]) within the title of the array.

With all that in thoughts, we sort the next:

jq ".folks[].title" astro.json

The "jq ".people[].name" astros.json" command in a terminal window.

This time, all of the title values print to the terminal window. What we requested jq to do was print the title worth for each object within the array. Fairly neat, huh?

We are able to retrieve the title of a single object if we put its place within the array within the brackets ([]) on the command line. The array makes use of zero-offset indexing, that means the item within the first place of the array is zero.

To entry the final object within the array you should use -1; to get the second to final object within the array, you should use -2, and so forth.

Generally, the JSON object supplies the variety of components within the array, which is the case with this one. Together with the array, it incorporates a key:title pair referred to as quantity with a price of six.

The next variety of objects are on this array:

jq ".folks[1].title" astro.json
jq ".folks[3].title" astro.json
jq ".folks[-1].title" astro.json
jq ".folks[-2].title" astro.json

The "jq ".people[1].name" astro.json," "jq ".people[3].name" astro.json," "jq ".people[-1].name" astro.json," and "jq ".people[-2].name" astro.json" in a terminal window.

You can even present a begin and finish object inside the array. That is referred to as “slicing,” and it may be just a little complicated. Bear in mind the array makes use of a zero-offset.

To retrieve the objects from index place two, as much as (however not together with) the item at index place 4, we sort the next command:

jq ".folks[2:4]" astro.json

The "jq ".people[2:4]" astro.json" command in a terminal window.

This prints the objects at array index two (the third object within the array) and three (the fourth object within the array). It stops processing at array index 4, which is the fifth object within the array.

The way in which to higher perceive that is to experiment on the command line. You’ll quickly see the way it works.

Easy methods to Use Pipes with Filters

You possibly can pipe the output from one filter to a different, and also you don’t should be taught a brand new image. The identical because the Linux command line, jq makes use of the vertical bar (|) to characterize a pipe.

We’ll inform jq to pipe the folks array into the .title filter, which ought to checklist the names of the astronauts within the terminal window.

We sort the next:

jq ".folks[] | .title" astro.json

The "jq ".people[] | .name" astros.json" command in a terminal window.

RELATED: How to Use Pipes on Linux

Creating Arrays and Modifying Outcomes

We are able to use jq to create new objects, equivalent to arrays. On this instance, we’ll extract three values and create a brand new array that incorporates these values. Be aware the opening ([) and shutting brackets (]) are additionally the primary and final characters within the filter string.

We sort the next:

jq "[.iss-position.latitude, iss_position.longitude, .timestamp]" iss.json

The "jq "[.iss-position.latitude, iss_position.longitude, .timestamp]" iss.json" command in a terminal window.

The output is wrapped in brackets and separated by commas, making it a accurately fashioned array.

Numeric values may also be manipulated as they’re retrieved. Let’s pull the timestamp from the ISS place file, after which extract it once more and alter the worth that’s returned.

To take action, we sort the next:

jq ".timestamp" iss.json
jq ".timestamp - 1570000000" iss.json

The "jq ".timestamp" iss.json" and "jq ".timestamp - 1570000000" iss.json" commands in a terminal window.

That is helpful if you must add or take away an ordinary offset from an array of values.

Let’s sort the next to remind ourselves what the iss.json file incorporates:

jq . iss.json

The "jq . iss.json" command in a terminal window.

Let’s say we wish to eliminate the message key:worth pair. It doesn’t have something to do with the place of the Worldwide House Station. It’s only a flag that signifies the situation was retrieved efficiently. If it’s surplus to necessities, we will dispense with it. (You could possibly additionally simply ignore it.)

We are able to use jq‘s delete perform, del(), to delete a key:worth pair. To delete the message key:worth pair, we sort this command:

jq "del(.message)" iss.json

The "jq "del(.message)" iss.json" command in a terminal window.

Be aware this doesn’t truly delete it from the “iss.json” file; it simply removes it from the output of the command. If you must create a brand new file with out the message key:worth pair in it, run the command, after which redirect the output into a brand new file.

Extra Difficult JSON Objects

Let’s retrieve some extra NASA knowledge. This time, we’ll use a JSON object that incorporates information on meteor impact sites from around the globe. This can be a larger file with a much more sophisticated JSON construction than these we’ve handled beforehand.

First, we’ll sort the next to redirect it to a file referred to as “strikes.json”:

curl -s https://knowledge.nasa.gov/useful resource/y77d-th95.json | jq . > strikes.json

The "curl -s https://data.nasa.gov/resource/y77d-th95.json | jq . > strikes.json" command in a terminal window.

To see what JSON appears to be like like, we sort the next:

much less strikes.json

The "less strikes.json" command in less in a terminal window.

As proven under, the file begins with a gap bracket ([), so your entire object is an array. The objects within the array are collections of key:worth pairs, and there’s a nested object referred to as geolocation. The geolocation object incorporates additional key:worth pairs, and an array referred to as coordinates.

Output from the "less strikes.json" command in less in a terminal window.

Let’s retrieve the names of the meteor strikes from the item at index place 995 by the top of the array.

We’ll sort the next to pipe the JSON by three filters:

jq ".[995:] | .[] | .title" strikes.json

The "jq ".[995:] | .[] | .name" strikes.json" command in a terminal window.

The filters perform within the following methods:

  • .[995:]: This tells jq to course of the objects from array index 995 by the top of the array. No quantity after the colon ( : ) is what tells jq to proceed to the top of the array.
  • .[]: This array iterator tells jq to course of every object within the array.
  • .title: This filter extracts the title worth.

With a slight change, we will extract the final 10 objects from the array. A “-10” instructs jq to begin processing objects 10 again from the top of the array.

We sort the next:

jq ".[-10:] | .[] | .title" strikes.json

The "jq ".[-10:] | .[] | .name" strikes.json" command in a terminal window.

Simply as we have now in earlier examples, we will sort the next to pick out a single object:

jq ".[650].title" strikes.json

The "jq ".[650].name" strikes.json" command in a terminal window.

We are able to additionally apply slicing to strings. To take action, we’ll sort the next to request the primary 4 characters of the title of the item at array index 234:

jq ".[234].title[0:4]" strikes.json

The "jq ".[234].name[0:4]" strikes.json" command in a terminal window.

We are able to additionally see a particular object in its entirety. To do that, we sort the next and embody an array index with none key:worth filters:

jq ".[234]" strikes.json

The "jq ".[234]" strikes.json" command in a terminal window.

If you wish to see solely the values, you are able to do the identical factor with out the important thing names.

For our instance, we sort this command:

jq ".[234][]" strikes.json

The "jq "[234][]" strikes.json" command in a terminal window.

To retrieve a number of values from every object, we separate them with commas within the following command:

jq ".[450:455] | .[] | .title, .mass" strikes.json

The "jq ".[450:455] | .[] | .name, .mass" strikes.json" command in a terminal window.

If you wish to retrieve nested values, you must establish the objects that kind the “path” to them.

For instance, to reference the coordinates values, we have now to incorporate the all-encompassing array, the geolocation nested object, and the nested coordinates array, as proven under.

The path to array in a nested JSON object highlighted in a terminal window.

To see the coordinates values for the item at index place 121 of the array, we sort the next command:

jq ".[121].geolocation.coordinates[]" strikes.json

The "jq ".[121].geolocation.coordinates[]" strikes.json" command in a terminal window.

The size Perform

The jq size perform offers totally different metrics based on what it’s been utilized, equivalent to:

  • Strings: The size of the string in bytes.
  • Objects: The variety of key:worth pairs within the object.
  • Arrays: The variety of array components within the array.

The next command returns the size of the title worth in 10 of the objects within the JSON array, beginning at index place 100:

jq ".[100:110] | .[].title | size" strikes.json

The "jq ".[100:110] | .[].name | length" strikes.json" command in a terminal window.

To see what number of key:worth pairs are within the first object within the array, we sort this command:

jq ".[0] | size" strikes.json

The "jq ".[0] | length" strikes.json" command in a terminal window.

The keys Perform

You should use the keys perform to seek out out concerning the JSON you’ve started working with. It will probably inform you what the names of the keys are, and what number of objects there are in an array.

To search out the keys within the folks object within the “astro.json” file, we sort this command:

jq ".folks.[0] | keys" astro.json

The "jq ".people.[0] | keys" astro.json" command in a terminal window.

To see what number of components are within the folks array, we sort this command:

jq ".folks | keys" astro.json

The "jq ".people | keys" astro.json" command in a terminal window.

This reveals there are six, zero-offset array components, numbered zero to 5.

The has() Perform

You should use the has() perform to interrogate the JSON and see whether or not an object has a selected key title. Be aware the important thing title have to be wrapped in citation marks. We’ll wrap the filter command in single quotes ('), as follows:

jq '.[] | has("nametype")' strikes.json

The "jq '.[] | has("nametype")' strikes.json" in a terminal window.

Every object within the array is checked, as proven under.

Output from the "jq '.[] | has("nametype")' strikes.json" command in a terminal window.

If you wish to verify a particular object, you embody its index place within the array filter, as follows:

jq '.[678] | has("nametype")' strikes.json

The "jq '.[678] | has("nametype")' strikes.json" command in a terminal window.

Don’t Go Close to JSON With out It

The jq utility is the proper instance of the skilled, highly effective, quick software program that makes dwelling within the Linux world such a pleasure.

This was only a temporary introduction to the frequent capabilities of this command—there’s an entire lot extra to it. Remember to try the excellent jq manual if you wish to dig deeper.





Source link

LEAVE A REPLY

Please enter your comment!
Please enter your name here