As test automation engineers we often touch many different tools, protocols, data, and systems. Whatever we need to get the job done. Wrangling data can be quite burdensome, for example when preparing data for a next test or to verify expectations against data retrieved. Data can be in the wrong format, have additional unwanted fields, lack fields, or the receiving system expects a different structure.

Nowadays a lot of the data that we deal with is JSON. Simple enough to deal with and widely supported by tons of tools. It gets annoying when some of the issues above pop-up and you want to change the data for a lot of tests, or it’s just a lot of data.

Here is where jq can help. Jq is an open source command line tool, that makes it easy to deal with json in all kinds of scenarios. Below I’m sharing a few of the most useful applications I’ve come across for jq.

For the examples in this article I’m using data and endpoints from https://fakestoreapi.com/.

Pretty print & color json

Often systems return minified json, without white space and line breaks. While a bit more efficient to transmit, it makes it a lot harder to read for humans. Jq can help. Given a piece of json on your clipboard simply run:

pbpaste | jq

If you pass in JSON to the jq command it will automatically pretty-print and color the input. Nice!

Select only one field from an array of objects

https://fakestoreapi.com/users returns a list of users and their details. To grab only the address fields of each entry:

curl https://fakestoreapi.com/users | jq '.[].address'

Slice the data

In order to grab the first 3 elements from an array you’d run:

curl https://fakestoreapi.com/users | jq '.[0:3]'

Construct a new object

Sometimes you want to shuffle the data you have into another structure. jq again makes this easy.

 curl https://fakestoreapi.com/users | jq '.[] | {user: "\(.name.firstname) \(.name.lastname)", phone: .phone}'

Count

Calling the length function gets you the number of items in an array or the number of key-value pairs in an object. Very useful if you want to run some quick verifications on the data.

curl https://fakestoreapi.com/users | jq length

Get a list of all users whose e-mail address contains ‘w’

curl https://fakestoreapi.com/users|jq 'map(select(.email | contains("w")))'

Count all users with a ‘w’ in their e-mail address

curl https://fakestoreapi.com/users|jq 'map(select(.email | contains("w")))' | jq length

There’s a lot more you can do with jq, check out the documentation for much, much more you can do with it.

Learn the fundamentals of Test Automation, in my course on Pluralsight- Test Automation: The Big Picture(affiliate link).