jq Command-Line JSON Recipes: Filter, Transform, Validate, and Compare API Data
jqJSONCommand LineAPIsShell ScriptingData Processing

jq Command-Line JSON Recipes: Filter, Transform, Validate, and Compare API Data

CCodeNScripts Editorial Team
2026-11-20
8 min read

Practical jq recipes for filtering, transforming, validating, exporting, and comparing JSON from APIs, logs, and recurring checks.

jq turns JSON inspection into a repeatable command-line workflow. This guide provides copy-ready recipes for reading API responses and log files, selecting and filtering fields, transforming nested data, validating assumptions, exporting CSV, and comparing responses safely. Keep it nearby when debugging an integration, preparing test fixtures, or checking how recurring API data changes over time.

Overview

jq is a command-line JSON processor designed for composing small filters in shell pipelines. It reads JSON from a file or standard input, applies an expression, and writes JSON or formatted text to standard output. Unlike searching a response with a text editor, a jq filter understands arrays, objects, strings, numbers, booleans, and null values.

Most jq workflows have the same shape:

command-that-returns-json | jq 'filter'

For example, this parses an API response and prints only its name field:

curl -sS https://api.example.test/users/42 | jq -r '.name'

The -r option emits raw strings without JSON quotes. Leave it out when you want valid JSON output for another program. The -sS options tell curl to run quietly while still showing errors; the same request patterns are covered in these curl API testing recipes.

Install jq through the package manager used by your operating system, or use the official distribution method for your environment. Verify the executable before building automation around it:

jq --version

Keep shell quoting in mind. In POSIX shells, single quotes around a jq filter prevent the shell from expanding characters such as $, *, and parentheses. If a filter must contain a shell variable, use a jq argument rather than interpolating untrusted text directly:

key="email"
jq --arg key "$key" '.[$key]' response.json

On Windows, PowerShell and Command Prompt have different quoting rules. Test a filter interactively before placing it in a script, and keep platform-specific examples in separate scripts when a team uses more than one shell.

What to track

A useful jq reference is less about memorizing every function and more about tracking the transformations you repeat. The following syntax covers the common path from raw API response to a stable, reviewable result.

Taskjq filterExample use
Select a field.user.emailRead a nested value
Select array items.items[]Process each item
Filter an array[.items[] | select(.active == true)]Keep matching objects
Provide a fallback.region // "unknown"Handle null or missing data
Build an object{id: .id, label: .name}Shape a smaller response
Format a value"\(.method) \(.path)"Create a readable line
Sort valuessort_by(.created_at)Order objects by a field
Count valueslengthCount an array or string

Selecting and filtering JSON

Given an object containing an items array, print the identifiers of enabled items with:

jq -r '.items[] | select(.enabled == true) | .id' response.json

Use a comparison appropriate to the data type. A numeric status code should be compared as a number, while a string status such as "ready" needs quotes:

jq '.items[] | select(.status == "ready" and .retry_count < 3)' response.json

When you need to preserve the result as one JSON array, wrap the iterator:

jq '[.items[] | select(.enabled == true)]' response.json

Handling missing fields and null values

API responses often differ between records or environments. The alternative operator, //, supplies a fallback for missing or null values:

jq '{id, owner: (.owner.name // "unassigned"), tags: (.tags // [])}' response.json

Use has("field") when the difference between a missing field and an explicitly null field matters:

jq 'if has("metadata") then .metadata else {} end' response.json

For optional nested paths, the optional operator can prevent a missing path from stopping the useful part of a pipeline:

jq '.account?.profile?.timezone // "UTC"' response.json

Transforming and flattening objects

Object construction is the foundation of reusable jq transformations. This filter keeps only fields needed by a downstream script:

jq '[.users[] | {id, email, role: (.role // "member")] ]' response.json

To flatten nested records into lines, combine interpolation with raw output:

jq -r '.events[] | "\(.timestamp)\t\(.type)\t\(.actor.id)"' response.json

For a nested array such as users with projects, map can transform every element while keeping the array structure:

jq '.users | map({id, projects: [.projects[] | .name]})' response.json

Grouping and counting

Sort before using group_by, because grouping expects equal keys to be adjacent:

jq '[.requests[]]
  | sort_by(.status)
  | group_by(.status)
  | map({status: .[0].status, count: length})' access.json

This produces a compact summary that is easier to compare during a recurring API or log review.

Converting JSON to CSV

CSV export works best when every row has the same columns. Add a header explicitly, then emit each row with @csv:

jq -r '(["id", "email", "status"] | @csv),
  (.users[] | [.id, .email, .status] | @csv)' response.json > users.csv

@csv handles quoting for commas and other special characters. Avoid manually joining fields with commas, because that produces invalid rows when a value contains a comma or newline.

Cadence and checkpoints

Use jq as a lightweight checkpoint tool whenever a response, fixture, or log shape is likely to change. A monthly or quarterly review is practical for stable integrations; review sooner when an API version, authentication flow, pagination rule, or deployment environment changes.

Inspect a response before writing a filter

Start with structure rather than guessing field paths:

jq 'keys' response.json
jq '.items[0] | keys' response.json
jq 'type' response.json

Pretty-printed output is jq's default, so it is suitable for a quick visual check. Use jq -C only when color is useful in an interactive terminal; avoid color in files or machine-readable pipelines.

Validate expected structure

Filtering can hide a broken assumption by returning no rows. Add explicit checks before a recurring job depends on the result:

jq -e '
  (.items | type == "array") and
  (all(.items[]; has("id") and (.id | type == "string")))
' response.json > /dev/null

The -e option makes jq's exit status reflect whether the final result is true, false, or unavailable. The exact status details are less important than treating a nonzero result as a checkpoint failure and recording it in the surrounding script. For more formal contracts, pair jq inspection with JSON Schema validation; this JSON Schema guide shows how to enforce rules in JavaScript and Python.

Compare normalized responses

Raw API responses often contain changing timestamps, ordering, request identifiers, or generated metadata. Normalize the fields that matter before comparing:

jq -S '{status, items: [.items[] | {id, name, enabled}]}' response-a.json > normalized-a.json
jq -S '{status, items: [.items[] | {id, name, enabled}]}' response-b.json > normalized-b.json
diff -u normalized-a.json normalized-b.json

The -S option sorts object keys, which makes textual comparison more stable. Sorting keys does not sort arrays; sort an array deliberately when order is not meaningful:

jq -S '.items |= sort_by(.id)' response.json

Do not normalize away fields that are operationally important. A change in item order may matter to a consumer, and a timestamp may be the exact signal you are tracking.

How to interpret changes

A changed jq result is a prompt to investigate, not proof of a defect. First decide whether the difference is structural, semantic, environmental, or expected.

  • Structural change: a field moved, changed type, disappeared, or became an array. Check the producer's response contract and update the filter only after confirming the intended shape.
  • Semantic change: the field remains present but values or meanings changed, such as a new status value or altered pagination behavior.
  • Environmental change: development and production return different records, permissions, feature flags, or defaults.
  • Expected change: counts, timestamps, inventory, or event data changed as part of normal operation.

Separate these categories by tracking both a compact summary and a raw sample. For example:

jq '{count: (.items | length), statuses: ([.items[].status] | sort | unique)}' response.json

If the count changes but the schema summary is stable, the result may simply reflect new data. If a status appears that your script does not recognize, treat it as a compatibility checkpoint. If a field changes from a number to a string, fail clearly rather than allowing an implicit conversion to spread through later steps.

When jq is part of a shell script, preserve errors and make temporary output safe. Write to a temporary file, validate the result, then replace the destination only after success. The patterns in Shell Script Reliability Patterns are useful for exit codes, cleanup, logging, and argument handling.

Also keep secrets out of captured output. API tokens, cookies, personal data, and authorization headers should not be committed as fixtures or printed into CI logs. For automated checks, use sanitized responses that retain the shape and edge cases your filter needs.

When to revisit

Revisit this jq workflow on a monthly or quarterly cadence for integrations that change gradually, and immediately after any event that can alter response data or script behavior:

  • an API version, endpoint, pagination model, or authentication method changes;
  • a deployment changes environment variables, permissions, feature flags, or shell runtime;
  • a scheduled job reports an empty result, a nonzero exit code, or an unexpected type;
  • a log format or event producer adds, removes, or renames fields;
  • a test fixture no longer represents production responses; or
  • a downstream consumer starts requiring a new field or output format.

At each checkpoint, save one sanitized representative response, rerun the structure checks, compare the normalized summary, and review filters that use optional fields. If the command is used in CI, run it against a small fixture as well as a current sample. A GitHub Actions workflow can make these checks repeatable; see the guide to CI workflows for JavaScript, TypeScript, and Python projects.

For a practical starting routine, create a directory containing fixtures/, filters/, and a short README that records the expected input shape. Put complex filters in files and invoke them with jq -f filters/summary.jq response.json so they can be reviewed and tested. Keep one command for inspection, one for validation, and one for the final transformation. This separation makes failures easier to diagnose than a single opaque pipeline.

Finally, rerun the smallest useful command before expanding it. Start with jq '.items[0]', confirm the path, add select, then add the output shape or export format. That incremental approach turns jq from a one-off JSON formatter into a dependable reusable script for API debugging, data comparison, and recurring operational checks.

Related Topics

#jq#JSON#Command Line#APIs#Shell Scripting#Data Processing
C

CodeNScripts Editorial Team

Developer Tools Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.