jq turns JSON inspection into a repeatable command-line workflow. This guide provides copy-ready recipes for reading API responses and log files, selecting and filtering fields, transforming nested data, validating assumptions, exporting CSV, and comparing responses safely. Keep it nearby when debugging an integration, preparing test fixtures, or checking how recurring API data changes over time.
Overview
jq is a command-line JSON processor designed for composing small filters in shell pipelines. It reads JSON from a file or standard input, applies an expression, and writes JSON or formatted text to standard output. Unlike searching a response with a text editor, a jq filter understands arrays, objects, strings, numbers, booleans, and null values.
Most jq workflows have the same shape:
command-that-returns-json | jq 'filter'
For example, this parses an API response and prints only its name field:
curl -sS https://api.example.test/users/42 | jq -r '.name'
The -r option emits raw strings without JSON quotes. Leave it out when you want valid JSON output for another program. The -sS options tell curl to run quietly while still showing errors; the same request patterns are covered in these curl API testing recipes.
Install jq through the package manager used by your operating system, or use the official distribution method for your environment. Verify the executable before building automation around it:
jq --version
Keep shell quoting in mind. In POSIX shells, single quotes around a jq filter prevent the shell from expanding characters such as $, *, and parentheses. If a filter must contain a shell variable, use a jq argument rather than interpolating untrusted text directly:
key="email"
jq --arg key "$key" '.[$key]' response.json
On Windows, PowerShell and Command Prompt have different quoting rules. Test a filter interactively before placing it in a script, and keep platform-specific examples in separate scripts when a team uses more than one shell.
What to track
A useful jq reference is less about memorizing every function and more about tracking the transformations you repeat. The following syntax covers the common path from raw API response to a stable, reviewable result.
| Task | jq filter | Example use |
|---|---|---|
| Select a field | .user.email | Read a nested value |
| Select array items | .items[] | Process each item |
| Filter an array | [.items[] | select(.active == true)] | Keep matching objects |
| Provide a fallback | .region // "unknown" | Handle null or missing data |
| Build an object | {id: .id, label: .name} | Shape a smaller response |
| Format a value | "\(.method) \(.path)" | Create a readable line |
| Sort values | sort_by(.created_at) | Order objects by a field |
| Count values | length | Count an array or string |
Selecting and filtering JSON
Given an object containing an items array, print the identifiers of enabled items with:
jq -r '.items[] | select(.enabled == true) | .id' response.json
Use a comparison appropriate to the data type. A numeric status code should be compared as a number, while a string status such as "ready" needs quotes:
jq '.items[] | select(.status == "ready" and .retry_count < 3)' response.json
When you need to preserve the result as one JSON array, wrap the iterator:
jq '[.items[] | select(.enabled == true)]' response.json
Handling missing fields and null values
API responses often differ between records or environments. The alternative operator, //, supplies a fallback for missing or null values:
jq '{id, owner: (.owner.name // "unassigned"), tags: (.tags // [])}' response.json
Use has("field") when the difference between a missing field and an explicitly null field matters:
jq 'if has("metadata") then .metadata else {} end' response.json
For optional nested paths, the optional operator can prevent a missing path from stopping the useful part of a pipeline:
jq '.account?.profile?.timezone // "UTC"' response.json
Transforming and flattening objects
Object construction is the foundation of reusable jq transformations. This filter keeps only fields needed by a downstream script:
jq '[.users[] | {id, email, role: (.role // "member")] ]' response.json
To flatten nested records into lines, combine interpolation with raw output:
jq -r '.events[] | "\(.timestamp)\t\(.type)\t\(.actor.id)"' response.json
For a nested array such as users with projects, map can transform every element while keeping the array structure:
jq '.users | map({id, projects: [.projects[] | .name]})' response.json
Grouping and counting
Sort before using group_by, because grouping expects equal keys to be adjacent:
jq '[.requests[]]
| sort_by(.status)
| group_by(.status)
| map({status: .[0].status, count: length})' access.json
This produces a compact summary that is easier to compare during a recurring API or log review.
Converting JSON to CSV
CSV export works best when every row has the same columns. Add a header explicitly, then emit each row with @csv:
jq -r '(["id", "email", "status"] | @csv),
(.users[] | [.id, .email, .status] | @csv)' response.json > users.csv
@csv handles quoting for commas and other special characters. Avoid manually joining fields with commas, because that produces invalid rows when a value contains a comma or newline.
Cadence and checkpoints
Use jq as a lightweight checkpoint tool whenever a response, fixture, or log shape is likely to change. A monthly or quarterly review is practical for stable integrations; review sooner when an API version, authentication flow, pagination rule, or deployment environment changes.
Inspect a response before writing a filter
Start with structure rather than guessing field paths:
jq 'keys' response.json
jq '.items[0] | keys' response.json
jq 'type' response.json
Pretty-printed output is jq's default, so it is suitable for a quick visual check. Use jq -C only when color is useful in an interactive terminal; avoid color in files or machine-readable pipelines.
Validate expected structure
Filtering can hide a broken assumption by returning no rows. Add explicit checks before a recurring job depends on the result:
jq -e '
(.items | type == "array") and
(all(.items[]; has("id") and (.id | type == "string")))
' response.json > /dev/null
The -e option makes jq's exit status reflect whether the final result is true, false, or unavailable. The exact status details are less important than treating a nonzero result as a checkpoint failure and recording it in the surrounding script. For more formal contracts, pair jq inspection with JSON Schema validation; this JSON Schema guide shows how to enforce rules in JavaScript and Python.
Compare normalized responses
Raw API responses often contain changing timestamps, ordering, request identifiers, or generated metadata. Normalize the fields that matter before comparing:
jq -S '{status, items: [.items[] | {id, name, enabled}]}' response-a.json > normalized-a.json
jq -S '{status, items: [.items[] | {id, name, enabled}]}' response-b.json > normalized-b.json
diff -u normalized-a.json normalized-b.json
The -S option sorts object keys, which makes textual comparison more stable. Sorting keys does not sort arrays; sort an array deliberately when order is not meaningful:
jq -S '.items |= sort_by(.id)' response.json
Do not normalize away fields that are operationally important. A change in item order may matter to a consumer, and a timestamp may be the exact signal you are tracking.
How to interpret changes
A changed jq result is a prompt to investigate, not proof of a defect. First decide whether the difference is structural, semantic, environmental, or expected.
- Structural change: a field moved, changed type, disappeared, or became an array. Check the producer's response contract and update the filter only after confirming the intended shape.
- Semantic change: the field remains present but values or meanings changed, such as a new status value or altered pagination behavior.
- Environmental change: development and production return different records, permissions, feature flags, or defaults.
- Expected change: counts, timestamps, inventory, or event data changed as part of normal operation.
Separate these categories by tracking both a compact summary and a raw sample. For example:
jq '{count: (.items | length), statuses: ([.items[].status] | sort | unique)}' response.json
If the count changes but the schema summary is stable, the result may simply reflect new data. If a status appears that your script does not recognize, treat it as a compatibility checkpoint. If a field changes from a number to a string, fail clearly rather than allowing an implicit conversion to spread through later steps.
When jq is part of a shell script, preserve errors and make temporary output safe. Write to a temporary file, validate the result, then replace the destination only after success. The patterns in Shell Script Reliability Patterns are useful for exit codes, cleanup, logging, and argument handling.
Also keep secrets out of captured output. API tokens, cookies, personal data, and authorization headers should not be committed as fixtures or printed into CI logs. For automated checks, use sanitized responses that retain the shape and edge cases your filter needs.
When to revisit
Revisit this jq workflow on a monthly or quarterly cadence for integrations that change gradually, and immediately after any event that can alter response data or script behavior:
- an API version, endpoint, pagination model, or authentication method changes;
- a deployment changes environment variables, permissions, feature flags, or shell runtime;
- a scheduled job reports an empty result, a nonzero exit code, or an unexpected type;
- a log format or event producer adds, removes, or renames fields;
- a test fixture no longer represents production responses; or
- a downstream consumer starts requiring a new field or output format.
At each checkpoint, save one sanitized representative response, rerun the structure checks, compare the normalized summary, and review filters that use optional fields. If the command is used in CI, run it against a small fixture as well as a current sample. A GitHub Actions workflow can make these checks repeatable; see the guide to CI workflows for JavaScript, TypeScript, and Python projects.
For a practical starting routine, create a directory containing fixtures/, filters/, and a short README that records the expected input shape. Put complex filters in files and invoke them with jq -f filters/summary.jq response.json so they can be reviewed and tested. Keep one command for inspection, one for validation, and one for the final transformation. This separation makes failures easier to diagnose than a single opaque pipeline.
Finally, rerun the smallest useful command before expanding it. Start with jq '.items[0]', confirm the path, add select, then add the output shape or export format. That incremental approach turns jq from a one-off JSON formatter into a dependable reusable script for API debugging, data comparison, and recurring operational checks.