CI/CD and automation

Using a reconciliation as a release gate, and the credentials an unattended caller needs

A reconciliation is a fact about your data, which makes it a good thing to block a release on. This page covers wiring one into a pipeline, and the credential constraints that apply when nobody is at the keyboard.

The shape of a gate

synq-recon trigger <suite-id> --wait --fail-on mismatched,failed

--wait blocks until the run completes and exits with a code your pipeline can act on:

Exit	Meaning
0	The run finished and no --fail-on condition was met.
1	A --fail-on condition was met — the data disagreed.
2	The run itself failed: a connection, a timeout, a query error.

These codes are data-driven and distinct, which is what makes them usable in a pipeline. Locally executed commands collapse both into 1 — see exit codes. If you need “the data is wrong” to be a different outcome from “the check could not run”, use a backend run.

Two more flags shape the wait:

--wait-timeout 45m      # give up waiting (default 30m); the run keeps going
--poll-interval 10s     # how often to check (default 5s)

Choosing what fails the build

--fail-on takes any combination of the four result statuses. Default: mismatched,failed.

Recipe	--fail-on	Use for
Standard gate	mismatched,failed	Block on a real difference. Tolerated aggregate drift passes.
Strict gate	mismatched,within_threshold,failed	Block on any difference at all, even inside thresholds.
Data-only gate	mismatched	The data must agree, but an infrastructure blip should not fail the build.
Inverted canary	passed	The two sides are supposed to differ. Failing to differ means something is misconfigured.

The inverted canary is more useful than it sounds. A reconciliation that compares a table against a deliberately-modified copy should always report a mismatch; if it ever passes, your reconciliation is not actually comparing what you think it is. It is a test for the test.

Where a gate belongs

1. Before a cutover

   Migrating a warehouse, or switching a consumer from one table to another. Promote a suite comparing old against new, and make the cutover step depend on it passing. This is the highest-value use: the alternative is a spot check and a hope.

2. After a backfill

   Run the reconciliation as the last step of the backfill job. A backfill that reports a mismatch has not finished, whatever the row count says.

3. On a schedule, with the pipeline reading the result

   Often the better pattern for a nightly pipeline: let the schedule own the run and have your job read the outcome. No coordination, no duplicate runs, and the run appears in the app whether or not the pipeline looked.

For the third pattern, read the result rather than triggering it:

# The most recent run's outcome for one suite
outcome=$(synq-recon audit-logs list --suite <suite-id> --limit 1 \
            --columns STATUS --no-headers 2>/dev/null)

case "$outcome" in
  *PASSED*)            echo "reconciliation passed" ;;
  *WITHIN_THRESHOLD*)  echo "within tolerance" ;;
  *)                   echo "reconciliation did not pass: $outcome"; exit 1 ;;
esac

The outcome is one of AUDIT_OUTCOME_PASSED, AUDIT_OUTCOME_MISMATCHED_WITHIN_THRESHOLD, AUDIT_OUTCOME_MISMATCHED or AUDIT_OUTCOME_FAILED. The same names, minus the prefix, are what --status and --fail-on accept.

Scopes and credentials

Each operation needs a scope, and each scope is carried by a role:

Operation	Scope	Role (as labelled in the app)
View suites, deployments, runs, audit logs	SCOPE_RECON_READ	Business User and up
Save a suite, edit it, run it ad hoc, cancel a run	SCOPE_RECON_EDIT	Developer and up
Promote, unpromote, trigger, pause, resume, edit a deployment	SCOPE_RECON_PROMOTE	Developer and up
Report a locally executed run’s results	SCOPE_RECON_EDIT	Developer and up

All three scopes are selectable when you create an API client or token, so every command works from an unattended pipeline with client credentials or QUALITY_TOKEN — nothing here requires a developer’s browser login.

Credentials resolve in a fixed order, so an automated pipeline never silently picks up a developer’s browser login:

1. Client credentials

   --client-id / --client-secret, or QUALITY_CLIENT_ID / QUALITY_CLIENT_SECRET, or the synq: block in the suite.

2. A pre-issued token

   QUALITY_TOKEN. No YAML equivalent, by design — it is a CI credential, not suite configuration.

3. The cached browser login

   From synq-recon auth login, refreshed automatically.

Choosing the deployment

--region names a deployment; --endpoint (or QUALITY_API_ENDPOINT) points at a specific host, for a staging or self-hosted one. Both are root flags, so they work on every command.

You rarely need either in CI. synq-recon auth login --region <region> records the deployment and later commands resolve to it; synq-recon auth use <region> switches between deployments you are already logged in to, and auth use --clear returns to the default.

Whichever you set also selects which region’s stored credential is used — a token minted for one region is never sent to another. The full resolution order:

--endpoint  >  --region  >  the suite's synq.endpoint  >  QUALITY_API_ENDPOINT
            >  QUALITY_REGION  >  the last login  >  the default region

The --synq--prefixed flags and SYNQ_-prefixed variables are the previous spellings. They still work, so nothing in an existing pipeline has to change, but the names above are the ones every Coalesce Quality CLI now shares.

Give a pipeline the narrowest set that covers what it does. A CI gate that only reads results needs SCOPE_RECON_READ alone; one that uploads a suite and triggers it needs SCOPE_RECON_EDIT and SCOPE_RECON_PROMOTE as well.

Running in a container

The published image is the artifact available without monorepo access:

docker run --rm \
  -v "$PWD:/work" -w /work \
  -e QUALITY_CLIENT_ID -e QUALITY_CLIENT_SECRET \
  -e PG_USER -e PG_PASSWORD \
  europe-docker.pkg.dev/synq-cicd-public/synq-public/synq-recon:latest \
  run-check suite.yaml -o json

Notes for a pipeline:

• Pass secrets as environment variables, and reference them in the suite as ${VAR}. Never bake a credentials file into an image.
• Pin the tag. :latest moves; a pipeline should name a version.
• type: duckdb works in the image, so a pipeline can validate against DuckDB fixtures with no warehouse. If it fails with a driver error, the image predates DuckDB support — pull a newer tag.
• AGENTS.md ships inside the image at /opt/synq-recon/AGENTS.md, so an agent working in a container has the operating guide without fetching anything.

Validating a suite in CI

Cheapest useful check, and the one to add first — it costs nothing and catches a renamed column before a scheduled run does:

synq-recon check-config suite.yaml                     # syntax and semantics, no network
synq-recon check-config suite.yaml --db                # ...and every query, against the databases

Set strict_time_references: true in suites you validate in CI. It turns the NOW() / CURRENT_DATE warning into an error, so the class of bug that produces phantom mismatches cannot reach production.

Triggering without the CLI

trigger is a thin wrapper over the public API, so any HTTP or gRPC client can do the same thing. The deployment must be active, triggerable by API, and not paused.

The relevant service is synq.agent.recon.v1.SuiteDeploymentService — TriggerDeployment to start a run, GetSuiteDeployment to inspect one — with AuditLogService and RunStateService for the results. See the gRPC API reference and scopes.

Make submissions idempotent by supplying your own invocation id, so a retried pipeline step does not start a second run:

synq-recon run-remote <suite-id> --invocation-id "$CI_PIPELINE_ID" --wait

Next

Workspace suites

Schedules, deployments, and what an edit preserves.

Agent workflow

The same operations, as an operating procedure for a coding agent.