Google Cloud Storage

Automatically discover and track GCS buckets in Coalesce Quality

Overview

The Coalesce Quality Google Cloud Storage integration discovers the GCS buckets in your Google Cloud project and publishes each one as a custom entity in the Coalesce Quality catalog.

It is a command-line tool that you run in your own environment, for example on a schedule. It is also a reference implementation for custom integrations built on the Coalesce Quality Public API.

Source code: github.com/getsynq/synq-google-cloud-storage

What it does

The integration:

• Discovers all GCS buckets in your Google Cloud project.
• Creates a custom entity for each bucket. The entity description holds the storage class, location, creation date, versioning status, lifecycle rules with their conditions, uniform bucket-level access setting and labels.
• Optionally creates lineage from a bucket to the Pub/Sub topics its notifications send events to. See Lineage to topics.
• Filters buckets with include and exclude patterns.
• Removes the entity of a bucket that was deleted.

Use cases

• Keep an up-to-date inventory of your GCS buckets in the catalog.
• Review storage configurations and lifecycle policies.
• Connect buckets with the pipelines and tables that use them.

Installation

Install with Go 1.25 or later:

go install github.com/getsynq/synq-google-cloud-storage@latest

Or build from source:

git clone https://github.com/getsynq/synq-google-cloud-storage.git
cd synq-google-cloud-storage
go build

Configuration

Authentication

The integration needs a Coalesce Quality credential and access to your GCP project. It resolves the credential in this order:

1. Client credentials, for scheduled runs. Set them in the environment or in a .env file:

   QUALITY_CLIENT_ID=your_client_id_here
   QUALITY_CLIENT_SECRET=your_client_secret_here
   

2. A pre-issued access token in QUALITY_TOKEN.

3. A browser login, for running the tool by hand:

   synq-google-cloud-storage auth login
   synq-google-cloud-storage auth status
   synq-google-cloud-storage auth logout
   

   The login is stored under ~/.synq/oauth/ and is shared with synqcli and synq-recon.

The credential needs the SCOPE_ENTITY_EDIT, SCOPE_ENTITY_TYPE_EDIT and SCOPE_LINEAGE_EDIT scopes. See Scopes.

For GCP, the tool uses Application Default Credentials. The account needs the storage.buckets.list permission. With relationships turned on, it also needs storage.buckets.get to read the notification configuration of each bucket.

Region

The integration connects to the EU region by default, or to the region you last logged in to with auth login. Select another region with --region:

synq-google-cloud-storage --region us   # eu, us or au

You can also set the region with region: in config.yaml or with QUALITY_REGION. --endpoint (or QUALITY_API_ENDPOINT) sets an explicit API endpoint and overrides the region.

GCP project

The tool finds the GCP project ID in this order:

1. The GCP_PROJECT_ID, GOOGLE_CLOUD_PROJECT or GCLOUD_PROJECT environment variable
2. The gcloud CLI configuration
3. The GCP metadata server, when the tool runs on GCP
4. gcp.project_id in config.yaml

Optional: config.yaml

All settings are optional. Command-line flags override the config file and environment variables.

# Coalesce Quality
quality:
  # region: "us"                      # eu (default), us or au
  # endpoint: "api.us.synq.io:443"    # overrides region

# GCP
gcp:
  user_agent: "synq-gcs-client-v1.0.0"
  # project_id: "your-project-id"
  # entity_group_id: "gcs::custom-group-id"  # Defaults to gcs::<project_id>

# Custom entity type IDs
types:
  bucket_type_id: 40
  # bucket_icon: "path/to/custom-bucket-icon.svg"

# Resource filters
filter:
  buckets:
    include: []  # Empty means include all
    exclude: []  # Regex patterns to exclude
    # include: ["prod-.*"]  # Only buckets starting with prod-
    # exclude: ["test-.*"]  # Skip buckets starting with test-

# Lineage from buckets to Pub/Sub topics (off by default)
relationships:
  enabled: false
  filter:
    include: []  # Format: "bucket_name->topic_id"
    exclude: []

Network requirements

The integration runs in your environment and makes only outbound HTTPS connections, to the Google Cloud APIs and to the Coalesce Quality API of your region on port 443:

Region	API endpoint
EU (default)	developer.synq.io
US	api.us.synq.io
AU	api.au.synq.io

If your network restricts outbound traffic, allow these hosts. Nothing connects in to your environment.

Running the integration

# Run with defaults
synq-google-cloud-storage

# Debug logging
LOG_LEVEL=DEBUG synq-google-cloud-storage

# JSON logging
LOG_FORMAT=json synq-google-cloud-storage

Dry-run mode

--dry-run scans the buckets and shows the entities it would create. It does not call the Coalesce Quality API and does not need a credential.

synq-google-cloud-storage --dry-run

Command-line options

• --dry-run: scan without publishing
• --region: eu, us or au
• --endpoint: API endpoint, overrides --region
• --client-id, --client-secret: client credentials
• --gcp.project-id: GCP project ID
• --filter.buckets.include, --filter.buckets.exclude: bucket name patterns
• --relationships.enabled: create lineage from buckets to Pub/Sub topics
• --types.bucket-type-id: custom entity type ID (default: 40)
• -c, --config: path to the config file (default: config.yaml)

Run synq-google-cloud-storage --help for all options.

How it works

1. Authenticates with the Coalesce Quality API.
2. Creates or updates the custom entity type for buckets.
3. Lists the buckets in your project.
4. Creates or updates an entity for each bucket.
5. Updates the entity group, so that Coalesce Quality removes the entities of deleted buckets.

Entity groups

Each run puts all the entities it creates in one entity group. When the group is updated, Coalesce Quality removes the entities that were in the previous group but are not in the current one.

Custom identifiers

Buckets use custom identifiers in the format gcs::<bucket_name>.

Lineage to topics

With relationships.enabled: true, the integration reads the notification configuration of each bucket and creates lineage from the bucket to each Pub/Sub topic it notifies. Run the Pub/Sub integration first. If the pubsub::<topic_id> entity does not exist, the relationship is skipped.

A run withdraws only the bucket-to-topic relationships that it published. Lineage from other sources stays as it is.

Scheduling with cron

# Every hour
0 * * * * cd /path/to/integration && synq-google-cloud-storage

# Every 6 hours
0 */6 * * * cd /path/to/integration && synq-google-cloud-storage

Troubleshooting

Authentication errors

Check the credential with synq-google-cloud-storage auth status. Make sure the tool connects to the region of your workspace (--region eu, us or au).

GCP project not detected

Set the GCP_PROJECT_ID environment variable, or set the project in the gcloud CLI:

gcloud config set project YOUR_PROJECT

Permission denied errors

Make sure your GCP credentials have the storage.buckets.list permission, and storage.buckets.get if relationships are turned on. For the Coalesce Quality API, make sure the credential has the scopes listed under Authentication.

Network connectivity issues

Make sure outbound HTTPS connections to the API endpoint of your region are allowed. See Network requirements.

Support

• GitHub Issues: github.com/getsynq/synq-google-cloud-storage/issues
• Coalesce Quality Support: support