Google Cloud Pub/Sub

Automatically discover and track Pub/Sub topics and subscriptions in Coalesce Quality

​

Overview

The Coalesce Quality Google Cloud Pub/Sub integration discovers the Pub/Sub topics and subscriptions in your Google Cloud project and publishes each one as a custom entity in the Coalesce Quality catalog.

It is a command-line tool that you run in your own environment, for example on a schedule. It is also a reference implementation for custom integrations built on the Coalesce Quality Public API.

Source code: github.com/getsynq/synq-google-cloud-pubsub

​

What it does

The integration:

• Discovers all Pub/Sub topics and subscriptions in your GCP project.
• Creates a custom entity for each topic and subscription.
• Optionally creates lineage from each topic to its subscriptions, and from a subscription to the BigQuery table or GCS bucket it delivers to. See Relationships.
• Filters topics and subscriptions with include and exclude patterns.
• Removes the entities of deleted topics and subscriptions.

​

Use cases

• Keep an up-to-date inventory of your messaging infrastructure in the catalog.
• Review topics and their subscriptions.
• Connect Pub/Sub topics with pipelines, BigQuery tables and other systems.

​

Installation

Download the archive for your platform from the releases page, or install with Go 1.25 or later:

go install github.com/getsynq/synq-google-cloud-pubsub@latest

To build from source:

git clone https://github.com/getsynq/synq-google-cloud-pubsub.git
cd synq-google-cloud-pubsub
go build

​

Configuration

​

Authentication

The integration needs a Coalesce Quality credential and access to your GCP project. It resolves the credential in this order:

1. Client credentials, for scheduled runs. Set them in the environment or in a .env file:

   QUALITY_CLIENT_ID=your_client_id_here
   QUALITY_CLIENT_SECRET=your_client_secret_here
   

2. A pre-issued access token in QUALITY_TOKEN.

3. A browser login, for running the tool by hand:

   synq-google-cloud-pubsub auth login
   synq-google-cloud-pubsub auth status
   synq-google-cloud-pubsub auth logout
   

   The login is stored under ~/.synq/oauth/ and is shared with synqcli and synq-recon.

The credential needs the SCOPE_ENTITY_EDIT, SCOPE_ENTITY_TYPE_EDIT and SCOPE_LINEAGE_EDIT scopes. See Scopes.

For GCP, the tool uses Application Default Credentials. The account needs the pubsub.topics.list and pubsub.subscriptions.list permissions.

​

Region

The integration connects to the EU region by default, or to the region you last logged in to with auth login. Select another region with --region:

synq-google-cloud-pubsub --region us   # eu, us or au

You can also set the region with region: in config.yaml or with QUALITY_REGION. --endpoint (or QUALITY_API_ENDPOINT) sets an explicit API endpoint and overrides the region.

​

GCP project

The tool finds the GCP project ID in this order:

1. The GCP_PROJECT_ID, GOOGLE_CLOUD_PROJECT or GCLOUD_PROJECT environment variable
2. The gcloud CLI configuration
3. The GCP metadata server, when the tool runs on GCP
4. gcp.project_id in config.yaml

​

Optional: config.yaml

All settings are optional. Command-line flags override the config file and environment variables.

# Coalesce Quality
quality:
  # region: "us"                      # eu (default), us or au
  # endpoint: "api.us.synq.io:443"    # overrides region

# GCP
gcp:
  user_agent: "synq-pubsub-client-v1.0.0"
  # project_id: "your-project-id"
  # entity_group_id: "pubsub::custom-group-id"  # Defaults to pubsub::<project_id>

# Custom entity type IDs
types:
  topic_type_id: 20
  subscription_type_id: 21
  # topic_icon: "path/to/custom-topic-icon.svg"
  # subscription_icon: "path/to/custom-subscription-icon.svg"

# Resource filters
filter:
  topics:
    include: []  # Empty means include all
    exclude: []  # Regex patterns to exclude

  subscriptions:
    include: []  # Empty means include all
    exclude:
      # Exclude auto-generated per-pod subscriptions
      - '-[a-z0-9]{9,10}-[a-z0-9]{5}\.subscription$'

# Relationships (off by default)
relationships:
  enabled: false
  filter:
    include: []  # Format: "topic_id->subscription_id"
    exclude: []
    # include: ["important-topic->.*"]
    # exclude: ["test-.*->.*"]

​

Network requirements

The integration runs in your environment and makes only outbound HTTPS connections, to the Google Cloud APIs and to the Coalesce Quality API of your region on port 443:

Region	API endpoint
EU (default)	developer.synq.io
US	api.us.synq.io
AU	api.au.synq.io

If your network restricts outbound traffic, allow these hosts. Nothing connects in to your environment.

​

Running the integration

# Run with defaults
synq-google-cloud-pubsub

# Debug logging
LOG_LEVEL=DEBUG synq-google-cloud-pubsub

# JSON logging
LOG_FORMAT=json synq-google-cloud-pubsub

​

Dry-run mode

--dry-run scans the topics and subscriptions and shows the entities it would create. It does not call the Coalesce Quality API and does not need a credential.

synq-google-cloud-pubsub --dry-run --relationships.enabled

​

Command-line options

• --dry-run: scan without publishing
• --region: eu, us or au
• --endpoint: API endpoint, overrides --region
• --client-id, --client-secret: client credentials
• --gcp.project-id: GCP project ID
• --relationships.enabled: create relationships
• --relationships.prune: withdraw the topic-to-subscription relationships this integration published, and create none
• --filter.topics.include, --filter.topics.exclude: topic name patterns
• --filter.subscriptions.include, --filter.subscriptions.exclude: subscription name patterns
• --types.topic-type-id: custom entity type ID for topics (default: 20)
• --types.subscription-type-id: custom entity type ID for subscriptions (default: 21)
• -c, --config: path to the config file (default: config.yaml)

Run synq-google-cloud-pubsub --help for all options.

​

How it works

1. Authenticates with the Coalesce Quality API.
2. Creates or updates the custom entity types for topics and subscriptions.
3. Lists the topics and subscriptions in your project.
4. Creates or updates an entity for each one.
5. Creates relationships, if they are turned on.
6. Updates the entity group, so that Coalesce Quality removes the entities of deleted resources.

​

Entity groups

Each run puts all the entities it creates in one entity group. When the group is updated, Coalesce Quality removes the entities that were in the previous group but are not in the current one.

​

Relationships

Relationships are off by default. Turn them on with --relationships.enabled or relationships.enabled: true. The integration then creates:

• Topic to subscription, for each subscription of a topic.
• Subscription to BigQuery table, for a BigQuery subscription. Connect your BigQuery integration to see the table.
• Subscription to GCS bucket, for a Cloud Storage subscription. Run the Google Cloud Storage integration first. If the gcs::<bucket_name> entity does not exist, the relationship is skipped.

Relationship filter patterns use the format topic_id->subscription_id. A BigQuery relationship has the key subscription_id->bq-<table>, and a GCS relationship subscription_id->gcs-<bucket>, so exclude: ['->gcs-.*'] skips all GCS relationships.

A run withdraws only the relationships that it published. Lineage from other sources stays as it is. A subscription that a filter excludes keeps its relationships. To remove the relationships this integration published, run it once with --relationships.prune.

​

Custom identifiers

Topics use custom identifiers in the format pubsub::<topic_id>. Subscriptions use pubsub::<topic_id>::<subscription_id>.

​

Scheduling with cron

# Every hour
0 * * * * cd /path/to/integration && synq-google-cloud-pubsub

# Every 6 hours, with relationships
0 */6 * * * cd /path/to/integration && synq-google-cloud-pubsub --relationships.enabled

​

Troubleshooting

Authentication errors

Check the credential with synq-google-cloud-pubsub auth status. Make sure the tool connects to the region of your workspace (--region eu, us or au).

GCP project not detected

Set the GCP_PROJECT_ID environment variable, or set the project in the gcloud CLI:

gcloud config set project YOUR_PROJECT

Permission denied errors

Make sure your GCP credentials have these permissions:

• pubsub.topics.list
• pubsub.subscriptions.list

For the Coalesce Quality API, make sure the credential has the scopes listed under Authentication.

Too many auto-generated subscriptions

The default filter excludes auto-generated per-pod subscriptions. Add your own patterns to it:

filter:
  subscriptions:
    exclude:
      - '-[a-z0-9]{9,10}-[a-z0-9]{5}\.subscription$'
      - 'test-.*'

Network connectivity issues

Make sure outbound HTTPS connections to the API endpoint of your region are allowed. See Network requirements.

​

Support

• GitHub Issues: github.com/getsynq/synq-google-cloud-pubsub/issues
• Coalesce Quality Support: support