Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.synq.io/llms.txt

Use this file to discover all available pages before exploring further.

This guide details the steps to integrate Coalesce Quality with Databricks using the Unity Catalog feature. This integration enables efficient data observability and management across your Databricks environment.Before proceeding, ensure you have:
  • Administrative access to your Databricks workspace
  • Permissions to manage Unity Catalog settings
⏱️ Estimated time to complete: 10 minutes.

​
Integration Overview

Integrating with Databricks through Unity Catalog allows you to monitor and manage data reliability and quality directly within your Databricks environment.

​
Prerequisites

  • Databricks Workspace URL: The URL of your Databricks workspace where Unity Catalog is configured.
  • OAuth Client ID and Client Secret / Access Token: A Databricks credentials with permissions to access to monitored catalogs.
  • Warehouse ID: The identifier for the SQL warehouse within Databricks that will be used to run monitoring queries. We recommend using Serverless SQL Warehouses.

​
Step-by-Step Guide

​
Step 1: Configuring Unity Catalog

  1. Log in to your Databricks workspace: Navigate to your Databricks workspace by entering the Workspace URL in your browser.
  2. Access the Unity Catalog: From the sidebar, select ‘Data’ and then ‘Unity Catalog’ to configure the data catalog settings.
  3. Create or Select a Catalog: Choose an existing catalog or create a new one for the integration to operate on.

​
Step 2: Create authentication method

  1. Create Service Principal: Navigate to the ‘Admin Console’ and select ‘Identity and access’ > ‘Service Principals’ > ‘Manage’. Click on ‘Add Service Principal’ > ‘Add new’ and provide the name of the service principal. Note the generated Service Principal ID.
  2. Generate OAuth Secret: Click on the created Service Principal, go to the Secrets tab and click on ‘Generate secret’. Note the generated OAuth token securely.
  3. Assign Permissions: Assign the necessary permissions to the Service Principal to access the Unity Catalog and other required resources.
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT SELECT ON CATALOG <catalog_name> TO `<service_principal_id>`;
  1. Navigate to the User Settings: Click on your profile at the bottom left corner and select ‘User Settings’.
  2. Access Tokens: Go to the ‘Access Tokens’ tab and click on ‘Generate New Token’. Enter a description, set the expiration according to your policy, and note the generated token securely.

​
Step 3: Configure the Integration

  1. Log in to Coalesce Quality: Access your dashboard.
  2. Add Databricks as an Integration: Navigate to ‘Data Sources’, select ‘Add integration’, and choose ‘Databricks’.
  3. Enter Integration Details: Provide the Workspace URL, Access Token, and Warehouse ID to establish the connection.
  4. Network Configuration: If your Databricks workspace has network access controls or firewall restrictions, you may need to whitelist our IP addresses. See Coalesce Quality IP Whitelist for the complete list of IP addresses by region.
  5. Set Exclusion Rules: Define any exclusion rules for catalogs, schemas, and tables to tailor the monitoring to your needs.
title title

​
Personal access (user OAuth)

In addition to the workspace service principal above, you can let each Coalesce Quality member authenticate as themselves against Databricks. Queries then run under the member’s own Databricks user and respect the Unity Catalog grants on that user. See Data Warehouse Access for the full model and when to enable it. To enable personal access, a Databricks account admin registers a custom OAuth app once at the account level, then pastes the client ID and secret into the Databricks integration’s edit page in Coalesce Quality. The same app can be reused across every workspace in the account.

​
Register the OAuth app in Databricks

  1. Sign in to the Databricks Account Console (accounts.cloud.databricks.com for AWS, accounts.azuredatabricks.net for Azure, accounts.gcp.databricks.com for GCP) as an account admin.
  2. Open Settings → App connections → Add connection.
  3. Fill in:
    • Name: e.g. Coalesce Quality.
    • Redirect URLs — add the URL(s) for the Coalesce Quality region(s) your members use, with no trailing slash:
      • EU: https://app.synq.io/auth/dwh-oauth/callback
      • US: https://app.us.synq.io/auth/dwh-oauth/callback
    • Access scopes: all-apis and offline_access.
    • Mark the app as Confidential (Coalesce Quality stores the client secret server-side).
  4. Save. Copy the Client ID and Client secret — the secret is shown only once.
If your Databricks tier exposes app connections at the workspace level as well (<workspace>/settings → Developer or Identity and access → App connections), either surface works — the resulting OAuth app is the same. Account-level registration is the recommended path because the app then covers every workspace in the account.Coalesce Quality builds the authorize URL automatically from the workspace URL you configured on the integration (<workspace>/oidc/v1/authorize) — you don’t need to enter the authorize endpoint anywhere.

​
Grant Unity Catalog access to your members

Personal access uses each member’s own Unity Catalog grants — the OAuth app itself does not need catalog grants. Make sure every member who will use this path has at least: 
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<member_email>`;
GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<member_email>`;
GRANT SELECT ON CATALOG <catalog_name> TO `<member_email>`;
Members also need access to the SQL warehouse configured on the integration — CAN USE on the warehouse is enough.

​
Paste credentials into Coalesce Quality

In Settings → Integrations → (your Databricks integration) → Warehouse access → Personal credentials, paste the Client ID and Client secret. Members will then see this Databricks integration listed under Settings → Warehouse access → My credentials and can connect their own Databricks user.

​
Conclusion

Once configured, the platform will begin monitoring the specified data assets within Databricks, leveraging Unity Catalog for enhanced data management and observability. For further assistance, contact our support team.