Databricks

Integrating Databricks with Coalesce Quality

This guide shows how to connect Coalesce Quality to a Databricks workspace with Unity Catalog.

Before proceeding, ensure you have:

• Administrative access to your Databricks workspace
• Permissions to manage Unity Catalog settings

⏱️ Estimated time to complete: 10 minutes.

​

Prerequisites

• The URL of your Databricks workspace where Unity Catalog is configured.
• Credentials with access to the monitored catalogs: an OAuth client ID and secret of a service principal (OAuth M2M), or a personal access token.
• The ID of the SQL warehouse that runs the monitoring queries. We recommend a serverless SQL warehouse.

​

Step-by-Step Guide

​

Step 1: Configuring Unity Catalog

1. Log in to your Databricks workspace: Navigate to your Databricks workspace by entering the Workspace URL in your browser.

2. Access the Unity Catalog: From the sidebar, select ‘Data’ and then ‘Unity Catalog’ to configure the data catalog settings.

3. Create or Select a Catalog: Choose an existing catalog or create a new one for the integration to operate on.

​

Step 2: Create authentication method

​

Option 1: Using Service Principal and OAuth (recommended)

1. Create Service Principal: Navigate to the ‘Admin Console’ and select ‘Identity and access’ > ‘Service Principals’ > ‘Manage’. Click on ‘Add Service Principal’ > ‘Add new’ and provide the name of the service principal. Note the generated Service Principal ID.
2. Generate OAuth Secret: Click on the created Service Principal, go to the Secrets tab and click on ‘Generate secret’. Note the generated OAuth token securely.
3. Assign Permissions: Assign the necessary permissions to the Service Principal to access the Unity Catalog and other required resources.

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT SELECT ON CATALOG <catalog_name> TO `<service_principal_id>`;

4. Give it the SQL warehouse: Open the SQL warehouse set in the integration → Permissions and give the service principal CAN USE. Every SQL query of the integration runs on this warehouse: tags, SHOW CREATE TABLE, ANALYZE TABLE, and your monitors.

These catalog grants are enough to read metadata and run monitoring queries. They are not enough to build lineage from query history. See Query log lineage permissions below.

​

Option 2: Using Personal Access Token (not recommended)

1. Navigate to the User Settings: Click on your profile at the bottom left corner and select ‘User Settings’.

2. Access Tokens: Go to the ‘Access Tokens’ tab and click on ‘Generate New Token’. Enter a description, set the expiration according to your policy, and note the generated token securely.

​

Step 3: Configure the Integration

1. Log in to Coalesce Quality: Access your dashboard.

2. Add Databricks as an Integration: Navigate to ‘Data Sources’, select ‘Add integration’, and choose ‘Databricks’.

3. Enter Integration Details: Provide the Workspace URL and the Warehouse ID. Under Authentication method, pick OAuth M2M and enter the service principal’s Client ID and Client secret, or pick Personal access token and enter the Access token.

4. Network Configuration: If your Databricks workspace has network access controls or firewall restrictions, you may need to whitelist our IP addresses. See Coalesce Quality IP Whitelist for the complete list of IP addresses by region.

5. Set the scope: Under Catalog scope and Metrics scope, add include and exclude rules for catalogs, schemas and tables. Catalog scope limits what is ingested into the catalog, Metrics scope limits which tables are scanned for metrics.

​

Data collection options

These switches on the integration control what Coalesce Quality reads beyond the Unity Catalog metadata. On the on-premise agent, each one is a key of the databricks connection.

Switch	Agent key	What it does	Access it needs
Run ANALYZE TABLE before reading statistics	refresh_table_metrics	Runs ANALYZE TABLE … COMPUTE STATISTICS NOSCAN on every table that changed since the last run, so the row count and size are current.	MODIFY on the tables. Without it the statement fails and the table keeps the statistics it had.
Run ANALYZE TABLE with a full scan	refresh_table_metrics_use_scan	Leaves out NOSCAN, so ANALYZE TABLE scans the table’s files and counts its rows. Takes more warehouse time.	Same as above.
Use SHOW CREATE TABLE to get DDLs of tables and views	use_show_create_table	Runs SHOW CREATE TABLE for every table to read its DDL.	SELECT on the table, granted above.
Retrieve table tags from INFORMATION_SCHEMA.*_TAGS	fetch_table_tags	See Unity Catalog tags.

All of them run SQL on the warehouse set in the integration.

The integration has three more switches:

• Fetch query logs (recommended): reads query history for anomaly monitors, usage, unused tables, costs and lineage. It also gives the statements that created your tables.
• Fetch table metrics (required for automatic monitors): on by default. Reads the row count, size and last change time of each table every 30 minutes. Automatic freshness and volume monitors stop working when it is off.
• Disable Unity Catalog table-to-table lineage retrieval: Coalesce Quality reads table-to-table lineage from Unity Catalog by default. Turn this on to use only the lineage from query logs.

Under dbt mappings, you can link a dbt integration and its project directory to this Databricks integration.

​

Workspace assets

Besides Unity Catalog, Coalesce Quality lists the workspace’s jobs and their runs, notebooks, AI/BI and legacy dashboards, apps, SQL warehouses and saved queries, with their lineage to tables. Databricks lists only the objects the service principal can see. Notebooks, AI/BI dashboards and apps need the Workspace access entitlement on the service principal. If you use a personal access token with limited scopes, add the scopes for dashboards and apps. Without them, the integration shows a recommendation and the rest is read as usual.

​

Platform users

With Fetch users and service accounts turned on (it shows once query logs are on), Coalesce Quality lists the workspace’s users and service principals through the SCIM API once a day, to tell people from service principals among the users in query history. Databricks lists them only to a workspace admin, so the service principal needs to be in the workspace admins group. A token limited to scopes also needs the scim scope.

​

Unity Catalog tags

Turn on Retrieve table tags from INFORMATION_SCHEMA.*_TAGS in the integration settings to read your Unity Catalog tags. Tags are only available through SQL, so this needs the SQL warehouse set in the integration.

• Coalesce Quality reads CATALOG_TAGS, SCHEMA_TAGS, TABLE_TAGS and COLUMN_TAGS from each catalog’s information_schema, which lists only the objects the service principal can see.
• A table gets its own tags and the tags of its catalog and schema. A column gets its own.
• A tag is read as a label with its name and its value, which may be empty. Table tags also become annotations named databricks.tag.<name>.
• If the tags of a catalog cannot be read, the rest of the catalog is still read, without its tags.

​

Query log lineage permissions

Besides the Unity Catalog lineage, Coalesce Quality derives cross-table and column-level lineage from your Databricks query history. It reads the executed statements (such as CREATE TABLE AS SELECT and MERGE) and parses the relationships between source and target tables. Views carry their own SQL definition, so they show lineage without query history.

Databricks controls two things separately: which queries the service principal can see, and whether it can read their text. Lineage needs both, so the catalog grants above are not sufficient on their own.

​

See the queries

A service principal (or user) sees:

• its own queries; plus
• queries that ran on a SQL warehouse it has at least CAN VIEW access to.

A workspace admin sees all queries. Without warehouse access, the integration sees only the queries it runs itself, so the statements that build your tables stay invisible.

Grant the service principal access to the query history of the warehouse(s) where your transformations run. Choose one of:

• Grant CAN VIEW (or CAN MANAGE) on each SQL warehouse that executes your transformations, via the warehouse’s Permissions settings, or
• Make the service principal a workspace admin.

​

Read the query text

Databricks masks the text of every query as <REDACTED> unless the reader is an account admin or a member of the account-level group databricks_pii_access. Workspace admin and warehouse permissions do not lift the mask.

The group does not exist until an account admin creates it:

1. Sign in to the Databricks Account Console as an account admin.
2. Open User management → Groups → Add group and name it exactly databricks_pii_access (the name is case-sensitive).
3. Add the integration’s service principal to the group as a member.

Queries whose text is masked still appear in Coalesce Quality with their user, warehouse, duration and metrics, but they are not attributed to the tables they read or wrote, and no lineage is produced from them. The integration’s health page raises a recommendation with the number of masked queries in each fetch until the service principal can read their text.

Query history only covers queries executed on SQL warehouses. Statements that run on all-purpose clusters, Jobs compute, or Delta Live Tables / Lakeflow pipelines do not appear in query history. Run the transformations you want lineage for on a SQL warehouse.

​

Personal access (user OAuth)

In addition to the workspace service principal above, you can let each Coalesce Quality member authenticate as themselves against Databricks. Queries then run under the member’s own Databricks user and respect the Unity Catalog grants on that user. See Data Warehouse Access for the full model and when to enable it.

To enable personal access, a Databricks account admin registers a custom OAuth app once at the account level, then pastes the client ID and secret into the Databricks integration’s edit page in Coalesce Quality. The same app can be reused across every workspace in the account.

​

Register the OAuth app in Databricks

1. Sign in to the Databricks Account Console (accounts.cloud.databricks.com for AWS, accounts.azuredatabricks.net for Azure, accounts.gcp.databricks.com for GCP) as an account admin.
2. Open Settings → App connections → Add connection.
3. Fill in:
   • Name: e.g. Coalesce Quality.
   • Redirect URLs: add the URL(s) for the Coalesce Quality region(s) your members use, with no trailing slash:
     • EU: https://app.synq.io/auth/dwh-oauth/callback
     • US: https://app.us.synq.io/auth/dwh-oauth/callback
     • AU: https://app.au.synq.io/auth/dwh-oauth/callback
   • Access scopes: all-apis and offline_access.
   • Mark the app as Confidential (Coalesce Quality stores the client secret server-side).
4. Save. Copy the Client ID and Client secret. The secret is shown only once.

If your Databricks tier also shows app connections at the workspace level (<workspace>/settings → Developer or Identity and access → App connections), you can register the app there too. We recommend the account level, as the app then covers every workspace in the account.

​

Grant Unity Catalog access to your members

Personal access uses each member’s own Unity Catalog grants. The OAuth app itself does not need catalog grants. Make sure every member who will use this path has at least:

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<member_email>`;
GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<member_email>`;
GRANT SELECT ON CATALOG <catalog_name> TO `<member_email>`;

Members also need CAN USE on the SQL warehouse configured on the integration.

​

Paste credentials into Coalesce Quality

In Settings → Integrations → (your Databricks integration) → Warehouse access → Personal credentials, paste the Client ID and Client secret. Members will then see this Databricks integration listed under Settings → Warehouse access → My credentials and can connect their own Databricks user.