BigQuery

Integrating BigQuery with SYNQ

This guide explains how to connect SYNQ to BigQuery securely.

We need this information so we can collect relevant data about your tables.

To be able to finish this guide, you’ll need the following:
→ Access to modify your BigQuery configuration

⏱️ Estimated time to finish: 10 minutes.

Data we collect

For the automated data anomaly testing, we collect the following:

• Number of rows in every table in the monitored dataset(s)
• Timestamp of the last change of data in all tables in the monitored dataset(s)

To provide out-of-the-box monitors for volume of data and freshness SYNQ doesn’t require access to your actual data. For custom monitors, SYNQ requires access to query your raw data

Permission overview:

Permission	Description	Included in BQ role	Purpose
bigquery.datasets.get	Get metadata about a dataset.	Data Viewer	automated / custom monitors
bigquery.datasets.getIamPolicy	Required by the Cloud Console to give the user the option of getting a dataset’s IAM permissions. Fails open. The ability to actually perform the operation of getting the permissions is gated by the bigquery.datasets.get permission.	Data Viewer	automated / custom monitors
bigquery.jobs.create	Run jobs (including queries) within the project.	Job User	automated / custom monitors
bigquery.jobs.get	Get data and metadata on any job.1
bigquery.jobs.list	List all jobs and retrieve metadata on any job submitted by any user. For jobs submitted by other users, details and metadata are redacted.	Resource Viewer	query logs
bigquery.jobs.listAll	List all jobs and retrieve metadata on any job submitted by any user.	Resource Viewer	query logs
bigquery.tables.get	Get table metadata.	Data Viewer	automated monitors
bigquery.tables.getData	Get table data.	Data Viewer	automated / custom monitors
bigquery.tables.list	List tables and metadata on tables.	Data Viewer	automated / custom monitors
bigquery.routines.get	To query data in INFORMATION_SCHEMA.TABLES.	Data Viewer	tables DDL
bigquery.routines.list	To query data in INFORMATION_SCHEMA.TABLES.	Data Viewer	tables DDL
resourcemanager.projects.get		Data/Jobs/Resource Viewer	All

Setup BigQuery access

Create a dedicated SYNQ role

1. Select the project with your BigQuery instance in project selection combobox

2. Go to IAM and Admin > Roles

3. Click the Create Role button on top.

4. Fill in information as follows

   1. Title: SYNQ Monitoring
   2. Description: SYNQ Monitoring role
   3. Role launch stage: General Availability
   4. Assigned permissions:
   bigquery.datasets.get
   bigquery.datasets.getIamPolicy
   bigquery.jobs.create
   bigquery.jobs.get
   bigquery.jobs.list
   bigquery.jobs.listAll
   bigquery.tables.get
   bigquery.tables.getData
   bigquery.tables.list
   bigquery.routines.get
   bigquery.routines.list
   resourcemanager.projects.get
   

5. Confirm and save

Create a service account

1. Go to IAM and Admin > Service Accounts

2. Click the Create Service Account button

3. Fill in information as follow:

   1. Service account name: synq-monitoring

   2. Service account description: SYNQ Monitoring Service Account

4. Click Create and continue

5. In section Grant this service account access to the project, select the previously created role

   1. Click Done

Create a service account key

1. Open your newly created Service Account

2. Switch to Keys tab

3. Create a new JSON key

4. Store the newly created JSON key securely.

Once you’ve run successfully completed these steps input the following data in SYNQ

Input data in the SYNQ UI

Integration name

For example BigQuery

Project ID

You can see all your projects and the associated IDs by clicking the drop-down and looking at the ID column of the project.

Service account key

The content of the JSON file you created earlier in the guide

Region

Location of your BigQuery instance (typically US or EU)

Assistant

Responses are generated using AI and may contain mistakes.