This guide explains how to connect Synq to BigQuery securely.

We need this information so we can collect relevant data about your tables.

To be able to finish this guide, you’ll need the following:
→ Access to modify your BigQuery configuration

⏱️ Estimated time to finish: 10 minutes.

​
Data we collect

For the automated data anomaly testing, we collect the following:

  • Number of rows in every table in the monitored dataset(s)
  • Timestamp of the last change of data in all tables in the monitored dataset(s)

To provide out-of-the-box monitors for volume of data and freshness Synq doesn’t require access to your actual data. For custom monitors, Synq requires access to query your raw data

Permission overview:

PermissionDescriptionIncluded in BQ rolePurpose
bigquery.datasets.getGet metadata about a dataset.Data Viewerautomated / custom monitors
bigquery.datasets.getIamPolicyRequired by the Cloud Console to give the user the option of getting a dataset’s IAM permissions. Fails open. The ability to actually perform the operation of getting the permissions is gated by the bigquery.datasets.get permission.Data Viewerautomated / custom monitors
bigquery.jobs.createRun jobs (including queries) within the project.Job Userautomated / custom monitors
bigquery.jobs.getGet data and metadata on any job.1
bigquery.jobs.listList all jobs and retrieve metadata on any job submitted by any user. For jobs submitted by other users, details and metadata are redacted.Resource Viewerquery logs
bigquery.jobs.listAllList all jobs and retrieve metadata on any job submitted by any user.Resource Viewerquery logs
bigquery.tables.getGet table metadata.Data Viewerautomated monitors
bigquery.tables.getDataGet table data.Data Viewerautomated / custom monitors
bigquery.tables.listList tables and metadata on tables.Data Viewerautomated / custom monitors
bigquery.routines.getTo query data in INFORMATION_SCHEMA.TABLES.Data Viewertables DDL
bigquery.routines.listTo query data in INFORMATION_SCHEMA.TABLES.Data Viewertables DDL
resourcemanager.projects.getData/Jobs/Resource ViewerAll

​
Setup BigQuery access

​
Create a dedicated Synq role

  1. Select the project with your BigQuery instance in project selection combobox

  2. Go to IAM and Admin > Roles

  3. Click the Create Role button on top.

  4. Fill in information as follows

    1. Title: Synq Monitoring
    2. Description: Synq Monitoring role
    3. Role launch stage: General Availability
    4. Assigned permissions:
    bigquery.datasets.get
bigquery.datasets.getIamPolicy
bigquery.jobs.create
bigquery.jobs.get
bigquery.jobs.list
bigquery.jobs.listAll
bigquery.tables.get
bigquery.tables.getData
bigquery.tables.list
bigquery.routines.get
bigquery.routines.list
resourcemanager.projects.get

  5. Confirm and save

​
Create a service account

  1. Go to IAM and Admin > Service Accounts

  2. Click the Create Service Account button

  3. Fill in information as follow:

    1. Service account name: synq-monitoring

    2. Service account description: Synq Monitoring Service Account

  4. Click Create and continue

  5. In section Grant this service account access to the project, select the previously created role

    1. Click Done

​
Create a service account key

  1. Open your newly created Service Account

  2. Switch to Keys tab

  3. Create a new JSON key

  4. Store the newly created JSON key securely.

Once you’ve run successfully completed these steps input the following data in Synq

​
Input data in the Synq UI

​
Integration name

For example BigQuery

​
Project ID

You can see all your projects and the associated IDs by clicking the drop-down and looking at the ID column of the project.

​
Service account key

The content of the JSON file you created earlier in the guide

​
Region

Location of your BigQuery instance (typically US or EU)

TableauClickHouse