Adding BigQuery source

To add a source, navigate to the Connections page and click the BigQuery button.

There are two ways of providing access to your BigQuery instance:

Option 1: Provide us with a service account that you manage in your Google Cloud account. This account should be provisioned with the following read-only roles:

• bigquery.dataViewer
• bigquery.jobUser
• bigquery.readSessionUser
• A custom role that provides bigquery.jobs.listAll and bigquery.jobs.list permissions (see Granting Permissions below for commands)

You'll need to provide us with the service account's email, JSON-formatted key, and the location of your BigQuery instance. You'll also need to provide a nickname (e.g., BigQuery Prod); a Metaplane-specific display name.

Option 2: Grant permissions to a Metaplane-provided service account. In this case, we'll create a service account for you within our Google Cloud account. Initially, this new service account won't have access your BigQuery instance so you'll have to grant it the appropriate roles.

When selecting this option you'll get prompted for a nickname (e.g. BigQuery Prod), your project_id, and region:

Granting permissions

Regardless of the option you choose—providing your own service account or using one of ours—the service account will need the appropriate read-only roles.

The easiest way to grant these roles is through the Google Cloud Shell.

First, we'll create a custom role for Metaplane-related permissions and then bind it to the service account that you're using. We'll also bind read-only BigQuery roles to the service account.

1. Create a Metaplane custom role

gcloud iam roles create MetaplaneMonitor \
  --project={{PROJECT_ID}} \
  --title=MetaplaneMonitor \
  --description="Metaplane specific permissions" \
  --permissions=bigquery.jobs.listAll,bigquery.jobs.list

Note that the {{PROJECT_ID}} placeholder needs to be replaced with your project id.

2. Bind the custom role to a service account and apply read-only BQ roles

gcloud projects add-iam-policy-binding {{PROJECT_ID}} \
  --member="serviceAccount:{{SERVICE_ACCOUNT}}" \
  --role="projects/{{PROJECT_ID}}/roles/MetaplaneMonitor"

gcloud projects add-iam-policy-binding {{PROJECT_ID}} \
  --member="serviceAccount:{{SERVICE_ACCOUNT}}" \
  --role="roles/bigquery.dataViewer"

gcloud projects add-iam-policy-binding {{PROJECT_ID}} \
  --member="serviceAccount:{{SERVICE_ACCOUNT}}" \
  --role="roles/bigquery.jobUser"

gcloud projects add-iam-policy-binding {{PROJECT_ID}} \
  --member="serviceAccount:{{SERVICE_ACCOUNT}}" \
  --role="roles/bigquery.readSessionUser"

Note that the {{SERVICE_ACCOUNT}} and {{PROJECT_ID}} placeholders needs to be replaced with your service account and project id, respectively.

Multiple projects

If you have multiple projects in BigQuery that you want Metaplane to monitor, follow the above instructions to grant permissions to each project_id. In addition, you will need to enable the Cloud Resource API in order for Metaplane to query the projects our service account has access to. You can enable the API here: https://console.cloud.google.com/apis/api/cloudresourcemanager.googleapis.com/metrics

After enabling permission in BigQuery, navigate to the connection in metaplane, select configure in actions, and then enable multiple projects. Enabling this flags tells us that the above API is enabled and we can search for all the projects we have access to.

---

Whitelisted IPs

Metaplane will always connect to your data warehouse from these static IP addresses:

• 44.197.96.121/32
• 34.206.79.174/32
• 107.22.42.246/32