Airbyte

Airbyte is a data movement platform with an expansive catalog of connectors, allowing users to seamlessly sync data between systems.

Generate Airbyte API credentials

Pick an Airbyte user for the Metaplane connection

The credentials you give Metaplane must be associated with some Airbyte user account. You can use your own account, or you can create a dedicated Metaplane account in Airbyte, which allows you finer-grained control over what resources you allow Metaplane to access.

Airbyte Cloud

If you use Airbyte Cloud, you can use API token authentication to connect Metaplane.

Generate an API token

Navigate to the Airbyte developer portal at https://portal.airbyte.com/, and log in using your normal Airbyte Cloud login.

Click New Api Key to generate a new key

Give your key a name - it can be anything - and then click Generate

You'll be able to view the generated API key - make sure to copy it now, or click Download to save it

Airbyte OSS

If you run a self-hosted open-source deployment of Airbyte, you can use username-password authentication to connect Metaplane. This could be the username and password of your personal Airbyte user, or of the Metaplane service you (optionally) created in the previous step.

Identify your base API URL

Usually of the format https://<host>/api/public/v1, but you can check with your Airbyte administrator to be sure.

Create an Airbyte connection in Metaplane

Head over Metaplane, add a new connection via Settings -> Data Stack -> Add connection, and select Airbyte.

Select your deployment type - Cloudif you use Airbyte Cloud, or Self hosted otherwise, and enter the corresponding credentials you generated in the previous step (username/password for Self hosted and api token for Cloud).

For Self hosted Airbyte instances, you'll also need to enter the Airbyte Public API base URL that you identified.

What to expect

Metaplane will populate with all of the Airbyte Workspaces, Connections, and Streams that it has access to. You'll see both upstream and downstream warehouse lineage for Streams. You'll also see some useful metrics (which you can create monitors on) about the syncs that Airbyte ran for each Connection, including:

  • Time since an Airbyte Connection last successfully synced
  • Bytes + rows written per sync
  • Duration of each sync