Guides
BlogTry for free

dbt Core

Suggest Edits

Metaplane is able to integrate with dbt Core by ingesting the manifest and run_result artifacts from a dbt run. We provide a CLI tool and Python script to make it easier to integrate and send the artifacts from whatever your orchestration environment might be.

Creating the Connection

Navigate to the connections page, click on "dbt Core", then follow the prompt.

Metaplane will associate all dbt Core-related monitoring results with this connection. We will use the connection id, taken from the url likeapp.metaplane.dev/s/<your connectionId UUID> in later steps.

Creating an API token

To send dbt Core artifacts to Metaplane, you will need to create an API token to authenticate requests. Create a new token following these steps. Save your token in a secure location since we will not be able to show it to you again.

Using the CLI

The CLI can be downloaded and installed via this command: 

/bin/bash -c "$(curl -fsSL https://cli.metaplane.dev/install.sh)"

The setup script will install a binary at ~/.local/bin and automatically add that directory to your PATH if it's not already there.

If you'd prefer to avoid having to download the binary on every run, you can instead download the binary directly and include it in your orchestration environment, e.g., in a Docker image.

To use the CLI, you will also need to set your API token as an environment variable like MP_API_TOKEN=<yourSecrectAPIToken>

After the dbt Core completes its run, you can send the resulting artifacts to Metaplane by invoking the CLI like this: 

metaplane import-dbt-run \
  --connection-id <yourDbtCoreConnectionId> \
  --project-name <yourProjectName> \
  --job-name <yourJobName> \
  --manifest <pathToManifest.json> \
  --run-results <pathToRunResults.json>

The connection ID can be found in the URL of the dbt Core connection page. You can get there by navigating to the Connections page, clicking on your dbt Core connection, then copying the UUID from the URL — it immediately follows /s/ in the path.

Both project-name and job-name are arbitrary strings that you can set to to help organize and group your different results.

Integrating with Git

dbt Core connections can take advantage of all of Metaplane's Data CI/CD features. To start, follow the docs for your Git provider.

Once your Git provider has been connected, you'll need to update your CI to notify Metaplane of PR-associated dbt Core runs. You can do this by providing a --commit-sha option to the import-dbt-run CLI command as well as providing a CI job name to --job-name. For example,

metaplane import-dbt-run \
  --connection-id <yourDbtCoreConnectionId> \
  --project-name <yourProjectName> \
  --job-name <yourCiJobName> \
  --manifest <pathToManifest.json> \
  --run-results <pathToRunResults.json> \
  --commit-sha <currentCommitSha>

The --commit-sha is the SHA associated a dbt run on a PR. Be sure to provide a different --job-name than the one you've chosen for production runs. More details below.

Full GitHub Actions example

Below is full example of using the Metaplane CLI from the context of a GitHub Action.

name: Build
on: [push]
jobs:
  run-dbt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
        with:
          fetch-depth: 1
      - uses: actions/setup-python@v1
        with:
          python-version: "3.7.x"

      - name: install dbt
        run: |
          pip install dbt-postgres
          dbt deps

      - name: run dbt
        run: dbt run
        env:
          DBT_POSTGRES_HOST: ${{ secrets.DBT_POSTGRES_HOST }}
          DBT_POSTGRES_USER: ${{ secrets.DBT_POSTGRES_USER }}
          DBT_POSTGRES_PASSWORD: ${{ secrets.DBT_POSTGRES_PASSWORD }}
          DBT_POSTGRES_PORT: ${{ secrets.DBT_POSTGRES_PORT }}
          DBT_POSTGRES_DB: ${{ secrets.DBT_POSTGRES_DB }}
          DBT_POSTGRES_SCHEMA: ${{ secrets.DBT_POSTGRES_SCHEMA }}

      - name: install metaplane cli
        run: /bin/bash -c "$(curl -fsSL https://cli.metaplane.dev/install.sh)"

      - name: report to metaplane
        if: github.ref == 'refs/heads/main' # only report main branch runs
        run: |
          metaplane import-dbt-run  \
            --connection-id 562b0248-2fc9-11ee-be56-0242ac120002 \
            --project-name my-dbt-core-project \
            --job-name production-run \
            --manifest target/manifest.json \
            --run-results target/run_results.json
        env:
          MP_API_TOKEN: ${{ secrets.MP_API_TOKEN }}

      - name: trigger metaplane pr comments
        if: github.ref != 'refs/heads/main' # provide --commit-sha on non-main branch run
        run: |
          metaplane import-dbt-run  \
            --connection-id 562b0248-2fc9-11ee-be56-0242ac120002 \
            --project-name my-dbt-core-project \
            --job-name branch-run \
            --manifest target/manifest.json \
            --run-results target/run_results.json \
            --commit-sha "${{ github.event.pull_request.head.sha || github.sha }}"
        env:
          MP_API_TOKEN: ${{ secrets.MP_API_TOKEN }}

Notes:

  • The names provided to --job-name are arbitrary and are used as display names in the Metaplane UI. For example, for main branch runs we used the job name production-run while for non-main branch runs we chose branch-run.
  • The job names should be static since static job names are required to configure the GitHub Integration. Also, keeping the job names static will allow you to see results over time grouped under a consistent name.
  • ${{ github.event.pull_request.head.sha || github.sha }} combines GitHub Actions-specific environment variables that represents the head sha of the current pull request being built.

Using a Python script

We've provided a Python script that you can use to import runs the same way the CLI would.

$ dbt run
$ curl -s https://cli.metaplane.dev/import_run.py | python - \
    --connection-id <connection-id> \
    --api-key <api-key> \
    --project-name <project-name \
    --job-name <job-name>

The script referenced above simply uploads the run_results.json and manifest.json artifacts found in the local target directory within your dbt project. The example command downloads the script and evaluates it using the global python binary.

Updated about 2 months ago

What’s Next