Data engineers are responsible for pipelines that move data from source to target systems, and discrepancies between those systems are a common data-quality issue.

Enter source-to-target monitors. Now data engineers can continuously monitor metrics across two databases and be the first to know about discrepancies. Setting up a source-to-target monitor can be done in minutes. Just like the standard monitors you know and love, source to target monitors will automatically learn the behavior of the data, incorporate seasonality trends, and detect anomalies over time.

Creating a Source to Target Monitor

You can create and configure Source-to-target monitors from the "Monitoring" tab or from any data connection monitoring page.

Select your source and target

Select the source and target data you want to compare over time. Metaplane supports monitoring both tables and columns across the same or different databases.

🚧

Please note

You can't use the same asset as both the source and the target

💡

Did you know?

You can monitor the difference between most standard metrics that Metaplane supports for tables and columns. For example: you might monitor the difference in row counts across two database tables or monitor the nullness of two columns across two databases.

To make it easier for teammates to find the monitor and see any related context, write a helpful name and description.

Select a format (optional)

By default, source-to-target monitors will monitor the difference in measured values between your two assets. You can also use a percent difference.

💡

When to Use Difference vs. Percent Difference

If you are comparing the row counts across two very large tables, you may want to monitor the absolute difference in row counts rather than percentage difference since small changes in percentage may miss large changes in absolute row count.

If you have two small tables and want to compare row count, a percent difference may be more effective in catching discrepancies even if the absolute row count is small.

Select a metric type

Just like our standard monitors, source-to-target monitors expect a metric type. With a source to target monitor, you can monitor the difference between any of our standard metric types.

Custom SQL

Custom SQL is slightly different from our standard metric types in that we require you to provide us with a query. When you select custom SQL as the metric type for your source to target monitor, an additional step will appear for you to define custom SQL for your source and target independently. Before creating the monitor, you'll have to validate that the SQL runs as expected.

Advanced configuration (optional)

Source-to-target monitors support the same advanced configuration as standard monitors including anomaly detection, model configuration, and incident threshold.

Group-by Columns

Any source-to-target monitor meeting the following criteria can also use group-by-columns:

• Non-custom-SQL metrics
• Source and Target are both tables

When using group-by columns for source-to-target monitors, both metrics must specify group-by columns. Values will only be computed when they can be produced for a group in both the source and the target.

Set up alerts

Source-to-target monitors have the same alerting rules as other monitors. If you have an alert rule that includes either the source or target table or column, the monitor will inherit those rules.

You can confirm where alerts will be sent by viewing the configuration section on the right hand side called "Sending alerts to"