Data engineers are responsible for pipelines that move data from source to target systems, and discrepancies between those systems are a common data-quality issue.

Enter Source-to-target monitors. Now data engineers can continuously monitor metrics across two databases and be the first to know about discrepancies. Setting up a source-to-target monitor can be done in minutes. Just like the standard monitors you know and love, source to target monitors will automatically learn the behavior of the data, incorporate seasonality trends, and detect anomalies over time.

Creating a Source to Target Monitor

You can create and configure Source-to-target monitors from the "Monitoring" tab or from any data connection monitoring page.

Select your source and target

Select the source and target data you want to compare over time. Metaplane supports monitoring both tables and columns across the same or different databases.

💡

Did you know?

You can monitor the difference between most standard metrics that Metaplane supports for tables and columns. For example: you might monitor the difference in row counts across two database tables or monitor the nullness of two columns across two databases.

To make it easier for teammates to find the monitor and see any related context, write a helpful name and description.

Select a Format (optional)

By default, source-to-target monitors will monitor the difference in measured values between your two assets. You can also use a percent difference.

💡

When to Use Difference vs. Percent Difference

If you are comparing the row counts across two very large tables, you may want to monitor the absolute difference in row counts rather than percentage difference since small changes in percentage may miss large changes in absolute row count.

If you have two small tables and want to compare row count, a percent difference may be more effective in catching discrepancies even if the absolute row count is small.

Advanced configuration (optional)

Source-to-target monitors support the same advanced configuration as standard monitors including anomaly detection, model configuration, and incident threshold.

Set up alerts

Source-to-target monitors have the same alerting rules as other monitors. If you have an alert rule that includes either the source or target table or column, the monitor will inherit those rules.

You can confirm where alerts will be sent by viewing the configuration section on the right hand side called "Sending alerts to"