Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
188 changes: 167 additions & 21 deletions v2.1.x-SNAPSHOT/how-to-guides/data-governance/metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,34 @@ The **Metrics** entity in OpenMetadata lets you define, document, and manage key

## Overview of Metrics

Metrics document calculated values based on data assets and are categorized under the **Governance** section in OpenMetadata. Each metric can be linked to glossary terms, data assets, and other metrics, providing comprehensive visibility into data quality and usage.
Metrics document calculated values based on data assets. OpenMetadata lists them under the **Governance** section.

A metric in OpenMetadata lets you document and share a business metric's definition. This includes its name, its meaning, and the formula or SQL used to calculate it. A metric is documentation only. OpenMetadata saves what you enter but doesn't run the formula or compute a value. Calculate the value in your own database or business intelligence (BI) tool, then record its definition here so everyone can find it. If instead you want OpenMetadata to track how many of your tables and other data assets have a description or an owner, use [Key Performance Indicators (KPIs)](/v2.1.x-SNAPSHOT/how-to-guides/data-insights/kpi). Data Insights calculates them for you.
You can link each metric to:

- Glossary terms
- Data assets
- Other metrics

These links show how the metric relates to your data quality and usage.

## What a Metric Stores

A metric lets you document and share a business metric's definition. It includes:

- **Name**: The metric's unique identifier.
- **Meaning**: A description of what the metric measures.
- **Calculation**: The formula or SQL used to calculate it.

## Calculation and Scheduling

Keep the following in mind about how metrics work:

- **No calculation**: A metric is documentation only. OpenMetadata saves what you enter but doesn't run the formula, compute a value, or schedule a run.
- **Your own tools**: Calculate the value in your own database or business intelligence (BI) tool, then record its definition here so everyone can find it.

<Tip>
If you want OpenMetadata to track how many of your tables and other data assets have a description or an owner, use [Key Performance Indicators (KPIs)](/v2.1.x-SNAPSHOT/how-to-guides/data-insights/kpi). Data Insights calculates them for you.
</Tip>

## Key Properties of a Metric

Expand All @@ -36,43 +61,164 @@ Database Table → Metric → Pipeline → Dashboard

View associated tables, pipelines, and dashboards to understand how each metric is defined and where it's used.

Metrics support column-level lineage too. A metric is usually calculated from one column, such as `Total Sales = SUM(sales.amount)`. You can link that column to the metric. For more information, see [Metric Column Lineage](/v2.1.x-SNAPSHOT/how-to-guides/data-lineage/column#metric-column-lineage).

## Creating Metrics Using the UI

To create a new metric in OpenMetadata using the user interface, follow these steps:

### 1. Navigate to the Metrics Section
1. From the left navigation bar, select **Govern** > **Metrics**.

<img src="/public/images/how-to-guides/governance/metrics/metric-create-navigate.png" alt="Navigate to Govern > Metrics" />

2. Click **Add Metric**.

<img src="/public/images/how-to-guides/governance/metrics/metric-create-add-metric.png" alt="Add Metric button on the Metrics page" />

3. On the **Add New Metric** page, fill in the fields:

- **Name** (required): Enter a unique identifier in camelCase. Start with a lowercase letter. Spaces, underscores, and dots are not supported (for example, `customerRetentionRate`).
- **Display Name** (Optional): Enter a human-readable name (for example, `Customer Retention Rate`).
- **Description** (Optional): Describe what the metric measures and how it is used.
- **Granularity** (Optional): Select the time scale, such as day, week, or month.
- **Metric Type** (Optional): Select the type of calculation, such as count, average, or percentage.
- **Measurement Unit** (Optional): Select the unit of the value, such as count, percentage, or dollar.
- **Language** and **Code**: Select the language (SQL, Python, Java, or JavaScript) and enter the code that documents how the metric is calculated. Both are required.

<img src="/public/images/how-to-guides/governance/metrics/metric-create-form.png" alt="Add New Metric form" />

4. Click **Create** to save the metric. The new metric appears in the **Metrics** list.

<img src="/public/images/how-to-guides/governance/metrics/metric-create-result.png" alt="New metric in the Metrics list" />

## Metric Details Page

To open the Metric Details page:

1. From the left navigation bar, select **Govern** > **Metrics**.
2. In the Metrics list, select the name of the metric you want to open.

The page shows everything about the metric in one place. It has five tabs:

- **Overview**: Description, hierarchy, definition, and related metrics.
- **Lineage**: Where the metric's data comes from and where it is used.
- **Assets**: The data assets linked to the metric.
- **Activity Feeds & Tasks**: Conversations and tasks about the metric.
- **Custom Properties**: Extra fields your organization added to metrics.

<img src="/public/images/how-to-guides/governance/metrics/metric-details-overview.png" alt="Overview tab of a metric" />

### Overview Tab

The top of the page shows the metric name, its status (for example, **Approved**), and the **Domains**, **Owners**, **Tier**, and **Certification** fields. Select the edit icon next to a field to change it.

The **Overview** tab has a main area and a set of side panels.

**Main area**

- **Description**: What the metric measures and how to use it. Select the edit icon to change it.
- **Metric Hierarchy**: Shows where the metric sits among other metrics. For more information, see [Metric Hierarchy](#metric-hierarchy).
- **Definition**: The metric's type, granularity, unit, and calculation code. For more information, see [Metric Definition](#metric-definition).

**Side panels**

- **Data Products**: The data products the metric belongs to.
- **Tags**: Tags applied to the metric.
- **Glossary Term**: Glossary terms linked to the metric.
- **Reviewers**: The people assigned to review the metric.
- **Related Metrics**: Other metrics that relate to this one. For more information, see [Related Metrics](#related-metrics).
- **Custom Properties**: Extra fields your organization added to metrics.

### Metric Hierarchy

Metrics often build on each other. For example, `Revenue` can be the parent of `Online Revenue` and `Store Revenue`. The **Metric Hierarchy** card shows these relationships as a tree, so you can see how a metric fits in without leaving its page.

The tree can show:

- **Metric Group**: The group the metric belongs to, with the number of metrics it holds.
- **Parent metrics**: The metrics above the current one, shown as a path from the top.
- **Peer metrics**: Other metrics that share the same parent.
- **Child metrics**: Variants that sit under the current metric.
- **You are here**: A label that marks the metric you are viewing.

How to use the tree:

- Select a metric in the tree to open its page.
- Select a **Metric Group** to open the Metrics list with that group highlighted.
- Select **Show more Metrics** or **Show more Variants** when a list is long.
- Select **Add Child Metric** to create a new metric under the current one. You see this button only if you have permission to create metrics.

If a metric has no group, parent, or child, the card says: "This metric is standalone, it belongs to no group and has no variants."

<img src="/public/images/how-to-guides/governance/metrics/metric-hierarchy-card.png" alt="Metric Hierarchy card" />

### Metric Definition

The **Definition** card holds the details that describe how the metric is calculated. It shows the code block first, then the metric type, measurement unit, and granularity.

- **Metric Type**: The kind of calculation, such as count, average, or percentage.
- **Granularity**: The time scale, such as daily, weekly, or monthly.
- **Measurement Unit**: The unit of the value, such as count, percentage, or dollar. Choose a listed unit or enter your own custom unit.
- **Language**: The language of the calculation code, such as SQL, Python, or Java.
- **Code**: The formula or query that documents the calculation. The card shows it in a block labeled **Expression**, with the language badge (for example, **SQL**).

To change any of these, select **Edit** on the card, update the fields, and select **Save**. If you choose a language, **Code** is required. If you pick a custom unit, you must enter it.

<img src="/public/images/how-to-guides/governance/metrics/metric-definition-card.png" alt="Metric Definition card" />

### Related Metrics

Use **Related Metrics** to point readers to metrics they should look at together, such as `Gross Revenue` and `Net Revenue`.

- Select the add icon to choose related metrics from the existing metrics.
- Select the edit icon to add or remove entries.
- Use **Show more** when the list is long.

## Link Data Assets to a Metric

The **Assets** tab lists the data assets that a metric uses or describes, such as tables, dashboards, and pipelines. Link assets so that anyone who finds the metric can see where it comes from and where it is used.

<img src="/public/images/how-to-guides/governance/metrics/metric-assets-tab.png" alt="Assets tab of a metric with the asset preview panel" />

### Add Assets

- Go to **Govern > Metrics** in the OpenMetadata UI.
Add assets to a metric from the metric's **Assets** tab.

<img src="/public/images/how-to-guides/governance/metrics/metrics-1.png" alt="Navigate to the Metrics Section" />
1. Open the metric and select the **Assets** tab.
2. Select **Add Assets** in the page header.
3. In the dialog, search for and select the assets you want to link.
4. Confirm your selection.

### 2. Add a New Metric
The linked assets now appear in the list, and the tab shows the number of assets.

- Click on **Add Metric** to initiate the metric creation process.
You see **Add Assets** only if you can edit the metric.

<img src="/public/images/how-to-guides/governance/metrics/metrics-2.png" alt="Add a New Metric" />
### Browse and Preview Assets

### 3. Enter Metric Details
The **Assets** tab lists every asset linked to the metric.

Provide the required information, including:
- The list shows each linked asset with its type and details, in the same layout as the glossary term and domain Assets tabs.
- Select an asset to open a preview panel next to the list. The panel shows the asset's summary without leaving the metric.
- Close the panel to return to the full list.
- If no assets are linked yet, the tab shows "No data assets are linked to this metric yet."

- **Metric Name**
- **Description**
- **Granularity** (time scale of the metric)
- **Metric Type** (e.g., count, average, ratio)
- **Language** and **Code**: the SQL, Python, Java, or JavaScript that documents how the metric is calculated. Both are required.
### Remove Assets

### 4. Create the Metric
Unlink an asset when it no longer relates to the metric. Removing a link does not delete the asset itself.

- After entering the details, click **Create** to finalize the metric.
To remove one asset:

<img src="/public/images/how-to-guides/governance/metrics/metrics-3.png" alt="Create the Metric" />
1. Open the metric and select the **Assets** tab.
2. On the asset, open the three-dot menu and select **Delete**.
3. In the confirmation dialog, select **Delete**.

### 5. View the Created Metric
To remove several assets at once:

- The newly created metric will now be available in the **Metrics** page for reference and further use.
1. On the **Assets** tab, select the checkbox next to each asset, or select **Select All**.
2. In the bar that shows the number of items selected, select **Delete**.
3. Confirm the removal.

<img src="/public/images/how-to-guides/governance/metrics/metrics-4.png" alt="View the Created Metric" />
You can remove assets only if you have the Edit All permission on the metric.

## Example JSON Schema for Metric

Expand Down
88 changes: 82 additions & 6 deletions v2.1.x-SNAPSHOT/how-to-guides/data-lineage/column.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,103 @@ sidebarTitle: How Column-Level Lineage Works

# How Column-Level Lineage Works

OpenMetadata supports rich column-level lineage for understanding the relationship between tables and to perform impact analysis. Users can manually edit both the table and column level lineage to capture any information that is not automatically surfaced.
Column-level lineage shows how individual columns move between tables. Use it to:

- Understand how one table relates to another.
- Run impact analysis before you change a column.
- Trace where a column comes from and what depends on it.

OpenMetadata can generate this lineage automatically. You can also edit it by hand to add anything that is not detected.

<img src="/public/images/how-to-guides/lineage/lineage1.png" alt="Column-Level Data Lineage in OpenMetadata" />

## View Column-Level Lineage

To see the column-level lineage of a table:

1. From the left navigation bar, select **Explore**, or use the search bar at the top.
2. Open the table you want to inspect.
3. Select the **Lineage** tab.

<img src="/public/images/how-to-guides/lineage/column-lineage-tab.png" alt="Lineage tab on a table page" />

4. Select the **Layers** button, and under **Level**, select **Field-level Lineage**. Each table in the graph now lists its columns.
5. Select a column to follow it through the graph.

<Tip>
**Quick Tip:** Drilldown to view all the available columns for a table when viewing column-level lineage.
**Quick Tip**: Expand a table in the graph to see all its columns.
</Tip>

You can generate the column-level lineage automatically by running the **Lineage Agent**.
If a table shows no column links, generate them with a Lineage Agent (see the next section) or add them by hand.

## Generate Column-Level Lineage Automatically

Run a **Lineage Agent** on the database service that holds the tables:

1. Open the database service, then select the **Agents** tab.
2. Select **Add Agent** > **Add Lineage Agent**.

<img src="/public/images/how-to-guides/lineage/column-lineage-add-agent.png" alt="Add Lineage Agent option in the Add Agent menu on a database service" />

3. Configure the agent, then save it.
4. On the **Agents** tab, find the Lineage Agent and select **Run**.

<img src="/public/images/how-to-guides/lineage/column-lineage-run-agent.png" alt="Run button for an agent on the Agents tab" />

<img src="/public/images/how-to-guides/lineage/ingestion.png" alt="Lineage Ingestion" />
When the run finishes, open the table's **Lineage** tab to see the new links.

## Manually Edit Column Level Lineage

OpenMetadata supports manual editing of both table and column level lineage. You can edit the lineage for the individual columns by clicking on the edit option on the top right. User the anchor points on either side of the columns to create links and trace individual columns through their lineage. You can also add new tables that have columns you want to trace. Connect the relevant columns to the current lineage.
OpenMetadata supports manual editing of both table-level and column-level lineage. Use it to add links that are not detected automatically.

<img src="/public/images/how-to-guides/lineage/column1.png" alt="Manually Edit Column Level Lineage" />
To link columns by hand:

1. Open the table and select the **Lineage** tab.
2. Click **Edit** (the pencil icon) in the lineage toolbar. The asset panel opens next to the graph.

<img src="/public/images/how-to-guides/lineage/column-lineage-edit-icon.png" alt="Edit icon in the lineage toolbar" />

3. If the table that holds the column you want to trace is not in the graph, add it first:
- In the asset panel, choose the asset type, such as **Tables**.
- Search for the table and add it to the graph.

<img src="/public/images/how-to-guides/lineage/column-lineage-asset-panel.png" alt="Asset panel in lineage edit mode listing Tables, Dashboards, Charts, Stored Procedures, Topics, and ML Models" />

4. Find the column in the source table. Each column has an anchor point on its edge.
5. Drag from the anchor point of the source column to the anchor point of the column it feeds in the other table.
6. Repeat for each column you want to link.
7. Optional: To remove a link, select the **x** icon on the connecting line.
8. Save your changes.

Watch the video on editing column-level lineage.
<iframe width="800" height="450" src="https://www.youtube.com/embed/HTkbTvi2H9c?start=0&end=51" frameBorder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowFullScreen></iframe>

## Metric Column Lineage

A metric is often calculated from a single column. Link the column to the metric so the lineage shows exactly which field feeds the metric. For example, link `sales.amount` to the metric `Total Sales`.

- **Direction**: Connect a column to a metric (column feeds the metric), or a metric to a column.
- **Endpoint**: A metric has no columns of its own. In column-level lineage, the metric appears as one connection point that carries the metric's own name.
- **Visibility**: The connection point appears on a metric node at the **Field-level Lineage** level or in edit mode. In the default view, a metric node shows no connection point.
- **Impact Analysis**: The column-to-metric link also appears in the column-level Impact Analysis view, so you can see which metrics depend on a column before you change it.

### Link a Column to a Metric

Before you start, make sure you can edit the metric and the table that holds the column.

1. Open the metric and select the **Lineage** tab.
2. Select **Edit** (the pencil icon) in the lineage toolbar.
3. If the table that holds the column is not in the graph, add it from the asset panel. For more information, see [Manually Edit Column Level Lineage](#manually-edit-column-level-lineage).
4. Select the **Layers** button and, under **Level**, select **Field-level Lineage** so the table's columns are listed.
5. Drag from the anchor point beside the column to the anchor point on the metric.
6. Save your changes.

To remove the link, select the **x** icon on the connecting line in edit mode.

<Note>
You can also create these links with the API by adding a `columnsLineage` entry whose `toColumn` is the metric's fully qualified name.
</Note>

<Card title="How to Manually Add or Edit Lineage" href="/v2.1.x-SNAPSHOT/how-to-guides/data-lineage/manual" >
Edit the table and column level lineage manually.
</Card>
Loading
Loading