Describe the feature
We are using cosmos + dbt-databricks in Airflow 3 to run DBT dags. In order to track costs of different dags, we are trying to take advantage of Databricks query tags. https://docs.databricks.com/aws/en/sql/user/queries/query-tags. However, the official supported way to add query tags in dbt is modifying the dbt_project.yml. However, doing it this way still cannot make sure every query has the tags instrumented since the yaml file is user controlled.
Describe alternatives you've considered
We have tried using query comments but it is still user controlled and the solution isn't as structured as query tags.
Who will this benefit?
Platform teams who want to have better auditing and cost attribution capabilities.
Describe the feature
We are using cosmos + dbt-databricks in Airflow 3 to run DBT dags. In order to track costs of different dags, we are trying to take advantage of Databricks query tags. https://docs.databricks.com/aws/en/sql/user/queries/query-tags. However, the official supported way to add query tags in dbt is modifying the dbt_project.yml. However, doing it this way still cannot make sure every query has the tags instrumented since the yaml file is user controlled.
Describe alternatives you've considered
We have tried using query comments but it is still user controlled and the solution isn't as structured as query tags.
Who will this benefit?
Platform teams who want to have better auditing and cost attribution capabilities.