Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# dbt
worktrees/
**/package-lock.yml
package-lock.yml
.dbt/
Expand Down
22 changes: 21 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,23 @@
# dbt_intercom v1.7.0

[PR #84](https://github.com/fivetran/dbt_intercom/pull/84) and [PR #85](https://github.com/fivetran/dbt_intercom/pull/85) includes the following updates:

## Schema/Data Change
**2 total changes • 1 possible breaking change**

| Data Model(s) | Change type | Old | New | Notes |
| ------------- | ----------- | --- | --- | ----- |
| **(Breaking)** `stg_intercom__tag`, `stg_intercom__company_tag_history`, `stg_intercom__contact_tag_history`, `stg_intercom__conversation_tag_history` | Changed data type | integer | string | Casts `tag_id` as a string so joins succeed regardless of source type. |
| `stg_intercom__company_history`, `stg_intercom__company_tag_history`, `stg_intercom__contact_company_history` | explicit data type cast | source data type | string | Casts `company_id` as a string for consistent join keys. |

## Bug Fix
- Resolves a `Database Error` in `intercom__company_enhanced` (and other tag-joined models) caused by mismatched `tag_id` data types. The fix casts all ID join keys to string in staging. See the Schema/Data Change table above.

## Under the Hood
- Migrates the `union_connections`, `apply_source_relation`, and `partition_by_source_relation` macros to the `dbt_fivetran_utils` package.
- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_intercom/#source-casing-for-case-sensitive-destinations) section of the README for details.


# dbt_intercom v1.6.1

[PR #82](https://github.com/fivetran/dbt_intercom/pull/82) includes the following updates:
Expand Down Expand Up @@ -363,4 +383,4 @@ As discussed within [Issue #40](https://github.com/fivetran/dbt_intercom/issues/
- Please note, if you are installing a version of `dbt_utils` in your `packages.yml` that is not in the range above then you will encounter a package dependency error.

# dbt_intercom v0.1.0 -> v0.3.1
Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
52 changes: 13 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,17 +68,15 @@ Include the following intercom package version in your `packages.yml` file:
```yaml
packages:
- package: fivetran/intercom
version: [">=1.6.0", "<1.7.0"]
version: [">=1.7.0", "<1.8.0"]
```
### Define database and schema variables

#### Option A: Single connection
By default, this package runs using your destination and the `intercom` schema. If this is not where your Intercom data is (for example, if your Intercom schema is named `intercom_fivetran`), add the following configuration to your root `dbt_project.yml` file:

```yml
vars:
intercom:
intercom_database: your_database_name
intercom_database: your_destination_name
intercom_schema: your_schema_name
```

Expand All @@ -102,42 +100,10 @@ vars:
name: connection_2_source_name
```

##### Recommended: Incorporate unioned sources into DAG
> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your Intercom connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each Intercom source rather than one set of unioned models.*

By default, this package defines one single-connection source, called `intercom`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your Intercom sources, though the package will run successfully.
#### Optional: Incorporate unioned sources into DAG

To properly incorporate all of your Intercom connections into your project's DAG:
1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_intercom.yml` [file](https://github.com/fivetran/dbt_intercom/blob/main/models/staging/src_intercom.yml).
If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Intercom connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_intercom/blob/main/models/staging/src_intercom.yml). Set the variable `has_defined_sources: true` under the Intercom namespace in your `dbt_project.yml`. Otherwise, your Intercom connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.

```yml
# a .yml file in your root project

version: 2

sources:
- name: <name> # ex: Should match name in intercom_sources
schema: <schema_name>
database: <database_name>
loader: fivetran
config:
loaded_at_field: _fivetran_synced
freshness: # feel free to adjust to your liking
warn_after: {count: 72, period: hour}
error_after: {count: 168, period: hour}

tables: # copy and paste from intercom/models/staging/src_intercom.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
```

> **Note**: If there are source tables you do not have (see [Additional configurations](https://github.com/fivetran/dbt_intercom?tab=readme-ov-file#optional-step-4-additional-configurations)), you may still include them, as long as you have set the right variables to `False`.

2. Set the `has_defined_sources` variable (scoped to the `intercom` package) to `True`, like such:
```yml
# dbt_project.yml
vars:
intercom:
has_defined_sources: true
```
### (Optional) Additional configurations
<details open><summary>Expand/Collapse details</summary>

Expand Down Expand Up @@ -197,6 +163,14 @@ vars:
intercom_<default_source_table_name>_identifier: your_table_name
```

#### Source casing for case-sensitive destinations
By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:

```yml
vars:
fivetran_using_source_casing: true
```

</details>

### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™
Expand Down Expand Up @@ -234,4 +208,4 @@ We highly encourage and welcome contributions to this package. Learn how to cont

## Are there any resources available?
- If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_intercom/issues/new/choose) section to find the right avenue of support for you.
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
config-version: 2
name: 'intercom'
version: '1.6.1'
version: '1.7.0'
require-dbt-version: [">=1.3.0", "<3.0.0"]
models:
intercom:
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

253 changes: 3 additions & 250 deletions docs/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

8 changes: 7 additions & 1 deletion integration_tests/ci/test_scenarios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,10 @@ test_scenarios:
- name: "help center history disabled"
vars:
intercom__using_help_center_history: false
include_incremental: false
include_incremental: false

- name: "MDLS"
vars:
fivetran_using_source_casing: true
include_incremental: false
include_dbt_seed: true
8 changes: 4 additions & 4 deletions integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'intercom_integration_tests'
version: '1.6.1'
version: '1.7.0'
profile: 'integration_tests'
config-version: 2

Expand Down Expand Up @@ -33,7 +33,7 @@ seeds:
+docs:
show: False
intercom_integration_tests:
+quote_columns: "{{ true if target.type == 'redshift' else false }}"
+quote_columns: "{{ target.type == 'redshift' or var('fivetran_using_source_casing', false) }}"
admin_data:
+column_types:
id: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string'}}"
Expand All @@ -56,7 +56,7 @@ seeds:
_fivetran_synced: timestamp
collection_history_data:
+quote_columns: true
+enabled: "{{ true if target.type != 'snowflake' else false }}"
+enabled: "{{ target.type != 'snowflake' or var('fivetran_using_source_casing', false) }}"
+column_types:
id: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string' }}"
help_center_id: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string' }}"
Expand All @@ -70,7 +70,7 @@ seeds:
_fivetran_synced: timestamp
collection_history_data_snowflake:
+quote_columns: true
+enabled: "{{ true if target.type == 'snowflake' else false }}"
+enabled: "{{ target.type == 'snowflake' and not var('fivetran_using_source_casing', false) }}"
+alias: "collection_history_data"
+column_types:
ID: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string' }}"
Expand Down
2 changes: 1 addition & 1 deletion macros/staging/get_company_tag_history_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
{"name": "_fivetran_synced", "datatype": dbt.type_string()},
{"name": "company_id", "datatype": dbt.type_string()},
{"name": "company_updated_at", "datatype": dbt.type_timestamp()},
{"name": "tag_id", "datatype": dbt.type_int()}
{"name": "tag_id", "datatype": dbt.type_string()}
] %}

{{ return(columns) }}
Expand Down
2 changes: 1 addition & 1 deletion macros/staging/get_contact_tag_history_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
{"name": "_fivetran_end", "datatype": dbt.type_timestamp()},
{"name": "contact_id", "datatype": dbt.type_string()},
{"name": "contact_updated_at", "datatype": dbt.type_timestamp()},
{"name": "tag_id", "datatype": dbt.type_int()}
{"name": "tag_id", "datatype": dbt.type_string()}
] %}

{{ return(columns) }}
Expand Down
2 changes: 1 addition & 1 deletion macros/staging/get_conversation_tag_history_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
{"name": "_fivetran_end", "datatype": dbt.type_timestamp()},
{"name": "conversation_id", "datatype": dbt.type_int()},
{"name": "conversation_updated_at", "datatype": dbt.type_timestamp()},
{"name": "tag_id", "datatype": dbt.type_int()}
{"name": "tag_id", "datatype": dbt.type_string()}
] %}

{{ return(columns) }}
Expand Down
2 changes: 1 addition & 1 deletion macros/staging/get_tag_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
{% set columns = [
{"name": "_fivetran_deleted", "datatype": "boolean"},
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
{"name": "id", "datatype": dbt.type_int()},
{"name": "id", "datatype": dbt.type_string()},
{"name": "name", "datatype": dbt.type_string()}
] %}

Expand Down
15 changes: 0 additions & 15 deletions macros/union/apply_source_relation.sql

This file was deleted.

15 changes: 0 additions & 15 deletions macros/union/partition_by_source_relation.sql

This file was deleted.

93 changes: 0 additions & 93 deletions macros/union/union_connections.sql

This file was deleted.

Loading
Loading