docs(postgres): document Preserve TOAST Values option by gauri-i · Pull Request #597 · datazip-inc/olake-docs

gauri-i · 2026-10-21T07:41:27Z

Documents Postgres preserve_toast_values, introduced in olake#1264.

Updates the UI/CLI docs and shared source.json example
Explains supported Iceberg pos/dv CDC modes & limitations
Advises against changing the setting for existing sources or jobs
Adds troubleshooting fixes: Preserve TOAST Values or REPLICA IDENTITY FULL

Both MDX files compile. Merge after olake#1264 ships.

CLAassistant · 2026-10-21T07:41:43Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Gauri seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

With preserve_toast_values on, a CDC stream on eq fails the sync instead of writing the placeholder.

Descriptions now match the spec: OLake reads unchanged TOAST column values from the Iceberg destination, not from an in-memory cache. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

nayanj98 · 2026-10-26T12:49:22Z

 | Replication Slot `required for CDC` | Logical replication slot (pgoutput) that retains WAL segments until read, ensuring consistent CDC.                                                                            | `olake-repl-slot`                                                                                        |
 | Publication `required for CDC`      | Name of the PostgreSQL publication that defines which tables and operations (INSERT, UPDATE, DELETE, TRUNCATE) to replicate. Must exist in the same database as the replication slot and include all tables for CDC capture. | `postgres_pub` |
 | Initial Wait Time                   | Idle timeout for pgoutput log reading                                                                                                                                         | `120`                                                                                                    |
+| Preserve TOAST Values               | Keeps large column values (TOAST values) that Postgres leaves out of an update by reading them from the destination. Works only with Iceberg destinations whose update type is positional deletes (`pos`) or deletion vectors (`dv`). Off by default. | Off |


When an UPDATE operation leaves a TOAST column unchanged, Postgres omits it. OLake Go writes __olake_unavailable_value__ for that column. With the toggle ON, OLake Go reads that record's previous value from the Iceberg destination and writes that value instead, so the unchanged column still shows its last real value. Works only with the Iceberg Destination whose update type is positional deletes or deletion vectors.

Also make the example value Off -> Off (Default)

nayanj98 · 2026-10-26T12:49:56Z

 | initial_wait_time `required for CDC` | Idle timeout for pgoutput log reading                                                                                                                               | `120`                                                                                                  | INTEGER                                                          |
 | replication_slot `required for CDC`  | Logical replication slot (pgoutput) that retains WAL segments until read, ensuring consistent CDC.                                                                  | `"olake-repl-slot"`                                                                                    | STRING                                                           |
 | publication `required for CDC`       | Name of the PostgreSQL publication that defines which tables and operations (INSERT, UPDATE, DELETE, TRUNCATE) to replicate. Must exist in the same database as the replication slot and include all tables for CDC capture.  | `"postgres_pub"`                        | STRING                                                           |
+| preserve_toast_values                | Keeps large column values (TOAST values) that Postgres leaves out of an update by reading them from the destination. Iceberg destination with update type `pos` or `dv` only. Optional; default `false`. | `true` | BOOLEAN |


Same description change as mentioned above

nayanj98 · 2026-10-26T12:51:17Z

+:::note[Preserve TOAST Values]
+- **Set it when you create the source.** This option can only be changed while creating the source. Once a source or job exists, it can't be changed from the UI.
+- **It limits the update types.** With this option on, Equality deletes (`eq`) is not offered.
+:::


no need to mention about equality deletes again here as we have already mentioned in the table that it only works with positional and deletion vector

For the other point we can directly inform:
This field can only be set while creating the source. Once the source or job exists, it can't be modified.

Also in the heading of note can be removed (just use :::note)

nayanj98 · 2026-10-26T13:07:55Z

+:::warning[Choose this once]
+Set `preserve_toast_values` before the first `discover` and keep it. Changing it might cause errors.
+:::


no need to add warning, we can add a note and mention:
This field can only be set before running the discover command. Changing it later might cause errors.

nayanj98 · 2026-10-26T13:14:00Z

+| Setup | Large values kept? |
+| ----- | ------------------ |
+| Iceberg destination, update type `pos` or `dv`, CDC or Strict CDC | ✅ Yes |
+| Iceberg destination, update type `eq` | ❌ No, placeholder written |
+| Stream in append mode | ❌ No, placeholder written |
+| Parquet / S3 destination | ❌ No, placeholder written |
+| Full refresh / incremental / backfill | Not needed. These read whole rows |
+| Table with `REPLICA IDENTITY FULL` | Not needed. Postgres already sends the value |
+| `wal2json` plugin | ❌ Not supported (deprecated) |


since we have already mentioned above that it only works for iceberg positional and deletion, Iceberg destination part can be removed (as it is redundant), parquet s3 destination can also be removed. Also check once if any other parameter is also necessary to be mentioned here

nayanj98 · 2026-10-26T13:17:43Z

+**Solution**: Use either of these:

-**Solution**: To ensure TOAST column values are always included in the WAL output, run the following on the affected table:
+- Turn on **Preserve TOAST Values** when creating the source (Iceberg destination with update type `pos` or `dv`).


Remove the iceberg requirement from here already mentioned above in the table

… section Remove emojis and add short inline reasons to each row. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

merlynm20 · 2026-10-27T17:28:29Z

+| Preserve TOAST Values               | When an UPDATE operation leaves a TOAST column unchanged, Postgres omits it. OLake Go writes **`__olake_unavailable_value__`** for that column. With the toggle **`ON`** , OLake Go reads that record's previous value from the Iceberg destination and writes that value instead, so the unchanged column still shows its last real value. Works only with the Iceberg Destination whose update type is positional deletes or deletion vectors. | Off (Default) |
+
+:::note Preserve TOAST Values
+This field can only be set while creating the source. Once the source or job exists, it can't be modified.


we should give reason here why it cannot be modified . user shud understand that changing this setting will cause data descrepencies

Added -

Changing it later causes data discrepancies — existing records already hold either real values or __olake_unavailable_value__ depending on what was set at sync time.

merlynm20 · 2026-10-27T17:29:39Z

 | initial_wait_time `required for CDC` | Idle timeout for pgoutput log reading                                                                                                                               | `120`                                                                                                  | INTEGER                                                          |
 | replication_slot `required for CDC`  | Logical replication slot (pgoutput) that retains WAL segments until read, ensuring consistent CDC.                                                                  | `"olake-repl-slot"`                                                                                    | STRING                                                           |
 | publication `required for CDC`       | Name of the PostgreSQL publication that defines which tables and operations (INSERT, UPDATE, DELETE, TRUNCATE) to replicate. Must exist in the same database as the replication slot and include all tables for CDC capture.  | `"postgres_pub"`                        | STRING                                                           |
+| preserve_toast_values                | When an UPDATE operation leaves a TOAST column unchanged, Postgres omits it. OLake Go writes **`__olake_unavailable_value__`** for that column. With the value set as **`true`**, OLake Go reads that record's previous value from the Iceberg destination and writes that value instead, so the unchanged column still shows its last real value. Works only with the Iceberg Destination whose **`update_type`** is **`pos`** or **`dv`** in selected streams. | `false` | BOOLEAN |


what happens if in cli user will set preserve toast as true and update_type as eq ?

Added -
Does not work with eq

merlynm20 · 2026-10-27T17:31:31Z

 | ssh_config                           | Configure OLake Go to connect through an SSH tunnel.  | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | OBJECT                                                           |

+:::note
+Set `preserve_toast_values` before the first `discover` and leave unchanged. Changing it might cause errors.


this will be a part of the source.json right ? so for first time user it will anyway be before discover , but source.json is still editable he might still go and change that. what happens then?

merlynm20 · 2026-10-27T17:32:50Z

+| Setup | Large values kept? |
+| ----- | ------------------ |
+| Only Iceberg destination and update type `pos` or `dv` | Yes — prior value read from destination |
+| Sync Mode: CDC or Strict CDC | Yes — required to detect missing TOAST columns |


i dont think it is available in strict cdc ? Can you once chcek with vaibhav

https://datazip.atlassian.net/wiki/spaces/olaketech/pages/800456707/Toast+documentation+updates

Vaibhav's documentation mentions that it is available under Strict CDC

merlynm20 · 2026-10-27T17:33:18Z

+| `wal2json` plugin | Not supported (deprecated) |
+
+:::info
+Performance: with the option on, a sync may take longer when many updates leave large values unchanged, because each missing value is read from the destination.


"with the option on" : please write better english

merlynm20 · 2026-10-27T17:35:16Z

+Performance: with the option on, a sync may take longer when many updates leave large values unchanged, because each missing value is read from the destination.
+:::
+
+### Do Not Switch It On or Off for an Existing Source or Job


also we are not allowing user to edit this setting

You can change this setting in cli , but if syncs have already run, it may cause data differences. This section explains the impact so you can make an informed decision before making any changes.

we can't block user from turning it on/off in cli, this is explanation for that . in ui this option will be blocked once the source is created.

merlynm20 · 2026-10-27T17:37:42Z

-**Solution**: To ensure TOAST column values are always included in the WAL output, run the following on the affected table:
+- Turn on **Preserve TOAST Values** when creating the source.
+- Run the following on the affected table. Postgres then sends the full row, at the cost of more WAL:



what happens if user upgrades the source version of a already created source ? the toast value toggle will still be OFF , so then assume if user wants to enable this setting what he needs to do ? we need to mention this as a seperate block or something.

…fter creation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…lues section Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… update type Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ata discrepancies Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

gauri-i · 2026-10-28T05:03:39Z

+:::note
+Set `preserve_toast_values` before the first `discover` and leave unchanged. Changing it causes data discrepancies.
+:::
+


@merlynm20

Source.json is editable
If changed after syncs have run, data becomes inconsistent — rows synced with it off hold olake_unavailable_value, rows synced with it on hold real values.

Updated the note to say "causes data discrepancies" instead of "might cause errors"

docs(postgres): document Preserve TOAST Values option

5d63bf0

gauri-i mentioned this pull request Oct 21, 2026

docs(postgres): document Preserve TOAST Values option #595

Closed

gauri-i closed this Oct 21, 2026

gauri-i reopened this Oct 21, 2026

Gauri and others added 5 commits October 21, 2026 16:10

docs(postgres): fix eq row in Preserve TOAST Values table

5f109ca

With preserve_toast_values on, a CDC stream on eq fails the sync instead of writing the placeholder.

docs(postgres): remove memory-first language from Preserve TOAST Values

765495a

Descriptions now match the spec: OLake reads unchanged TOAST column values from the Iceberg destination, not from an in-memory cache. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(postgres): align Preserve TOAST Values content with spec

413b08a

docs(postgres): simplify Preserve TOAST Values field descriptions

13b41d9

docs(postgres): fix Known limits label and remove extra cross-reference

4499c2f

nayanj98 reviewed Oct 26, 2026

View reviewed changes

Comment thread docs/connectors/postgres/index.mdx

nayanj98 reviewed Oct 26, 2026

View reviewed changes

Gauri and others added 2 commits October 26, 2026 18:56

docs(postgres): address PR review comments on Preserve TOAST Values

1a04c98

docs(postgres): clean up When It Works table in Preserve TOAST Values…

5d01920

… section Remove emojis and add short inline reasons to each row. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

merlynm20 requested changes Oct 27, 2026

View reviewed changes

Gauri and others added 4 commits October 28, 2026 10:05

docs(postgres): explain why Preserve TOAST Values cannot be changed a…

a5e1df9

…fter creation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(postgres): improve performance note wording in Preserve TOAST Va…

454cd46

…lues section Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(postgres): note that preserve_toast_values does not work with eq…

373a978

… update type Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(postgres): clarify preserve_toast_values note: changing causes d…

ee58020

…ata discrepancies Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

gauri-i commented Oct 28, 2026

View reviewed changes

Conversation

gauri-i commented Oct 21, 2026

Uh oh!

CLAassistant commented Oct 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nayanj98 Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gauri-i Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nayanj98 Oct 26, 2026 •

edited

Loading

gauri-i Oct 28, 2026 •

edited

Loading