Skip to content

docs: Azure Parquet docs #607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sahitya-datazip merged 4 commits into from
Oct 26, 2026
Merged

docs: Azure Parquet docs #607

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
857dc54
add connection string and shared key authentication
nayanj98 Oct 26, 2026
342a8a4
add sas token authentication
nayanj98 Oct 26, 2026
64c00a9
add prerequisites
nayanj98 Oct 26, 2026
9c23bba
update based on review
nayanj98 Oct 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
244 changes: 241 additions & 3 deletions docs/writers/parquet/config.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,28 @@
---
title: "OLake Parquet S3 Writer Setup & Configuration Guide"
description: Configure OLake to write Parquet files to AWS S3, GCS, or MinIO with detailed setup, IAM permissions, HMAC keys, and endpoint configurations
title: "OLake Parquet Writer Setup & Configuration Guide"
description: Configure the Amazon S3 connector to write Parquet files to S3-compatible storage (AWS S3, GCS, or MinIO) or Azure.
sidebar_label: Config and SetUp
sidebar_position: 2
---

import TOCTabLinker from '@site/src/components/TOCTabLinker';

# S3 Parquet

OLake Go writes Parquet through the Amazon S3 connector. **Storage Type** selects S3-compatible storage (AWS S3, GCS, or MinIO) or Azure. Before proceeding, review the [Getting Started](/docs/getting-started/quickstart) and [Installation](/docs/install/olake-ui) sections.

<TOCTabLinker
queryParam="storage-type"
headingMap={{
's3': ['prerequisites', 'configuration'],
'azure': ['prerequisites-1', 'configuration-1', 'authentication'],
}}
/>

<Tabs queryString="storage-type" groupId="parquet-storage">

<TabItem value="s3" label="S3" default>

OLake Go supports writing in parquet format directly to S3. Before proceeding with the S3 Parquet destination, we recommend reviewing the [Getting Started](/docs/getting-started/quickstart) and [Installation](/docs/install/olake-ui) sections.

---
Expand Down Expand Up @@ -50,4 +66,226 @@ OLake Go supports writing in parquet format directly to S3. Before proceeding wi
S3 write test
```
This is used to verify that you have the necessary permissions to write to S3.
:::
:::

</TabItem>

<TabItem value="azure" label="Azure">

OLake Go writes Parquet to Azure when **Storage Type** is Azure on the Amazon S3 connector. Azure supports Connection String, Shared Key, and SAS Token authentication.
Comment thread
merlynm20 marked this conversation as resolved.

---

## Prerequisites

- The identity must be able to list, read, write, and delete blobs in the container.

---

## Configuration

Configure the following fields to write Parquet files to Azure.

### Authentication

It supports multiple authentication types. Select the authentication mechanism your storage account uses.

<Tabs queryString="azure-auth-type" groupId="parquet-azure-auth">

<TabItem value="connection-string" label="Connection String" default>

To use **Connection String** authentication, pass one connection string to OLake Go. The key can be an account key or a SAS key.

- The string must contain `AccountName` or `BlobEndpoint`, and `AccountKey` or `SharedAccessSignature`.
- When the string contains an `AccountKey`, OLake Go pulls that key out and signs each Blob request with it.
- A string with `SharedAccessSignature` is not signed with an `AccountKey`.
- `UseDevelopmentStorage=true` is rejected unless `AccountName`, `AccountKey`, and `BlobEndpoint` are all set.

Follow the steps in the [Manage storage account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) documentation to access the key.

<Tabs queryString="tab">

<TabItem value="azure-ui" label="OLake UI" default>

![OLake S3 destination config fields and reference guide.](/img/docs/s3/azure-conn-string.webp)

**Connection String Configuration Parameters:**

| Parameter | Sample Value | Description |
|-----------|--------------|-------------|
| **Storage Type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. |
| **Azure Auth Type** <br/> `required` | `Connection String` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. |
| **Azure Storage Container Name** <br/> `required` | `olake-data` | Name of an existing Azure storage container with permission to store output files. |
| **Azure Path** | `/data` | Path inside the container where data files are written. |
Comment thread
sahitya-datazip marked this conversation as resolved.
| **Azure Connection String** <br/> `required` | `DefaultEndpointsProtocol=https;AccountName=<ACCOUNT_NAME>;AccountKey=<ACCOUNT_KEY>;EndpointSuffix=core.windows.net` | Connection string OLake Go uses to authenticate Blob requests. The key inside it can be an Account key or a SAS key. |
| **Max File Size (MB)** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. |

</TabItem>

<TabItem value="azure-cli" label="OLake CLI">

```json title="destination.json"
{
"type": "PARQUET",
"writer": {
"storage_type": "Azure",
"azure_auth_type": "connection-string",
"azure_container_name": "olakeauthtest",
"azure_path": "/data",
"azure_connection_string": "",
"max_file_size_mb": ""
}
}
```

**Connection String Configuration Parameters:**

| Parameter | Sample Value | Description |
|-----------|--------------|-------------|
| **storage_type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. |
| **azure_auth_type** <br/> `required` | `connection-string` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. |
| **azure_container_name** <br/> `required` | `olakeauthtest` | Name of an existing Azure storage container with permission to store output files. |
| **azure_path** | `/data` | Path inside the container where data files are written. |
| **azure_connection_string** <br/> `required` | `DefaultEndpointsProtocol=https;AccountName=<ACCOUNT_NAME>;AccountKey=<ACCOUNT_KEY>;EndpointSuffix=core.windows.net` | Connection string OLake Go uses to authenticate Blob requests. The key inside it can be an Account key or a SAS key. |
| **max_file_size_mb** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. |

</TabItem>

</Tabs>

</TabItem>

<TabItem value="shared-key" label="Shared Key">

To use **Shared Key** authentication, copy a storage account access key in Azure and give OLake Go the raw key and the account name as separate fields. OLake Go uses the key to sign each Blob request, and Azure checks the signature. Follow the steps in the [Manage storage account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) documentation to access the key.

<Tabs queryString="tab">

<TabItem value="azure-ui" label="OLake UI" default>

![OLake S3 destination config fields and reference guide.](/img/docs/s3/azure-shared-key.webp)

**Shared Key Configuration Parameters:**

| Parameter | Sample Value | Description |
|-----------|--------------|-------------|
| **Storage Type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. |
| **Azure Auth Type** <br/> `required` | `Shared Key` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. |
| **Azure Storage Account Name** <br/> `required` | `olakedata` | Name of the Azure storage account that contains the container. |
| **Azure Storage Container Name** <br/> `required` | `olake-data` | Name of an existing Azure storage container with permission to store output files. |
| **Azure Path** | `/data` | Path inside the container where data files are written. |
| **Azure Endpoint** | `https://<ACCOUNT_NAME>.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. |
| **Azure Storage Account Key** <br/> `required` | `<ACCOUNT_KEY>` | Raw storage account access key, passed separately from the account name. OLake Go uses the key to sign each Blob request. |
| **Max File Size (MB)** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. |

</TabItem>

<TabItem value="azure-cli" label="OLake CLI">

```json title="destination.json"
{
"type": "PARQUET",
"writer": {
"storage_type": "Azure",
"azure_auth_type": "shared-key",
"azure_storage_account_name": "olakeadlsauthtest",
"azure_container_name": "warehouse",
"azure_path": "",
"azure_endpoint": "",
"azure_storage_account_key": "",
"max_file_size_mb": ""
}
}
```

**Shared Key Configuration Parameters:**

| Parameter | Sample Value | Description |
|-----------|--------------|-------------|
| **storage_type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. |
| **azure_auth_type** <br/> `required` | `shared-key` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. |
| **azure_storage_account_name** <br/> `required` | `olakeadlsauthtest` | Name of the Azure storage account that contains the container. |
| **azure_container_name** <br/> `required` | `warehouse` | Name of an existing Azure storage container with permission to store output files. |
| **azure_path** | `/data` | Path inside the container where data files are written. |
| **azure_endpoint** | `https://<ACCOUNT_NAME>.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. |
| **azure_storage_account_key** <br/> `required` | `<ACCOUNT_KEY>` | Raw storage account access key, passed separately from the account name. OLake Go uses the key to sign each Blob request. |
| **max_file_size_mb** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. |

</TabItem>

</Tabs>

</TabItem>

<TabItem value="sas-token" label="SAS Token">

To use **SAS Token** authentication, generate a shared access signature for your container in Azure and give OLake Go the token and the account name as separate fields. The token limits what OLake Go can do: it is scoped to a container, and its permissions can be set directly on the token or through a stored access policy. OLake Go appends the token to each Blob request, and Azure checks the signature and the permissions before allowing the request.

- The token must be the query string (`sv=...&sig=...`), with or without a leading `?`. It must not be the full blob URL.
- The expiry is whatever was set when the SAS was created.

Follow the steps in the [Grant limited access to Azure Storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) resources using shared access signatures (SAS) documentation to create the token.

<Tabs queryString="tab">

<TabItem value="azure-ui" label="OLake UI" default>

![OLake S3 destination config fields and reference guide.](/img/docs/s3/azure-sas-token.webp)

**SAS Token Configuration Parameters:**

| Parameter | Sample Value | Description |
|-----------|--------------|-------------|
| **Storage Type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. |
| **Azure Auth Type** <br/> `required` | `SAS Token` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. |
| **Azure Storage Account Name** <br/> `required` | `olakedata` | Name of the Azure storage account that contains the container. |
| **Azure Storage Container Name** <br/> `required` | `olake-data` | Name of an existing Azure storage container the SAS token is scoped to. The container must already exist and the token must allow writes. |
| **Azure Path** | `/data` | Path inside the container where data files are written. |
| **Azure Endpoint** | `https://<ACCOUNT_NAME>.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. |
| **Azure SAS Token** <br/> `required` | `sv=2024-11-04&sr=c&sig=<SIGNATURE>` | SAS query string OLake Go appends to each Blob request. Pass `sv=...&sig=...`, not the full blob URL. |
| **Max File Size (MB)** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. |

</TabItem>

<TabItem value="azure-cli" label="OLake CLI">

```json title="destination.json"
{
"type": "PARQUET",
"writer": {
"storage_type": "Azure",
"azure_auth_type": "sas-token",
"azure_storage_account_name": "olakedata",
"azure_endpoint": "https://olakedata.blob.core.windows.net/",
"azure_path": "/data",
"azure_sas_token": "sv=2024-11-04&sr=c&sig=<SIGNATURE>",
"azure_container_name": "warehouse",
"max_file_size_mb": 512
}
}
```

**SAS Token Configuration Parameters:**

| Parameter | Sample Value | Description |
|-----------|--------------|-------------|
| **storage_type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. |
| **azure_auth_type** <br/> `required` | `sas-token` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. |
| **azure_storage_account_name** <br/> `required` | `olakedata` | Name of the Azure storage account that contains the container. |
| **azure_container_name** <br/> `required` | `warehouse` | Name of an existing Azure storage container the SAS token is scoped to. The container must already exist and the token must allow writes. |
| **azure_path** | `/data` | Path inside the container where data files are written. |
| **azure_endpoint** | `https://olakedata.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. |
| **azure_sas_token** <br/> `required` | `sv=2024-11-04&sr=c&sig=<SIGNATURE>` | SAS query string OLake Go appends to each Blob request. Pass `sv=...&sig=...`, not the full blob URL. |
| **max_file_size_mb** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. |

</TabItem>

</Tabs>

</TabItem>

</Tabs>

</TabItem>

</Tabs>
Binary file added static/img/docs/s3/azure-conn-string.webp
View file
Open in desktop
Comment thread
sahitya-datazip marked this conversation as resolved.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/docs/s3/azure-sas-token.webp
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/docs/s3/azure-shared-key.webp
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading