diff --git a/docs/writers/parquet/config.mdx b/docs/writers/parquet/config.mdx index cc06057fc..45981d096 100644 --- a/docs/writers/parquet/config.mdx +++ b/docs/writers/parquet/config.mdx @@ -1,12 +1,28 @@ --- -title: "OLake Parquet S3 Writer Setup & Configuration Guide" -description: Configure OLake to write Parquet files to AWS S3, GCS, or MinIO with detailed setup, IAM permissions, HMAC keys, and endpoint configurations +title: "OLake Parquet Writer Setup & Configuration Guide" +description: Configure the Amazon S3 connector to write Parquet files to S3-compatible storage (AWS S3, GCS, or MinIO) or Azure. sidebar_label: Config and SetUp sidebar_position: 2 --- +import TOCTabLinker from '@site/src/components/TOCTabLinker'; + # S3 Parquet +OLake Go writes Parquet through the Amazon S3 connector. **Storage Type** selects S3-compatible storage (AWS S3, GCS, or MinIO) or Azure. Before proceeding, review the [Getting Started](/docs/getting-started/quickstart) and [Installation](/docs/install/olake-ui) sections. + + + + + + + OLake Go supports writing in parquet format directly to S3. Before proceeding with the S3 Parquet destination, we recommend reviewing the [Getting Started](/docs/getting-started/quickstart) and [Installation](/docs/install/olake-ui) sections. --- @@ -50,4 +66,226 @@ OLake Go supports writing in parquet format directly to S3. Before proceeding wi S3 write test ``` This is used to verify that you have the necessary permissions to write to S3. -::: \ No newline at end of file +::: + + + + + +OLake Go writes Parquet to Azure when **Storage Type** is Azure on the Amazon S3 connector. Azure supports Connection String, Shared Key, and SAS Token authentication. + +--- + +## Prerequisites + +- The identity must be able to list, read, write, and delete blobs in the container. + +--- + +## Configuration + +Configure the following fields to write Parquet files to Azure. + +### Authentication + +It supports multiple authentication types. Select the authentication mechanism your storage account uses. + + + + + +To use **Connection String** authentication, pass one connection string to OLake Go. The key can be an account key or a SAS key. + +- The string must contain `AccountName` or `BlobEndpoint`, and `AccountKey` or `SharedAccessSignature`. +- When the string contains an `AccountKey`, OLake Go pulls that key out and signs each Blob request with it. +- A string with `SharedAccessSignature` is not signed with an `AccountKey`. +- `UseDevelopmentStorage=true` is rejected unless `AccountName`, `AccountKey`, and `BlobEndpoint` are all set. + +Follow the steps in the [Manage storage account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) documentation to access the key. + + + + + +![OLake S3 destination config fields and reference guide.](/img/docs/s3/azure-conn-string.webp) + +**Connection String Configuration Parameters:** + +| Parameter | Sample Value | Description | +|-----------|--------------|-------------| +| **Storage Type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. | +| **Azure Auth Type**
`required` | `Connection String` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. | +| **Azure Storage Container Name**
`required` | `olake-data` | Name of an existing Azure storage container with permission to store output files. | +| **Azure Path** | `/data` | Path inside the container where data files are written. | +| **Azure Connection String**
`required` | `DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net` | Connection string OLake Go uses to authenticate Blob requests. The key inside it can be an Account key or a SAS key. | +| **Max File Size (MB)** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. | + + + + + +```json title="destination.json" +{ + "type": "PARQUET", + "writer": { + "storage_type": "Azure", + "azure_auth_type": "connection-string", + "azure_container_name": "olakeauthtest", + "azure_path": "/data", + "azure_connection_string": "", + "max_file_size_mb": "" + } +} +``` + +**Connection String Configuration Parameters:** + +| Parameter | Sample Value | Description | +|-----------|--------------|-------------| +| **storage_type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. | +| **azure_auth_type**
`required` | `connection-string` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. | +| **azure_container_name**
`required` | `olakeauthtest` | Name of an existing Azure storage container with permission to store output files. | +| **azure_path** | `/data` | Path inside the container where data files are written. | +| **azure_connection_string**
`required` | `DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net` | Connection string OLake Go uses to authenticate Blob requests. The key inside it can be an Account key or a SAS key. | +| **max_file_size_mb** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. | + + + + + + + + + +To use **Shared Key** authentication, copy a storage account access key in Azure and give OLake Go the raw key and the account name as separate fields. OLake Go uses the key to sign each Blob request, and Azure checks the signature. Follow the steps in the [Manage storage account access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) documentation to access the key. + + + + + +![OLake S3 destination config fields and reference guide.](/img/docs/s3/azure-shared-key.webp) + +**Shared Key Configuration Parameters:** + +| Parameter | Sample Value | Description | +|-----------|--------------|-------------| +| **Storage Type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. | +| **Azure Auth Type**
`required` | `Shared Key` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. | +| **Azure Storage Account Name**
`required` | `olakedata` | Name of the Azure storage account that contains the container. | +| **Azure Storage Container Name**
`required` | `olake-data` | Name of an existing Azure storage container with permission to store output files. | +| **Azure Path** | `/data` | Path inside the container where data files are written. | +| **Azure Endpoint** | `https://.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. | +| **Azure Storage Account Key**
`required` | `` | Raw storage account access key, passed separately from the account name. OLake Go uses the key to sign each Blob request. | +| **Max File Size (MB)** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. | + + + + + +```json title="destination.json" +{ + "type": "PARQUET", + "writer": { + "storage_type": "Azure", + "azure_auth_type": "shared-key", + "azure_storage_account_name": "olakeadlsauthtest", + "azure_container_name": "warehouse", + "azure_path": "", + "azure_endpoint": "", + "azure_storage_account_key": "", + "max_file_size_mb": "" + } +} +``` + +**Shared Key Configuration Parameters:** + +| Parameter | Sample Value | Description | +|-----------|--------------|-------------| +| **storage_type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. | +| **azure_auth_type**
`required` | `shared-key` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. | +| **azure_storage_account_name**
`required` | `olakeadlsauthtest` | Name of the Azure storage account that contains the container. | +| **azure_container_name**
`required` | `warehouse` | Name of an existing Azure storage container with permission to store output files. | +| **azure_path** | `/data` | Path inside the container where data files are written. | +| **azure_endpoint** | `https://.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. | +| **azure_storage_account_key**
`required` | `` | Raw storage account access key, passed separately from the account name. OLake Go uses the key to sign each Blob request. | +| **max_file_size_mb** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. | + + + + + + + + + +To use **SAS Token** authentication, generate a shared access signature for your container in Azure and give OLake Go the token and the account name as separate fields. The token limits what OLake Go can do: it is scoped to a container, and its permissions can be set directly on the token or through a stored access policy. OLake Go appends the token to each Blob request, and Azure checks the signature and the permissions before allowing the request. + +- The token must be the query string (`sv=...&sig=...`), with or without a leading `?`. It must not be the full blob URL. +- The expiry is whatever was set when the SAS was created. + +Follow the steps in the [Grant limited access to Azure Storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) resources using shared access signatures (SAS) documentation to create the token. + + + + + +![OLake S3 destination config fields and reference guide.](/img/docs/s3/azure-sas-token.webp) + +**SAS Token Configuration Parameters:** + +| Parameter | Sample Value | Description | +|-----------|--------------|-------------| +| **Storage Type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. | +| **Azure Auth Type**
`required` | `SAS Token` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. | +| **Azure Storage Account Name**
`required` | `olakedata` | Name of the Azure storage account that contains the container. | +| **Azure Storage Container Name**
`required` | `olake-data` | Name of an existing Azure storage container the SAS token is scoped to. The container must already exist and the token must allow writes. | +| **Azure Path** | `/data` | Path inside the container where data files are written. | +| **Azure Endpoint** | `https://.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. | +| **Azure SAS Token**
`required` | `sv=2024-11-04&sr=c&sig=` | SAS query string OLake Go appends to each Blob request. Pass `sv=...&sig=...`, not the full blob URL. | +| **Max File Size (MB)** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. | + + + + + +```json title="destination.json" +{ + "type": "PARQUET", + "writer": { + "storage_type": "Azure", + "azure_auth_type": "sas-token", + "azure_storage_account_name": "olakedata", + "azure_endpoint": "https://olakedata.blob.core.windows.net/", + "azure_path": "/data", + "azure_sas_token": "sv=2024-11-04&sr=c&sig=", + "azure_container_name": "warehouse", + "max_file_size_mb": 512 + } +} +``` + +**SAS Token Configuration Parameters:** + +| Parameter | Sample Value | Description | +|-----------|--------------|-------------| +| **storage_type** | `Azure` | Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage. | +| **azure_auth_type**
`required` | `sas-token` | Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement. | +| **azure_storage_account_name**
`required` | `olakedata` | Name of the Azure storage account that contains the container. | +| **azure_container_name**
`required` | `warehouse` | Name of an existing Azure storage container the SAS token is scoped to. The container must already exist and the token must allow writes. | +| **azure_path** | `/data` | Path inside the container where data files are written. | +| **azure_endpoint** | `https://olakedata.blob.core.windows.net/` | Blob service URL. Leave empty to use `https://{account}.blob.core.windows.net/`. Set it only when using a custom blob URL. | +| **azure_sas_token**
`required` | `sv=2024-11-04&sr=c&sig=` | SAS query string OLake Go appends to each Blob request. Pass `sv=...&sig=...`, not the full blob URL. | +| **max_file_size_mb** | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. | + + + + + + + + + + + + diff --git a/static/img/docs/s3/azure-conn-string.webp b/static/img/docs/s3/azure-conn-string.webp new file mode 100644 index 000000000..08f4cbbbf Binary files /dev/null and b/static/img/docs/s3/azure-conn-string.webp differ diff --git a/static/img/docs/s3/azure-sas-token.webp b/static/img/docs/s3/azure-sas-token.webp new file mode 100644 index 000000000..51d2bd27a Binary files /dev/null and b/static/img/docs/s3/azure-sas-token.webp differ diff --git a/static/img/docs/s3/azure-shared-key.webp b/static/img/docs/s3/azure-shared-key.webp new file mode 100644 index 000000000..a5886ffd3 Binary files /dev/null and b/static/img/docs/s3/azure-shared-key.webp differ