Skip to content

feat: Azure blob storage support in Parquet - #1247

Merged
vikaxsh merged 75 commits into
stagingfrom
azure-blob-storage-parquet
Oct 26, 2026
Merged

feat: Azure blob storage support in Parquet#1247
vikaxsh merged 75 commits into
stagingfrom
azure-blob-storage-parquet

Conversation

@sahitya-datazip

@sahitya-datazip sahitya-datazip commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Added azure blob storage support (ADLS Gen2 via Blob API) as Parquet destination similar to S3.
Azure fields:

  • Azure storage account name
  • Azure storage account key
  • Azure container name
  • Azure Path
    Azure and S3 share objectStore Interface

Dependencies: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How Has This Been Tested?

  • Locally tested - full refresh + CDC
  • Unit tests for azure and s3

Screenshots or Recordings

image image

Documentation

Related PR's (If Any):

github-actions[bot]
github-actions Bot reviewed Oct 20, 2026
View reviewed changes
Comment thread destination/iceberg/local-test/docker-compose.yml Outdated
Comment thread tests/testutils/azurite.go
Comment thread tests/testutils/integration/azurite.go
Comment thread tests/testutils/integration/azurite.go
vikaxsh
vikaxsh reviewed Oct 20, 2026
View reviewed changes
Comment thread destination/parquet/config.go
Comment thread destination/parquet/interface.go Outdated
Comment thread destination/parquet/interface.go Outdated
Comment thread destination/parquet/s3/s3.go Outdated
Comment thread destination/parquet/s3/s3.go
Comment thread destination/parquet/azure/azure.go
Comment thread destination/parquet/azure/azure.go Outdated
Comment thread destination/parquet/azure/azure.go
Comment thread destination/parquet/azure/azure.go
Comment thread tests/testutils/integration/parquet_rolling.go Outdated
github-actions[bot]
github-actions Bot reviewed Oct 20, 2026
View reviewed changes
Comment thread .github/workflows/tests.yml
Comment thread Makefile
Comment thread tests/testutils/integration/azurite.go Outdated
Comment thread tests/testutils/integration/azurite.go
Comment thread tests/testutils/integration/sync.go
@sahitya-datazip
test: fix kafka and db2 for azurite
820ee02
@sahitya-datazip
test: parquet rolling for azurite
1425f84
@sahitya-datazip
fix: trim s3 prefix
8743c34
@sahitya-datazip
chore: rename azure field names
483cd51
@sahitya-datazip
chore: move kind and error checks to object store
70e40c0
@sahitya-datazip
Merge branch 'staging' into azure-blob-storage-parquet
4c7a02a
@sahitya-datazip
chore: go mod tidy for azure dependencies
012cd77
@sahitya-datazip
Merge branch 'staging' into azure-blob-storage-parquet
f6deadb
@sahitya-datazip
fix: skip ADLS Gen2 folders when listing azure blobs
389ed48
@sahitya-datazip
fix: delete ADLS files first then folders
8b69e3d
@sahitya-datazip
chore: use startCopyFromURl for azurite
bc4eb30
@sahitya-datazip
fix: validate azure connection string
9dd6ca5
@sahitya-datazip
chore: UseDevelopmentStorage=true error in connection string
8930c3c
@sahitya-datazip
fix: azure blob HTTP attempt to 5 minutes
cbca3b3
@sahitya-datazip
fix: delete azure prefix files concurrently
e6706ee
@sahitya-datazip
chore: set Azure blob User-Agent to olake
a94b58d
@sahitya-datazip
chore: fix comments
75d48e8
@sahitya-datazip
chore: address comments
5933ea1
@sahitya-datazip
Merge branch 'staging' into azure-blob-storage-parquet
663db3b 
Signed-off-by: Sahitya <sahitya@datazip.io>
@sahitya-datazip
fix: parquet rolling merge
b1a62fb
vikaxsh
vikaxsh reviewed Oct 26, 2026
View reviewed changes
Comment thread destination/iceberg/iceberg.go Outdated
github-actions[bot]
github-actions Bot reviewed Oct 26, 2026
View reviewed changes
Comment thread destination/parquet/azure/azure.go
Comment thread destination/parquet/azure/azure.go
Comment thread destination/parquet/config.go
Comment thread destination/parquet/parquet_2pc.go
Comment thread destination/parquet/parquet.go
Comment thread tests/testutils/integration/azurite.go
saksham-datazip
saksham-datazip reviewed Oct 26, 2026
View reviewed changes

@saksham-datazip saksham-datazip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch was successfully deployed

2 active and 1 inactive deployments
Publish Driver — 24f52c69 Deployed Oct 26, 2026 by sahitya-datazip via Publish driver mysql [manual] #289
integration_tests — 24f52c69 Deployed Oct 26, 2026 by sahitya-datazip via Approve Test Deployment #342
ai-code-review — 24f52c69 Deployed Oct 26, 2026 by sahitya-datazip via OpenCodeReview #713
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments
@saksham-datazip saksham-datazip saksham-datazip left review comments
@vikaxsh vikaxsh vikaxsh approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sahitya-datazip @vikaxsh @saksham-datazip @prashant-shahi