Skip to content

Build benchmarks on VMs instead of metal instances which we use for runs #9289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AdamGS wants to merge 2 commits into
base: develop
Choose a base branch
from
+123 −5
Open

Build benchmarks on VMs instead of metal instances which we use for runs #9289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a0a7e66
Build benchmarks on VMs instead of metal instances which we use for runs
AdamGS Aug 7, 2026
9315df0
Merge branch 'develop' into adamg/build-vm
AdamGS Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 62 additions & 1 deletion .github/workflows/develop-bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,53 @@ jobs:
bash scripts/commit-json.sh > new-commit.json
bash scripts/cat-s3.sh vortex-ci-benchmark-results commits.json new-commit.json

build:
timeout-minutes: 60
runs-on: >-
${{ github.repository == 'vortex-data/vortex'
&& format('runs-on={0}/runner=bench-dedicated/family=c6id.8xlarge/extras=s3-cache/tag=build-{1}', github.run_id, matrix.benchmark.id)
|| 'ubuntu-latest' }}
strategy:
fail-fast: false
matrix:
benchmark:
- id: random-access-bench
build_args: "--features lance"
- id: compress-bench
build_args: "--features lance"
steps:
- uses: runs-on/action@v2
if: github.repository == 'vortex-data/vortex'
with:
sccache: s3
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.github/actions/setup-rust
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
enable-sccache: ${{ github.repository == 'vortex-data/vortex' && 'true' || 'false' }}
- name: Build binary
shell: bash
env:
RUSTFLAGS: "-C target-cpu=native -C force-frame-pointers=yes"
run: |
cargo build --bin ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} --features unstable_encodings
- name: Pre-upload benchmark debuginfo to Polar Signals
uses: ./.github/actions/upload-parca-debuginfo
continue-on-error: true
with:
paths: |
target/release_debug/${{ matrix.benchmark.id }}
polarsignals-cloud-token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
project-id: "e5d846e1-b54c-46e7-9174-8bf055a3af56"
- name: Upload benchmark binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ${{ matrix.benchmark.id }}-binary
path: target/release_debug/${{ matrix.benchmark.id }}

bench:
needs: build
if: ${{ !cancelled() }}

@onursatici onursatici Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean we would run bench even if build fails? I mean as long as it is not cancelled we will dispatch the metal bench job only for it to fail soon?

@AdamGS AdamGS Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think needs takes care of that?

@onursatici onursatici Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think needs only is for dependency order, but if is evaluated to determine when to run the child. If we omit if the default is success but we are overriding it to be only to run if not cancelled afaik

timeout-minutes: 120
runs-on: >-
${{ github.repository == 'vortex-data/vortex'
Expand All @@ -57,11 +103,12 @@ jobs:
v4_ingest: false
steps:
- uses: runs-on/action@v2
if: github.repository == 'vortex-data/vortex'
if: github.repository == 'vortex-data/vortex' && matrix.benchmark.id == 'string-bench'
with:
sccache: s3
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.github/actions/setup-rust
if: matrix.benchmark.id == 'string-bench'

@robert3005 robert3005 Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this mean?

@robert3005 robert3005 Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the string-bench special in here?

@AdamGS AdamGS Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good question, let me fix that

with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
enable-sccache: ${{ github.repository == 'vortex-data/vortex' && 'true' || 'false' }}
Expand All @@ -75,13 +122,15 @@ jobs:
- uses: ./.github/actions/system-info

- name: Build binary
if: matrix.benchmark.id == 'string-bench'
shell: bash
env:
RUSTFLAGS: "-C target-cpu=native -C force-frame-pointers=yes"
run: |
cargo build --bin ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} --features unstable_encodings

- name: Pre-upload benchmark debuginfo to Polar Signals
if: matrix.benchmark.id == 'string-bench'
uses: ./.github/actions/upload-parca-debuginfo
continue-on-error: true
with:
Expand All @@ -90,6 +139,18 @@ jobs:
polarsignals-cloud-token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
project-id: "e5d846e1-b54c-46e7-9174-8bf055a3af56"

- name: Download benchmark binary
if: matrix.benchmark.id != 'string-bench'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: ${{ matrix.benchmark.id }}-binary
path: target/release_debug/

- name: Make benchmark binary executable
if: matrix.benchmark.id != 'string-bench'
shell: bash
run: chmod +x target/release_debug/${{ matrix.benchmark.id }}

- name: Setup Polar Signals
uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1
# Profiling is observability only, so a flaky agent download must not fail the benchmark.
Expand Down
63 changes: 60 additions & 3 deletions .github/workflows/pr-bench-runner.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,22 +28,66 @@ permissions:
id-token: write # enables AWS-GitHub OIDC

jobs:
build:
if: inputs.benchmark_id != 'string-bench'
timeout-minutes: 60
runs-on: >-
${{ github.repository == 'vortex-data/vortex'
&& format('runs-on={0}/runner=bench-dedicated/family=c6id.8xlarge/tag=build-{1}{2}', github.run_id, inputs.benchmark_id, github.event.pull_request.head.repo.fork == false && '/extras=s3-cache' || '')
|| 'ubuntu-latest' }}
steps:
- uses: runs-on/action@v2
if: github.event.pull_request.head.repo.fork == false
with:
sccache: s3
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
fetch-depth: 0
- uses: ./.github/actions/setup-rust
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
enable-sccache: ${{ github.event.pull_request.head.repo.fork == false && 'true' || 'false' }}
- name: Build binary
shell: bash
env:
RUSTFLAGS: "-C target-cpu=native -C force-frame-pointers=yes"
run: |
cargo build --package ${{ inputs.benchmark_id }} --profile release_debug \
--features ${{ inputs.with_lance && 'lance,' || '' }}unstable_encodings
- name: Pre-upload benchmark debuginfo to Polar Signals
if: github.event.pull_request.head.repo.fork == false
uses: ./.github/actions/upload-parca-debuginfo
continue-on-error: true
with:
paths: |
target/release_debug/${{ inputs.benchmark_id }}
polarsignals-cloud-token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
project-id: "e5d846e1-b54c-46e7-9174-8bf055a3af56"
- name: Upload benchmark binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ${{ inputs.benchmark_id }}-binary
path: target/release_debug/${{ inputs.benchmark_id }}

bench:
needs: build
if: ${{ !cancelled() && (inputs.benchmark_id == 'string-bench' || needs.build.result == 'success') }}
timeout-minutes: 120
runs-on: >-
${{ github.repository == 'vortex-data/vortex'
&& format('runs-on={0}/runner=bench-dedicated/family=c6id.metal/tag={1}{2}', github.run_id, inputs.benchmark_id, github.event.pull_request.head.repo.fork == false && '/extras=s3-cache' || '')
|| 'ubuntu-latest' }}
steps:
- uses: runs-on/action@v2
if: github.event.pull_request.head.repo.fork == false
if: github.event.pull_request.head.repo.fork == false && inputs.benchmark_id == 'string-bench'
with:
sccache: s3
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
fetch-depth: 0
- uses: ./.github/actions/setup-rust
if: inputs.benchmark_id == 'string-bench'
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
enable-sccache: ${{ github.event.pull_request.head.repo.fork == false && 'true' || 'false' }}
Expand All @@ -57,6 +101,7 @@ jobs:
- uses: ./.github/actions/system-info

- name: Build binary
if: inputs.benchmark_id == 'string-bench'
shell: bash
env:
RUSTFLAGS: "-C target-cpu=native -C force-frame-pointers=yes"
Expand All @@ -65,7 +110,7 @@ jobs:
--features ${{ inputs.with_lance && 'lance,' || '' }}unstable_encodings

- name: Pre-upload benchmark debuginfo to Polar Signals
if: github.event.pull_request.head.repo.fork == false
if: github.event.pull_request.head.repo.fork == false && inputs.benchmark_id == 'string-bench'
uses: ./.github/actions/upload-parca-debuginfo
continue-on-error: true
with:
Expand All @@ -74,6 +119,18 @@ jobs:
polarsignals-cloud-token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
project-id: "e5d846e1-b54c-46e7-9174-8bf055a3af56"

- name: Download benchmark binary
if: inputs.benchmark_id != 'string-bench'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: ${{ inputs.benchmark_id }}-binary
path: target/release_debug/

- name: Make benchmark binary executable
if: inputs.benchmark_id != 'string-bench'
shell: bash
run: chmod +x target/release_debug/${{ inputs.benchmark_id }}

- name: Setup Polar Signals
if: github.event.pull_request.head.repo.fork == false
uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sql-bench-matrix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
FLAT_LAYOUT_INLINE_ARRAY_NODE: "1"
runs-on: >-
${{ github.repository == 'vortex-data/vortex'
&& format('runs-on={0}/runner=bench-dedicated/family={1}/tag=build{2}', github.run_id, inputs.machine_type, (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && '/extras=s3-cache' || '')
&& format('runs-on={0}/runner=bench-dedicated/family=c6id.8xlarge/tag=build{1}', github.run_id, (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && '/extras=s3-cache' || '')
|| 'ubuntu-latest' }}
steps:
- uses: runs-on/action@v2
Expand Down
Loading