Skip to content

fix(persistence): bump gorqlite to avoid panic on empty rqlite write results - #5994

Open
kriscoleman wants to merge 1 commit into
mainfrom
bump-gorqlite-empty-write-panic
Open

fix(persistence): bump gorqlite to avoid panic on empty rqlite write results#5994
kriscoleman wants to merge 1 commit into
mainfrom
bump-gorqlite-empty-write-panic

Conversation

@kriscoleman

@kriscoleman kriscoleman commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

Bumps the vendored gorqlite client so a successful-but-empty write result no longer panics kotsadm, and bumps schemahero to a compatible release.

Problem

[sc-139123]

Older gorqlite versions read the first element of the write-results slice without checking that the slice is non-empty. When rqlite answers a write with HTTP 200 and an empty results array, gorqlite panics:

panic: runtime error: index out of range [0] with length 0
github.com/rqlite/gorqlite.(*Connection).WriteOne  write.go:68

KOTS issues writes during startup migrations (migrateSkippedPreflights) and from the app-status monitor (SetAppStatus), so this can crash kotsadm and leave it in CrashLoopBackOff whenever rqlite returns that empty-but-successful response, for example when the embedded rqlite cluster is in a degraded state.

Fix

  • Bump github.com/rqlite/gorqlite to a revision whose result handling guards the empty case (it returns an empty result rather than indexing out of range).
  • Bump github.com/schemahero/schemahero to a release compatible with that gorqlite revision.
  • gorqlite's Open now returns a *Connection, so MustGetDBSession is adjusted to match.

Testing

  • go build ./cmd/... and the gorqlite-importing packages (pkg/persistence, pkg/store, pkg/tasks, pkg/app, pkg/filestore, pkg/version) compile cleanly.
  • Reproduced the original panic against the previous gorqlite by driving the exact migrateSkippedPreflights write at an endpoint returning 200 {"results":[]}, then confirmed the bumped gorqlite returns a result and does not panic under the same scenario.

panic(err)
}
db = &newDB
db = newDB

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the upstream gorqlite version had a breaking change with this type-signature, so this was changed due to that

@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown

Reviews (1): Last reviewed commit: "fix(persistence): bump gorqlite to avoid..." | Re-trigger Greptile

…results

Older gorqlite versions index results[0] in WriteOne/WriteOneParameterized
without a length check, so a write that returns HTTP 200 with an empty
results array panics the process ("index out of range [0] with length 0")
instead of returning an error. This can crash kotsadm on startup migrations
and on app-status writes when rqlite returns an empty-but-successful result.

Bump gorqlite to a revision whose result handling guards the empty case, and
bump schemahero to a compatible release. gorqlite's Open now returns a
*Connection, so adjust MustGetDBSession and the rqlite migration test to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kriscoleman
kriscoleman force-pushed the bump-gorqlite-empty-write-panic branch from d810f92 to 9379b93 Compare July 28, 2026 22:01
@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown

Reviews (2): Last reviewed commit: "fix(persistence): bump gorqlite to avoid..." | Re-trigger Greptile

@kriscoleman

Copy link
Copy Markdown
Member Author

Flagging a blocker on this bump before it goes further, since the fix is not a clean version bump.

The coupling. The empty-result guard we want only exists on gorqlite revisions that also changed Open to return *Connection (the guard and the signature change landed together upstream). Because the whole module graph resolves a single gorqlite version, bumping gorqlite for pkg/persistence also recompiles schemahero's in-tree rqlite driver against the new signature, which is why schemahero had to move too:

schemahero@v0.22.1/pkg/database/rqlite/connection.go: cannot use &db (**Connection) as *Connection

The wall. The only schemahero releases built against the new gorqlite API are v0.23.0+, and those moved the database drivers to a plugin architecture (hashicorp/go-plugin): postgres/rqlite are now plugin-only, launched as separate schemahero-<driver> binaries from /var/lib/schemahero/plugins, with no in-tree fallback. KOTS calls schemahero in-process (Database{}.PlanSyncFromFile / ApplySync), so v0.25 builds fine but fails at runtime, which is the CI failure here:

--- FAIL: TestMigrateFromPostgresToRqlite
    rqlite_migration_test.go: Failed to update postgres schema: ... postgres driver requires plugin - install schemahero-postgres plugin

So there is no schemahero version that is both compatible with the fixed gorqlite and usable in-process:

schemahero rqlite driver gorqlite API usable in-process by KOTS
v0.22.x (current) in-tree old (value Open) yes, but will not compile with fixed gorqlite
v0.23.0+ plugin binary new (*Connection) no, requires launching plugin binaries

Options I can see:

  1. Adopt the schemahero plugin model: wire a PluginManager at the KOTS call sites and ship schemahero-postgres and schemahero-rqlite (multi-arch, airgap-safe) in the kotsadm and migrations images. This is the forward-looking fix but touches the build/release pipeline and the airgap path.
  2. Carry the guard via a Replicated-maintained gorqlite fork (old base plus the empty-result guard, keeping the value Open) referenced through a go.mod replace. Keeps schemahero untouched and is a small change, at the cost of a maintained fork. I am not in favor of this one: I would rather not take on maintaining a gorqlite fork.
  3. Leave gorqlite pinned for now and guard at the call sites another way.

I do not want to force the version bump given the plugin dependency, so converting to a discussion of direction. Happy to take whichever route the maintainers prefer. For context, the underlying behavior is the panic described above: an rqlite write that returns HTTP 200 with an empty results array makes gorqlite index results[0] and crash kotsadm rather than return an error.

@xavpaice

Copy link
Copy Markdown
Member

Shortcut: [sc-139123]

@banjoh

banjoh commented Jul 29, 2026

Copy link
Copy Markdown
Member

For future travellers

PR: rqlite/gorqlite#62
Upstream issue: rqlite/gorqlite#41

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants