auth-source-secrets-plus

Make Emacs' built-in Secret Service backend for auth-source actually reliable — so smtpmail, mu4e, erc, forge, and anything else that calls auth-source-search can read passwords from KeePassXC, GNOME Keyring, KWallet, or any other freedesktop Secret Service provider without silently failing.

Why this exists

Emacs already ships a secrets backend for auth-source. In theory you enable it with one line:

(setq auth-sources '("secrets:My Database"))

In practice it very often just returns nil, and the reasons are genuinely hard to find: the failures are silent, they stick around for hours, and they depend on whether Emacs or your keyring started first. Worse, once it has failed, fixing your configuration appears to change nothing — so people conclude the backend is broken and go write a shell-out to secret-tool.

The backend isn't broken. It has five specific problems, and this package fixes all five. It does not implement a new backend — it heals the stock one, so entry creation, auth-source-search, auth-info-password and everything else keep working exactly as documented.

Problems 1–3 make lookups return nil. Problems 4 and 5 are worse in different ways: 4 can hang Emacs for 25 seconds and then abort your init file, and 5 makes auto-unlock silently never fire, precisely when it is needed. Both were found the hard way, under EXWM, and both are described in enough detail below to be fixed upstream by someone who wants to.

Problem 1 — secrets-enabled is decided once, at load time

secrets.el sets secrets-enabled exactly once, when the library loads, and only if the Secret Service already answers on D-Bus. Emacs normally starts before your keyring agent, so the flag stays nil for the whole session.

The critical detail: auth-source consults that flag while parsing auth-sources, so the backend is discarded before any search function runs.

auth-source-search: found 0 backends matching (:host "smtp.gmail.com" ...)

This is also why the obvious workaround — advising the backend's search function — cannot work. By the time a search function would run, the backend is already gone.

Problem 2 — the D-Bus session handle goes stale

secrets-session-path is cached for the life of the Emacs session. When your provider restarts, or its database is locked and reopened, that handle becomes invalid and every subsequent read fails:

D-Bus error: "org.freedesktop.Secret.Error.NoSession"

Without this package, recovery means re-evaluating your config by hand.

A nasty subtlety: setting secrets-enabled to t without opening a session is worse than doing nothing — the backend is kept, but every read raises NoSession. Both steps are required, together.

Problem 3 — auth-source caches failures for two hours

This is the one that makes the whole thing feel unfixable. auth-source remembers that a credential was not found, for auth-source-cache-expiry seconds (7200 by default):

auth-source-search: found 0 CACHED results matching (:host "smtp.gmail.com" ...)

So one transient failure — keyring not up yet, database locked — makes every later lookup return nil straight from cache. You correct your configuration, re-evaluate, and nothing changes, because you are reading a cached negative result instead of querying at all.

Problem 4 — loading secrets.el can hang Emacs for 25 seconds, then abort your init

This one doesn't return nil — it drops you in the debugger with a broken Emacs. Under EXWM, where Emacs is the window manager, that means no window manager.

Debugger entered--Lisp error:
  (dbus-error "org.freedesktop.DBus.Error.NoReply" "Call timed out")
  dbus-call-method(:session "org.freedesktop.secrets" ... "OpenSession" ...)
  secrets-open-session()
  require(secrets)
  load-with-code-conversion(".../init.el" ...)

The last form in secrets.el is:

(when (dbus-ping :session secrets-service 100)
  (secrets-open-session)
  ...)

Two independent defects combine here:

dbus-ping does not test what it looks like it tests. It asks the bus daemon whether anyone owns the name — never whether that owner is answering. It returns t in at least three situations where the next call will block:

Situation	dbus-ping	Reality
Provider owns the name, but its service thread is blocked	t	every call hangs
Provider is starting up, name claimed, not yet serving	t	hangs until ready
DBUS_SESSION_BUS_ADDRESS points at a socket that does not exist	t	nothing there at all

That last row is not a typo — verified with emacs -Q --batch:

(setenv "DBUS_SESSION_BUS_ADDRESS" "unix:path=/nonexistent-bus-sock")
(dbus-ping :session "org.freedesktop.secrets" 100)   ;; => t
(require 'secrets)                                   ;; => hangs 25s, then signals

The call it guards has no timeout. secrets-open-session issues its OpenSession with no :timeout, so it waits out D-Bus's 25-second default and then signals dbus-error — out of require, out of your init file. Nor is this the only such call: ReadAlias, SearchItems, GetSecret and Unlock in secrets.el are all unbounded too, so the same stall can happen later, mid lookup.

The 100 in (dbus-ping ... 100) is a red herring. It bounds the ping, not the OpenSession beneath it.

Why this is easy to hit, and why it isn't just "start the keyring first"

The obvious guess — Emacs starts before the keyring, so add a delay — does not work, and the reason matters: a delay assumes the check is honest and merely early. It isn't early, it's wrong. No amount of waiting repairs a question that never tested the thing you care about.

KeePassXC makes this concrete. It answers Ping from a separate Qt D-Bus thread, while the Secret Service itself is implemented on its GUI thread. Block the GUI thread — a modal dialog, an unlock prompt, a stall — and you get a provider that pings healthy and serves nothing. Observed directly:

$ dbus-send --session --dest=org.freedesktop.DBus ... GetNameOwner \
      string:org.freedesktop.secrets
   string ":1.85"                       # name is owned

$ dbus-send --session --dest=org.freedesktop.secrets ... Peer.Ping
   method return ...                    # answers instantly

$ dbus-send --session --dest=org.freedesktop.secrets ... Service.OpenSession ...
   Error org.freedesktop.DBus.Error.NoReply    # after 25s

Anything trusting the ping is affected, not just Emacs — secret-tool was timing out on the same machine at the same time:

mbsync[51930]: secret-tool: Timeout was reached

Fix. Load secrets.el with dbus-ping temporarily stubbed to nil, so its load-time block is skipped entirely — it can neither block nor open a session. That leaves exactly the state this package already heals ("keyring not up yet", Problem 1), and reachability is then decided later by a real bounded call rather than a ping:

(cl-letf (((symbol-function 'dbus-ping) (lambda (&rest _) nil)))
  (require 'secrets))

Every otherwise-unbounded Secret Service call is then wrapped so it carries auth-source-secrets-plus-ping-timeout. Calls on the prompt interface are deliberately left unbounded — a human answers a passphrase dialog, so there is no sensible deadline.

Measured on a provider in the wedged state above: 25 s hard failure → 0.05 s load, with lookups bounded at ~2 s.

Fixing this upstream would take two small changes to secrets.el: give the load-time OpenSession an explicit :timeout, and treat a ping as necessary-but-not-sufficient (or drop it in favour of a bounded real call). Neither needs new API. Passing :timeout to dbus-call-method is enough, though note dbus-get-property hardcodes :timeout 500 and accepts no override, so property reads need dbus-call-method directly.

Problem 5 — a locked collection can't be found, so it can't be unlocked

A genuine chicken-and-egg, and the reason auth-source-secrets-plus-auto-unlock could appear to do nothing at all.

secrets.el finds a collection by comparing your configured name against each collection's Label property. But a locked KeePassXC database will not report its real label — while locked it publishes a placeholder derived from the filename, and only switches to the configured label once unlocked:

State	Label reported	secrets-collection-path "Keepass Database"
Locked	"KeePassDB" (from KeePassDB.kdbx)	nil
Unlocked	"Keepass Database"	/org/freedesktop/secrets/collection/KeePassDB

So while the database is locked, the collection you configured does not exist as far as a label lookup is concerned. The consequences cascade:

• secrets-collection-path returns nil
• the locked check therefore answers nil — reporting "not locked" when it actually means "cannot see it"
• so auto-unlock concludes there is nothing to unlock, and never fires — at exactly the moment it is needed
• and diagnostics report the collection as missing, inviting you to go and "fix" a name that was correct all along

Note the API is circular here, not merely awkward: unlocking requires identifying the collection, and identifying it requires a label that only becomes available after unlocking.

"Resolving by path rather than by label" — what that actually means

Every object on D-Bus has two quite different identifiers, and conflating them is the bug:

• A path is the object's permanent address, like a filesystem path: /org/freedesktop/secrets/collection/KeePassDB. It is assigned when the collection is exposed and it does not change. It is how you talk to the thing. (In this example your keepass database is KeePassDB.kdbx)
• A label is a mutable, human-facing property of that object — the display name shown in a UI. It is data the object hands out when asked, and it can change at runtime. Which is precisely what happens on unlock.

secrets.el only accepts a label, then searches every collection asking each for its Label until one matches. That is a lookup by mutable display name — and it fails whenever the display name is withheld or changes.

Resolving by path means: get the object's address once, then use the address for the actual operations, instead of re-deriving it from a name that might have changed underneath you. In this package:

1. Try the normal label lookup first (correct and unambiguous when it works).
2. If that finds nothing, fall back to the collection's path — and when the provider exposes exactly one collection, that is unambiguously the one meant, whatever it currently calls itself.
3. If the label doesn't match and there are several collections, return nil rather than guessing. Picking the wrong keyring is worse than failing.
4. Unlock via that path, rather than via secrets-unlock-collection — which would just redo the same broken label lookup.

Concretely, instead of "find the collection named Keepass Database" (which fails while locked), it becomes "unlock the object at /org/freedesktop/secrets/collection/KeePassDB" — which works regardless of what that object is currently calling itself.

The path stays stable across the transition, which is what makes this reliable: after unlocking, the label becomes "Keepass Database" (or whatever you called it in the Keepassxc GUI) while the path remains /collection/KeePassDB. The filename-derived path is not a bug to route around — it's the one identifier that doesn't move.

Fixing this upstream would mean having secrets.el accept an object path anywhere it currently accepts a label, and having secrets-unlock-collection take either. A secrets-collection-locked-p that distinguishes "not locked" from "cannot determine" would help too — collapsing those two into nil is what hid this.

Installation

Not yet on MELPA — see melpa-recipe for the submission recipe. Until then, use one of the methods below.

Manual

(add-to-list 'load-path "/path/to/auth-source-secrets-plus")
(require 'auth-source-secrets-plus)

use-package with straight.el

(use-package auth-source-secrets-plus
  :straight (auth-source-secrets-plus
             :type git :host github
             :repo "xpusostomos/auth-source-secrets-plus")
  :custom (auth-source-secrets-plus-collection "My Database")
  :config (auth-source-secrets-plus-enable))

use-package with Emacs 30's built-in :vc

(use-package auth-source-secrets-plus
  :vc (:url "https://github.com/xpusostomos/auth-source-secrets-plus" :rev :newest)
  :custom (auth-source-secrets-plus-collection "My Database")
  :config (auth-source-secrets-plus-enable))

Requires Emacs 26.1+ built with D-Bus support, and a running Secret Service provider.

Quick start

(require 'auth-source-secrets-plus)
(setq auth-source-secrets-plus-collection "My Database")
(auth-source-secrets-plus-enable)

Don't know your collection's exact label? Run M-x auth-source-secrets-plus-list-collections.

Then confirm it works — this reports the secret's length, never the secret:

M-x auth-source-secrets-plus-check

Storing a credential so auth-source can find it

This is the single most common reason lookups fail, and it has nothing to do with the bugs above. auth-source matches on an item's attributes, not its title. A query for :host/:user/:port only finds an item that carries host, user and port attributes with those exact values. An entry that merely looks right by name will never be found.

Store one with secret-tool:

secret-tool store --label="Gmail SMTP" \
    host smtp.gmail.com \
    user you@gmail.com \
    port 465

In KeePassXC, add these as custom attributes on the entry (and make sure the database is exposed via Settings → Secret Service Integration).

To see what attributes your existing items actually have:

M-x auth-source-secrets-plus-list-items

That prints every item with its attributes and never prints secrets.

Gotcha: the port attribute must match the port you search with. If you store port 465 and later switch to 587, the lookup silently returns nil. Either update the attribute or omit :port from your queries.

Examples

Gmail SMTP with smtpmail

A complete, working outgoing-mail setup:

(require 'auth-source-secrets-plus)
(setq auth-source-secrets-plus-collection "My Database")
(auth-source-secrets-plus-enable)

(setq send-mail-function        'smtpmail-send-it
      message-send-mail-function 'smtpmail-send-it
      smtpmail-smtp-server  "smtp.gmail.com"
      smtpmail-smtp-service 465
      smtpmail-stream-type  'ssl
      smtpmail-smtp-user    "you@gmail.com"
      user-mail-address     "you@gmail.com")

For KeepassXC the database or collection name is the name you see in Database ->Database Settings-> Database name

Matching keyring entry:

secret-tool store --label="Gmail SMTP" \
    host smtp.gmail.com user you@gmail.com port 465

Note that Gmail requires an app password, not your account password.

A database that locks on idle

KeePassXC can lock itself after a period of inactivity. By default a lookup against a locked database fails; enable auto-unlock to be prompted instead:

(setq auth-source-secrets-plus-auto-unlock t)

This is off by default because unlocking raises a graphical passphrase prompt that blocks Emacs until you answer it. You can also unlock deliberately with M-x auth-source-secrets-plus-unlock.

Note that with KeePassXC the collection's reported name changes while locked (see Problem 5), so don't be alarmed if auth-source-secrets-plus-list-collections shows something like KeePassDB rather than your database name — it switches to the real label on unlock, and this package handles both. Keep auth-source-secrets-plus-collection set to the unlocked name, which is the one under Database → Database Settings → Database name.

Combining with .authinfo.gpg

Leave auth-source-secrets-plus-collection at nil and the package will not touch auth-sources at all — it only installs the reliability fixes. Useful when you want the keyring first and an encrypted file as fallback:

(setq auth-source-secrets-plus-collection nil)
(auth-source-secrets-plus-enable)
(setq auth-sources '("secrets:My Database" "~/.authinfo.gpg"))

Reading a password in your own code

Nothing special is needed — this is stock auth-source:

(auth-source-pick-first-password :host "smtp.gmail.com" :user "you@gmail.com")

Or with full control over the entry:

(let ((info (car (auth-source-search :host "smtp.gmail.com"
                                     :user "you@gmail.com"
                                     :port "465" :max 1))))
  (when info
    (funcall (plist-get info :secret))))

Configuration

Variable	Default	Description
auth-source-secrets-plus-collection	nil	Collection label to search. When nil, auth-sources is left untouched and only the fixes are installed.
auth-source-secrets-plus-auto-unlock	nil	Unlock a locked collection during lookup. Raises a blocking passphrase prompt.
auth-source-secrets-plus-suppress-negative-cache	t	Don't cache failed lookups. Strongly recommended — see Problem 3.
auth-source-secrets-plus-ping-timeout	1000	Milliseconds bounding every D-Bus call this package makes, so it is the longest Emacs can block on an absent or wedged keyring. Raise if your keyring is slow to start; lower to fail faster. Don't set it to nil — that selects D-Bus's 25-second default, which looks like a hang (Problem 4).
auth-source-secrets-plus-verbose	nil	Report healing actions via message. Never logs secrets.

Commands

Command	Purpose
auth-source-secrets-plus-enable	Install the fixes and wire up auth-sources. Idempotent.
auth-source-secrets-plus-disable	Undo everything.
auth-source-secrets-plus-doctor	Step-by-step diagnosis of why lookups fail. Start here.
auth-source-secrets-plus-check	Try a real lookup; reports secret length, never the secret.
auth-source-secrets-plus-list-collections	Show available collections and their lock state.
auth-source-secrets-plus-list-items	Show items and their attributes, without secrets.
auth-source-secrets-plus-unlock	Unlock the configured collection.

Troubleshooting

Run M-x auth-source-secrets-plus-doctor first. It checks each link in the chain and tells you which one is broken.

Symptom	Likely cause
found 0 backends matching	Problem 1. Confirm auth-source-secrets-plus-enable has run.
found 0 CACHED results	Problem 3. You're reading a cached miss — M-x auth-source-forget-all-cached, then retry.
NoSession D-Bus error	Problem 2. Should self-heal; if not, check the provider is still running.
Lookup returns nil, doctor all green	Attribute mismatch. Run auth-source-secrets-plus-list-items and compare against your query.
Secret Service unreachable	No provider on the bus, or the database is locked.
Works in emacs -Q, not in your config	Something else sets auth-sources later. Check M-x auth-source-secrets-plus-doctor for the effective value.
Emacs hangs ~25 s at startup, then dbus-error ... "Call timed out" in the debugger	Problem 4, and it means secrets is being loaded by something before this package. Ensure auth-source-secrets-plus is required first — it is what makes loading secrets safe.
Doctor says OWNED BUT NOT ANSWERING	Problem 4. The provider holds the bus name but isn't serving it — usually a blocked UI (modal dialog, unlock prompt) or a slow start. Restart the keyring agent.
Auto-unlock never prompts, though the database is clearly locked	Problem 5. Pre-fix, the locked check couldn't see a locked collection. Confirm via doctor, which now reports LOCKED plus the placeholder label.
Doctor says collection NOT FOUND but the name looks right	If it's locked, that's Problem 5 and the name is probably fine — the doctor now says LOCKED (provider reports ...) instead. Otherwise compare against auth-source-secrets-plus-list-collections.

When testing configuration changes, always clear the cache first (M-x auth-source-forget-all-cached), or you will be measuring a stale cached answer rather than your change. auth-source-secrets-plus-check does this for you.

For deeper debugging, turn on the stock tracing:

(setq auth-source-debug 'trivia
      auth-source-secrets-plus-verbose t)

How it works

A guarded load, plus advice — all narrow:

• At load time — secrets is required with dbus-ping stubbed to nil, so its load-time block is skipped and cannot block or signal (Problem 4). The signal handlers that block would otherwise have installed — the ones that invalidate cached session and collection handles when the provider restarts — are registered separately, since skipping them would reintroduce Problem 2. Registering them talks to the bus daemon rather than the provider, so it neither blocks nor requires the provider to be up.

• auth-source-search — before each lookup, re-check the bus with a bounded real call (not a ping), force secrets-enabled on, and open a session. This has to happen before auth-sources is parsed, which is why it wraps auth-source-search rather than the backend's own search function. If the call then dies on a stale session, all cached handles are dropped and it retries exactly once. A persistent failure returns nil rather than signalling, because callers like smtpmail treat nil as "prompt me" and would break on an error.

• dbus-call-method, only for the duration of a lookup — adds auth-source-secrets-plus-ping-timeout to Secret Service calls that carry no timeout of their own, so no single call can stall Emacs for D-Bus's 25-second default. Prompt-interface calls are exempt: a human answers those. The advice is removed on exit, including when the body signals.

• auth-source-remember — skip caching when the result is empty, so a transient failure doesn't persist for two hours.

• Collection resolution — by path with a label fallback, so a locked collection that hides its label can still be found and unlocked (Problem 5).

Testing

emacs -Q --batch -L . -l test/auth-source-secrets-plus-test.el \
      -f ert-run-tests-batch-and-exit

Tests needing a live Secret Service are skipped unless you opt in:

ASSP_TEST_COLLECTION="My Database" \
ASSP_TEST_HOST="smtp.gmail.com" \
ASSP_TEST_USER="you@gmail.com" \
ASSP_TEST_PORT="465" \
  emacs -Q --batch -L . -l test/auth-source-secrets-plus-test.el \
        -f ert-run-tests-batch-and-exit

The live tests deliberately reproduce each failure mode — disabled backend, stale session, poisoned negative cache — and assert that a lookup still succeeds. The offline tests cover the rest, including the two that need no provider to demonstrate: that availability is not decided by dbus-ping (Problem 4), that unbounded Secret Service calls acquire a timeout while prompt calls do not, that the bounding advice is removed even when the body signals, and that a collection hiding its label is still found and reported as locked (Problem 5) — while several collections with no label match are not guessed between.

Reproducing Problem 4 by hand, without needing a wedged provider:

;; dbus-ping says the service is there; nothing is.
(setenv "DBUS_SESSION_BUS_ADDRESS" "unix:path=/nonexistent-bus-sock")
(dbus-ping :session "org.freedesktop.secrets" 100)   ;; => t
(require 'secrets)                                   ;; => 25s, then dbus-error

Credits

Inspired by auth-source-1password and auth-source-gopass. The problem here is the inverse of theirs: the backend already exists and works well, it just needed to stop failing quietly.

License

GPL-3.0-or-later. See LICENSE.