Skip to content

perf(apply-extras): stop sleeping and re-minting a token for every item - #376

Open
jlinenkohl wants to merge 1 commit into
microsoft:mainfrom
jlinenkohl:fix-apply-extras-perf
Open

jlinenkohl wants to merge 1 commit into
microsoft:mainfrom
jlinenkohl:fix-apply-extras-perf

Conversation

@jlinenkohl

Copy link
Copy Markdown

Applying one real agent's extras took about two hours, almost none of it spent talking to the service. Two causes, measured per knowledge item:

jq 26ms · base64 6ms · body 8ms · token 1053ms · PUT 377ms · sleep 5000ms

_dp_token shelled out to the Azure CLI on every call -- once per skill, per subagent, per hook and per knowledge item -- for a token valid about an hour. It is now fetched once and reused. The cache is a file rather than a variable because callers use TOKEN=$(_dp_token), which runs the function in a subshell, so a variable assigned inside it is discarded as soon as the substitution closes and every call re-mints regardless.

The larger cost was an unconditional five-second pause between knowledge items: 79 minutes of sleeping for an agent with 949 of them. It cannot simply be removed -- the service answers a burst of PUTs with 405, and dropping the pause took a 20-item sample from 20 successes down to 8 -- so the transient codes are now retried with backoff and the interval is configurable, defaulting to one second instead of five.

Measured on a ten-item sample against a clean target, counting only item outcomes:

unconditional 5s 74s, 10/10
1s + retry (default) 36s, 10/10
0s + retry 16s, 10/10

Extrapolated to 949 knowledge items, that is roughly two hours down to under an hour at the default, or about twenty-five minutes with SRE_APPLY_ITEM_DELAY=0. Correctness is unchanged in every sample.

Applying one real agent's extras took about two hours, almost none of it spent
talking to the service. Two causes, measured per knowledge item:

  jq 26ms · base64 6ms · body 8ms · token 1053ms · PUT 377ms · sleep 5000ms

_dp_token shelled out to the Azure CLI on every call -- once per skill, per
subagent, per hook and per knowledge item -- for a token valid about an hour.
It is now fetched once and reused. The cache is a file rather than a variable
because callers use TOKEN=$(_dp_token), which runs the function in a subshell,
so a variable assigned inside it is discarded as soon as the substitution
closes and every call re-mints regardless.

The larger cost was an unconditional five-second pause between knowledge items:
79 minutes of sleeping for an agent with 949 of them. It cannot simply be
removed -- the service answers a burst of PUTs with 405, and dropping the pause
took a 20-item sample from 20 successes down to 8 -- so the transient codes are
now retried with backoff and the interval is configurable, defaulting to one
second instead of five.

Measured on a ten-item sample against a clean target, counting only item
outcomes:

  unconditional 5s      74s, 10/10
  1s + retry (default)  36s, 10/10
  0s + retry            16s, 10/10

Extrapolated to 949 knowledge items, that is roughly two hours down to under an
hour at the default, or about twenty-five minutes with
SRE_APPLY_ITEM_DELAY=0. Correctness is unchanged in every sample.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant