test, wallet: the fee-market acceptance suite, and a fee bump that spent confidential coins - #197
Open
albertodeluigi wants to merge 5 commits into
Open
albertodeluigi wants to merge 5 commits into
albertodeluigi wants to merge 5 commits into
Conversation
Every fee-market behaviour worth testing is invisible on an idle chain, because on an idle chain the answer is always yes at the relay floor. A wallet cannot be shown to lose an auction that nobody else is bidding in, so "does my transaction get in", "does RBF lift it over the queue" and "does CPFP rescue the parent" have no observable answer today outside feature_any_asset_fee_congestion.py, which builds its wall by hand inside one test. tx-spammer.py manufactures the competition: it splits a wallet into a working set of UTXOs and keeps the mempool holding a few blocks of transactions that all pay a rate the operator chose, so the next block is full and the price of entry is a known number rather than an accident. It is standalone by design -- plain JSON-RPC, no test framework import -- because the node it has to drive is usually a server that has the daemon and not the source tree. The budget, not the clock, is what ends a run, and that is the part worth stating up front: fees are paid to whoever produces the block, which on a shared testnet is somebody else's node, so the money is gone. --plan prints the arithmetic and refuses a duration the wallet cannot afford, naming the rate that would have fitted. At the block sizes the live testnet runs, an hour of full blocks costs six million vB times the rate, which is five figures of value at Bitcoin-mainnet fee levels and a few thousand at ordinary ones. Discovering that from a drained wallet halfway through a test is the failure this guards against. run-local-testnet.py gains the three options that make the free version of the same experiment possible on one machine: --blockmaxweight to shrink the block until a handful of transactions fills it, --initialfreecoins so a wallet has something to bid with, and --client-conf to write the consensus block out again for a non-producing node -- the wallet under test -- which otherwise computes a different genesis and fails to connect in a way that reads as a networking fault. That node gets the same block ceiling as the producers, since its own congestion projection is about the block it would build.
Every question worth asking about the open fee market has the same, useless answer on an idle chain -- yes, at the relay floor -- because while there is room in the next block nothing competes for it. A wallet, a fee estimator or the block assembler can therefore be badly wrong with the whole suite green. feature_any_asset_fee_congestion.py showed the ordering rules hold under congestion; this asks the operator's question instead: does MY transaction confirm, and who decided. Nine cases, in one file so they share one wall: an asset nobody prices; an under-bidder that waits and lands in the gap; a transaction under the producer's own -blockmintxfee reserve, which is refused even against a half-empty block; RBF over the wall; CPFP; and both of those again while switching fee asset, since value rather than atom count is what the mempool compares. Two of the nine are the reason the file exists. Which assets a node accepts fees in is POLICY: producers with different price servers may disagree about what a fee is worth, or whether it is a fee at all, and must still build one chain. A regression there surfaces as a fork rather than as a rejected transaction, so the suite drives the disagreement all the way through -- producer A refuses the transaction from its mempool, producer B mines it, and A's best block becomes B's block. The wall is made by shrinking the block rather than by generating load: -blockmaxweight=8000 leaves about 1 kvB, so three transactions fill a block and a run takes under a minute. The capacity is measured from the suite's own filler at runtime rather than hard-coded, and every exclusion is asserted against a block that was checked to be full first -- a fee-market test that quietly stops congesting is one that passes for the rest of its life while measuring nothing. Three traps cost a false result each while writing it, and the harness now closes them, because the next fee test will meet them too. Coin selection left to itself spends the wallet's own unconfirmed change, so a filler becomes the CHILD of the transaction under test and drags it in as an accidental CPFP -- the suite passed or failed by coin selection. A blinded input, which setup change is on a custom chain, forces the whole transaction to be blinded and fails as "output has nonce, but is not blinded", a confidentiality error wearing a fee error's clothes. And change has to be change: an output paid to a receive address is a payment, so bumpfee cannot shrink it and refuses. doc/sequentia/fee-market-testing.md explains the suite case by case, records those traps, and says which cases a shared testnet can reproduce -- everything about one node's policy, nothing about disagreement between two, which needs a private network or a partner.
Driving the desktop wallet against this suite's own regtest turned up a failure the suite cannot see, and fixing the harness to look for it turned up a second one it had been surviving by luck. The tenth case is the shape that broke: an RBF that switches fee asset, where the new asset has to come from an input of its own and the wallet holds confidential outputs beside explicit ones. It passes, and it stays as the regression guard for that shape -- but it does NOT reproduce the real failure, and says so. A test that passes for the wrong reason is worse than no test, so the measured evidence goes in the documentation instead: bumping across assets produced a transaction the wallet's own node refused as bad-txns-in-ne-out, and decoding the pair shows the fee asset's side failing to balance around an input that is blinded. A bump rebuilds every output explicitly and then leaves coin selection free, where a confidential coin arrives with its value and asset at zero -- so it can pick an input the accounting cannot see. feebumper already refuses a transaction whose OUTPUTS are blinded; the same invariant is not applied to the inputs it goes and fetches. Reproducing it needs an output that is genuinely blinded, which ignoreblindfail quietly prevents here, so the fix is left until it can be verified rather than guessed. The flake is mine. take_confirmed handed out outputs from a cached list, and bumpfee picks its own inputs: one of those can be an entry still sitting in the cache, so a later case builds on an already-spent input and dies as bad-txns-inputs-missingorspent -- in whichever case happens to draw it. Four clean runs had said "deterministic" and were wrong about it. The list is now asked for fresh each time, which listunspent answers authoritatively because it already excludes what the mempool spends.
Bumping the fee into a different asset produced a replacement the wallet's own node refused as bad-txns-in-ne-out -- and it did so AFTER recording the bump and reporting success, which leaves the user with a stuck transaction and a wallet that believes it replaced it. Nothing in the interface says anything is wrong. It was found by driving the desktop wallet against the fee-market suite's own regtest. CreateRateBumpTransaction builds an entirely explicit replacement: it rebuilds every recipient without a blinding key and asks for the new fee asset's change with add_blinding_key = false. It then sets fAllowOtherInputs and leaves coin selection free. A confidential coin reaches selection as a CInputCoin whose effective_value, value and asset are all left at zero, so the wallet can fund the transaction with an input the explicit amount accounting cannot see, and the sums do not match. The same function already refuses to bump a transaction whose OUTPUTS are blinded; the invariant was simply never applied to the inputs the bump goes and fetches. CCoinControl::m_only_explicit_inputs, honoured in AvailableCoins and set by the bumper, closes that. A bump now either funds itself explicitly or fails, and can no longer emit a transaction that cannot be relayed. Failing has to say the true thing. "Insufficient funds" is a lie to someone looking at the balance in their own wallet, so when the fee asset is present but held only in confidential outputs the bumper says that instead, and names the way out: bump in another asset, or unblind the balance first. Two functional cases cover it. The first is the one that matters: a wallet holding the fee asset BOTH ways -- the ordinary state of a wallet that has been used -- must take the explicit coin, and every input of the replacement is checked. Without this change that case fails, so the bug was never confined to wallets whose funds are all confidential. Two things had to be got right for the cases to test anything at all, and both had already produced a green test of nothing. Blinding does not happen merely because the destination is confidential: with ignoreblindfail on, a transaction that cannot balance its blinding drops it silently and pays explicitly, and a single confidential output can never balance -- so the funding uses two, and then asserts the coins are really blinded. And filler paid to the suite's own wallet left 0.001 coins behind, one of which is worth exactly the fee a bump first estimates: branch-and-bound takes it as an exact match, the transaction then grows by an input and a change output, and the bump dies as "Could not cover fee" in whichever case happens to draw it. Filler is now paid out of the wallet. That last one exposed a separate weakness left alone here: a cross-asset bump whose added input makes the transaction larger than the estimate it selected against does not retry. It fails loudly rather than corrupting anything, so it is recorded in the documentation rather than fixed in the same change.
Executable bit on the two new scripts, which both carry a shebang; encoding="utf8" on the one open() that lacked it; and "trough of a wave" reworded, because codespell reads it as a misspelling of "through" and a warning nobody can act on is worse than a plainer word.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits, in the order they happened: a tool to make the fee market
observable, the standard suite that measures it, and the wallet bug that
suite found once a real wallet was pointed at it.
Why any of this exists
Every question worth asking about the open fee market has the same,
useless answer on an idle chain -- yes, at the relay floor -- because
while there is room in the next block nothing competes for it. A wallet,
a fee estimator or the block assembler can be badly wrong with the whole
test suite green. The fee market only exists when the next block is full.
contrib/sequentia/tx-spammer.pyHolds a chain at a chosen fee floor: it splits a wallet into independent
UTXOs and keeps the mempool a few blocks deep at a rate the operator
picks, on a closed loop against
getmempoolcongestion. Standalone plainJSON-RPC, because the node it has to drive is usually a server with the
daemon and not the source tree.
The budget, not the clock, ends a run -- fees go to whoever produces the
block, which on a shared testnet is somebody else's node.
--planpricesthe run against the wallet that would pay for it and refuses a duration
it cannot afford, naming the rate that would have fitted.
run-local-testnet.pygains--blockmaxweight,--initialfreecoinsand--client-conf, the last of which writes the configuration of anon-producing node -- the wallet under test -- which otherwise computes a
different genesis and fails to connect in a way that reads as a
networking fault.
test/functional/feature_any_asset_fee_market.pyEleven cases sharing one wall, built by shrinking the block rather than
generating load: about 1 kvB of room, so three transactions fill a block
and a run takes under a minute. Capacity is measured from the suite's own
filler at runtime, and every exclusion is asserted against a block first
checked to be full -- a fee-market test that quietly stops congesting
passes for the rest of its life while measuring nothing.
An asset nobody prices; an under-bidder that waits and lands in the gap;
a transaction under the producer's own
-blockmintxfeereserve, refusedeven against a half-empty block; RBF; CPFP; and both again while
switching fee asset.
Two of them carry the claim the market rests on. Which assets a node
accepts fees in is POLICY, not consensus: producers with different price
servers may disagree about what a fee is worth, or whether it is a fee at
all, and must still build one chain. A regression there surfaces as a
fork rather than as a rejected transaction, so the suite drives the
disagreement all the way through -- one producer refuses the transaction
from its mempool, the other mines it, and the first accepts the block.
doc/sequentia/fee-market-testing.mdexplains the suite case by case andsays which cases a shared testnet can reproduce: everything about one
node's policy, nothing about disagreement between two.
wallet: a fee bump must not spend a confidential coinFound by driving the desktop wallet against that regtest. Bumping the fee
into a different asset produced a replacement the wallet's own node
refused as
bad-txns-in-ne-out-- after recording the bump and reportingsuccess. The user is left with a stuck transaction and a wallet that
believes it replaced it, and nothing in the interface says otherwise.
A bump builds an entirely explicit replacement and then leaves coin
selection free, where a confidential coin arrives as a
CInputCoinwithits value and asset at zero: the wallet can fund the transaction with an
input the amount accounting cannot see. The same function already refuses
to bump a transaction whose OUTPUTS are blinded.
CCoinControl::m_only_explicit_inputsapplies the invariant to theinputs the bump fetches, and the failure path says the true thing rather
than "insufficient funds".
The wallet holding the fee asset both ways -- the ordinary state of a
wallet that has been used -- is enough to hit it, so this was never
confined to wallets whose funds are all confidential.
Verification
The suite is green over four consecutive runs, and case 10 fails without
the wallet change. Cases 1, 2, 4, 5, 7 and 9 were also walked through the
desktop GUI against a two-producer regtest, which is where the bug turned
up.