Bump nixpkgs from cf5e765 to cd4ea8d

.gitignore

@@ -1,2 +1,3 @@
 book/
 result
+.ipynb_checkpoints/

.typos.toml

@@ -0,0 +1,14 @@
+[default.extend-identifiers]
+# Mermaid class for Alice's cluster in the intersection diagram.
+aci = "aci"
+# Author of The Byzantine Generals Problem.
+Pease = "Pease"
+# Author of Asymmetric Distributed Trust.
+Cachin = "Cachin"
+# Single-use reply blocks.
+SURB = "SURB"
+SURBs = "SURBs"
+
+[default.extend-words]
+# Groth16, the proof system named after Jens Groth.
+Groth = "Groth"

README.md

@@ -1,41 +1,50 @@
 # Fungi
 
-This repository contains documents that describe a family collaborative Bitcoin transaction construction protocols. The protocols form a progression towards decentralized and market based privacy preserving transactions.
+Fungi is a protocol for collaborative transactions designed to make using Bitcoin more private. Participants can also pay each other interactively in these transactions, which helps not just privacy but also blockspace efficiency.
+
+A software development kit (SDK) is [in development](https://github.com/fungi-protocol/fungi). This repository contains the specifications and the analysis behind them.
 
 > [!NOTE]
-> **Work in progress**: most chapters are still under review. See the [open pull requests](https://github.com/fungi-protocol/docs/pulls) for material being drafted.
+> **Work in progress**: most chapters are drafts under review. See the [open pull requests](https://github.com/fungi-protocol/docs/pulls) for material being drafted.
+
+## Background
+
+Censorship resistance and fungibility are among Bitcoin's foundational properties. Without privacy, both can be undermined largely outside the protocol's rules, whether by targeting specific individuals or through chilling effects.
+
+Bitcoin transactions are public data, and they are connected in a graph structure. Chain analysis is the application of heuristics to this graph to infer which coins share an owner. While no heuristic is perfect, their effectiveness has been shown in numerous studies. Several heuristics can be combined for a more accurate analysis. Analysis might also draw on auxiliary information not recorded on the blockchain, such as which coins an exchange sent to which customer.
+
+CoinJoin transactions seek to undermine some of the heuristics used to cluster coins across the transaction graph. They combine several users' inputs and outputs in one transaction in order to obscure ownership links within it. Several CoinJoin implementations exist with different approaches to their transaction structure.
 
-## Fungi protocol suite
+## Problem statement
 
-### Honest setting
+Unfortunately, CoinJoin privacy is often stated in terms of the number of indistinguishable outputs in a transaction. This tacitly ignores the fact that those coins will go on to be spent. The information revealed by the spending transactions, which are themselves related to other transactions, can dramatically affect privacy, in some cases making the CoinJoin inputs linkable to the outputs.
 
-The *honest* setting assumes all participants can be trusted in two ways:
+Recurring relationships give rise to patterns too. Transacting with the same employers, shops and friends leaves a structure in the transaction graph, even if techniques like PayJoin might hide the individual amounts and confound some of the clustering heuristics. Obscuring this structure necessitates transacting with unrelated users, chosen at random, so users who don't know or trust each other need a way to find each other and agree on a transaction. Random selection must additionally be Sybil resistant, so that the cost of an $n-1$ deanonymization attack, where the adversary controls all of the other (apparent) participants, is a sufficient deterrent.
 
-- not to disrupt the execution of the protocol
-- with privacy, which means not revealing to the adversary, ideally forgetting, any information that may be revealed by participants in the course of the protocol's execution
+In general, deanonymizing a single user always makes it easier to deanonymize the other users. But sometimes the adversary might progress not just by process of elimination, but more like divide and conquer. This happens when deanonymizing that one user splits the problem of deanonymizing the rest into several smaller problems, as opposed to merely making it incrementally smaller.
 
-In this as well as subsequent protocol, trust never means  giving up control over one's funds. Input signatures in transactions enforce this safety by letting each party ensure its , requiring unanimous consent by all parties.
+The [problem definition](docs/collaborative_txn_privacy.md) chapter discusses these on-chain privacy leaks in depth, and describes a particular transaction graph structure that resists deanonymization more robustly.
 
-### Semi-honest setting
+## How Fungi works
 
-In the *semi-honest* setting, also known as *honest-but-curious*, we no longer assume peers have to be trusted to keep a secret. Instead this relies on privacy preserving transports for the communication, that allow a peer to broadcast its messages anonymously to the other participants. Because Bitcoin transactions are transparent, privacy further demands a more careful approach to transaction construction, or the actions of a peer may be revealing even if not directly attributable.
+A user first tells their wallet which payments they intend to make, and the wallet keeps these in a queue. Intents can vary in their time, privacy or cost sensitivity. Making a payment quickly, cheaply and privately is a balancing act, and improving one usually means compromising on another. The wallet's job is to construct transactions that make the queued payments, balancing these goals in line with the user's preferences.
 
-The semi-honest setting can provide some degree of confidentiality, especially against 3rd party observers, through the construction of net settlement transactions. But not anonymity. Transactions between peers that are somehow related have inherent privacy issues because such relationships constitute a social network, exhibiting community structure.
+The wallet schedules and batches payments, negotiating with other participants to find terms that suit the user's priorities. The more flexible a payment's timing, the more cost-effective it can be, because there is more time for compatible proposals to be combined, so that its cost can be shared with more participants. The protocol and the strategy are designed together, and the SDK implements both.
 
-### Byzantine fault tolerant setting
+Avoiding links between coins is central to the strategy. The input and output amounts can imply such links if there aren't enough possibilities for how to group inputs and outputs so their totals match. But as mentioned above, a partition of a transaction's coins can also be evaluated based on the surrounding context in the transaction graph, not just the amounts associated with that transaction. So the strategy must consider not just whether the transaction's internal structure is sufficiently ambiguous, but also how a transaction sits in the wider transaction graph.
 
-Overcoming the community analysis concern requires a Sybil resistant, permissionless protocol where peers with no meaningful economic relationship can find each other. This involves an additional *coalition formation* protocol that sets up transaction construction using a market based mechanism.
+In Fungi, a *co-spend proposal* sets out the terms for spending a set of coins together. Proposals are made automatically according to the privacy optimization strategy. The transaction-wide terms of a co-spend proposal are public, constraining things like the range of acceptable feerates or whether or not non-SegWit inputs can be used. Proposals can also include fees, which are kept confidential. A proposal involving three or more parties additionally hides which of them made the proposal. Once accepted by all parties, co-spend proposals are publicly broadcast.
 
-While it is beneficial for privacy, interacting with strangers in a permissionless protocol invites disruption or attempts at surveillance. For honest peers to be able to reliably transact, especially with the unanimity requirement, demands a byzantine fault tolerance from the transaction construction protocol.
+Any participant can collect fully accepted co-spend proposals from multiple parties, and then make a *coalition formation proposal* which aggregates them together, so long as the terms are compatible. If everyone accepts the coalition formation proposal, then a coalition has been formed, and the parties can proceed with collaborative transaction construction. Privacy is brittle in small transactions. A critical mass of participants is therefore necessary for privacy to be robust and cost-effective, and coalition formation is designed to facilitate such a critical mass forming.
 
-In the course of a BFT interaction with untrusted peers, a participant may still use the other variants of the protocol to coordinate activity with the subset of peers they do know or trust, for instance by performing net-settlement within a larger BFT CoinJoin.
+During collaborative transaction construction, participants add their intended inputs and outputs anonymously. This preserves the unlinkability of inputs and outputs so long as it isn't undermined in some other way (for example by reusing an output address). Interactive payments additionally provide privacy against the payment counterparty. In addition to hiding the payment amount from third parties, the sender and receiver do not learn about each other's coins. Since many interactive payments can be settled in one collaborative transaction, this creates the opportunity for blockspace savings.
 
-## Motivation
+In order to ensure funds are secure throughout, every participant retains control at all times and therefore signs their own inputs. Unfortunately, this means that as the number of participants grows, reliability will necessarily degrade. Even if the probability of any one participant causing a fault is low, that compounds when considering the probability that none of them will. Transaction construction is designed to be fault-tolerant, so that even when a valid transaction has not successfully been produced, participants have a basis to refine the coalition structure. Fault tolerance includes malicious disruption.
 
-Wallet clustering is the broad term for a variety of techniques that can be used to deanonymize on chain activity. Clustering involves linking particular coins to one another, attributing their ownership to the same entity. Over the course of Bitcoin's existence many papers have been published on this subject, also about related privacy issues issues, such as Lightning network, or the transport layer.
+## Compared to JoinMarket
 
-This body of work has, time and again, shown that misconceptions about privacy invariably err on the side of optimism. Moreover, depending on individual users' threat models, their adversary may have access to comprehensive auxiliary information. Such auxiliary information doesn't even need to be about the target user, information about other users may be useful for deanonymization of the target user. In cryptography, rotating long lived keys or favoring ephemeral ones can limit the scope of a compromise, but an individual's identity is who they are.
+Fungi could be described as a spiritual successor to JoinMarket (as implemented by [JoinMarket](https://github.com/JoinMarket-Org/joinmarket-clientserver) and [JoinMarket-NG](https://github.com/joinmarket-ng/joinmarket-ng)). Like JoinMarket, it is peer-to-peer and market-based, with no centralized coordinator, but differs substantially in how it approaches the order book mechanism as well as transaction construction.
 
-The fungi protocol suite is designed to make arbitrary transactions possible between mutually consenting but distrusting peers. This means safety, liveness, and unlinkability of protocol messages. Within such a protocol peers are free to pursue whatever outcomes they desire.
+JoinMarket distinguishes two roles: in each transaction, a user in the role of the taker selects several makers from the order book. Makers advertise their willingness to participate in CoinJoin transactions. Generally, the taker will pay the on-chain fees as well as any maker fees. The taker learns which coins belong to the makers it pays, so makers get no privacy from the taker. And because there is a single taker, a transaction can only include as many participants as one taker is willing to pay for.
 
-Privacy does not simply emerge naturally from collaborative transactions. Peers must actively optimize for that. Optimization involves the use of a cost function, an extension or generalization of the kind used for coin selection. This repository makes specific recommendations with which the protocols are designed to be compatible. The recommended privacy related terms address Sybil resistance, the transaction structure (for example how output values are chosen in relation to others' input values) and the graph structure.
+Fungi makes no distinction between the taker and maker roles. Instead, a participant can both send and accept proposals, and proposals involving multiple users can be aggregated, allowing a single transaction to be negotiated more as a group buy of blockspace. As the number of participants grows, the privacy benefits compound for everyone, which means each individual user needs to pay for proportionately less blockspace in order to meet their individual privacy requirements. Finally, the proposed transaction structure aims to address several deanonymization attacks that were not yet understood at the time JoinMarket's design was put in place.

book.toml

@@ -6,6 +6,11 @@ src = "src"
 [build]
 create-missing = false
 
+[preprocessor.jupyter]
+command = "python3 contrib/jupyter_nb.py"
+after = ["links"]
+before = ["graphviz", "mermaid", "katex"]
+
 [preprocessor.graphviz]
 command = "mdbook-graphviz"
 after = ["links"]
@@ -17,8 +22,11 @@ after = ["links", "graphviz"]
 [preprocessor.katex]
 command = "mdbook-katex"
 after = ["links"]
+# mdbook-graphviz re-serializes the markdown, escaping the TeX source.
+before = ["graphviz"]
 
 [output.html]
 additional-css = ["theme/diagrams.css"]
 git-repository-url = "https://github.com/fungi-protocol/docs"
-site-url = "/docs/"
+site-url = "/docs-draft/"
+additional-js = ["theme/nb-table.js"]

contrib/book.py

@@ -4,14 +4,19 @@
 docs/index.md, the bundle's OKF index, states the chapter order: its
 headings become the parts of SUMMARY.md and its entries the chapters,
 without their descriptions, so an entry must name a chapter, by a path
-relative to docs/, and every chapter must have an entry. The README is
-the introduction, and the index, with its descriptions, is the contents
-page. Frontmatter is checked and then dropped, as mdbook would render it
-as text. The bundle's other files are copied as they are.
+relative to docs/, and every chapter must have an entry. An entry
+indented two spaces more than the entry above it is a sub-chapter of
+that entry. The README is the introduction, and the index, with its
+descriptions, is the contents page. Frontmatter is checked and then
+dropped, as mdbook would render it as text. Chapters are markdown pages
+and Jupyter notebooks, whose frontmatter is a raw cell that opens them.
+The bundle's other files are copied as they are, except for Jupyter's
+autosaves.
 
     python3 book.py ROOT
 """
 import argparse
+import json
 import posixpath
 import re
 import shutil
@@ -21,20 +26,43 @@
 import yaml
 
 FRONTMATTER = re.compile(r"\A---\n(.*?)\n---[ \t]*(?:\n+|\Z)", re.DOTALL)
-ENTRY = re.compile(r"\* \[([^\]]+)\]\(([^)\s]+)\)(?: - .*)?")
+ENTRY = re.compile(r"( *)\* \[([^\]]+)\]\(([^)\s]+)\)(?: - .*)?")
 DOCS = "docs"
 # As set in book.toml.
 SRC = "src"
 CONTENTS = "contents.md"
 GENERATED = ("README.md", "SUMMARY.md", CONTENTS)
 STATUSES = ("draft", "stable", "deprecated")
 RESERVED = ("index.md", "log.md")
+NOTEBOOK = ".ipynb"
+CHECKPOINTS = ".ipynb_checkpoints"
+# A sub-chapter's entry is indented this much more than its chapter's.
+INDENT = "  "
 
 
 def strip_frontmatter(text):
     return FRONTMATTER.sub("", text, count=1)
 
 
+def notebook_frontmatter(text):
+    """A notebook's frontmatter, as a markdown page would have it, if a
+    raw cell holding nothing else opens it, and the notebook without it."""
+    nb = json.loads(text)
+    cells = nb.get("cells", [])
+    if cells and cells[0].get("cell_type") == "raw":
+        head = "".join(cells[0].get("source", []))
+        if FRONTMATTER.fullmatch(head):
+            nb["cells"] = cells[1:]
+            return head, json.dumps(nb, indent=1, ensure_ascii=False) + "\n"
+    return "", text
+
+
+def body(name, text):
+    if name.endswith(NOTEBOOK):
+        return notebook_frontmatter(text)[1]
+    return strip_frontmatter(text)
+
+
 def frontmatter(text):
     m = FRONTMATTER.match(text)
     fields = yaml.safe_load(m.group(1)) if m else None
@@ -47,6 +75,8 @@ def check(name, text):
     has passed is not checked, so that a build does not depend on when it
     runs."""
     try:
+        if name.endswith(NOTEBOOK):
+            text = notebook_frontmatter(text)[0]
         fields = frontmatter(text)
     except (yaml.YAMLError, ValueError) as e:
         raise ValueError(f"{name}: {e}") from e
@@ -71,20 +101,35 @@ def check(name, text):
 def summary(index, pages):
     """SUMMARY.md from an OKF index: `# Part` headings and
     `* [Title](file.md) - description` entries, naming each of `pages`,
-    the paths in docs/ of the pages the book is given."""
+    the paths in docs/ of the pages the book is given. Indented entries
+    are nested under the entry above them."""
     out = ["# Summary", "", "[Introduction](README.md)", f"[Contents]({CONTENTS})"]
     listed = set()
+    # The depth of the entry above, or -1 at the start of a part, where
+    # there is none to nest under.
+    above = -1
     for line in strip_frontmatter(index).splitlines():
         if line.startswith("# "):
             out += ["", line, ""]
+            above = -1
         elif m := ENTRY.fullmatch(line):
-            chapter = posixpath.normpath(m.group(2))
+            depth, odd = divmod(len(m.group(1)), len(INDENT))
+            if odd:
+                raise ValueError(f"index entry has odd indentation: {line!r}")
+            if depth > above + 1:
+                if above < 0:
+                    raise ValueError(f"index entry has no entry above: {line!r}")
+                raise ValueError(
+                    f"index entry is nested more than one level deeper: {line!r}"
+                )
+            above = depth
+            chapter = posixpath.normpath(m.group(3))
             if chapter not in pages:
                 raise ValueError(f"index entry is not a chapter: {line!r}")
             if chapter in listed:
                 raise ValueError(f"index entry repeats a chapter: {line!r}")
             listed.add(chapter)
-            out.append(f"- [{m.group(1)}]({chapter})")
+            out.append(f"{INDENT * depth}- [{m.group(2)}]({chapter})")
         elif line.strip():
             raise ValueError(f"unexpected line in index: {line!r}")
     if unlisted := sorted(set(pages) - listed):
@@ -103,8 +148,8 @@ def assemble(root):
     docs, out = root / DOCS, root / SRC
     index = (docs / "index.md").read_text()
     pages = {}
-    for page in sorted(docs.rglob("*.md")):
-        if page.name not in RESERVED:
+    for page in sorted([*docs.rglob("*.md"), *docs.rglob("*" + NOTEBOOK)]):
+        if page.name not in RESERVED and CHECKPOINTS not in page.parts:
             name = page.relative_to(docs).as_posix()
             pages[name] = page.read_text()
             check(name, pages[name])
@@ -136,14 +181,14 @@ def assemble(root):
     shutil.copytree(
         docs,
         out,
-        ignore=shutil.ignore_patterns("*.md", ".gitignore"),
+        ignore=shutil.ignore_patterns("*.md", "*" + NOTEBOOK, CHECKPOINTS, ".gitignore"),
         dirs_exist_ok=True,
     )
     for name, text in generated.items():
         (out / name).write_text(text)
     for name, text in pages.items():
         (out / name).parent.mkdir(parents=True, exist_ok=True)
-        (out / name).write_text(strip_frontmatter(text))
+        (out / name).write_text(body(name, text))
 
 
 def main(argv=None):