diff --git a/bench/ffi/README.md b/bench/ffi/README.md index 819320db3..19e0a0099 100644 --- a/bench/ffi/README.md +++ b/bench/ffi/README.md @@ -120,86 +120,92 @@ uses the stock public binding APIs. ## rust2-ffi -Measured 2026-11-10 at rust2 `7383790d` plus this harness, moq `ffcb7cd9`, on an Intel -Core Ultra 7 270K Plus (24 threads), Linux, **on a loaded shared host**: the 1-minute load -average, in parentheses beside each number, ran 22 to 29 throughout, so each cell is the -median of 5 rounds. CPU time leaves out time the program waited preempted, so it moves less -under load than wall time, but caches and SMT siblings stay shared; take the CPU tables -first. Rust 1.101.0-nightly 2026-10-14 (the export's default `release` profile, no LTO), GCC 16.2.0, -CPython 3.14.8, OpenJDK 21.0.12.1 with Kotlin 2.4.21 and kotlinx-coroutines 1.10.2, -Swift 6.3.3, Go 1.26.10, Dart 3.13.0. +Measured 2026-11-13 at rust2 `534ef663`, moq `ffcb7cd9`, on an Intel Core Ultra 7 270K +Plus (24 threads), Linux, on a quiet host: the 1-minute load average, in parentheses +beside each number, ran 2 to 3, and each cell is the median of 3 rounds. CPU time leaves +out time the program waited preempted, so take the CPU tables first. Rust +1.101.0-nightly 2026-10-14 (the export's default `release` profile, no LTO), GCC 16.2.0, +CPython 3.14.10, OpenJDK 21.0.13 with Kotlin 2.4.21 and kotlinx-coroutines 1.10.2, Swift +6.4, Go 1.27.3, Dart 3.13.0. + +Against main (`7008d1d3`) at the same load, measured in turn four times, a 100 B write +costs C 4 to 6 ns less, and Swift 6 and Dart 10 more in every pair: per-call cost, since +each copies a 100 B frame as often as main does. Kotlin and Go each moved within the 15 to +25 ns its own runs spread, so no row resolves the copy Kotlin lost at 100 B or 1 KiB. +Bytes Rust returned and a binding passes back cross with no copy, which `ffi/tests` +checks by address and by drop count. CPU time counts every thread of the process: the JVM's JIT and GC threads, Go's GC, and the Swift global executor's workers. So Kotlin's CPU column runs above its wall column, and -Swift's woken read spends 16.6 us of CPU for 5.7 us of wall. Throughput from 16 KiB up is -bound by memory bandwidth, which the host's load shares: its cells moved 2x between runs, -the Rust floor's included, so read only the 100 B and 1 KiB rows as the bindings' cost. +Swift's woken read spends 14.1 us of CPU for 4.8 us of wall. Throughput from 16 KiB up is +bound by memory bandwidth: its cells moved up to 3x between runs even on this quiet host, +so read only the 100 B and 1 KiB rows as the bindings' cost. ### Calls (ns per call, CPU time; load average) | call | Rust | C | C++ | moq-c | Kotlin | Python | Swift | Go | Dart | |---|---|---|---|---|---|---|---|---|---| -| `track.is_closed()`, a handle call | 9.1 (26) | 18 (25) | 18 (26) | | 60 (26) | 53 (25) | 26 (27) | 49 (26) | 47 (26) | -| `demand.sequence()`, a u64 getter | 0.4 (26) | 10 (25) | 10 (26) | | 50 (26) | 47 (25) | 18 (27) | 41 (26) | 29 (26) | -| `track.largest()`, a record return | 18 (26) | 31 (25) | 31 (26) | | 230 (26) | 97 (25) | 36 (27) | 76 (26) | 56 (26) | -| `track.info()`, an object return | 1.1 (26) | 25 (25) | 25 (26) | | 210 (26) | 97 (25) | 50 (27) | 263 (26) | 242 (26) | -| `group.write_frame(at, frame)`, 100 B | 132 (26) | 148 (25) | 161 (26) | | 230 (26) | 196 (25) | 194 (27) | 270 (26) | 209 (26) | -| the same, with a new `Timestamp` each | 133 (26) | 169 (25) | 177 (26) | 267 (28) | 480 (26) | 336 (25) | 221 (27) | 526 (26) | 454 (26) | -| a 100 B frame written and read, ready | 205 (26) | 303 (25) | 337 (26) | | 690 (26) | 560 (25) | 509 (27) | 841 (26) | 735 (26) | -| the same, read after a wake | 329 (26) | 427 (25) | 461 (26) | | 1,630 (26) | 6,912 (25) | 16,580 (27) | 4,711 (26) | 1,776 (26) | -| `frame.payload`, 100 B | 11 (26) | 23 (25) | 24 (26) | | 70 (26) | 150 (25) | 96 (27) | 165 (26) | 67 (26) | +| `track.is_closed()`, a handle call | 9.0 (3) | 18 (3) | 18 (3) | | 60 (3) | 55 (3) | 26 (3) | 47 (3) | 45 (3) | +| `demand.sequence()`, a u64 getter | 0.4 (3) | 10 (3) | 10 (3) | | 50 (3) | 49 (3) | 9.9 (3) | 38 (3) | 28 (3) | +| `track.largest()`, a record return | 18 (3) | 28 (3) | 29 (3) | | 150 (3) | 99 (3) | 36 (3) | 69 (3) | 55 (3) | +| `track.info()`, an object return | 1.1 (3) | 31 (3) | 29 (3) | | 190 (3) | 104 (3) | 38 (3) | 195 (3) | 239 (3) | +| `group.write_frame(at, frame)`, 100 B | 121 (3) | 143 (3) | 148 (3) | | 240 (3) | 198 (3) | 179 (3) | 285 (3) | 214 (3) | +| the same, with a new `Timestamp` each | 122 (3) | 162 (3) | 170 (3) | 219 (3) | 390 (3) | 339 (3) | 208 (3) | 470 (3) | 455 (3) | +| a 100 B frame written and read, ready | 194 (3) | 301 (3) | 327 (3) | | 650 (3) | 552 (3) | 471 (3) | 721 (3) | 739 (3) | +| the same, read after a wake | 308 (3) | 444 (3) | 450 (3) | | 1,590 (3) | 6,917 (3) | 14,061 (3) | 4,765 (3) | 1,665 (3) | +| `frame.payload`, 100 B | 10 (3) | 20 (3) | 25 (3) | | 70 (3) | 155 (3) | 85 (3) | 128 (3) | 68 (3) | ### Throughput (MB per CPU second, write plus ready read; load average) | frame | Rust | C | C++ | Kotlin | Python | Swift | Go | Dart | |---|---|---|---|---|---|---|---|---| -| 100 B | 472 (26) | 303 (25) | 300 (26) | 105 (26) | 138 (25) | 162 (27) | 106 (26) | 120 (26) | -| 1 KiB | 3,444 (26) | 2,394 (25) | 2,267 (26) | 1,341 (26) | 1,240 (25) | 1,135 (27) | 713 (26) | 991 (26) | -| 16 KiB | 7,324 (26) | 14,822 (25) | 6,923 (26) | 6,554 (26) | 9,460 (25) | 2,156 (27) | 1,820 (26) | 5,536 (26) | -| 64 KiB | 2,552 (26) | 10,612 (25) | 6,441 (26) | 3,277 (26) | 9,468 (25) | 2,206 (27) | 2,382 (26) | 6,574 (26) | -| 1 MiB | 3,647 (26) | 3,596 (25) | 5,546 (26) | 3,146 (26) | 3,464 (25) | 3,354 (27) | 2,767 (26) | 3,812 (26) | +| 100 B | 563 (3) | 326 (3) | 297 (3) | 118 (3) | 138 (3) | 170 (3) | 114 (3) | 126 (3) | +| 1 KiB | 4,148 (3) | 2,823 (3) | 2,589 (3) | 1,341 (3) | 1,319 (3) | 1,218 (3) | 765 (3) | 1,104 (3) | +| 16 KiB | 11,331 (3) | 10,982 (3) | 11,715 (3) | 6,554 (3) | 8,784 (3) | 2,434 (3) | 1,648 (3) | 5,682 (3) | +| 64 KiB | 2,679 (3) | 8,637 (3) | 8,763 (3) | 2,185 (3) | 9,079 (3) | 2,522 (3) | 2,351 (3) | 6,767 (3) | +| 1 MiB | 3,742 (3) | 3,785 (3) | 6,300 (3) | 3,146 (3) | 3,586 (3) | 3,531 (3) | 4,357 (3) | 3,926 (3) | ### Calls (ns per call, wall time; load average) | call | Rust | C | C++ | moq-c | Kotlin | Python | Swift | Go | Dart | |---|---|---|---|---|---|---|---|---|---| -| `track.is_closed()`, a handle call | 9.1 (26) | 18 (25) | 18 (26) | | 32 (26) | 53 (25) | 26 (27) | 48 (26) | 42 (26) | -| `demand.sequence()`, a u64 getter | 0.4 (26) | 10 (25) | 10 (26) | | 24 (26) | 47 (25) | 18 (27) | 41 (26) | 28 (26) | -| `track.largest()`, a record return | 18 (26) | 31 (25) | 31 (26) | | 113 (26) | 97 (25) | 37 (27) | 73 (26) | 56 (26) | -| `track.info()`, an object return | 1.1 (26) | 25 (25) | 25 (26) | | 137 (26) | 97 (25) | 50 (27) | 255 (26) | 243 (26) | -| `group.write_frame(at, frame)`, 100 B | 132 (26) | 148 (25) | 161 (26) | | 239 (26) | 196 (25) | 194 (27) | 274 (26) | 208 (26) | -| the same, with a new `Timestamp` each | 133 (26) | 172 (25) | 178 (26) | 233 (27) | 331 (26) | 336 (25) | 224 (27) | 539 (26) | 447 (26) | -| a 100 B frame written and read, ready | 205 (26) | 303 (25) | 341 (26) | | 544 (26) | 560 (25) | 517 (27) | 903 (26) | 731 (26) | -| the same, read after a wake | 330 (26) | 431 (25) | 474 (26) | | 1,600 (26) | 6,912 (25) | 5,697 (27) | 5,061 (26) | 1,766 (26) | -| `frame.payload`, 100 B | 11 (26) | 23 (25) | 24 (26) | | 70 (26) | 150 (25) | 96 (27) | 150 (26) | 67 (26) | +| `track.is_closed()`, a handle call | 9.0 (3) | 18 (3) | 18 (3) | | 31 (3) | 55 (3) | 26 (3) | 46 (3) | 41 (3) | +| `demand.sequence()`, a u64 getter | 0.4 (3) | 10 (3) | 10 (3) | | 24 (3) | 49 (3) | 9.9 (3) | 38 (3) | 28 (3) | +| `track.largest()`, a record return | 18 (3) | 28 (3) | 29 (3) | | 74 (3) | 99 (3) | 36 (3) | 65 (3) | 54 (3) | +| `track.info()`, an object return | 1.1 (3) | 31 (3) | 29 (3) | | 89 (3) | 104 (3) | 38 (3) | 181 (3) | 238 (3) | +| `group.write_frame(at, frame)`, 100 B | 122 (3) | 143 (3) | 148 (3) | | 236 (3) | 198 (3) | 179 (3) | 280 (3) | 211 (3) | +| the same, with a new `Timestamp` each | 122 (3) | 162 (3) | 170 (3) | 219 (3) | 331 (3) | 339 (3) | 208 (3) | 445 (3) | 445 (3) | +| a 100 B frame written and read, ready | 194 (3) | 301 (3) | 327 (3) | | 540 (3) | 552 (3) | 471 (3) | 694 (3) | 732 (3) | +| the same, read after a wake | 308 (3) | 444 (3) | 450 (3) | | 1,574 (3) | 6,918 (3) | 4,757 (3) | 3,740 (3) | 1,652 (3) | +| `frame.payload`, 100 B | 10 (3) | 20 (3) | 25 (3) | | 67 (3) | 155 (3) | 85 (3) | 115 (3) | 67 (3) | ### Throughput (MB/s, write plus ready read; load average) | frame | Rust | C | C++ | Kotlin | Python | Swift | Go | Dart | |---|---|---|---|---|---|---|---|---| -| 100 B | 472 (26) | 299 (25) | 290 (26) | 170 (26) | 138 (25) | 162 (27) | 105 (26) | 121 (26) | -| 1 KiB | 3,444 (26) | 2,394 (25) | 2,267 (26) | 1,384 (26) | 1,239 (25) | 1,135 (27) | 769 (26) | 1,019 (26) | -| 16 KiB | 7,324 (26) | 14,824 (25) | 6,923 (26) | 8,822 (26) | 9,464 (25) | 2,156 (27) | 2,072 (26) | 5,618 (26) | -| 64 KiB | 2,552 (26) | 10,612 (25) | 5,466 (26) | 6,048 (26) | 9,469 (25) | 2,206 (27) | 2,622 (26) | 6,679 (26) | -| 1 MiB | 3,647 (26) | 3,591 (25) | 5,253 (26) | 3,369 (26) | 3,464 (25) | 3,353 (27) | 2,693 (26) | 3,861 (26) | +| 100 B | 563 (3) | 326 (3) | 297 (3) | 166 (3) | 138 (3) | 169 (3) | 121 (3) | 127 (3) | +| 1 KiB | 4,144 (3) | 2,823 (3) | 2,589 (3) | 1,404 (3) | 1,319 (3) | 1,218 (3) | 872 (3) | 1,130 (3) | +| 16 KiB | 11,330 (3) | 10,982 (3) | 11,715 (3) | 6,096 (3) | 8,784 (3) | 2,433 (3) | 2,196 (3) | 5,729 (3) | +| 64 KiB | 2,679 (3) | 8,637 (3) | 8,762 (3) | 6,651 (3) | 9,079 (3) | 2,522 (3) | 2,429 (3) | 6,971 (3) | +| 1 MiB | 3,742 (3) | 3,785 (3) | 6,298 (3) | 3,346 (3) | 3,586 (3) | 3,530 (3) | 4,847 (3) | 3,995 (3) | ### Size | binding | files | raw | gzip | Rust glue, raw / gzip | |---|---|---|---|---| -| C | 1 | 1,604 KiB | 235 KiB | 2,346 KiB / 125 KiB | -| C++ | 1 | 2,664 KiB | 299 KiB | none | -| Kotlin | 2 | 2,373 KiB | 318 KiB | 1,649 KiB / 72 KiB | -| Python | 1 | 1,424 KiB | 217 KiB | 3,678 KiB / 282 KiB | -| Swift | 3 | 2,231 KiB | 295 KiB | none | -| Go | 19 | 3,429 KiB | 404 KiB | none | -| Dart | 1 | 3,921 KiB | 360 KiB | 1,606 KiB / 59 KiB | +| C | 1 | 1,633 KiB | 242 KiB | 2,416 KiB / 134 KiB | +| C++ | 1 | 2,865 KiB | 320 KiB | none | +| Kotlin | 2 | 2,498 KiB | 361 KiB | 1,720 KiB / 79 KiB | +| Python | 1 | 1,454 KiB | 225 KiB | 3,762 KiB / 294 KiB | +| Swift | 3 | 2,147 KiB | 302 KiB | none | +| Go | 23 | 3,554 KiB | 435 KiB | none | +| Dart | 1 | 4,034 KiB | 376 KiB | 1,643 KiB / 62 KiB | | native library | as built | stripped | |---|---|---| -| `libmoq.a`, the C ABI that C, C++, Swift, and Go link | 191,403 KiB | 102,985 KiB | -| `libmoq.so`, with the Kotlin and Dart glue | 60,222 KiB | 45,126 KiB | -| `python/libmoq.so`, the CPython extension | 54,694 KiB | 41,023 KiB | +| `libmoq.a`, the C ABI that C, C++, Swift, and Go link | 193,455 KiB | 104,224 KiB | +| `libmoq.so`, with the Kotlin and Dart glue | 61,361 KiB | 46,029 KiB | +| `python/libmoq.so`, the CPython extension | 56,145 KiB | 42,166 KiB | The stand-in exports all of moq-net, kio, and moq-pattern, and hang, moq-mux, and moq-tokio's session side, far more than moq-ffi's UniFFI interface, so its sizes do not @@ -211,7 +217,8 @@ the export's default `release` build, which links all of moq's crates; "stripped ### What a call spends -From the CPU tables, read against the code each call runs. In order of what it costs moq's +From the CPU tables of 2026-11-10, measured at rust2 `7383790d` at a load average of 22 to +29 and since replaced by the tables above, read against the code each call runs. In order of what it costs moq's frame path: 1. **A handle per frame timestamp.** `Timestamp` is `Copy` in Rust, but its private fields @@ -251,8 +258,8 @@ frame path: For [checks-raise](/quest/m1/ffi-export/checks-raise.md)'s one accounting: the u64 getter is one borrowed handle and nothing else, so its row less Rust's 0.4 ns is what a -handle costs a call. C, C++: 10 ns, the runtime's two atomics. Swift: 18, its hold count -adding two more. Kotlin: 24 wall, Go 41, Dart 29, Python 47, each with its call's own +handle costs a call. C, C++: 10 ns, the runtime's two atomics. Swift: 9.9, with no hold +count since the change below. Kotlin: 24 wall, Go 38, Dart 28, Python 49, each with its call's own transition besides. Swift has since made each object a noncopyable struct and dropped its hold count, but for @@ -265,10 +272,10 @@ same amount; measure both, CPU first, at the load beside them. ### Against the targets Where the cases match, on wall time: Kotlin's getter costs 24 ns against BoltFFI's 5.8, -its record return 113 against 61, its write 107 ns over Rust against BoltFFI's 56, and 100 B -throughput 170 MB/s against 105. Python's getter costs 47 ns against 63, its record return -97 against 1,120, and 100 B throughput 138 MB/s against 12.1. So Python meets BoltFFI's -and Kotlin's calls trail it by 2 to 4 times, behind JNI's per-call cost and the hold count. +its record return 74 against 61, its write 114 ns over Rust against BoltFFI's 56, and 100 B +throughput 166 MB/s against 105. Python's getter costs 49 ns against 63, its record return +99 against 1,120, and 100 B throughput 138 MB/s against 12.1. So Python meets BoltFFI's +and Kotlin's calls trail it by 1.2 to 4 times, behind JNI's per-call cost and the hold count. ## Consumer release profile diff --git a/ffi/README.md b/ffi/README.md index a56985326..35504b96f 100644 --- a/ffi/README.md +++ b/ffi/README.md @@ -108,7 +108,7 @@ refused. | `u128` | `_u128`: its low and high 64 bits | | `NonZero` of an integer but 128-bit | `T`, in every binding too; a zero passed in aborts, and `Option>` is `T`'s option | | `[u8; N]` | `_array_u8_`: a struct of its `N` bytes, `items`, copied across; an empty one, or one of any other element, is refused | -| `&str`, `String`, `&[u8]`, `Vec`, `bytes::Bytes` | `_buf`: a view passed in, an owned buffer returned | +| `&str`, `String`, `&[u8]`, `Vec`, `bytes::Bytes` | `_buf`: an owned buffer returned; passed in, a view, or a buffer Rust returned, which Rust takes back without a copy unless it sits behind a pointer or in an array; `_buf_clone` shares one the caller keeps, which a `Vec`, `String`, or `PathBuf` parameter then copies, since only a sole owner's allocation becomes a `Vec` | | `&Path`, `PathBuf` | `_buf` of the path's bytes as the OS holds them on Unix, which need not be UTF-8, else UTF-8 | | `Duration` | `_duration` | | `SocketAddr`, `SocketAddrV4`, `SocketAddrV6` | `_socket_addr`: its IP's bytes, port, and whether it is IPv6; one of the other family passed to `V4` or `V6` aborts | @@ -488,18 +488,20 @@ out, with only its other fields encoded. - **Records** are data classes, compared by their fields, and **enums** an `enum class`, or for one with data a sealed class of a data class per variant. `Display` is `toString()`, `Debug` `debug()`, and `Default` a companion's `default()`. -- **Bytes** passed in are a `ByteArray`, copied, as are those Rust lends a closure or trait - method. Bytes Rust returns, or a record holds, are a `Bytes`, read only through `size`, +- **Bytes** are a `Bytes`. Passed in, Rust's own are shared with no copy, and Kotlin's, + which `Bytes(ByteArray)` holds without a copy, are copied once into a buffer Rust takes, + at a call's top level or in a record; those Rust lends a closure or trait method are + copied. A `Bytes` is read only through `size`, `get`, `slice`, `copyInto`, and `toByteArray()`, so no view of Rust's memory escapes and no read meets it freed. Up to 1 KiB are copied, which costs less than the GC's cleanup; longer ones stay in Rust's buffer, each read a JNI call that passes the buffer's owner, which JNI keeps reachable until the call returns, and are freed once the GC finds no `Bytes` reaching them, slices included. The GC sees none of Rust's buffers, so at most 64 MiB are lent at once, freed or not; past that, longer ones are copied too, and that - garbage in Kotlin's heap runs the GC, which frees them. `Bytes(ByteArray)` holds - Kotlin's for a record passed in. A `Bytes` equals and hashes by its contents, wherever each keeps them, so a + garbage in Kotlin's heap runs the GC, which frees them. A `Bytes` equals and hashes by its contents, wherever each keeps them, so a record holding bytes does too; Rust's lent ones are copied across in one call to compare. - Strings are `String`, `usize` `ULong` and `isize` `Long`, which abort past a 32-bit + Strings are `String`, passed in as their UTF-16, which Rust encodes once into UTF-8 as + `encodeToByteArray` does, a lone surrogate as `?`; `usize` `ULong` and `isize` `Long`, which abort past a 32-bit target's range, `Option` `T?`, a tuple `Pair` or `Triple`, or else a generated `Tuple`, and also `Tuple2` or `Tuple3` when it holds a numeric array or a float anywhere, records included, since `Pair` and `Triple` cannot @@ -688,9 +690,13 @@ and, for `Swift.Duration`, macOS 13 or iOS 16, and builds in either language mod - **Records** are structs and **enums** Swift enums, with a variant's data as associated values, all `Sendable`. `PartialEq` is `==`, through Rust's; `Display` is `description`, `Debug` `debugDescription`, and `Default` a static `default()`. -- **Bytes** are `Data`. Passed in at a call's top level, Rust reads Swift's own; nested in - another value, they are copied in. Bytes Rust returns lend its buffer until Swift - releases the `Data`, though `Data` copies a few bytes inline, freeing Rust's at once. +- **Bytes** are the runtime's `Bytes`, a `RandomAccessCollection` of `UInt8` whose slices + are `Bytes` too. Bytes Rust returns lend its buffer until Swift releases the last `Bytes` + or slice reading it, and pass back to Rust as a second owner of it, with no copy, except + as an array's items or behind a pointer, which Rust only reads and so copies. Swift's + own, made with `Bytes(data)` or from any sequence of bytes, Rust copies once where it + keeps them. `bytes.data` is a `Data`: Swift's as they are, and Rust's copied, so nothing + writes into a buffer Rust shares. Strings are `String`, copied, `char` `Unicode.Scalar`, `usize` `UInt`, `Option` `T?`, a tuple a Swift tuple, though a tuple of one is its element, `u128` a `U128` of its `high` and `low` halves until the deployment target allows `UInt128`, and `Duration` @@ -895,8 +901,10 @@ the library's own URI, mapped to the library by a build hook or by `--native-ass - **Integers** are `int`. A narrower one passed in throws a `RangeError` when out of its range, and a `u64` passes its 64 bits, so one past `i64`'s range is negative in Dart. A `char` is a `String` of one Unicode scalar. -- **Bytes** a call takes are a `Uint8List`, and text a `String`, each copied into a - buffer the call frees. Bytes Rust returns are a `Uint8List` owned by Dart's memory +- **Bytes** a call takes are a `Uint8List`, and text a `String`. Rust keeps Dart's own + with one copy from 1 KiB, into a buffer Rust takes, and with two below, since allocating + Rust's buffer costs more there than the second copy; a lent `Uint8List` Rust returned + passes back as a second owner of Rust's buffer, with no copy. Bytes Rust returns are a `Uint8List` owned by Dart's memory manager: up to 64 KiB copied, with Rust's freed at the call, and a longer one an unmodifiable view of Rust's buffer, freed once neither it nor any view of it is reachable, so no view outlives its memory. A lent view costs a finalizer and charges the diff --git a/ffi/moq/bench/export.go b/ffi/moq/bench/export.go index 74fc100bb..360139300 100644 --- a/ffi/moq/bench/export.go +++ b/ffi/moq/bench/export.go @@ -11,6 +11,7 @@ import ( "time" "moq/net" + "moq/rt" ) const ( @@ -67,7 +68,7 @@ var sink int // CPU second. func throughput(ctx context.Context, track *net.TrackProducer, subscriber *net.TrackSubscriber, at *net.Timestamp, size int) span { count := min(max((64<<20)/(size*frames), 3), 2000) - frame := make([]byte, size) + frame := rt.NewBytes(make([]byte, size)) start := now() for range count { group := must(track.AppendGroup()) @@ -112,7 +113,7 @@ func round() []result { micro := net.TimescaleMICRO() track := must(broadcast.CreateTrack("bench", must(net.TrackInfoDefault().WithTimescale(micro)))) micro.Close() - frame := make([]byte, 100) + frame := rt.NewBytes(make([]byte, 100)) start := now() for range n { diff --git a/ffi/moq/bench/export.kt b/ffi/moq/bench/export.kt index 19e9b48a3..85fdc2f61 100644 --- a/ffi/moq/bench/export.kt +++ b/ffi/moq/bench/export.kt @@ -36,7 +36,7 @@ fun per(start: Span, n: Double = N.toDouble()): Span { // byte touched, in about 64 MiB and at least 300 frames: megabytes a wall and a CPU second. suspend fun throughput(track: TrackProducer, subscriber: TrackSubscriber, at: Timestamp, size: Int): Span { val groups = ((64 shl 20) / (size * FRAMES)).coerceIn(3, 2000) - val frame = ByteArray(size) + val frame = Bytes(ByteArray(size)) var sum = 0 val start = now() repeat(groups) { @@ -61,7 +61,7 @@ fun round(): Map { val track = Timescale.MICRO.use { micro -> broadcast.createTrack("bench", TrackInfo.default().withTimescale(micro)) } - val frame = ByteArray(100) + val frame = Bytes(ByteArray(100)) var sum = 0 var start = now() diff --git a/ffi/moq/bench/export.swift b/ffi/moq/bench/export.swift index e3e010ba0..37667ef0b 100644 --- a/ffi/moq/bench/export.swift +++ b/ffi/moq/bench/export.swift @@ -86,7 +86,7 @@ struct Reads: ~Copyable { /// byte touched, in about 64 MiB and at least 300 frames: megabytes a wall and a CPU second. func throughput(_ track: TrackProducer, _ subscriber: inout TrackSubscriber, _ at: Timestamp, _ size: Int) async throws -> (Span, Int) { let count = min(max((64 << 20) / (size * frames), 3), 2000) - let frame = Data(count: size) + let frame = Bytes(Data(count: size)) var sum = 0 let start = now() for _ in 0.. ([(String, Span)], Int) { var out: [(String, Span)] = [] let broadcast = BroadcastInfo.new().produce() let track = try broadcast.createTrack("bench", TrackInfo.default().withTimescale(Timescale.MICRO)) - let frame = Data(count: 100) + let frame = Bytes(Data(count: 100)) var sum = 0 var start = now() @@ -273,7 +273,7 @@ func round() throws -> ([(String, Span)], Int) { precondition(done.value == true) out.append(("drained", per(start))) - // The last frame's 100 bytes, copied into a Data, and one of them read. + // The last frame's 100 bytes, returned as Bytes, and one of them read. start = now() for _ in 0.. String? { +func text(_ data: Bytes?) -> String? { data.map { String(decoding: $0, as: UTF8.self) } } @@ -243,7 +243,7 @@ func script() throws { } q.drain() expect(frame.isEmpty, "a read waits for a frame") - try writer.writeFrame(nil, Data("hello".utf8)) + try writer.writeFrame(nil, Bytes("hello".utf8)) expect(q.size == 1, "a frame's arrival wakes the read") q.drain() let hello = frame.take()! @@ -257,7 +257,7 @@ func script() throws { let at = try Timestamp.fromMillis(1500) expect(at.asSecs() == 1, "1500 ms is 1 s") do { - try writer.writeFrame(at, Data("late".utf8)) + try writer.writeFrame(at, Bytes("late".utf8)) expect(false, "a timestamped frame on an untimed track is an error") } catch let e as Moq.Error { expect("\(e)" == "frame timestamp doesn't fit its track", "the late frame's error is Rust's") @@ -265,7 +265,7 @@ func script() throws { expect(at.asSecs() == 1, "a copied timestamp stays usable") // The last frame, bytes that are not UTF-8, then the group's end, which reads as nil. - let world = Data([0xff]) + Data("world".utf8) + let world = Bytes(Data([0xff]) + Data("world".utf8)) try writer.writeFrame(nil, world) try writer.finish() try block { @@ -292,7 +292,7 @@ func script() throws { // moq-mux reads an OpusHead into hang's catalog config: 44.1 kHz in, stereo, which the // catalog lists at the decoder's 48 kHz, keeping the head as its description. - let head = Data("OpusHead".utf8) + Data([1, 2, 0x38, 0x01, 0x44, 0xac, 0, 0, 0, 0, 0]) + let head = Bytes(Data("OpusHead".utf8) + Data([1, 2, 0x38, 0x01, 0x44, 0xac, 0, 0, 0, 0, 0])) let audio = try muxCodecOpusConfig(head) expect(audio.sampleRateGet == 48000, "Opus decodes at 48 kHz") expect(audio.channelCountGet == 2, "the head is stereo") @@ -428,7 +428,7 @@ func script() throws { try live.announce(OriginRoute.default()) let liveTrack = try live.broadcast().createTrack("video", nil) var liveGroup = try liveTrack.appendGroup() - try liveGroup.writeFrame(nil, Data("hello".utf8)) + try liveGroup.writeFrame(nil, Bytes("hello".utf8)) try liveGroup.finish() var serverConfig = TokioServerConfig.default() diff --git a/ffi/runtime/ffi_cgo.go b/ffi/runtime/ffi_cgo.go index 8aa59149d..44dc0022a 100644 --- a/ffi/runtime/ffi_cgo.go +++ b/ffi/runtime/ffi_cgo.go @@ -29,6 +29,23 @@ func bufBytes_(b []byte) cBuf { return cBuf{ptr: (*C.uint8_t)(unsafe.Pointer(unsafe.SliceData(b))), len: C.size_t(len(b))} } +// bufTake_ is b as a buffer Rust takes: Rust's own shared without a copy, or Go's memory, +// which C reads during the call and Rust copies. +func bufTake_(b rt.Bytes) cBuf { + ptr, n, owner := rt.BytesTake_(b) + if n == 0 { + return cBuf{ptr: (*C.uint8_t)(emptyPtr_())} + } + return cBuf{ptr: (*C.uint8_t)(ptr), len: C.size_t(n), owner: owner} +} + +func bufTakeOption_(b *rt.Bytes) cBuf { + if b == nil { + return cBuf{} + } + return bufTake_(*b) +} + func bufStringOption_(s *string) cBuf { if s == nil { return cBuf{} @@ -239,12 +256,24 @@ type arena_ struct { held []held_ // The owners of Rust's buffers C reads. kept []any + // Whether a buffer lowered now is passed by value, which Rust takes: set once the + // receiver, which Rust only reads, is lowered, and unset inside an array. + take bool + // Second owners of Rust's buffers lowered for Rust to take, freed here unless the + // call was made. + taken []cBuf + made bool // The first object the call takes that was closed, which fails the call before it is // made. err error } func (a *arena_) finish() { + if !a.made { + for _, buf := range a.taken { + cBufFree(buf) + } + } for _, held := range a.held { held.h.Release() } @@ -264,8 +293,15 @@ func (a *arena_) bufString(s string) cBuf { } // bufView is the buffer of b, which C reads during the call: Go's memory pinned, and -// Rust's kept reachable. +// Rust's kept reachable, or where a.take, a second owner of it, which Rust takes. func (a *arena_) bufView(b rt.Bytes) cBuf { + if a.take && b.Len() != 0 { + if ptr, n, owner := rt.BytesTake_(b); owner != nil { + buf := cBuf{ptr: (*C.uint8_t)(ptr), len: C.size_t(n), owner: owner} + a.taken = append(a.taken, buf) + return buf + } + } ptr, n, owner := rt.BytesBuf_(b) if n == 0 { return cBuf{ptr: (*C.uint8_t)(emptyPtr_())} @@ -332,9 +368,13 @@ func list_[T, CT any](a *arena_, items []T, lower func(T) CT) (*CT, C.size_t) { return (*CT)(emptyPtr_()), 0 } out := make([]CT, len(items)) + // Rust only reads an array's items. + take := a.take + a.take = false for i, item := range items { out[i] = lower(item) } + a.take = take a.pins.Pin(&out[0]) return &out[0], C.size_t(len(items)) } diff --git a/ffi/runtime/jni.rs b/ffi/runtime/jni.rs index 1dd4dd15f..4554a5d26 100644 --- a/ffi/runtime/jni.rs +++ b/ffi/runtime/jni.rs @@ -244,12 +244,85 @@ pub fn new_block(env: *mut JNIEnv, list: rt::List) -> jarray { } } -/// A view of `bytes`, or a null view for none. -pub fn view(bytes: &Option>) -> rt::Buf { - match bytes { - Some(bytes) => rt::Buf::view(bytes), - None => rt::Buf::NONE, +/// A `String`'s UTF-8, encoded once into a buffer Rust takes, or none for null. +pub fn text(env: *mut JNIEnv, text: jstring) -> rt::Buf { + match text.is_null() { + true => rt::Buf::NONE, + false => rt::Buf::vec(utf8(env, text)), + } +} + +/// A live `String`'s UTF-8, as Kotlin's `encodeToByteArray` encodes it: a lone surrogate +/// is `?`. JNI's own UTF-8 is modified, which differs for NUL and past the BMP, so the +/// UTF-16 is read a chunk at a time and encoded here. +fn utf8(env: *mut JNIEnv, text: jstring) -> Vec { + const CHUNK: usize = 1024; + // SAFETY: `text` is a live `String` the call was given. + let len = unsafe { ((**env).v1_2.GetStringLength)(env, text) } as usize; + let mut out = Vec::with_capacity(len); + let mut chunk = [0u16; CHUNK]; + let mut at = 0; + while at < len { + let n = (len - at).min(CHUNK); + // SAFETY: `at + n` is within the string, and `chunk` holds `n`. + unsafe { ((**env).v1_2.GetStringRegion)(env, text, at as jint, n as jint, chunk.as_mut_ptr()) }; + let mut units = &chunk[..n]; + // A pair split across chunks is read whole with the next. + if at + n < len && (0xD800..0xDC00).contains(&units[n - 1]) { + units = &units[..n - 1]; + } + at += units.len(); + for c in char::decode_utf16(units.iter().copied()) { + match c { + Ok(c) if c.is_ascii() => out.push(c as u8), + Ok(c) => out.extend_from_slice(c.encode_utf8(&mut [0; 4]).as_bytes()), + Err(_) => out.push(b'?'), + } + } + } + out +} + +/// `len` bytes of a live `byte[]` from `start`, copied, failing loud past its end, which +/// Kotlin's `Bytes` checks first. +fn region(env: *mut JNIEnv, array: jbyteArray, start: jint, len: jint) -> Vec { + let Ok(n) = usize::try_from(len) else { + rt::fail("jni", "bytes passed have a negative length") + }; + let mut bytes = Vec::::with_capacity(n); + // SAFETY: `array` is a live `byte[]`; JNI checks the range, throwing past its end. + unsafe { + ((**env).v1_2.GetByteArrayRegion)(env, array, start, len, bytes.as_mut_ptr().cast()); + if ((**env).v1_2.ExceptionCheck)(env) { + fail(env, "jni", "bytes passed are past their array's end"); + } + bytes.set_len(n); + } + bytes +} + +/// Bytes Kotlin passes as a `Bytes` holds them, for Rust to take: none where `len` is -1, +/// else those from `start` of what a non-zero `owner` holds, shared without a copy, or of +/// the `byte[]` `held`, copied once. +/// +/// # Safety +/// A non-zero `owner` came from [`hand`], and `held` keeps it from being freed. +pub unsafe fn passed(env: *mut JNIEnv, held: jobject, owner: jlong, start: jint, len: jint) -> rt::Buf { + if len == -1 { + return rt::Buf::NONE; + } + if owner != 0 { + // SAFETY: the caller's contract. + let bytes = unsafe { range(owner, start, len) }; + let buf = rt::Buf { + ptr: bytes.as_ptr(), + len: bytes.len(), + owner: owner as usize as _, + }; + // SAFETY: as above. + return unsafe { buf.share("jni") }; } + rt::Buf::vec(region(env, held, start, len)) } /// A new `byte[]` holding `bytes`. @@ -769,11 +842,58 @@ pub unsafe fn drop_returned(raw: jlong) { pub struct Reader<'a> { bytes: &'a [u8], arrays: Vec>, + /// The call's thread, and the arrays and text the encoding names by place. + env: *mut JNIEnv, + refs: jobjectArray, + /// Whether the value read is one Rust takes, so Rust's own bytes in it, outside an + /// array, cross as a second owner of them rather than a view. + take: bool, } impl<'a> Reader<'a> { - pub fn new(bytes: &'a [u8]) -> Self { - Self { bytes, arrays: Vec::new() } + /// A reader of a value Rust only reads, as through a pointer, whose encoding names + /// the objects of `refs` by place. + pub fn new(env: *mut JNIEnv, bytes: &'a [u8], refs: jobjectArray) -> Self { + Self { + bytes, + arrays: Vec::new(), + env, + refs, + take: false, + } + } + + /// A reader of a value passed by value, which Rust takes. + pub fn taking(env: *mut JNIEnv, bytes: &'a [u8], refs: jobjectArray) -> Self { + Self { + take: true, + ..Self::new(env, bytes, refs) + } + } + + /// Bytes made once by `make` from the object at `at` of `refs`: taken where the reader + /// takes, else kept until it drops and read in place. + fn referred(&mut self, at: i32, make: impl FnOnce(*mut JNIEnv, jobject) -> Vec) -> rt::Buf { + let env = self.env; + if self.refs.is_null() { + fail(env, "jni", "an encoded value names an object, but none were passed"); + } + // SAFETY: `refs` is the live `Object[]` the call was given; JNI throws past its end. + let object = unsafe { ((**env).v1_2.GetObjectArrayElement)(env, self.refs, at) }; + // SAFETY: as above. + if object.is_null() || unsafe { ((**env).v1_2.ExceptionCheck)(env) } { + fail(env, "jni", "an encoded value names no object"); + } + let bytes = make(env, object); + // SAFETY: a local reference made here. + unsafe { ((**env).v1_2.DeleteLocalRef)(env, object) }; + if self.take { + return rt::Buf::vec(bytes); + } + let view = rt::Buf::view(&bytes); + // The vector's heap block, which the view reads, stays put as the box moves. + self.arrays.push(Box::new(bytes)); + view } fn take(&mut self, n: usize) -> &'a [u8] { @@ -951,11 +1071,41 @@ impl Wire for rt::Waker { } impl Wire for rt::Buf { - /// A view of the encoded bytes, which outlive the call. + /// A view of the encoded bytes, or after [`LENT`] of those Rust lent Kotlin, in place, + /// either of which outlives the call. fn read(r: &mut Reader<'_>) -> Self { - match usize::try_from(r.get::()) { - Ok(len) => rt::Buf::view(r.take(len)), - Err(_) => rt::Buf::NONE, + match r.get::() { + LENT => { + let owner = r.get::(); + let (start, len) = (r.get::(), r.get::()); + // SAFETY: an owner Kotlin keeps from being freed until the call returns. + let bytes = unsafe { range(owner, start, len) }; + let view = rt::Buf::view(bytes); + match r.take { + false => view, + // SAFETY: as above. + true => unsafe { + rt::Buf { + owner: owner as usize as _, + ..view + } + .share("jni") + }, + } + } + KOTLIN => { + let at = r.get::(); + let (start, len) = (r.get::(), r.get::()); + r.referred(at, |env, array| region(env, array, start, len)) + } + TEXT => { + let at = r.get::(); + r.referred(at, utf8) + } + len => match usize::try_from(len) { + Ok(len) => rt::Buf::view(r.take(len)), + Err(_) => rt::Buf::NONE, + }, } } @@ -982,6 +1132,14 @@ pub struct Owned(pub T); /// What precedes a lent buffer's owner where an encoded one's length would. const LENT: i32 = -2; +/// What precedes the place in a call's objects of Kotlin's `byte[]`, then where its bytes +/// start and how many, where an encoded buffer's length would. +const KOTLIN: i32 = -3; + +/// What precedes the place in a call's objects of Kotlin's `String` where an encoded +/// buffer's length would. +const TEXT: i32 = -4; + impl Wire for Owned { fn read(_: &mut Reader<'_>) -> Self { rt::fail("jni", "Kotlin passes bytes, never their owner") @@ -1091,7 +1249,10 @@ impl Wire for rt::List { let Ok(len) = usize::try_from(r.get::()) else { return Self::NONE; }; + // Rust only reads an array's items. + let take = std::mem::replace(&mut r.take, false); let items: Box<[T]> = (0..len).map(|_| r.get()).collect(); + r.take = take; let list = Self { ptr: items.as_ptr(), len, diff --git a/ffi/runtime/python.rs b/ffi/runtime/python.rs index 95bf862e2..2dcc8ab5e 100644 --- a/ffi/runtime/python.rs +++ b/ffi/runtime/python.rs @@ -378,6 +378,8 @@ pub struct Scope { nodes: Vec, /// Awaitables given to Rust, freed unless the call happens. futures: Vec, + /// Buffers handed to Rust, which takes them, freed unless the call happens. + bufs: Vec, } impl Scope { @@ -402,6 +404,7 @@ impl Scope { } self.nodes.clear(); self.futures.clear(); + self.bufs.clear(); } /// The handles held, which a future keeps until it ends. @@ -413,6 +416,10 @@ impl Scope { // Popping rather than draining, as each call drops a scope whose lists are mostly empty. impl Drop for Scope { fn drop(&mut self) { + while let Some(buf) = self.bufs.pop() { + // SAFETY: a buffer the call never took. + unsafe { buf.free() }; + } while let Some(waker) = self.nodes.pop() { // SAFETY: a waker the call never took. unsafe { waker.release() }; @@ -732,10 +739,12 @@ pub fn str_out(buf: rt::Buf) -> R { } } -/// The bytes of `bytes`, read in place, or of any other object with the buffer protocol, -/// such as a `bytearray` or a `memoryview`, copied: the call runs without the GIL, so -/// another thread could write storage that is not immutable while Rust reads it. -pub fn bytes_in(obj: *mut PyObject, s: &mut Scope) -> R { +/// The bytes of `bytes`, or of a `memoryview` of Rust's, read in place, or of any other +/// object with the buffer protocol, such as a `bytearray`, copied: the call runs without +/// the GIL, so another thread could write storage that is not immutable while Rust reads +/// it. Where Rust takes the buffer, `take`, Rust's bytes cross shared and a copy crosses +/// as Rust's own, so neither is copied again. +pub fn bytes_in(obj: *mut PyObject, s: &mut Scope, take: bool) -> R { // SAFETY: the GIL is held, and `obj` is live. unsafe { if PyBytes_Check(obj) != 0 { @@ -751,26 +760,63 @@ pub fn bytes_in(obj: *mut PyObject, s: &mut Scope) -> R { owner: null_mut(), }); } - // SAFETY: `Py_buffer` is plain data, which the call fills. + // SAFETY: `Py_buffer` is plain data, which the call fills. Its format, which a + // `memoryview` passes on, marks Rust's bytes. let mut view: Py_buffer = std::mem::zeroed(); - if PyObject_GetBuffer(obj, &mut view, PyBUF_SIMPLE) != 0 { + if PyObject_GetBuffer(obj, &mut view, PyBUF_FORMAT | PyBUF_ND) != 0 { return Err(Raised); } - let copy = match view.len { - 0 => Vec::new(), - len => std::slice::from_raw_parts(view.buf.cast_const().cast::(), len as usize).to_vec(), + let ptr = view.buf.cast_const().cast::(); + let len = view.len as usize; + let buf = match (std::ptr::eq(view.format, RUST_FORMAT.as_ptr()), take) { + // Rust's, which never change, read in place: a second owner of them kept for + // the call, since releasing `obj`, a view, mid-call could free them. + (true, false) => { + let owner = (*view.internal.cast::()).buf.owner.cast::(); + s.arrays.push(Box::new((*owner).clone())); + rt::Buf { + ptr, + len, + owner: null_mut(), + } + } + (true, true) => { + let shared = (*view.internal.cast::()).buf.share("python"); + s.bufs.push(shared); + rt::Buf { ptr, len, ..shared } + } + (false, _) => { + let copy = match len { + 0 => Vec::new(), + len => std::slice::from_raw_parts(ptr, len).to_vec(), + }; + match take { + true => { + let owned = rt::Buf::vec(copy); + s.bufs.push(owned); + owned + } + false => { + let buf = rt::Buf { + ptr: copy.as_ptr(), + len: copy.len(), + owner: null_mut(), + }; + s.arrays.push(Box::new(copy)); + buf + } + } + } }; PyBuffer_Release(&mut view); - let buf = rt::Buf { - ptr: copy.as_ptr(), - len: copy.len(), - owner: null_mut(), - }; - s.arrays.push(Box::new(copy)); Ok(buf) } } +/// The format of a view of Rust's bytes: unsigned bytes, at an address no other exporter +/// uses, which marks them, its `internal` then their owner. +static RUST_FORMAT: [c_char; 2] = [b'B' as c_char, 0]; + /// Bytes Rust returned, as a read-only `memoryview` that lends them without a copy until /// it and every view of it are released. pub fn bytes_out(buf: rt::Buf) -> R { @@ -1804,7 +1850,13 @@ unsafe extern "C" fn bytes_getbuffer(obj: *mut PyObject, view: *mut Py_buffer, f // SAFETY: Python asks a live `Bytes` for a view. unsafe { let buf = (*obj.cast::()).buf; - PyBuffer_FillInfo(view, obj, buf.ptr.cast_mut().cast(), buf.len as Py_ssize_t, 1, flags) + let filled = PyBuffer_FillInfo(view, obj, buf.ptr.cast_mut().cast(), buf.len as Py_ssize_t, 1, flags); + // A `memoryview` keeps both in each view of it, its slices included. + if filled == 0 && !(*view).format.is_null() { + (*view).format = RUST_FORMAT.as_ptr().cast_mut(); + (*view).internal = obj.cast(); + } + filled } } diff --git a/ffi/runtime/rt.dart b/ffi/runtime/rt.dart index f4c9973a9..2136476c7 100644 --- a/ffi/runtime/rt.dart +++ b/ffi/runtime/rt.dart @@ -420,6 +420,12 @@ final class _Arena implements ffi.Allocator { List<_Object>? _more; List? _lent; + /// The buffers it gave Rust to take, freed unless the call took them. + List<_Buf>? _given; + + /// The lent bytes it passed a view of, kept reachable until the call returns. + List? _viewed; + _Arena() : _stack = _n.stack, _mark = _n.stack.mark(); @override @@ -451,8 +457,18 @@ final class _Arena implements ffi.Allocator { return relay; } + /// Note that the call returned, so Rust took each buffer it was given. + void took() => _given = null; + /// Give back what the call took. void end() { + final given = _given; + if (given != null) { + for (final buf in given) { + _bufFree(buf); + } + } + _viewed = null; final lent = _lent; if (lent != null) { for (var i = 0; i < lent.length; i += 2) { @@ -470,24 +486,50 @@ final class _Arena implements ffi.Allocator { _stack.reset(_mark); } - /// `bytes`, copied: never a null pointer, so an empty one is told apart from none. - void bytes(_Buf dst, typed.Uint8List? bytes) { + /// `bytes`, never a null pointer, so an empty one is told apart from none. Where Rust + /// [take]s them, a buffer it owns: a second owner of bytes Rust lent, else from + /// [_takeFrom] bytes a copy into a buffer of its own. Else a view: of bytes Rust lent, in + /// place, else of a copy here, which Rust copies again where it keeps them. + void bytes(_Buf dst, typed.Uint8List? bytes, bool take) { if (bytes == null) { dst.ptr = ffi.nullptr; dst.len = 0; - } else { - final ptr = this(bytes.isEmpty ? 1 : bytes.length); - ptr.asTypedList(bytes.length).setAll(0, bytes); - dst.ptr = ptr; + dst.owner = ffi.nullptr; + return; + } + // Only a list longer than [_lend] can be Rust's. + final lent = bytes.length > _lend ? _n.lentBytes[bytes] : null; + if (lent != null) { + dst.ptr = ffi.Pointer.fromAddress(lent.ptr); dst.len = bytes.length; + dst.owner = ffi.nullptr; + // Reachable through the call, so its finalizer cannot free the owner under Rust, + // even while [_bufClone] shares it. + (_viewed ??= []).add(bytes); + if (take) { + dst.owner = ffi.Pointer.fromAddress(lent.owner); + _give(dst, _bufClone(dst)); + } + return; + } + if (take && bytes.length >= _takeFrom) { + final buf = _dartBuf(bytes.length); + buf.ptr.asTypedList(bytes.length).setAll(0, bytes); + _give(dst, buf); + return; } + final ptr = this(bytes.isEmpty ? 1 : bytes.length); + ptr.asTypedList(bytes.length).setAll(0, bytes); + dst.ptr = ptr; + dst.len = bytes.length; dst.owner = ffi.nullptr; } - /// `text`'s UTF-8, copied. - void string(_Buf dst, String? text) { + /// `text`'s UTF-8, copied: into a buffer Rust owns where it [take]s it, from [_takeFrom] + /// bytes. + void string(_Buf dst, String? text, bool take) { if (text == null) { - bytes(dst, null); + bytes(dst, null, take); return; } final units = text.codeUnits; @@ -501,7 +543,13 @@ final class _Arena implements ffi.Allocator { } } if (!ascii) { - bytes(dst, convert.utf8.encode(text)); + bytes(dst, convert.utf8.encode(text), take); + return; + } + if (take && n >= _takeFrom) { + final buf = _dartBuf(n); + buf.ptr.asTypedList(n).setAll(0, units); + _give(dst, buf); return; } final ptr = this(n == 0 ? 1 : n); @@ -512,6 +560,30 @@ final class _Arena implements ffi.Allocator { } } +/// Dart's own bytes Rust takes from this long are copied into a buffer Rust allocates, and +/// shorter ones onto the arena's stack, which Rust copies again: on the binding benchmark, +/// the call to allocate Rust's buffer costs more than the second copy of 100 bytes, and +/// less than that of 1 KiB. +const _takeFrom = 1024; + +extension on _Arena { + /// Write `buf`, Rust's own, to `dst`, freed unless the call takes it. + void _give(_Buf dst, _Buf buf) { + dst.ptr = buf.ptr; + dst.len = buf.len; + dst.owner = buf.owner; + (_given ??= []).add(dst); + } +} + +/// Where a list of bytes Rust lent reads them, and their owner. +final class _Lent { + final int ptr; + final int owner; + + _Lent(this.ptr, this.owner); +} + /// The isolate's share of the runtime: checked against the library before any other /// call, then its stack, its port, and what Rust holds of its Dart values by token. final _n = _Native(); @@ -528,6 +600,9 @@ final class _Native { /// The closures lent for a call, and the implementations Rust holds, by token. final Map lent = {}; + /// Each list of bytes Rust lent, which passes back to Rust in place. + final lentBytes = Expando<_Lent>(); + /// The calls pending in Rust, by their future. final Map, _Call> calls = {}; int _token = 0; @@ -707,7 +782,7 @@ final class _Native { final a = _Arena(); try { final view = a<_Buf>(); - a.string(view.ref, text); + a.string(view.ref, text, false); return _bufCopy(view.ref); } finally { a.end(); @@ -741,7 +816,7 @@ final class _Native { Never fail(String text) { final a = _Arena(); final buf = a<_Buf>(); - a.string(buf.ref, text); + a.string(buf.ref, text, false); _dartAbort(buf.ref); throw StateError('unreachable'); } @@ -756,7 +831,9 @@ final class _Native { _bufFree(buf); return copy; } - return buf.ptr.asTypedList(len, finalizer: _ownerFree, token: buf.owner).asUnmodifiableView(); + final list = buf.ptr.asTypedList(len, finalizer: _ownerFree, token: buf.owner).asUnmodifiableView(); + lentBytes[list] = _Lent(buf.ptr.address, buf.owner.address); + return list; } typed.Uint8List? bytesOrNull(_Buf buf) => buf.ptr == ffi.nullptr ? null : bytes(buf); diff --git a/ffi/runtime/rt.go b/ffi/runtime/rt.go index 97883c97c..947cab826 100644 --- a/ffi/runtime/rt.go +++ b/ffi/runtime/rt.go @@ -314,6 +314,27 @@ func LiftBytes_(p unsafe.Pointer) Bytes { return Bytes{owner: lend_(buf), ptr: unsafe.Pointer(buf.ptr), len: int(buf.len)} } +// BytesTake_ is what b reads as a buffer Rust takes: a second owner of Rust's buffer, +// sharing it without a copy, narrowed to b, or a nil owner for Go's memory, which C reads +// during the call and Rust copies. For the generated packages. +func BytesTake_(b Bytes) (unsafe.Pointer, int, unsafe.Pointer) { + if b.owner == nil { + return b.ptr, b.len, nil + } + // A Bytes of Rust's is one of its batch's buffers, or a range of one. + batch := b.owner.batch + p := uintptr(b.ptr) + for _, buf := range batch.bufs[:batch.n.Load()] { + start := uintptr(unsafe.Pointer(buf.ptr)) + if p >= start && p+uintptr(b.len) <= start+uintptr(buf.len) { + owner := cBufClone(buf).owner + runtime.KeepAlive(b.owner) + return b.ptr, b.len, owner + } + } + panic("a Bytes of Rust's is in none of its batch's buffers") +} + // BytesBuf_ is what b reads, and what keeps it alive, nil for Go's memory, which a call // pins instead. For the generated packages. func BytesBuf_(b Bytes) (unsafe.Pointer, int, any) { diff --git a/ffi/runtime/rt.hpp b/ffi/runtime/rt.hpp index 60e6f1b14..3b00b84c7 100644 --- a/ffi/runtime/rt.hpp +++ b/ffi/runtime/rt.hpp @@ -263,6 +263,7 @@ inline const ::std::uint8_t *bytes(const void *data, ::std::size_t len) noexcept } // namespace detail class buf; +class input; // An object Rust lends a closure as `&mut`: its methods through `->`, but neither a move // nor a copy of it, which would keep the loan past the call. @@ -318,7 +319,8 @@ class view { }; // Owned bytes: returned by Rust without a copy, or copied from C++. They outlive the object -// they came from; a view or span of them lives only as long as they do. +// they came from; a view or span of them lives only as long as they do. A copy of Rust's +// shares its bytes, and one moved into a call's `input` goes back to Rust without a copy. class buf { // Rust's when `owner` is set, else C++'s, allocated with `new[]` unless empty. detail::c_buf raw_; @@ -344,7 +346,14 @@ class buf { raw_ = detail::c_buf{copy, bytes.size(), nullptr}; } } - buf(const buf &other) : buf(view(other)) {} + // A second owner of Rust's bytes, shared, or a copy of C++'s. + buf(const buf &other) : buf() { + if (other.raw_.owner != nullptr) { + raw_ = detail::c_buf_clone(other.raw_); + } else { + *this = buf(view(other)); + } + } buf(buf &&other) noexcept : raw_(::std::exchange(other.raw_, detail::c_buf{nullptr, 0, nullptr})) {} buf &operator=(const buf &other) { if (this != &other) { @@ -379,6 +388,46 @@ class buf { return view(*this).str(); } + friend struct detail::access; + friend class input; +}; + +// Bytes passed to a call: a view of anything a view takes, read during the call, or a `buf` +// moved in, which Rust takes without a copy if it made it, else reads and C++ frees after. +// Made for its call alone, so it is neither copied nor kept. +class input { + detail::c_buf raw_; + // C++'s bytes moved in, freed once the call returns. + buf held_; + +public: + input(const char *text) noexcept : input(view(text)) {} + template && + !::std::is_pointer_v<::std::decay_t>>> + input(const R &range RUST2_FFI_LIFETIMEBOUND) noexcept { + view v(range); + raw_ = detail::c_buf{v.data(), v.size(), nullptr}; + } + input(buf &&bytes) noexcept { + if (bytes.raw_.owner != nullptr) { + raw_ = ::std::exchange(bytes.raw_, detail::c_buf{nullptr, 0, nullptr}); + } else { + held_ = ::std::move(bytes); + raw_ = detail::c_buf{held_.data(), held_.size(), nullptr}; + } + } + input(input &&other) noexcept + : raw_(::std::exchange(other.raw_, detail::c_buf{nullptr, 0, nullptr})), held_(::std::move(other.held_)) {} + input(const input &) = delete; + input &operator=(const input &) = delete; + input &operator=(input &&) = delete; + // What no call took: Rust's bytes go back to it. + ~input() { + if (raw_.owner != nullptr) { + detail::c_buf_free(raw_); + } + } + friend struct detail::access; }; @@ -532,12 +581,13 @@ class waker { namespace detail { -// `f`, any future of nothing, such as a binding's `future` of `void`, as the future Rust -// polls: from any thread, with a waker of its own each time, until its `poll` returns -// true, and destroyed once Rust drops it, finished or not. +// `f`, any future of nothing, such as a binding's `future` of `void`, allocated by the +// caller, as the future Rust polls: from any thread, with a waker of its own each time, +// until its `poll` returns true, and destroyed once Rust drops it, finished or not. +// Allocation is the caller's, so a caller that may throw first can hold `f` until the call. template -c_foreign_future foreign_future(F f) { - static_assert(::std::is_same_v())), bool>, +c_foreign_future foreign_future(F *f) noexcept { + static_assert(::std::is_same_vpoll(::std::declval())), bool>, "a future Rust polls has `bool poll(const waker &)`, as a future of void does"); struct calls { static bool poll(void *data, c_waker w) noexcept { @@ -547,7 +597,7 @@ c_foreign_future foreign_future(F f) { delete static_cast(data); } }; - return c_foreign_future{&calls::poll, &calls::free, new F(::std::move(f))}; + return c_foreign_future{&calls::poll, &calls::free, f}; } } // namespace detail @@ -665,6 +715,20 @@ struct access { static c_buf raw(const view &v) noexcept { return c_buf{v.data_, v.size_, nullptr}; } + // What a call passes Rust, which takes Rust's bytes from then on. + static c_buf take(input &in) noexcept { + c_buf raw = in.raw_; + in.raw_.owner = nullptr; + return raw; + } + // A value passed for `&mut`, which takes what Rust writes back: Rust's bytes move to + // Rust, and C++'s it reads. + static c_buf take(buf &b) noexcept { + if (b.raw_.owner == nullptr) { + return raw(view(b)); + } + return ::std::exchange(b.raw_, c_buf{nullptr, 0, nullptr}); + } static buf adopt(c_buf raw) noexcept { return buf(raw); } diff --git a/ffi/runtime/rt.kt b/ffi/runtime/rt.kt index 67c4c1365..9a35ab0a3 100644 --- a/ffi/runtime/rt.kt +++ b/ffi/runtime/rt.kt @@ -46,17 +46,6 @@ class Writer { b.putDouble(v) } - /** Bytes as their length, or -1 for none, then the bytes. */ - fun bytes(v: ByteArray?) { - if (v == null) { - i32(-1) - return - } - room(4 + v.size) - b.putInt(v.size) - b.put(v) - } - /** Bytes of a length both sides know, such as an array's. */ fun raw(v: ByteArray) { room(v.size) @@ -105,19 +94,55 @@ class Writer { b.position(b.position() + v.size * 8) } - /** Bytes a record holds, as [bytes] writes a `ByteArray`. */ + /** + * Bytes a record holds, or -1 for none: Kotlin's as [KOTLIN], their array's place in + * [refs], where they start, and their length, which Rust copies once from the array, and + * Rust's as [LENT], their owner, where they start, and their length, which Rust reads in + * place, kept until [release]. + */ fun bytes(v: Bytes?) { if (v == null) { i32(-1) return } - room(4 + v.size) + val owner = v.owner + room(20) + if (owner != 0L) { + b.putInt(LENT) + b.putLong(owner) + b.putInt(v.start) + b.putInt(v.size) + (kept ?: java.util.ArrayList().also { kept = it }).add(v.held) + return + } + b.putInt(KOTLIN) + b.putInt(ref(v.held)) + b.putInt(v.start) b.putInt(v.size) - v.copyInto(b.array(), b.position()) - b.position(b.position() + v.size) } - fun string(v: String?) = bytes(v?.encodeToByteArray()) + /** Text as [TEXT] and its place in [refs], which Rust encodes once from, or -1 for none. */ + fun string(v: String?) { + if (v == null) { + i32(-1) + return + } + room(8) + b.putInt(TEXT) + b.putInt(ref(v)) + } + + private var objects: java.util.ArrayList? = null + + // The place in [refs] of [v], added to it. + private fun ref(v: Any?): Int { + val objects = objects ?: java.util.ArrayList().also { objects = it } + objects.add(v) + return objects.size - 1 + } + + /** The arrays and text the encoding refers to, which Rust reads in the call, or null for none. */ + fun refs(): kotlin.Array? = objects?.toArray() /** An address as Rust's `SocketAddr` holds it, which an unresolved one has no form of. */ fun socketAddr(v: java.net.InetSocketAddress) { @@ -161,6 +186,9 @@ class Writer { private var held: java.util.ArrayList? = null + // What holds Rust's bytes encoded in place, which the GC must not free before [release]. + private var kept: java.util.ArrayList? = null + /** An object's handle, held until [release], or 0 for none. */ fun handle(v: Handle?) { if (v == null) { @@ -176,6 +204,7 @@ class Writer { /** Release the handles encoded, once the call that reads them returns. */ fun release() { held?.forEach { it.release() } + java.lang.ref.Reference.reachabilityFence(kept) } /** Run [block] with this writer, releasing its handles once it returns. */ @@ -247,6 +276,9 @@ class Reader(private val b: java.nio.ByteBuffer) { return v } + /** Bytes Rust lent a call, copied, or null for none. */ + fun lent(): Bytes? = bytes()?.let { Bytes(it) } + /** Bytes Rust handed over, copied or lent by their owner, or null for none. */ fun owned(): Bytes? = when (val n = b.getInt()) { -1 -> null @@ -467,6 +499,12 @@ private class Free(lent: Lent, private val owner: Long) : Cleanup(lent) { /** What precedes a lent buffer's owner and length where an encoded one's length would. */ const val LENT = -2 +/** What precedes the place of Kotlin's array of bytes in a writer's `refs`, where an encoded one's length would. */ +const val KOTLIN = -3 + +/** What precedes the place of Kotlin's text in a writer's `refs`, where an encoded one's length would. */ +const val TEXT = -4 + /** Bytes a call returned: copied, lent by the owner and length a `LongArray` holds, or null for none. */ fun owned(given: Any?): Bytes? = when (given) { null -> null diff --git a/ffi/runtime/rt.rs b/ffi/runtime/rt.rs index e37c9834f..7e1da1895 100644 --- a/ffi/runtime/rt.rs +++ b/ffi/runtime/rt.rs @@ -81,8 +81,10 @@ impl Opaque { } } -/// Bytes across the ABI. Passed in, a view Rust copies or borrows for the call and never -/// frees. Returned, an owned buffer the caller frees once. A null `ptr` is `None`. +/// Bytes across the ABI. Returned, an owned buffer the caller frees once. Passed in by +/// value, one with an owner is Rust's own, which it takes without a copy, and one without +/// is a view it copies or borrows for the call; behind a pointer, as an array's items are, +/// it is only read. A null `ptr` is `None`. #[repr(C)] #[derive(Clone, Copy)] pub struct Buf { @@ -116,6 +118,113 @@ impl Buf { } } + /// The bytes a caller passed in: taken without a copy where `take` and it has an + /// owner, else copied, leaving the caller's buffer as it was. + /// + /// # Safety + /// As [`Self::bytes`], and an owner, where `take`, came from [`Self::owned`] and is + /// freed by nothing else. + pub unsafe fn bytes_in(self, what: &str, take: bool) -> bytes::Bytes { + match take && !self.owner.is_null() { + // SAFETY: the caller's contract. + true => unsafe { self.take(what) }, + // SAFETY: the caller's contract. + false => bytes::Bytes::copy_from_slice(unsafe { self.bytes(what) }), + } + } + + /// As [`Self::bytes_in`], as a `Vec`, which reuses an owner's allocation it alone holds. + /// + /// # Safety + /// As [`Self::bytes_in`]. + pub unsafe fn vec_in(self, what: &str, take: bool) -> Vec { + match take && !self.owner.is_null() { + // SAFETY: the caller's contract. + true => Vec::from(unsafe { self.take(what) }), + // SAFETY: the caller's contract. + false => unsafe { self.bytes(what) }.to_vec(), + } + } + + /// As [`Self::vec_in`], checked to be UTF-8. + /// + /// # Safety + /// As [`Self::bytes_in`]. + pub unsafe fn string_in(self, what: &str, take: bool) -> String { + // SAFETY: the caller's contract. + match String::from_utf8(unsafe { self.vec_in(what, take) }) { + Ok(s) => s, + Err(_) => fail(what, "a string is not UTF-8"), + } + } + + /// As [`Self::vec_in`], as a path: its bytes as the OS holds them on Unix, else UTF-8. + /// + /// # Safety + /// As [`Self::bytes_in`]. + pub unsafe fn path_in(self, what: &str, take: bool) -> std::path::PathBuf { + #[cfg(unix)] + { + use std::os::unix::ffi::OsStringExt; + // SAFETY: the caller's contract. + std::path::PathBuf::from(std::ffi::OsString::from_vec(unsafe { self.vec_in(what, take) })) + } + #[cfg(not(unix))] + { + // SAFETY: the caller's contract. + std::path::PathBuf::from(unsafe { self.string_in(what, take) }) + } + } + + /// The bytes a caller passed in for the call alone: taken where it has an owner, which + /// they free after, else borrowed. + /// + /// # Safety + /// As [`Self::bytes_in`] taking. + pub unsafe fn hold<'a>(self, what: &str) -> Hold<'a> { + match self.owner.is_null() { + // SAFETY: the caller's contract; a view lives as long as the call. + true => Hold::Lent(unsafe { std::slice::from_raw_parts(self.bytes(what).as_ptr(), self.len) }), + // SAFETY: the caller's contract. + false => Hold::Owned(unsafe { self.take(what) }), + } + } + + /// The bytes an owned buffer holds, taken from its owner: `ptr` and `len` may be any + /// range of them, as a caller slices a buffer Rust returned. + /// + /// # Safety + /// `owner` came from [`Self::owned`], and is freed by nothing else. + unsafe fn take(self, what: &str) -> bytes::Bytes { + // SAFETY: the caller's contract. + let bytes = *unsafe { Box::from_raw(self.owner.cast::()) }; + let start = (self.ptr as usize).wrapping_sub(bytes.as_ptr() as usize); + match self.len { + 0 => bytes::Bytes::new(), + // Whole, as is: a slice of bytes made from a `Vec` would allocate to share them. + len if start == 0 && len == bytes.len() => bytes, + _ if start <= bytes.len() && self.len <= bytes.len() - start => bytes.slice(start..start + self.len), + _ => fail(what, "a buffer's bytes are not its owner's"), + } + } + + /// A second owner of an owned buffer's bytes, sharing them: what a caller passes Rust + /// while it keeps the buffer. + /// + /// # Safety + /// `owner` came from [`Self::owned`], and is not yet freed. + pub unsafe fn share(self, what: &str) -> Self { + if self.owner.is_null() { + fail(what, "a buffer has no owner to share"); + } + // SAFETY: the caller's contract. + let bytes = unsafe { &*self.owner.cast::() }; + Self { + owner: Box::into_raw(Box::new(bytes.clone())).cast(), + ..self + } + } + /// The string a caller passed in, which it keeps. /// /// # Safety @@ -202,11 +311,7 @@ impl Buf { fail(what, "a returned buffer is not one buf_copy made"); } // SAFETY: the caller's contract. - let bytes = *unsafe { Box::from_raw(self.owner.cast::()) }; - match String::from_utf8(Vec::from(bytes)) { - Ok(string) => string, - Err(_) => fail(what, "a string is not UTF-8"), - } + unsafe { self.string_in(what, true) } } /// Free a buffer Rust returned; a view or `None` frees nothing. @@ -221,6 +326,42 @@ impl Buf { } } +/// Bytes passed in for a call: a view borrowed for it, or a buffer Rust took, freed after. +pub enum Hold<'a> { + Lent(&'a [u8]), + Owned(bytes::Bytes), +} + +impl Hold<'_> { + pub fn bytes(&self) -> &[u8] { + match self { + Self::Lent(bytes) => bytes, + Self::Owned(bytes) => bytes, + } + } + + pub fn str(&self, what: &str) -> &str { + match std::str::from_utf8(self.bytes()) { + Ok(s) => s, + Err(_) => fail(what, "a string is not UTF-8"), + } + } + + /// The path these hold: its bytes as the OS holds them on Unix, else UTF-8. + pub fn path(&self, what: &str) -> &std::path::Path { + #[cfg(unix)] + { + use std::os::unix::ffi::OsStrExt; + let _ = what; + std::path::Path::new(std::ffi::OsStr::from_bytes(self.bytes())) + } + #[cfg(not(unix))] + { + std::path::Path::new(self.str(what)) + } + } +} + /// `u128`, as its two halves. #[repr(C)] #[derive(Clone, Copy)] diff --git a/ffi/runtime/rt.swift b/ffi/runtime/rt.swift index f18e25dd3..02a8ae7d4 100644 --- a/ffi/runtime/rt.swift +++ b/ffi/runtime/rt.swift @@ -21,6 +21,71 @@ public struct SocketAddr: Hashable, Sendable { } } +/// Bytes as Rust holds them: Rust's stay in the buffer it returned, shared without a copy +/// by every `Bytes` and slice reading them and passed back to Rust uncopied, and Swift's are +/// a `Data`. Rust's are never lent as a `Data`, so nothing writes into the buffer Rust shares. +public struct Bytes: Hashable, Sendable, RandomAccessCollection { + // The bytes: Swift's, or a view of Rust's buffer that keeps `owner` alive. + let storage: Data + // Rust's buffer holding the bytes, or nil for Swift's. + let owner: Native.Owner? + + init(storage: Data, owner: Native.Owner?) { + self.storage = storage + self.owner = owner + } + + /// No bytes. + public init() { + self.init(storage: Data(), owner: nil) + } + + /// `data`'s bytes, sharing its storage as `Data` does: a call passing them lends them to + /// Rust, which copies them where it keeps them. + public init(_ data: Data) { + self.init(storage: data, owner: nil) + } + + /// `bytes`, copied. + public init(_ bytes: some Swift.Sequence) { + self.init(Data(bytes)) + } + + /// The bytes as a `Data`: Swift's as they are, and Rust's copied. + public var data: Data { + owner == nil ? storage : Data(storage) + } + + /// Indices count from the bytes sliced from, as `Data`'s do. + public var startIndex: Swift.Int { storage.startIndex } + public var endIndex: Swift.Int { storage.endIndex } + + public subscript(i: Swift.Int) -> Swift.UInt8 { + storage[i] + } + + /// The bytes in `bounds`, sharing these without a copy. + public subscript(bounds: Swift.Range) -> Bytes { + let slice = storage[bounds] + // `Data` may copy a short slice inline: those bytes are Swift's. + return Bytes(storage: slice, owner: owner.flatMap { $0.holds(slice) ? $0 : nil }) + } + + /// Run `body` with the bytes in place, which it must not keep. + public func withUnsafeBytes(_ body: (UnsafeRawBufferPointer) throws -> R) rethrows -> R { + try storage.withUnsafeBytes(body) + } + + /// Whether `a` and `b` hold the same bytes, wherever each keeps them. + public static func == (a: Bytes, b: Bytes) -> Swift.Bool { + a.storage == b.storage + } + + public func hash(into hasher: inout Swift.Hasher) { + storage.hash(into: &hasher) + } +} + /// A Rust `u128`, as its two halves, until the deployment target allows Swift's `UInt128`. public struct U128: Hashable, Comparable, Sendable, CustomStringConvertible { public var high: UInt64 @@ -173,20 +238,19 @@ extension Native { static var none: Buf { Buf(ptr: nil, len: 0, owner: nil) } - /// `bytes` as a view C reads during `body`, without a copy. Foundation's closure - /// returns only a copyable result, so `body`'s leaves through an optional. + /// `bytes` as a view C reads during `body`, without a copy. `Data`'s closure returns + /// only a copyable result, so `body`'s leaves through an optional. @inline(__always) - static func view(_ bytes: Data, _ body: (Buf) throws -> R) rethrows -> R { + static func view(_ bytes: Bytes, _ body: (Buf) throws -> R) rethrows -> R { var result: R? = nil - try bytes.withUnsafeBytes { raw in - let ptr = raw.baseAddress?.assumingMemoryBound(to: UInt8.self) ?? UnsafePointer(empty) - result = try body(Buf(ptr: ptr, len: raw.count, owner: nil)) + try bytes.storage.withUnsafeBytes { raw in + result = try body(view(raw)) } return result.take()! } @inline(__always) - static func view(_ bytes: Data?, _ body: (Buf) throws -> R) rethrows -> R { + static func view(_ bytes: Bytes?, _ body: (Buf) throws -> R) rethrows -> R { guard let bytes else { return try body(none) } return try view(bytes, body) } @@ -197,7 +261,7 @@ extension Native { var text = text var result: R? = nil try text.withUTF8 { utf8 in - result = try body(Buf(ptr: utf8.baseAddress ?? UnsafePointer(empty), len: utf8.count, owner: nil)) + result = try body(view(UnsafeRawBufferPointer(utf8))) } return result.take()! } @@ -208,28 +272,96 @@ extension Native { return try view(text, body) } - /// Bytes Rust returned, lent to `Data` without a copy and freed with it. - static func data(_ buf: Buf) -> Data { + /// A view of `raw`, never null, so empty bytes read as present. + @inline(__always) + static func view(_ raw: UnsafeRawBufferPointer) -> Buf { + Buf( + ptr: raw.baseAddress?.assumingMemoryBound(to: UInt8.self) ?? UnsafePointer(empty), len: raw.count, + owner: nil) + } + + /// The buffer Rust returned that `Bytes` read, freed once no `Bytes` does. + final class Owner: @unchecked Swift.Sendable { + let buf: Buf + + init(_ buf: Buf) { + self.buf = buf + } + + /// Whether `data`'s bytes lie in this buffer rather than in a copy. + func holds(_ data: Data) -> Swift.Bool { + guard let ptr = buf.ptr else { return false } + return data.withUnsafeBytes { raw in + guard let base = raw.baseAddress else { return false } + let start = UnsafeRawPointer(ptr) + return base >= start && base + raw.count <= start + buf.len + } + } + + deinit { + Native.free(buf) + } + + /// A second owner of `buf`, narrowed to `raw`, its bytes some `Bytes` reads, which + /// Rust takes. + func share(_ raw: UnsafeRawBufferPointer) -> Buf { + var shared = Native.clone(buf) + shared.ptr = raw.baseAddress!.assumingMemoryBound(to: UInt8.self) + shared.len = raw.count + return shared + } + } + + /// Bytes Rust returned, lent to `Bytes` without a copy and freed with the last one. + static func bytes(_ buf: Buf) -> Bytes { guard let ptr = buf.ptr, buf.len > 0 else { free(buf) - return Data() + return Bytes() } - return Data( + let owner = Owner(buf) + let storage = Data( bytesNoCopy: UnsafeMutableRawPointer(mutating: ptr), count: buf.len, - deallocator: .custom { _, _ in free(buf) }) + deallocator: .custom { _, _ in Swift.withExtendedLifetime(owner) {} }) + // `Data` copies a few bytes inline, freeing Rust's buffer at once: those are Swift's. + return Bytes(storage: storage, owner: owner.holds(storage) ? owner : nil) + } + + static func bytes(option buf: Buf) -> Bytes? { + buf.ptr == nil ? nil : bytes(buf) } - static func data(option buf: Buf) -> Data? { - buf.ptr == nil ? nil : data(buf) + /// `bytes` as a buffer Rust takes during `body`: a second owner of Rust's own, or a view + /// of Swift's, which Rust copies. + @inline(__always) + static func take(_ bytes: Bytes, _ body: (Buf) throws -> R) rethrows -> R { + guard let owner = bytes.owner, !bytes.isEmpty else { return try view(bytes, body) } + return try body(bytes.storage.withUnsafeBytes { owner.share($0) }) + } + + @inline(__always) + static func take(_ bytes: Bytes?, _ body: (Buf) throws -> R) rethrows -> R { + guard let bytes else { return try body(none) } + return try take(bytes, body) + } + + /// Text Rust takes during `body`: a view, which Rust copies. + @inline(__always) + static func take(_ text: Swift.String, _ body: (Buf) throws -> R) rethrows -> R { + try view(text, body) + } + + @inline(__always) + static func take(_ text: Swift.String?, _ body: (Buf) throws -> R) rethrows -> R { + try view(text, body) } /// Bytes C lends a callback for its call, copied. - static func data(lent buf: Buf) -> Data { - buf.len == 0 ? Data() : Data(bytes: buf.ptr!, count: buf.len) + static func bytes(lent buf: Buf) -> Bytes { + buf.len == 0 ? Bytes() : Bytes(Data(bytes: buf.ptr!, count: buf.len)) } - static func data(lentOption buf: Buf) -> Data? { - buf.ptr == nil ? nil : data(lent: buf) + static func bytes(lentOption buf: Buf) -> Bytes? { + buf.ptr == nil ? nil : bytes(lent: buf) } /// Text Rust returned, copied, then freed. @@ -475,6 +607,9 @@ extension Native { final class Arena { private var blocks: [UnsafeMutableRawPointer] = [] private var held: [Handle] = [] + /// Whether bytes lowered now are passed by value, which Rust takes: set once the + /// receiver, which Rust only reads, is lowered, and unset inside an array. + var take = false func copy(_ bytes: UnsafeRawBufferPointer) -> Buf { guard bytes.count > 0 else { @@ -486,16 +621,25 @@ extension Native { return Buf(ptr: block.assumingMemoryBound(to: UInt8.self), len: bytes.count, owner: nil) } - func buf(_ bytes: Data) -> Buf { - bytes.withUnsafeBytes { copy($0) } + /// `bytes` for Rust to read, copied here, or where `take`, Rust's own: a second owner + /// of a buffer it returned, else a copy it owns, which it takes without another. + func buf(_ bytes: Bytes) -> Buf { + bytes.storage.withUnsafeBytes { raw in + guard take else { return copy(raw) } + if let owner = bytes.owner, raw.count > 0 { return owner.share(raw) } + return Native.copy(view(raw)) + } } func buf(_ text: Swift.String) -> Buf { var text = text - return text.withUTF8 { copy(UnsafeRawBufferPointer($0)) } + return text.withUTF8 { utf8 in + guard take else { return copy(UnsafeRawBufferPointer(utf8)) } + return Native.copy(view(UnsafeRawBufferPointer(utf8))) + } } - func buf(_ bytes: Data?) -> Buf { + func buf(_ bytes: Bytes?) -> Buf { bytes.map { buf($0) } ?? none } @@ -518,6 +662,10 @@ extension Native { func list(_ items: [T], _ lower: (T) throws -> C) rethrows -> (UnsafePointer, Int) { let array = UnsafeMutablePointer.allocate(capacity: Swift.max(items.count, 1)) blocks.append(UnsafeMutableRawPointer(array)) + // Rust only reads an array's items. + let take = self.take + self.take = false + defer { self.take = take } for (i, item) in items.enumerated() { (array + i).initialize(to: try lower(item)) } @@ -538,6 +686,10 @@ extension Native { ) rethrows -> (UnsafePointer, Int) { let array = UnsafeMutablePointer.allocate(capacity: Swift.max(items.count, 1)) blocks.append(UnsafeMutableRawPointer(array)) + // Rust only reads an array's items. + let take = self.take + self.take = false + defer { self.take = take } for i in 0.. Abi<'a> { "foreign_future", "foreign_reader", "buf_copy", + "buf_clone", "poll", "view", "borrow", @@ -935,6 +936,7 @@ impl<'a> Abi<'a> { "detail", "native", "bytes", + "input", "mut", "atomic_load", "atomic_store", diff --git a/ffi/src/c.rs b/ffi/src/c.rs index 1a58c464c..38cb2b8f0 100644 --- a/ffi/src/c.rs +++ b/ffi/src/c.rs @@ -79,7 +79,7 @@ impl Header<'_> { "/* The index of the library's target among those this header serves:{targets} */\nuint32_t {p}_target(void);\n" )); self.line(format!( - "/* Bytes. Passed in, a view Rust reads during the call and never frees. Returned,\n * an owned buffer to free once with {p}_buf_free. A null ptr is None. */\ntypedef struct {p}_buf {{\n\tconst uint8_t *ptr;\n\tsize_t len;\n\tvoid *owner;\n}} {p}_buf;\n\nvoid {p}_buf_free({p}_buf buf);\n\n/* An owned copy of view's bytes: what a closure or reader C implements returns Rust,\n * which then owns it. */\n{p}_buf {p}_buf_copy({p}_buf view);\n" + "/* Bytes. Returned, an owned buffer to free once with {p}_buf_free, or pass back to\n * Rust by value, ptr and len narrowed to any range of its bytes. Passed by value, one\n * whose owner is set is Rust's from then on, taken without a copy, and one whose owner\n * is NULL is a view Rust reads during the call and never frees; in a value passed by\n * pointer, or an array's item, either is only read. A null ptr is None. */\ntypedef struct {p}_buf {{\n\tconst uint8_t *ptr;\n\tsize_t len;\n\tvoid *owner;\n}} {p}_buf;\n\nvoid {p}_buf_free({p}_buf buf);\n\n/* An owned copy of view's bytes: what a closure or reader C implements returns Rust,\n * which then owns it. */\n{p}_buf {p}_buf_copy({p}_buf view);\n\n/* A second owned buffer of an owned buffer's bytes, sharing them without a copy: what\n * a caller passes Rust while keeping buf. */\n{p}_buf {p}_buf_clone({p}_buf buf);\n" )); self.line(format!( "/* A u128: lo holds its low 64 bits and hi its high 64. */\ntypedef struct {p}_u128 {{\n\tuint64_t lo;\n\tuint64_t hi;\n}} {p}_u128;\n" diff --git a/ffi/src/cpp.rs b/ffi/src/cpp.rs index 5515f9e72..8d8dabf9f 100644 --- a/ffi/src/cpp.rs +++ b/ffi/src/cpp.rs @@ -198,11 +198,12 @@ impl<'a> Header<'a> { { return format!("{} &", self.owned(inner)); } + // Passed by value, so a `buf` Rust made moves back without a copy. if Abi::is_buf(ty) { - return format!("::{}::view", self.ns); + return format!("::{}::input", self.ns); } match ty { - Type::Option(inner) if Abi::is_buf(inner) => format!("::std::optional<::{}::view>", self.ns), + Type::Option(inner) if Abi::is_buf(inner) => format!("::std::optional<::{}::input>", self.ns), // Moved out of, each handle as a handle taken by value is. Type::Option(inner) if matches!(inner.base(), Type::List(_, elem) if moves(abi, elem)) => { format!("::std::optional<{}>", self.items(inner.base())) @@ -301,6 +302,30 @@ impl<'a> Header<'a> { } } + /// Whether [`Self::arg`] takes Rust's bytes from a parameter of type `ty`. + fn takes(ty: &Type) -> bool { + match ty { + _ if Abi::is_buf(ty) => true, + Type::Option(inner) => Abi::is_buf(inner), + Type::Ref(_, inner) | Type::Box(inner) => matches!(inner.base(), Type::Option(o) if Abi::is_buf(o)), + _ => false, + } + } + + /// The C value of parameter `e` of type `ty`: a buffer's `input`, or a value passed for + /// `&mut`, hands Rust's bytes over. + fn arg(&self, ty: &Type, e: &str) -> String { + let d = self.d(); + match ty { + _ if Abi::is_buf(ty) => format!("{d}::access::take({e})"), + Type::Option(inner) if Abi::is_buf(inner) => format!("({e} ? {d}::access::take(*{e}) : {d}::none())"), + Type::Ref(_, inner) | Type::Box(inner) if matches!(inner.base(), Type::Option(o) if Abi::is_buf(o)) => { + self.arg(inner.base(), e) + } + _ => self.lower(ty, e), + } + } + /// The C value of the C++ value `e` of type `ty`, as a parameter or a field passes it. fn lower(&self, ty: &Type, e: &str) -> String { let abi = self.abi; @@ -501,7 +526,7 @@ impl<'a> Header<'a> { ); self.line(format!("namespace {ns} {{\n")); self.line(format!( - "namespace detail {{\n\nusing c_buf = ::{p}_buf;\nusing c_waker = ::{p}_waker;\nusing c_foreign_future = ::{p}_foreign_future;\nusing c_foreign_reader = ::{p}_foreign_reader;\n\ninline void c_buf_free(c_buf buf) noexcept {{\n\t::{p}_buf_free(buf);\n}}\n\ninline c_buf c_buf_copy(c_buf view) noexcept {{\n\treturn ::{p}_buf_copy(view);\n}}\n\n}} // namespace detail\n" + "namespace detail {{\n\nusing c_buf = ::{p}_buf;\nusing c_waker = ::{p}_waker;\nusing c_foreign_future = ::{p}_foreign_future;\nusing c_foreign_reader = ::{p}_foreign_reader;\n\ninline void c_buf_free(c_buf buf) noexcept {{\n\t::{p}_buf_free(buf);\n}}\n\ninline c_buf c_buf_copy(c_buf view) noexcept {{\n\treturn ::{p}_buf_copy(view);\n}}\n\ninline c_buf c_buf_clone(c_buf buf) noexcept {{\n\treturn ::{p}_buf_clone(buf);\n}}\n\n}} // namespace detail\n" )); self.out.push_str(RUNTIME); self.line(format!( @@ -1381,6 +1406,14 @@ impl<'a> Header<'a> { _ => &[], }; let mut names: Vec = Vec::new(); + // Whether an argument takes Rust's bytes from its parameter, and the arguments that + // may throw, by allocating. + let mut taken = false; + let mut throwing = Vec::new(); + // The allocations among them that Rust takes (a kept closure, a future), each by its + // argument, its type, its `new`, and what passes the pointer, which a hoisted one holds + // in a `unique_ptr` until the call, so a later throw frees it. + let mut owned: Vec<(usize, String, String, Option)> = Vec::new(); for param in &function.params { let (_, c_ty) = c_params.next().expect("each parameter has its C one"); let name = parameter(¶m.name, |name| { @@ -1391,7 +1424,11 @@ impl<'a> Header<'a> { if let Type::Future = param.ty { let f = format!("F{}", templates.len()); params.push(format!("{f} {name}")); - args.push(format!("{d}::foreign_future(::std::move({name}))")); + let new = format!("new {f}(::std::move({name}))"); + let wrap = format!("{d}::foreign_future"); + throwing.push(args.len()); + args.push(format!("{wrap}({new})")); + owned.push((args.len() - 1, f.clone(), new, Some(wrap))); templates.push(format!("class {f}")); continue; } @@ -1417,7 +1454,11 @@ impl<'a> Header<'a> { // Moved or copied into a callable Rust owns, which it destroys through `free`. true => { c_params.next().expect("a kept closure has its free"); - args.push(format!("new ::std::decay_t<{f}>(::std::forward<{f}>({name}))")); + let ty = format!("::std::decay_t<{f}>"); + let new = format!("new {ty}(::std::forward<{f}>({name}))"); + throwing.push(args.len()); + owned.push((args.len(), ty, new.clone(), None)); + args.push(new); args.push(format!( "[](void *data_) noexcept {{ delete static_cast<::std::decay_t<{f}> *>(data_); }}" )); @@ -1430,7 +1471,32 @@ impl<'a> Header<'a> { continue; } params.push(declare(&self.param(¶m.ty), &name)); - args.push(self.lower(¶m.ty, &name)); + if Self::takes(¶m.ty) { + taken = true; + } else if self.arena(¶m.ty) { + throwing.push(args.len()); + } + args.push(self.arg(¶m.ty, &name)); + } + // Arguments evaluate in no set order, so those that may throw run first, and the call + // itself throws nothing: Rust's bytes are taken, and an allocation released to Rust, + // only once nothing left before the call can throw and leak them. + if taken || throwing.len() > 1 { + for &i in &throwing { + match owned.iter().find(|(k, ..)| *k == i) { + Some((_, ty, new, wrap)) => { + prelude.push(format!("::std::unique_ptr<{ty}> c_arg{i}_({new});")); + args[i] = match wrap { + Some(wrap) => format!("{wrap}(c_arg{i}_.release())"), + None => format!("c_arg{i}_.release()"), + }; + } + None => { + prelude.push(format!("auto c_arg{i}_ = {};", args[i])); + args[i] = format!("c_arg{i}_"); + } + } + } } let outs: Vec<(String, Ty)> = c_params.cloned().collect(); // Each value passed for `&mut` takes what Rust wrote back, on every path. diff --git a/ffi/src/dart.rs b/ffi/src/dart.rs index 072149693..922cfefb2 100644 --- a/ffi/src/dart.rs +++ b/ffi/src/dart.rs @@ -134,6 +134,8 @@ pub fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result", @@ -238,6 +240,7 @@ pub fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result format!("_{p}_{name}"), "buf_free" => "_bufFree".to_string(), "buf_copy" => "_bufCopy".to_string(), + "buf_clone" => "_bufClone".to_string(), other => format!("_{}", camel(other)), }; let symbol = match name.strip_prefix("dart") { @@ -286,6 +289,7 @@ pub fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result *mut c_void;\n\tfn free(ptr: *mut c_void);\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_alloc(len: usize) -> *mut c_void {{\n\tlet ptr = unsafe {{ malloc(len) }};\n\tif ptr.is_null() {{\n\t\tstd::alloc::handle_alloc_error(std::alloc::Layout::array::(len).expect(\"a length\"));\n\t}}\n\tptr\n}}\n\n\ + #[unsafe(no_mangle)]\npub extern \"C\" fn {space}_buf(len: usize) -> rt::Buf {{\n\trt::Buf::vec(vec![0; len])\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_free(ptr: *mut c_void) {{\n\tunsafe {{ free(ptr) }}\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_abort(text: rt::Buf) {{\n\trt::fail(\"dart\", unsafe {{ text.str(\"{space}_abort\") }})\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_reader_read(data: *mut c_void, buf: *mut u8, len: usize, n: *mut usize, err: *mut rt::Buf) -> bool {{\n\tunsafe {{ runtime::reader_read(data, buf, len, n, err) }}\n}}\n\n\ @@ -888,16 +892,17 @@ impl<'a> Gen<'a> { } /// Statements writing the Dart value `e` of type `ty` into the C place `dst`, a - /// struct's field or element: bytes copied into the arena `a_`, and handles held by it. - fn write(&mut self, ty: &Type, e: &str, dst: &str) -> String { + /// struct's field or element: handles held by the arena `a_`, and bytes given to Rust + /// where `take`, the Dart `bool` whether Rust takes the value, else viewed. + fn write(&mut self, ty: &Type, e: &str, dst: &str, take: &str) -> String { let abi = self.abi; let c = abi.ty(ty); if Self::scalar(&c) { return format!("{dst} = {};", self.lower(ty, e)); } match ty.base() { - Type::Str | Type::String | Type::Path | Type::PathBuf => format!("a_.string({dst}, {e});"), - Type::Slice | Type::Vec | Type::Bytes => format!("a_.bytes({dst}, {e});"), + Type::Str | Type::String | Type::Path | Type::PathBuf => format!("a_.string({dst}, {e}, {take});"), + Type::Slice | Type::Vec | Type::Bytes => format!("a_.bytes({dst}, {e}, {take});"), Type::U128 => format!("_n.u128({dst}, {e});"), Type::Duration => format!("_n.duration({dst}, {e});"), Type::Array(_) => format!("{e}._write({dst});"), @@ -910,17 +915,19 @@ impl<'a> Gen<'a> { format!("_n.addr({dst}, {e}, '{family}');") } Type::Option(inner) => match inner.base() { - Type::Str | Type::String | Type::Path | Type::PathBuf => format!("a_.string({dst}, {e});"), - Type::Slice | Type::Vec | Type::Bytes => format!("a_.bytes({dst}, {e});"), + Type::Str | Type::String | Type::Path | Type::PathBuf => { + format!("a_.string({dst}, {e}, {take});") + } + Type::Slice | Type::Vec | Type::Bytes => format!("a_.bytes({dst}, {e}, {take});"), Type::List(..) => { let c = self.c_name(inner); let w = self.writer(&c, inner.base()); - format!("{w}({e}, {dst}, a_);") + format!("{w}({e}, {dst}, a_, {take});") } _ => { let c = abi.option(inner); let w = self.writer(&c, ty.base()); - format!("{w}({e}, {dst}, a_);") + format!("{w}({e}, {dst}, a_, {take});") } }, Type::Record(_) @@ -932,7 +939,7 @@ impl<'a> Gen<'a> { | Type::Bounds(_) => { let c = self.c_name(ty.base()); let w = self.writer(&c, ty.base()); - format!("{w}({e}, {dst}, a_);") + format!("{w}({e}, {dst}, a_, {take});") } other => unreachable!("{other:?} is never nested"), } @@ -970,7 +977,7 @@ impl<'a> Gen<'a> { for field in &record.fields { let e = format!("v.{}", field_name(&field.name)); let dst = format!("c.${}", abi::escape(&field.name)); - lines.push(self.write(&field.ty, &e, &dst)); + lines.push(self.write(&field.ty, &e, &dst, "t")); } lines.join("\n\t") } @@ -990,7 +997,7 @@ impl<'a> Gen<'a> { for field in &variant.fields { let e = format!("v.{}", field_name(&field.name)); let dst = format!("c.$u.${member}.${}", abi::escape(&field.name)); - body.push(self.write(&field.ty, &e, &dst)); + body.push(self.write(&field.ty, &e, &dst, "t")); } lines.push(format!( "if (v is {case}) {{\n\t\t{}\n\t\treturn;\n\t}}", @@ -1000,7 +1007,7 @@ impl<'a> Gen<'a> { lines.join("\n\t") } Type::Option(inner) => { - let some = self.write(inner, "v", "c.$value"); + let some = self.write(inner, "v", "c.$value", "t"); format!("if (v == null) {{\n\t\tc.$some = false;\n\t}} else {{\n\t\tc.$some = true;\n\t\t{some}\n\t}}") } // A block is copied whole into the arena. @@ -1015,7 +1022,8 @@ impl<'a> Gen<'a> { let nt = self.nt(&elem_c); let item = match Self::scalar(&elem_c) { true => format!("p[i] = {};", self.lower(elem, "v[i]")), - false => self.write(elem, "v[i]", "p[i]"), + // An array's items are behind its pointer, which Rust only reads. + false => self.write(elem, "v[i]", "p[i]", "false"), }; format!( "if (v == null) {{\n\t\tc.$ptr = ffi.nullptr;\n\t\tc.$len = 0;\n\t\tc.$owner = ffi.nullptr;\n\t\treturn;\n\t}}\n\tfinal p = a_<{nt}>(v.isEmpty ? 1 : v.length);\n\tfor (var i = 0; i < v.length; i++) {{\n\t\t{item}\n\t}}\n\tc.$ptr = p;\n\tc.$len = v.length;\n\tc.$owner = ffi.nullptr;" @@ -1024,7 +1032,7 @@ impl<'a> Gen<'a> { Type::Tuple(elements) => { let mut lines = Vec::new(); for (n, element) in elements.iter().enumerate() { - lines.push(self.write(element, &format!("v.${}", n + 1), &format!("c.$_{n}"))); + lines.push(self.write(element, &format!("v.${}", n + 1), &format!("c.$_{n}"), "t")); } lines.join("\n\t") } @@ -1045,8 +1053,8 @@ impl<'a> Gen<'a> { let bound = Type::Bound(*int); format!( "{}\n\t{}", - self.write(&bound, "v.start", "c.$start"), - self.write(&bound, "v.end", "c.$end") + self.write(&bound, "v.start", "c.$start", "t"), + self.write(&bound, "v.end", "c.$end", "t") ) } other => unreachable!("{other:?} has no writer"), @@ -1057,7 +1065,7 @@ impl<'a> Gen<'a> { }; self.helpers.insert( name.clone(), - format!("void {name}({dt}{nullable} v, {class} c, _Arena a_) {{\n\t{body}\n}}\n\n"), + format!("void {name}({dt}{nullable} v, {class} c, _Arena a_, bool t) {{\n\t{body}\n}}\n\n"), ); name } @@ -1444,7 +1452,8 @@ impl<'a> Gen<'a> { let nt = self.nt(c); let write = match Self::scalar(c) { true => format!("{s}.value = {};", self.lower(ty, e)), - false => self.write(ty, e, &format!("{s}.ref")), + // Read through a pointer, so Rust takes nothing of it. + false => self.write(ty, e, &format!("{s}.ref"), "false"), }; (format!("final {s} = a_<{nt}>();\n\t\t\t{write}\n\t\t\t"), s) } @@ -2008,6 +2017,9 @@ impl<'a> Gen<'a> { // so every other one that may throw is lowered before, and a call that throws gives // Rust nothing. let mut gave = false; + // Whether the call is given a value Rust takes, whose buffers the arena frees + // unless the call returns. + let mut gives = false; let names = abi::distinct(function.params.iter().map(|param| param_name(¶m.name))); let mut writes = Vec::new(); for (k, (param, name)) in function.params.iter().zip(names).enumerate() { @@ -2022,7 +2034,8 @@ impl<'a> Gen<'a> { false => { let v = format!("c{k}_"); let nt = self.nt(c); - let write = self.write(inner, &value, &format!("{v}.ref")); + gives = true; + let write = self.write(inner, &value, &format!("{v}.ref"), "true"); pre.push(format!("final {v} = a_<{nt}>();\n{write}")); args.push(format!("{v}.ref")); } @@ -2078,7 +2091,14 @@ impl<'a> Gen<'a> { (ty, c) => { let v = format!("c{k}_"); let nt = self.nt(c); - let write = self.write(ty, &name, &format!("{v}.ref")); + // Rust takes what a call is passed by value, and only reads a borrow. + let take = match ty { + Type::Ref(..) => "false", + Type::Option(inner) if matches!(**inner, Type::Ref(..)) => "false", + _ => "true", + }; + gives |= take == "true"; + let write = self.write(ty, &name, &format!("{v}.ref"), take); pre.push(format!("final {v} = a_<{nt}>();\n{write}")); args.push(format!("{v}.ref")); } @@ -2148,11 +2168,11 @@ impl<'a> Gen<'a> { } inner.push("final o_f_ = a_>();".to_string()); start_args.push("o_f_".to_string()); - inner.push(format!( - "switch ({start_fn}({})) {{\n{}\n}}", - start_args.join(", "), - indent(&started.join("\n")) - )); + inner.push(format!("final started_ = {start_fn}({});", start_args.join(", "),)); + if gives { + inner.push("a_.took();".to_string()); + } + inner.push(format!("switch (started_) {{\n{}\n}}", indent(&started.join("\n")))); inner.extend(after.clone()); body.push(format!("final c_ = _Call<{ok_dt}>._({free_fn}, {poll_name});")); body.push(format!( @@ -2169,9 +2189,13 @@ impl<'a> Gen<'a> { let mut call_args = args.clone(); call_args.extend(outs.iter().map(|(n, _)| format!("o_{n}"))); let call = format!("{call_fn}({})", call_args.join(", ")); + let took = match gives { + true => "\na_.took();", + false => "", + }; match (&err, &ok) { (Some(err), ok) => { - inner.push(format!("final ok_ = {call};")); + inner.push(format!("final ok_ = {call};{took}")); inner.extend(writes); inner.extend(after.clone()); let ready = match ok { @@ -2187,19 +2211,19 @@ impl<'a> Gen<'a> { )); } (None, Type::Unit) => { - inner.push(format!("{call};")); + inner.push(format!("{call};{took}")); inner.extend(writes); inner.extend(after.clone()); } // A value C returns beside what Rust wrote back is its return, as with none. (None, ok) if outs.len() > writes.len() => { - inner.push(format!("{call};")); + inner.push(format!("{call};{took}")); inner.extend(writes); inner.extend(after.clone()); inner.push(format!("return {};", self.lift_outs(&function.ok, ok))); } (None, ok) => { - inner.push(format!("final r_ = {call};")); + inner.push(format!("final r_ = {call};{took}")); inner.extend(writes); inner.extend(after.clone()); inner.push(format!("return {};", self.lift(ok, "r_"))); diff --git a/ffi/src/glue.rs b/ffi/src/glue.rs index 77952a49e..bd0d7d6b4 100644 --- a/ffi/src/glue.rs +++ b/ffi/src/glue.rs @@ -580,8 +580,11 @@ impl Gen<'_> { } } - /// The Rust value of type `ty` that the C value `e` holds, copied out of it. - fn lift(&self, ty: &Type, e: &str) -> String { + /// The Rust value of type `ty` that the C value `e` holds. `take` is the Rust `bool` + /// whether `e` was handed over, as a value passed by value is, so a buffer in it with an + /// owner is taken without a copy; else what it holds is copied out, and it stays the + /// caller's. + fn lift(&self, ty: &Type, e: &str, take: &str) -> String { match ty { Type::Const(_) => unreachable!("a const argument is no value"), Type::Unit => e.to_string(), @@ -590,10 +593,10 @@ impl Gen<'_> { format!("::std::num::NonZero::new({e}).unwrap_or_else(|| rt::fail(w, \"a NonZero is zero\"))") } Type::Char => format!("rt::char({e}, w)"), - Type::Str | Type::String => format!("unsafe {{ {e}.str(w) }}.to_owned()"), - Type::Slice | Type::Vec => format!("unsafe {{ {e}.bytes(w) }}.to_vec()"), - Type::Bytes => format!("::bytes::Bytes::copy_from_slice(unsafe {{ {e}.bytes(w) }})"), - Type::Path | Type::PathBuf => format!("unsafe {{ {e}.path(w) }}.to_path_buf()"), + Type::Str | Type::String => format!("unsafe {{ {e}.string_in(w, {take}) }}"), + Type::Slice | Type::Vec => format!("unsafe {{ {e}.vec_in(w, {take}) }}"), + Type::Bytes => format!("unsafe {{ {e}.bytes_in(w, {take}) }}"), + Type::Path | Type::PathBuf => format!("unsafe {{ {e}.path_in(w, {take}) }}"), Type::U128 => format!("{e}.lift()"), Type::Array(_) => format!("{e}.items"), Type::Duration | Type::SocketAddr(Family::Any) => format!("{e}.lift(w)"), @@ -612,7 +615,8 @@ impl Gen<'_> { ), Type::List(c, inner) => format!( "unsafe {{ {e}.items(w) }}.iter().map(|c| {}).collect::<{}>()", - self.lift(inner, "(*c)"), + // An array's items are behind its pointer, so only read. + self.lift(inner, "(*c)", "false"), match c { Collection::BTreeMap | Collection::JsonMap => format!("{}<_, _>", collection(*c)), Collection::Box => format!("{}<[_]>", collection(*c)), @@ -623,24 +627,30 @@ impl Gen<'_> { // A foreign enum's option converts into its filter, as `tracing::LevelFilter`. Type::Option(inner) if self.filtered(inner) => format!( "::std::convert::From::from(if {e}.some {{ Some({}) }} else {{ None }})", - self.lift(inner, &format!("{e}.value")) + self.lift(inner, &format!("{e}.value"), take) ), Type::Option(inner) => match inner.base() { - Type::Object(_) => format!("if {e}.is_null() {{ None }} else {{ Some({}) }}", self.lift(inner, e)), + Type::Object(_) => format!( + "if {e}.is_null() {{ None }} else {{ Some({}) }}", + self.lift(inner, e, take) + ), base if Abi::nullable(base) => { - format!("if {e}.is_none() {{ None }} else {{ Some({}) }}", self.lift(inner, e)) + format!( + "if {e}.is_none() {{ None }} else {{ Some({}) }}", + self.lift(inner, e, take) + ) } _ => format!( "if {e}.some {{ Some({}) }} else {{ None }}", - self.lift(inner, &format!("{e}.value")) + self.lift(inner, &format!("{e}.value"), take) ), }, - Type::Box(inner) => format!("::std::boxed::Box::new({})", self.lift(inner, e)), - Type::Ref(_, inner) => self.lift(inner, e), - Type::Record(i) => format!("lift_{}(&{e}, w)", self.abi.record(*i)), + Type::Box(inner) => format!("::std::boxed::Box::new({})", self.lift(inner, e, take)), + Type::Ref(_, inner) => self.lift(inner, e, take), + Type::Record(i) => format!("lift_{}(&{e}, w, {take})", self.abi.record(*i)), Type::Enum(i) => match self.abi.api.enums[*i].plain() { true => format!("lift_{}({e}, w)", self.abi.enumeration(*i)), - false => format!("lift_{}(&{e}, w)", self.abi.enumeration(*i)), + false => format!("lift_{}(&{e}, w, {take})", self.abi.enumeration(*i)), }, Type::Object(i) if self.abi.api.objects[*i].opaque => { format!("unsafe {{ rt::Object::::take({e}, w) }}.arc(w)") @@ -660,7 +670,7 @@ impl Gen<'_> { Type::Tuple(elements) => { // The trailing comma makes `(x,)` a tuple, not parentheses. let lifted: Vec = (elements.iter().enumerate()) - .map(|(n, ty)| format!("{},", self.lift(ty, &format!("{e}._{n}")))) + .map(|(n, ty)| format!("{},", self.lift(ty, &format!("{e}._{n}"), take))) .collect(); format!("({})", lifted.concat()) } @@ -755,7 +765,7 @@ impl Gen<'_> { "{{ let r = {e}; if r.is_null() {{ None }} else {{ Some({}) }} }}", self.returned(inner, "r") ), - ty => self.lift(ty, e), + ty => self.lift(ty, e, "true"), } } @@ -932,6 +942,9 @@ impl Gen<'_> { self.line(format!( "#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {p}_buf_copy(view: rt::Buf) -> rt::Buf {{\n\trt::Buf::vec(unsafe {{ view.bytes(\"{p}_buf_copy\") }}.to_vec())\n}}\n" )); + self.line(format!( + "#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {p}_buf_clone(buf: rt::Buf) -> rt::Buf {{\n\tunsafe {{ buf.share(\"{p}_buf_clone\") }}\n}}\n" + )); self.line(format!( "#[unsafe(no_mangle)]\npub extern \"C\" fn {p}_contract() -> u64 {{\n\t0x{:016x}\n}}\n", self.contract @@ -1065,9 +1078,10 @@ impl Gen<'_> { /// The functions every value type has: its std traits, read through a pointer, and /// `free`. fn value_traits(&mut self, name: &str, path: &str, ty: &Type, traits: Traits) { - let lift = self.lift(ty, "unsafe { *this }"); - let lift_a = self.lift(ty, "unsafe { *a }"); - let lift_b = self.lift(ty, "unsafe { *b }"); + // Each is read through a pointer, so the caller keeps it. + let lift = self.lift(ty, "unsafe { *this }", "false"); + let lift_a = self.lift(ty, "unsafe { *a }", "false"); + let lift_b = self.lift(ty, "unsafe { *b }", "false"); let c = rs(&self.abi.ty(ty)); if traits.eq { self.export(format!( @@ -1115,7 +1129,7 @@ impl Gen<'_> { .fields .iter() .enumerate() - .map(|(n, f)| self.lift(&f.ty, &format!("c.f{n}"))) + .map(|(n, f)| self.lift(&f.ty, &format!("c.f{n}"), "take")) .collect(); let build = match record.tuple { true => format!("{path}({})", lifted.join(", ")), @@ -1130,7 +1144,7 @@ impl Gen<'_> { } }; self.item(format!( - "fn lift_{name}(c: &{name}, w: &str) -> {path} {{\n\tlet _ = w;\n\t{build}\n}}\n" + "fn lift_{name}(c: &{name}, w: &str, take: bool) -> {path} {{\n\tlet _ = (w, take);\n\t{build}\n}}\n" )); let bindings: Vec = (0..record.fields.len()).map(|n| format!("f{n}")).collect(); let pattern = match record.tuple { @@ -1277,7 +1291,7 @@ impl Gen<'_> { .fields .iter() .enumerate() - .map(|(k, f)| self.lift(&f.ty, &format!("p.f{k}"))) + .map(|(k, f)| self.lift(&f.ty, &format!("p.f{k}"), "take")) .collect(); let build = match variant.tuple { true => format!("{path}::{}({})", variant.name, lifted.join(", ")), @@ -1321,7 +1335,7 @@ impl Gen<'_> { } } self.item(format!( - "fn lift_{name}(c: &{name}, w: &str) -> {path} {{\n\tmatch c.tag {{{lift}\n\t\t_ => rt::fail(w, \"an enum tag is invalid\"),\n\t}}\n}}\n" + "fn lift_{name}(c: &{name}, w: &str, take: bool) -> {path} {{\n\tlet _ = take;\n\tmatch c.tag {{{lift}\n\t\t_ => rt::fail(w, \"an enum tag is invalid\"),\n\t}}\n}}\n" )); self.item(format!( "fn lower_{name}(v: {path}) -> {name} {{\n\tmatch v {{{lower}{wildcard}\n\t}}\n}}\n" @@ -1384,7 +1398,7 @@ impl Gen<'_> { params.iter().map(|p| format!(", {p}")).collect::(), self.rust(&method.ok), args.join(", "), - self.lift(&method.ok, "result"), + self.lift(&method.ok, "result", "true"), ); } self.line(format!("#[repr(C)]\npub struct {name}_vtable {{\n{table}}}\n")); @@ -1448,15 +1462,19 @@ impl Gen<'_> { }; (format!("{binding} {local} = {borrow};"), pass) } - (Type::Str, Mutability::Not, false) => { - (format!("let {local} = unsafe {{ {c}.str(w) }};"), local.to_string()) - } - (Type::Slice, Mutability::Not, false) => { - (format!("let {local} = unsafe {{ {c}.bytes(w) }};"), local.to_string()) - } - (Type::Path, Mutability::Not, false) => { - (format!("let {local} = unsafe {{ {c}.path(w) }};"), local.to_string()) - } + // Borrowed for the call: held, freeing a buffer it took once the call returns. + (Type::Str, Mutability::Not, false) => ( + format!("let {local} = unsafe {{ {c}.hold(w) }};"), + format!("{local}.str(w)"), + ), + (Type::Slice, Mutability::Not, false) => ( + format!("let {local} = unsafe {{ {c}.hold(w) }};"), + format!("{local}.bytes()"), + ), + (Type::Path, Mutability::Not, false) => ( + format!("let {local} = unsafe {{ {c}.hold(w) }};"), + format!("{local}.path(w)"), + ), // A slice of a block is borrowed in place, as bytes are. (list @ Type::List(Collection::Slice, _), Mutability::Not, false) if list.block().is_some() => { (format!("let {local} = unsafe {{ {c}.items(w) }};"), local.to_string()) @@ -1466,7 +1484,7 @@ impl Gen<'_> { Mutability::Mut => ("let mut", format!("&mut {local}")), Mutability::Not => ("let", format!("&{local}")), }; - (format!("{binding} {local} = {};", self.lift(inner, c)), pass) + (format!("{binding} {local} = {};", self.lift(inner, c, "true")), pass) } }, // `Some` of a borrow, lent as the borrow alone is: a null handle or buffer is `None`. @@ -1485,20 +1503,18 @@ impl Gen<'_> { } (Type::Str | Type::Slice | Type::Path, false) => { let view = match &**inner { - Type::Str => "str", - Type::Slice => "bytes", - _ => "path", + Type::Str => "v.str(w)", + Type::Slice => "v.bytes()", + _ => "v.path(w)", }; ( - format!( - "let {local} = if {c}.is_none() {{ None }} else {{ Some(unsafe {{ {c}.{view}(w) }}) }};" - ), - local.to_string(), + format!("let {local} = if {c}.is_none() {{ None }} else {{ Some(unsafe {{ {c}.hold(w) }}) }};"), + format!("{local}.as_ref().map(|v| {view})"), ) } // A future owns its arguments, so it lends a buffer's owned form. (Type::Str | Type::Slice | Type::Path, true) => ( - format!("let {local} = {};", self.lift(ty, c)), + format!("let {local} = {};", self.lift(ty, c, "true")), format!("{local}.as_deref()"), ), // A lifted `Vec` lends itself or, by coercion to the parameter's type, its slice, @@ -1509,16 +1525,19 @@ impl Gen<'_> { Mutability::Not => ("let", "&"), }; ( - format!("{binding} {local} = {};", self.lift(ty, c)), + format!("{binding} {local} = {};", self.lift(ty, c, "true")), format!("match {borrow} {local} {{ Some(v) => Some(v), None => None }}"), ) } _ => ( - format!("let {local} = {};", self.lift(ty, c)), + format!("let {local} = {};", self.lift(ty, c, "true")), format!("{local}.as_ref()"), ), }, - _ => (format!("let {local} = {};", self.lift(ty, c)), local.to_string()), + _ => ( + format!("let {local} = {};", self.lift(ty, c, "true")), + local.to_string(), + ), } } @@ -1633,7 +1652,11 @@ impl Gen<'_> { args.push(arg); } value => { - stmts.push(format!("let a_self = {};", self.lift(value, "unsafe { *this }"))); + // Read through a pointer, so the caller keeps it. + stmts.push(format!( + "let a_self = {};", + self.lift(value, "unsafe { *this }", "false") + )); args.push(match receiver { Receiver::Owned => "a_self".to_string(), _ => "&a_self".to_string(), @@ -1678,7 +1701,7 @@ impl Gen<'_> { ) } _ if param.interned => ( - format!("let a{n} = rt::intern(unsafe {{ p{n}.str(w) }});"), + format!("let a{n} = rt::intern(unsafe {{ p{n}.hold(w) }}.str(w));"), format!("a{n}"), ), // A future made of a `Poll` owns a future C implements, and lends it pinned to diff --git a/ffi/src/go.rs b/ffi/src/go.rs index 10e1f421d..8c6812512 100644 --- a/ffi/src/go.rs +++ b/ffi/src/go.rs @@ -592,6 +592,8 @@ impl<'a> Gen<'a> { func cBufFree(buf cBuf) {{\n\tC.{p}_buf_free(buf)\n}}\n\n\ // cBufCopy is a copy of s Rust owns, as a reader's error returns it.\n\ func cBufCopy(s string) cBuf {{\n\tif len(s) == 0 {{\n\t\treturn C.{p}_buf_copy(cBuf{{}})\n\t}}\n\treturn C.{p}_buf_copy(cBuf{{ptr: (*C.uint8_t)(unsafe.Pointer(unsafe.StringData(s))), len: C.size_t(len(s))}})\n}}\n\n\ + // cBufClone is a second owner of buf, which Rust returned, sharing its bytes.\n\ + func cBufClone(buf cBuf) cBuf {{\n\treturn C.{p}_buf_clone(buf)\n}}\n\n\ // cBufsFree frees bufs, which are not empty, in one call.\n\ func cBufsFree(bufs []cBuf) {{\n\tC.{go}_bufs_free(&bufs[0], C.size_t(len(bufs)))\n}}\n\n\ func cWakerWake(w cWaker) {{\n\tC.{go}_waker_wake(w)\n}}\n\n\ @@ -747,9 +749,19 @@ impl<'a> Gen<'a> { } } - /// The Go type of a parameter of type `ty`: bytes alone or in an `Option`, which C passes - /// as a buffer, are a `[]byte` either way, read in place when Go passes them and copied - /// when Rust lends them, and anything else as [`Self::go`] has it. + /// The Go type of a parameter of a function Go calls: as [`Self::go`] has it, so bytes + /// Rust returned pass back without a copy. + fn go_arg(&mut self, ty: &Type) -> String { + match ty.buffer() { + Some(Type::Bytes) => self.go(&Type::Bytes), + Some(Type::Option(inner)) if *inner == Type::Bytes => format!("*{}", self.go(&Type::Bytes)), + _ => self.go_param(ty), + } + } + + /// The Go type of a parameter Rust lends a callback or closure: bytes alone or in an + /// `Option`, which C passes as a buffer, are a `[]byte`, copied, and anything else as + /// [`Self::go`] has it. fn go_param(&mut self, ty: &Type) -> String { match ty.buffer() { Some(Type::Bytes) => "[]byte".to_string(), @@ -2323,6 +2335,12 @@ impl<'a> Gen<'a> { } } } + // Past the receiver, which Rust only reads, Rust takes each buffer passed by value. + let take_at = prelude.len(); + // Whether an argument is a parameter's own buffer, and the arguments converted from + // Go values, which may panic, such as on a nil enum. + let mut bufs = false; + let mut lowered = Vec::new(); let mut params = Vec::new(); if function.future { params.push(("ctx".to_string(), "context.Context".to_string())); @@ -2342,10 +2360,11 @@ impl<'a> Gen<'a> { let inner = param.ty.base(); params.push((name.clone(), format!("*{}", self.go(inner)))); arena |= self.arena(inner); + lowered.push(args.len()); args.push(self.lower(inner, &format!("(*{name})"))); continue; } - params.push((name.clone(), self.go_param(¶m.ty))); + params.push((name.clone(), self.go_arg(¶m.ty))); if moved(abi, ¶m.ty) { after.push(self.close(¶m.ty, &name, 0)); } @@ -2375,6 +2394,7 @@ impl<'a> Gen<'a> { } (ty @ Type::Callback(i, _), _) => { self.convert(ty); + lowered.push(args.len()); args.push(format!("lower_{}({name})", abi.callback(*i))) } (Type::Waker, _) => args.push(format!("waker_({name})")), @@ -2389,13 +2409,14 @@ impl<'a> Gen<'a> { (ty, Ty::Buf) => { let call = match ty.buffer() { Some(Type::String | Type::PathBuf) => format!("bufString_({name})"), - Some(Type::Bytes) => format!("bufBytes_({name})"), + Some(Type::Bytes) => format!("bufTake_({name})"), Some(Type::Option(inner)) => match *inner { - Type::Bytes => format!("bufBytesOption_({name})"), + Type::Bytes => format!("bufTakeOption_({name})"), _ => format!("bufStringOption_({name})"), }, other => unreachable!("a buffer holds {other:?}"), }; + bufs = true; args.push(call); } (ty, Ty::Handle { name: c, .. }) => { @@ -2421,6 +2442,7 @@ impl<'a> Gen<'a> { } (ty, _) => { arena |= self.arena(ty); + lowered.push(args.len()); args.push(self.lower(ty, &name)); } } @@ -2429,11 +2451,24 @@ impl<'a> Gen<'a> { let mut body = Vec::new(); if arena { body.push("a_ := &arena_{}\ndefer a_.finish()".to_string()); + prelude.insert(take_at, "a_.take = true".to_string()); } body.extend(prelude); if checked { body.push(format!("if a_.err != nil {{\n\t{}\n}}", fail("a_.err"))); } + // Converted first, as a conversion may panic: then the arena frees Rust's buffers it + // lowered, and no parameter's own buffer is shared yet. + if arena || bufs { + for &i in &lowered { + body.push(format!("l{i}_ := {}", args[i])); + args[i] = format!("l{i}_"); + } + } + // Nothing from here fails before the call, which Rust's buffers given it go to. + if arena { + body.push("a_.made = true".to_string()); + } if !after.is_empty() { match &after[..] { [one] if one.ends_with("Close()") && !one.contains(' ') => body.push(format!("defer {one}")), diff --git a/ffi/src/kotlin.rs b/ffi/src/kotlin.rs index 8a731c103..fd7ed0c72 100644 --- a/ffi/src/kotlin.rs +++ b/ffi/src/kotlin.rs @@ -315,6 +315,15 @@ class Bytes private constructor( \t\treturn destination \t} +\t/** What holds the bytes, kept alive across a call passing them: Kotlin's array or Rust's buffer. */ +\tinternal val held: Any? get() = array ?: lent + +\t/** Rust's owner of the bytes, which a call passing them shares without a copy, or 0 for Kotlin's. */ +\tinternal val owner: Long get() = lent?.owner ?: 0L + +\t/** Where the bytes start in what holds them. */ +\tinternal val start: Int get() = offset + \t/** The bytes, copied. */ \tfun toByteArray(): ByteArray = array?.copyOfRange(offset, offset + size) ?: copyInto(ByteArray(size)) @@ -561,26 +570,6 @@ impl<'a> Gen<'a> { } } - /// The Kotlin parameter value of the value `e` of type `ty` read back: bytes, which a - /// parameter takes as a `ByteArray`, copied out of each `Bytes` read, wherever they are. - fn to_param(&self, ty: &Type, e: &str) -> String { - match ty { - _ if !holds_bytes(ty) => e.to_string(), - Type::Slice | Type::Vec | Type::Bytes => format!("{e}.toByteArray()"), - Type::Box(inner) | Type::Ref(_, inner) => self.to_param(inner, e), - Type::Option(inner) => format!("{e}?.let {{ v -> {} }}", self.to_param(inner, "v")), - Type::List(_, inner) => format!("{e}.map {{ v -> {} }}", self.to_param(inner, "v")), - Type::Tuple(elements) => { - let converted: Vec = (elements.iter().enumerate()) - .map(|(k, ty)| self.to_param(ty, &format!("t.{}", ordinal(k)))) - .collect(); - let make = self.tuple_name(elements); - format!("{e}.let {{ t -> {make}({}) }}", converted.join(", ")) - } - _ => e.to_string(), - } - } - /// [`Gen::tuple_name`], declaring the `TupleN` it names. fn tuple_class(&mut self, elements: &[Type]) -> String { let name = self.tuple_name(elements); @@ -637,11 +626,10 @@ impl<'a> Gen<'a> { } } - /// The Kotlin type of a parameter of type `ty`, either way: its bytes, alone or in an - /// `Option` or a list, are a `ByteArray`, copied. + /// The Kotlin type of a parameter of type `ty`, either way: one Rust writes back is a + /// `Mut`. fn kt_param(&mut self, ty: &Type) -> String { match ty { - Type::Slice | Type::Vec | Type::Bytes => "ByteArray".to_string(), Type::List(..) if let Some(inner) = ty.block() => block(inner).kt.to_string(), Type::List(_, inner) => format!("kotlin.collections.List<{}>", self.kt_param(inner)), Type::Option(inner) => format!("{}?", self.kt_param(inner)), @@ -737,7 +725,7 @@ impl<'a> Gen<'a> { Type::Str | Type::String => format!("{r}.string()!!"), Type::Path | Type::PathBuf => format!("{r}.path()!!"), Type::Slice | Type::Vec | Type::Bytes if owned => format!("{r}.owned()!!"), - Type::Slice | Type::Vec | Type::Bytes => format!("{r}.bytes()!!"), + Type::Slice | Type::Vec | Type::Bytes => format!("{r}.lent()!!"), Type::U128 => format!("{r}.u128()"), Type::Array(n) => format!("{}({r}.raw({n}))", Abi::array_class(*n)), Type::Duration => format!("{r}.duration()"), @@ -769,7 +757,7 @@ impl<'a> Gen<'a> { ) } base if Abi::is_buf(base) && owned => format!("{r}.owned()"), - base if Abi::is_buf(base) => format!("{r}.bytes()"), + base if Abi::is_buf(base) => format!("{r}.lent()"), _ => format!("(if ({r}.bool()) {} else null)", self.read(inner, r, owned)), }, Type::Box(inner) | Type::Ref(_, inner) => self.read(inner, r, owned), @@ -813,8 +801,8 @@ impl<'a> Gen<'a> { Ty::Buf => match (decode(ty), matches!(ty.base(), Type::Option(_)), lent) { (Some(decode), true, _) => format!("{e}?.{decode}"), (Some(decode), false, _) => format!("{e}!!.{decode}"), - (None, true, true) => e.to_string(), - (None, false, true) => format!("{e}!!"), + (None, true, true) => format!("{e}?.let {{ Bytes(it) }}"), + (None, false, true) => format!("Bytes({e}!!)"), (None, true, false) => format!("Native.owned({e})"), (None, false, false) => format!("Native.owned({e})!!"), }, @@ -904,15 +892,42 @@ impl<'a> Gen<'a> { arg.exprs.push(h); arg.rust_args.push(format!("{p} as usize as _")); } + // Text crosses as the `String`, encoded once into a buffer Rust takes. + Ty::Buf if string(ty) => { + *arg.jni.last_mut().expect("pushed above") = "jstring"; + *arg.native.last_mut().expect("pushed above") = "String?".to_string(); + arg.exprs.push(e.to_string()); + arg.rust_pre.push(format!("let b{p} = rt::text(env, {p});")); + arg.rust_args.push(format!("b{p}")); + } + // Bytes cross as what holds them, Rust's owner or 0, where they start, and how + // many, or -1 for none: Rust takes a copy of Kotlin's, or shares its own. Ty::Buf => { - let option = matches!(ty.base(), Type::Option(_)); - arg.exprs.push(match (string(ty), option) { - (true, true) => format!("{e}?.encodeToByteArray()"), - (true, false) => format!("{e}.encodeToByteArray()"), - (false, _) => e.to_string(), + arg.jni + .splice(arg.jni.len() - 1.., ["jobject", "jlong", "jint", "jint"]); + arg.native.splice( + arg.native.len() - 1.., + ["Any?", "Long", "Int", "Int"].map(str::to_string), + ); + let (q, r, s) = (format!("p{n}"), format!("p{}", *n + 1), format!("p{}", *n + 2)); + *n += 3; + arg.exprs.extend(match matches!(ty.base(), Type::Option(_)) { + true => [ + format!("{e}?.held"), + format!("{e}?.owner ?: 0L"), + format!("{e}?.start ?: 0"), + format!("{e}?.size ?: -1"), + ], + false => [ + format!("{e}.held"), + format!("{e}.owner"), + format!("{e}.start"), + format!("{e}.size"), + ], }); - arg.rust_pre.push(format!("let b{p} = rt::bytes(env, {p});")); - arg.rust_args.push(format!("rt::view(&b{p})")); + arg.rust_pre + .push(format!("let b{p} = unsafe {{ rt::passed(env, {p}, {q}, {r}, {s}) }};")); + arg.rust_args.push(format!("b{p}")); } Ty::Struct(name) if matches!(ty.base(), Type::Callback(..)) => { arg.exprs.push(e.to_string()); @@ -924,7 +939,7 @@ impl<'a> Gen<'a> { // Only while the call's JVM parameters stay within the 255 slots a method takes. Ty::Struct(_) | Ty::In(_) if let Type::Record(i) = ty.base() - && let Some(left) = self.room.checked_sub(weight(abi, *i).saturating_sub(1)) => + && let Some(left) = self.room.checked_sub(weight(abi, *i).saturating_sub(c_slots(c))) => { self.room = left; arg.jni.clear(); @@ -934,15 +949,23 @@ impl<'a> Gen<'a> { let mut writes = Vec::new(); let value = self.flat_in(*i, e, (&mut arg, n), (&w, &mut writes), &r); if rest(abi, *i) { - let b = format!("p{n}"); - *n += 1; - arg.jni.push("jbyteArray"); - arg.native.push("ByteArray".to_string()); + // Then the arrays and text the rest names, which Rust copies from once. + let (b, q) = (format!("p{n}"), format!("p{}", *n + 1)); + *n += 2; + arg.jni.extend(["jbyteArray", "jobjectArray"]); + arg.native + .extend(["ByteArray", "kotlin.Array?"].map(str::to_string)); arg.open = Some(format!("Native.Writer().use {{ {w} ->")); arg.inside.extend(writes); arg.exprs.push(format!("{w}.finish()")); + arg.exprs.push(format!("{w}.refs()")); arg.rust_pre.push(format!( - "let b{p} = rt::bytes(env, {b}).unwrap_or_default();\n\t\tlet mut {r} = rt::Reader::new(&b{p});" + "let b{p} = rt::bytes(env, {b}).unwrap_or_default();\n\t\tlet mut {r} = rt::Reader::{}(env, &b{p}, {q});", + // Read through a pointer, Rust keeps nothing of it. + match c { + Ty::In(_) => "new", + _ => "taking", + } )); } arg.rust_pre.push(format!("let a{p} = {value};")); @@ -961,9 +984,20 @@ impl<'a> Gen<'a> { arg.open = Some(format!("Native.Writer().use {{ {w} ->")); let value = self.write_value(ty, &w, e); arg.inside.push(value); + // Then the arrays and text it names, which Rust copies from once. + arg.jni.push("jobjectArray"); + arg.native.push("kotlin.Array?".to_string()); + let q = format!("p{n}"); + *n += 1; arg.exprs.push(format!("{w}.finish()")); + arg.exprs.push(format!("{w}.refs()")); arg.rust_pre.push(format!( - "let b{p} = rt::bytes(env, {p}).unwrap_or_default();\n\t\tlet mut r{p} = rt::Reader::new(&b{p});\n\t\tlet a{p} = r{p}.get();" + "let b{p} = rt::bytes(env, {p}).unwrap_or_default();\n\t\tlet mut r{p} = rt::Reader::{}(env, &b{p}, {q});\n\t\tlet a{p} = r{p}.get();", + // Read through a pointer, Rust keeps nothing of it. + match c { + Ty::In(_) => "new", + _ => "taking", + } )); arg.rust_args.push(match c { Ty::In(_) => format!("&a{p}"), @@ -1173,10 +1207,9 @@ impl<'a> Gen<'a> { let bare = self.bare(c).to_string(); let object = matches!(ty, Type::Object(_)); // The receiver as the glue takes it: a handle, held for the call, or encoded. - let (jni_self, native_self, open, arg, rust_self) = match object { + let (params, open, arg, rust_self): (&[(&str, &str)], _, _, _) = match object { true => ( - "jlong", - "Long", + &[("jlong", "Long")], "handle.hold { h ->".to_string(), "h".to_string(), "p0 as usize as _".to_string(), @@ -1184,18 +1217,17 @@ impl<'a> Gen<'a> { false => { let write = self.write_value(ty, "w", "this"); ( - "jbyteArray", - "ByteArray", + &[("jbyteArray", "ByteArray"), ("jobjectArray", "kotlin.Array?")], format!("Native.Writer().use {{ w ->\n\t\t{write}"), - "w.finish()".to_string(), - "&rt::Reader::new(&rt::bytes(env, p0).unwrap_or_default()).get()".to_string(), + "w.finish(), w.refs()".to_string(), + "&rt::Reader::new(env, &rt::bytes(env, p0).unwrap_or_default(), p1).get()".to_string(), ) } }; let unary = |g: &mut Self, suffix: &str, sig: &str| { g.native( &format!("{bare}_{suffix}"), - &[(jni_self, native_self)], + params, ("jbyteArray", "ByteArray?"), &format!("rt::owned(env, crate::{c}_{suffix}({rust_self}))"), ); @@ -2332,13 +2364,11 @@ impl<'a> Gen<'a> { let factory = format!("{bare}__return"); self.dispatchers.push(factory.clone()); let mut lines = Vec::new(); + // What Rust wrote back comes first, read whether or not the call failed. for (k, (_, ty)) in written.iter().enumerate() { let read = self.read_value(ty, "r"); let kt = self.kt_param(ty); - lines.push(format!( - "(back!![{k}] as Mut<{kt}>).value = {}", - self.to_param(ty, &read) - )); + lines.push(format!("(back!![{k}] as Mut<{kt}>).value = {read}")); } lines.extend(self.lift_result(&ok, err.as_ref(), exception.as_deref(), "r")); if ok == Type::Unit { @@ -2786,16 +2816,17 @@ fn rest(abi: &Abi<'_>, i: usize) -> bool { } /// The JVM parameter slots the record `i` takes flattened: two for each `long` or `double` -/// scalar, one for any other, and one for the `byte[]` of its other fields, if any. +/// scalar, one for any other, and two for its other fields, if any: their `byte[]` and the +/// arrays and text they name. fn weight(abi: &Abi<'_>, i: usize) -> usize { let fields = abi.api.records[i].fields.iter().map(|f| match &f.ty { - Type::Record(j) => weight(abi, *j) - usize::from(rest(abi, *j)), + Type::Record(j) => weight(abi, *j) - 2 * usize::from(rest(abi, *j)), Type::F64 => 2, Type::Int(int) | Type::NonZero(int) if jni_int(*int) == "jlong" => 2, ty if scalar(abi, ty) => 1, _ => 0, }); - fields.sum::() + usize::from(rest(abi, i)) + fields.sum::() + 2 * usize::from(rest(abi, i)) } /// The JVM slots of the 255 a method takes left past what each C parameter takes @@ -2806,13 +2837,17 @@ fn room<'a>(params: impl Iterator, future: bool, written: bool) - 255usize.saturating_sub(unflattened + usize::from(future) + usize::from(written)) } -/// The JVM parameter slots the C parameter `c` takes as a native method's: two for a -/// `long` or `double`, none for what never crosses, and one for anything else. +/// The JVM parameter slots the C parameter `c` takes as a native method's at most: two for +/// a `long` or `double`, none for what never crosses, five for bytes (what holds them, a +/// `long` owner, and two `int`s; text takes one), two for an encoded value's `byte[]` and +/// the arrays and text it names (a callback takes one), and one for anything else. fn c_slots(c: &Ty) -> usize { match c { Ty::Void | Ty::Out(_) | Ty::Data | Ty::Poll => 0, Ty::Int(int) if jni_int(*int) == "jlong" => 2, Ty::F64 | Ty::Handle { .. } | Ty::Future(_) => 2, + Ty::Buf => 5, + Ty::U128 | Ty::Duration | Ty::SocketAddr | Ty::Struct(_) | Ty::In(_) => 2, _ => 1, } } @@ -2889,16 +2924,6 @@ fn block(elem: &Type) -> Block { } } -/// Whether `ty` holds bytes anywhere a parameter's form of it differs from a returned one's. -fn holds_bytes(ty: &Type) -> bool { - match ty { - Type::Slice | Type::Vec | Type::Bytes => true, - Type::Option(inner) | Type::Box(inner) | Type::Ref(_, inner) | Type::List(_, inner) => holds_bytes(inner), - Type::Tuple(elements) => elements.iter().any(holds_bytes), - _ => false, - } -} - /// The glue's statement writing the C value `e` of IR type `ty`, which Rust returns: its /// bytes cross as their owner. fn put(ty: &Type, e: &str) -> String { diff --git a/ffi/src/python.rs b/ffi/src/python.rs index 8d00cde6a..6b1725b8f 100644 --- a/ffi/src/python.rs +++ b/ffi/src/python.rs @@ -538,8 +538,10 @@ impl<'a> Gen<'a> { /// The glue's expression, an `R`, reading the Python object `o` as the C value of /// `ty`, with what it borrows held in the scope `s`; `m` is the glue's `bool`, whether - /// the position moves an object's value out, closing it. - fn lift(&mut self, ty: &Type, o: &str, m: &str) -> String { + /// the position moves an object's value out, closing it, and `t`, whether Rust takes + /// what the position holds, as it takes a value passed by value but not one read + /// through a pointer or an array's item, so a buffer crosses there as its own. + fn lift(&mut self, ty: &Type, o: &str, m: &str, t: &str) -> String { let abi = self.abi; match ty { Type::Const(_) => unreachable!("a const argument is no value"), @@ -553,13 +555,13 @@ impl<'a> Gen<'a> { Type::F64 => format!("rt::float_in({o})"), Type::Char => format!("rt::char_in({o})"), Type::Str | Type::String => format!("rt::str_in({o})"), - Type::Slice | Type::Vec | Type::Bytes => format!("rt::bytes_in({o}, s)"), + Type::Slice | Type::Vec | Type::Bytes => format!("rt::bytes_in({o}, s, {t})"), Type::Path | Type::PathBuf => format!("rt::path_in({o}, s)"), Type::U128 => format!("rt::u128_in({o})"), Type::Duration => format!("rt::duration_in({o})"), Type::SocketAddr(family) => format!("rt::addr_in({o}, {})", family_of(*family)), Type::List(_, inner) if ty.block().is_some() => format!("rt::block_in::<{}>({o}, s)", number(inner)), - Type::List(..) => format!("r_{}({o}, s, {m})", self.converter(ty)), + Type::List(..) => format!("r_{}({o}, s, {m}, {t})", self.converter(ty)), Type::Option(inner) => match inner.base() { Type::Object(i) => { let m = match abi.api.objects[*i].copy || matches!(**inner, Type::Ref(..)) { @@ -569,29 +571,29 @@ impl<'a> Gen<'a> { format!("rt::hold_opt({o}, {}, s, {m})", self.t(&abi.object(*i))) } base if Abi::is_buf(base) => { - let some = self.lift(inner, o, m); + let some = self.lift(inner, o, m, t); format!("(if rt::is_none({o}) {{ Ok(crate::rt::Buf::NONE) }} else {{ {some} }})") } Type::List(..) => { - let some = self.lift(inner, o, m); + let some = self.lift(inner, o, m, t); format!("(if rt::is_none({o}) {{ Ok(crate::rt::List::NONE) }} else {{ {some} }})") } - _ => format!("r_{}({o}, s, {m})", self.converter(ty)), + _ => format!("r_{}({o}, s, {m}, {t})", self.converter(ty)), }, - Type::Box(inner) => self.lift(inner, o, m), - Type::Ref(_, inner) => self.lift(inner, o, "false"), - Type::Record(_) => format!("r_{}({o}, s)", self.c_struct(ty)), + Type::Box(inner) => self.lift(inner, o, m, t), + Type::Ref(_, inner) => self.lift(inner, o, "false", t), + Type::Record(_) => format!("r_{}({o}, s, {t})", self.c_struct(ty)), Type::Enum(i) if abi.api.enums[*i].variants.iter().any(|v| !abi.api.everywhere(v.on)) => { // Checks the member a value is against the library's target. let c = abi.enumeration(*i); match abi.api.enums[*i].plain() { true => format!("rt::member_in({o}, {}).and_then(on_{c})", self.t(&c)), - false => format!("r_{c}({o}, s)"), + false => format!("r_{c}({o}, s, {t})"), } } Type::Enum(i) => match abi.api.enums[*i].plain() { true => format!("rt::member_in({o}, {})", self.t(&abi.enumeration(*i))), - false => format!("r_{}({o}, s)", abi.enumeration(*i)), + false => format!("r_{}({o}, s, {t})", abi.enumeration(*i)), }, Type::Object(i) => { let m = match abi.api.objects[*i].copy { @@ -612,7 +614,7 @@ impl<'a> Gen<'a> { Type::Future => format!("rt::future_in({o}, s)"), Type::Range(int) => format!("rt::range_in::<{}>({o})", int.rust()), Type::Bound(_) | Type::Bounds(_) => format!("r_{}({o}, s)", abi.range(ty)), - Type::Tuple(_) => format!("r_{}({o}, s, {m})", self.converter(ty)), + Type::Tuple(_) => format!("r_{}({o}, s, {m}, {t})", self.converter(ty)), Type::Unit | Type::Closure(..) => unreachable!("{ty:?} is no value Python passes"), } } @@ -859,11 +861,11 @@ impl<'a> Gen<'a> { /// The glue of an option struct `c` holding `inner`, as `r_` and `w_`. fn option(&mut self, name: &str, c: &str, inner: &Type) { - let read = self.lift(inner, "o", "m"); + let read = self.lift(inner, "o", "m", "take"); let write = self.lower(inner, "v.value"); let _ = write!( self.glue, - "fn r_{name}(o: *mut PyObject, s: &mut rt::Scope, m: bool) -> R {{\n\tmatch rt::is_none(o) {{\n\t\ttrue => Ok(crate::{c} {{ some: false, value: unsafe {{ std::mem::zeroed() }} }}),\n\t\tfalse => Ok(crate::{c} {{ some: true, value: {}? }}),\n\t}}\n}}\n\nfn w_{name}(v: crate::{c}) -> R {{\n\tmatch v.some {{\n\t\ttrue => {write},\n\t\tfalse => rt::none(),\n\t}}\n}}\n\n", + "fn r_{name}(o: *mut PyObject, s: &mut rt::Scope, m: bool, take: bool) -> R {{\n\tmatch rt::is_none(o) {{\n\t\ttrue => Ok(crate::{c} {{ some: false, value: unsafe {{ std::mem::zeroed() }} }}),\n\t\tfalse => Ok(crate::{c} {{ some: true, value: {}? }}),\n\t}}\n}}\n\nfn w_{name}(v: crate::{c}) -> R {{\n\tmatch v.some {{\n\t\ttrue => {write},\n\t\tfalse => rt::none(),\n\t}}\n}}\n\n", read ); } @@ -875,14 +877,14 @@ impl<'a> Gen<'a> { let mut fields = Vec::new(); let mut writes = Vec::new(); for (k, ty) in elements.iter().enumerate() { - let lift = self.lift(ty, &format!("x{k}"), "m"); + let lift = self.lift(ty, &format!("x{k}"), "m", "take"); reads.push(format!("let x{k} = rt::tuple_item(o, {k}, {n})?;")); fields.push(format!("_{k}: {lift}?")); writes.push(self.lower(ty, &format!("v._{k}"))); } let _ = write!( self.glue, - "fn r_{name}(o: *mut PyObject, s: &mut rt::Scope, m: bool) -> R {{\n\t{}\n\tOk(crate::{c} {{ {} }})\n}}\n\nfn w_{name}(v: crate::{c}) -> R {{\n\trt::tuple(vec![{}])\n}}\n\n", + "fn r_{name}(o: *mut PyObject, s: &mut rt::Scope, m: bool, take: bool) -> R {{\n\t{}\n\tOk(crate::{c} {{ {} }})\n}}\n\nfn w_{name}(v: crate::{c}) -> R {{\n\trt::tuple(vec![{}])\n}}\n\n", reads.join("\n\t"), fields.join(", "), writes.join(", ") @@ -891,11 +893,12 @@ impl<'a> Gen<'a> { /// The glue of a `Vec`'s array `c`, as `r_` and `w_`. fn list(&mut self, name: &str, c: &str, elem: &Type) { - let read = self.lift(elem, "o", "m"); + // An array's items are behind its pointer, which Rust only reads. + let read = self.lift(elem, "o", "m", "false"); let write = self.lower(elem, "e"); let _ = write!( self.glue, - "fn r_{name}(o: *mut PyObject, s: &mut rt::Scope, m: bool) -> R {{\n\trt::list_in(o, s, |o, s| {read})\n}}\n\nfn w_{name}(v: crate::{c}) -> R {{\n\trt::list_out(v, |e| {write})\n}}\n\n" + "fn r_{name}(o: *mut PyObject, s: &mut rt::Scope, m: bool, take: bool) -> R {{\n\trt::list_in(o, s, |o, s| {read})\n}}\n\nfn w_{name}(v: crate::{c}) -> R {{\n\trt::list_out(v, |e| {write})\n}}\n\n" ); } @@ -912,8 +915,9 @@ impl<'a> Gen<'a> { &Type::Ref(crate::ir::Mutability::Not, Box::new(ty.clone())), "a[0]", "false", + "false", ), - false => self.lift(ty, "a[0]", "false"), + false => self.lift(ty, "a[0]", "false", "false"), }; let pass = match object { true => "this", @@ -957,6 +961,7 @@ impl<'a> Gen<'a> { &Type::Ref(crate::ir::Mutability::Not, Box::new(ty.clone())), "a[1]", "false", + "false", ); self.native( &format!("{bare}_eq"), @@ -1091,7 +1096,10 @@ impl<'a> Gen<'a> { _ => format!("rt::attr(o, {key}, s)?"), }; reads.push(format!("let x{n} = {read};")); - fields.push(format!("f{n}: {}?", self.lift(&field.ty, &format!("x{n}"), "false"))); + fields.push(format!( + "f{n}: {}?", + self.lift(&field.ty, &format!("x{n}"), "false", "take") + )); writes.push(format!("({key}, {})", self.lower(&field.ty, &format!("v.f{n}")))); } if record.fields.is_empty() { @@ -1099,7 +1107,7 @@ impl<'a> Gen<'a> { } let _ = write!( self.glue, - "fn r_{c}(o: *mut PyObject, s: &mut rt::Scope) -> R {{\n\t{}\n\tOk(crate::{c} {{ {} }})\n}}\n\nfn w_{c}(v: crate::{c}) -> R {{\n\trt::record({t}, vec![{}])\n}}\n\n", + "fn r_{c}(o: *mut PyObject, s: &mut rt::Scope, take: bool) -> R {{\n\t{}\n\tOk(crate::{c} {{ {} }})\n}}\n\nfn w_{c}(v: crate::{c}) -> R {{\n\trt::record({t}, vec![{}])\n}}\n\n", reads.join("\n\t"), fields.join(", "), writes.join(", ") @@ -1224,7 +1232,10 @@ impl<'a> Gen<'a> { for (k, f) in variant.fields.iter().enumerate() { let key = self.key(&field_name(&f.name)); binds.push(format!("let x{k} = rt::attr(o, {key}, s)?;")); - fields.push(format!("f{k}: {}?", self.lift(&f.ty, &format!("x{k}"), "false"))); + fields.push(format!( + "f{k}: {}?", + self.lift(&f.ty, &format!("x{k}"), "false", "take") + )); lowers.push(format!("({key}, {})", self.lower(&f.ty, &format!("p.f{k}")))); } reads.push(format!( @@ -1241,7 +1252,7 @@ impl<'a> Gen<'a> { } let _ = write!( self.glue, - "fn r_{c}(o: *mut PyObject, s: &mut rt::Scope) -> R {{\n\t{}\n\trt::check(o, {t})?;\n\tcrate::rt::fail(\"python\", \"a variant is of no class of its enum\")\n}}\n\nfn w_{c}(v: crate::{c}) -> R {{\n\tmatch v.tag {{\n{}\n\t\t_ => crate::rt::fail(\"python\", \"an enum's tag is invalid\"),\n\t}}\n}}\n\n", + "fn r_{c}(o: *mut PyObject, s: &mut rt::Scope, take: bool) -> R {{\n\t{}\n\trt::check(o, {t})?;\n\tcrate::rt::fail(\"python\", \"a variant is of no class of its enum\")\n}}\n\nfn w_{c}(v: crate::{c}) -> R {{\n\tmatch v.tag {{\n{}\n\t\t_ => crate::rt::fail(\"python\", \"an enum's tag is invalid\"),\n\t}}\n}}\n\n", reads.join("\n\t"), writes.join("\n") ); @@ -1430,7 +1441,7 @@ impl<'a> Gen<'a> { let some = self.given(inner, o); format!("(if rt::is_none({o}) {{ Ok(std::ptr::null()) }} else {{ {some} }})") } - _ => self.lift(ty, o, "false"), + _ => self.lift(ty, o, "false", "false"), } } @@ -1619,11 +1630,18 @@ impl<'a> Gen<'a> { Receiver::Ref => Type::Ref(crate::ir::Mutability::Not, Box::new(ty)), Receiver::Mut => Type::Ref(crate::ir::Mutability::Mut, Box::new(ty)), }; - stmts.push(format!("let p{n} = {}?;", self.lift(&ty, &format!("a[{n}]"), "true"))); + stmts.push(format!( + "let p{n} = {}?;", + self.lift(&ty, &format!("a[{n}]"), "true", "true") + )); c_args.push(format!("p{n}")); } _ => { - stmts.push(format!("let p{n} = {}?;", self.lift(&ty, &format!("a[{n}]"), "false"))); + // Read through a pointer, so Rust takes nothing of it. + stmts.push(format!( + "let p{n} = {}?;", + self.lift(&ty, &format!("a[{n}]"), "false", "false") + )); c_args.push(format!("&p{n}")); } } @@ -1646,7 +1664,7 @@ impl<'a> Gen<'a> { (Type::Ref(_, inner), _) if param.ty.written() => { stmts.push(format!( "let h{n} = rt::mut_value(a[{n}], s)?;\n\t\tlet p{n} = {}?;", - self.lift(inner, &format!("h{n}"), "true") + self.lift(inner, &format!("h{n}"), "true", "true") )); c_args.push(format!("p{n}")); written.push((n, (**inner).clone())); @@ -1674,7 +1692,7 @@ impl<'a> Gen<'a> { _ => { stmts.push(format!( "let p{n} = {}?;", - self.lift(¶m.ty, &format!("a[{n}]"), "true") + self.lift(¶m.ty, &format!("a[{n}]"), "true", "true") )); c_args.push(format!("p{n}")); } diff --git a/ffi/src/swift.rs b/ffi/src/swift.rs index c43c1a973..4bd5ba40c 100644 --- a/ffi/src/swift.rs +++ b/ffi/src/swift.rs @@ -16,9 +16,10 @@ //! [`runtime/rt.swift`](../runtime/rt.swift), holds the classes' handles, the arena a call //! lowers nested values into, and the signal a future parks on. //! -//! A call passes bytes and text at its top level as views of Swift's own memory, and -//! returned bytes are `Data` lending Rust's buffer, so neither is copied; nested in another -//! value, bytes passed in are copied into the call's arena. +//! Bytes are the runtime's `Bytes`: those Rust returned lend its buffer, and pass back to +//! Rust as a second owner of it, so neither way copies them. Swift's own bytes and text +//! pass as views of Swift's memory at a call's top level, which Rust copies once where it +//! keeps them; nested in another value, they are copied once into a buffer Rust takes. use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::fmt::Write; @@ -94,6 +95,8 @@ pub fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result Buf {{\n\t\t{p}_buf_copy(view)\n\t}}\n\n\ + \t/// A second owner of `buf`, which Rust returned, sharing its bytes.\n\ + \tstatic func clone(_ buf: Buf) -> Buf {{\n\t\t{p}_buf_clone(buf)\n\t}}\n\n\ \tstatic func load(_ state: UnsafeMutablePointer) -> Swift.Int {{\n\t\t{p}_atomic_load(state)\n\t}}\n\n\ \tstatic func store(_ state: UnsafeMutablePointer, _ value: Swift.Int) {{\n\t\t{p}_atomic_store(state, value)\n\t}}\n\n\ \tstatic func add(_ state: UnsafeMutablePointer, _ value: Swift.Int) -> Swift.Int {{\n\t\t{p}_atomic_add(state, value)\n\t}}\n\n\ @@ -135,11 +138,12 @@ pub fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result_native` and `_socket_addr`. +/// runtime's own are the C ABI's: `Native`, `Bytes`, and `SocketAddr` are claimed as +/// `_native`, `_bytes`, and `_socket_addr`. const TAKEN: &[&str] = &[ "Array", "Bool", + "Bytes", "CancellationError", "Comparable", "CustomDebugStringConvertible", @@ -152,6 +156,7 @@ const TAKEN: &[&str] = &[ "Never", "OpaquePointer", "Optional", + "RandomAccessCollection", "Sendable", "SeekOrigin", "SeekableReader", @@ -175,6 +180,7 @@ const TAKEN: &[&str] = &[ "UnsafeMutableRawPointer", "UnsafePointer", "UnsafeRawBufferPointer", + "UnsafeRawPointer", "UniqueArray", "Void", "fatalError", @@ -321,7 +327,7 @@ impl<'a> Gen<'a> { Type::F64 => "Swift.Double".to_string(), Type::Char => "Swift.Unicode.Scalar".to_string(), Type::Str | Type::String | Type::Path | Type::PathBuf => "Swift.String".to_string(), - Type::Slice | Type::Vec | Type::Bytes => "Data".to_string(), + Type::Slice | Type::Vec | Type::Bytes => "Bytes".to_string(), Type::U128 => "U128".to_string(), Type::Duration => "Swift.Duration".to_string(), Type::SocketAddr(_) => "SocketAddr".to_string(), @@ -508,7 +514,7 @@ impl<'a> Gen<'a> { match ty { Type::Str | Type::String => format!("Native.string({e})"), Type::Path | Type::PathBuf => format!("Native.path({e})"), - Type::Slice | Type::Vec | Type::Bytes => format!("Native.data({e})"), + Type::Slice | Type::Vec | Type::Bytes => format!("Native.bytes({e})"), Type::Bool | Type::F32 | Type::F64 => e.to_string(), Type::Int(int) | Type::NonZero(int) => lift_int(*int, e), Type::Char => format!("Native.scalar({e})"), @@ -543,7 +549,7 @@ impl<'a> Gen<'a> { Type::Object(_) => format!("{e}.map {{ {}(raw: $0) }}", self.sw(inner)), Type::Str | Type::String => format!("Native.string(option: {e})"), Type::Path | Type::PathBuf => format!("Native.path(option: {e})"), - Type::Slice | Type::Vec | Type::Bytes => format!("Native.data(option: {e})"), + Type::Slice | Type::Vec | Type::Bytes => format!("Native.bytes(option: {e})"), Type::List(..) => format!( "({e}.ptr == nil ? nil : {} as {}?)", self.lift(inner, e), @@ -581,11 +587,11 @@ impl<'a> Gen<'a> { match ty { Type::Str | Type::String => format!("Native.string(lent: {e})"), Type::Path | Type::PathBuf => format!("Native.path(lent: {e})"), - Type::Slice | Type::Vec | Type::Bytes => format!("Native.data(lent: {e})"), + Type::Slice | Type::Vec | Type::Bytes => format!("Native.bytes(lent: {e})"), Type::Option(inner) => match inner.base() { Type::Str | Type::String => format!("Native.string(lentOption: {e})"), Type::Path | Type::PathBuf => format!("Native.path(lentOption: {e})"), - Type::Slice | Type::Vec | Type::Bytes => format!("Native.data(lentOption: {e})"), + Type::Slice | Type::Vec | Type::Bytes => format!("Native.bytes(lentOption: {e})"), _ => self.lift(ty, e), }, Type::Ref(_, inner) => self.lift_lent(inner, e), @@ -1233,7 +1239,7 @@ impl<'a> Gen<'a> { (Type::Future, _) => arg.exprs.push(format!("Native.future({e})")), (ty, Ty::Buf) if Abi::is_buf(ty) || matches!(ty.base(), Type::Option(inner) if Abi::is_buf(inner)) => { let b = format!("b{k}_"); - arg.open = Some((format!("{tr}Native.view({e}) {{ {b} in"), "}".to_string())); + arg.open = Some((format!("{tr}Native.take({e}) {{ {b} in"), "}".to_string())); arg.exprs.push(b); } // A struct the call borrows or consumes, so it lives until the call returns. @@ -1550,6 +1556,8 @@ impl<'a> Gen<'a> { 1.min(prelude.len()), "let a_ = Native.Arena()\ndefer { a_.finish() }".to_string(), ); + // Past the receiver, which Rust only reads, Rust takes each value passed. + prelude.push("a_.take = true".to_string()); } let after: Vec = args.iter().flat_map(|a| a.after.clone()).collect(); if !after.is_empty() { diff --git a/ffi/tests/api.c b/ffi/tests/api.c index 1c9632dce..60fdda00a 100644 --- a/ffi/tests/api.c +++ b/ffi/tests/api.c @@ -364,6 +364,40 @@ static bool broken_seek(void *data, int whence, int64_t offset, uint64_t *positi return true; } +/* The payload sent last, ready at the first poll. */ +static api_buf received(api_Channel *channel) { + api_Channel_recv_future *recv = api_Channel_recv(channel); + api_buf payload = {0}; + bool ready = api_Channel_recv_poll(recv, (api_waker){0}, &payload) == API_POLL_READY; + api_Channel_recv_future_free(recv); + return ready ? payload : (api_buf){0}; +} + +/* A payload Rust returned goes back to Rust in place: taken whole, taken narrowed to a + * range of it, shared by a second owner while kept, and carried in a record. Its bytes + * stay where Rust put them, and drop once, with its last owner. */ +static bool forwarding(void) { + uint32_t before = api_dropped(); + api_Channel *channel = api_Channel_new(); + api_buf payload = api_token_bytes(3, 64); + const uint8_t *at = payload.ptr; + api_Channel_send(channel, payload); + payload = received(channel); + bool same = payload.ptr == at && payload.len == 64; + api_Channel_send(channel, (api_buf){.ptr = payload.ptr + 1, .len = 62, .owner = payload.owner}); + payload = received(channel); + same = same && payload.ptr == at + 1 && payload.len == 62; + api_Channel_send(channel, api_buf_clone(payload)); + api_buf shared = received(channel); + same = same && shared.ptr == payload.ptr && shared.len == 62; + api_buf_free(payload); + api_Packet packet = api_next_packet((api_Packet){.seq = 1, .payload = shared}); + same = same && packet.payload.ptr == at + 1 && api_dropped() == before; + api_Packet_free(&packet); + api_Channel_free(channel); + return same && api_dropped() == before + 1; +} + static void tail(api_foreign_reader reader) { api_buf bytes; uint64_t at; @@ -1326,7 +1360,8 @@ static int script(void) { /* A path's bytes as the OS holds them, which need not be UTF-8. */ api_buf stray = api_stray(); - api_buf in_stray = api_file(stray, str("f")); + /* A second owner of the buffer goes to Rust, which takes it, so stray stays ours. */ + api_buf in_stray = api_file(api_buf_clone(stray), str("f")); if (in_stray.len != 3 || memcmp(in_stray.ptr, "\xff/f", 3) != 0) { return 1; } @@ -1379,12 +1414,11 @@ static int script(void) { api_Counter_free(two); api_Counter_free(three); - /* A value passed for &mut: Rust writes back what the call left, which the caller frees, - * and keeps what it passed. */ + /* A value passed for &mut: Rust writes back what the call left, which the caller frees. + * A view passed stays the caller's, and a buffer Rust returned Rust takes back. */ api_buf encoded, appended; api_put_number(1, str("0,"), &encoded); api_put_number(23, encoded, &appended); - api_buf_free(encoded); print("put_number ", appended, "\n"); api_tuple_bytes_u32 stamped; uint32_t stamps = api_push_counted((api_tuple_bytes_u32){._0 = str("ab"), ._1 = 1}, 'c', &stamped); @@ -1395,11 +1429,9 @@ static int script(void) { uint8_t first_digit, last_digit; api_Error digit_error; api_digit(str("7x9"), &digits_left, &first_digit, &digit_error); - /* The failed call took a byte too. */ + /* The failed call took a byte too. Each buffer Rust returned it takes back. */ bool skipped = !api_digit(digits_left, &digits_rest, &last_digit, &digit_error); - api_buf_free(digits_left); api_digit(digits_rest, &digits_left, &last_digit, &digit_error); - api_buf_free(digits_rest); printf("digit %u %s %u %zu\n", first_digit, skipped ? "true" : "false", last_digit, digits_left.len); api_buf_free(digits_left); api_Point nudged = {.x = 1, .y = 2}; @@ -1775,7 +1807,7 @@ static int script(void) { if (drops != 12) { return 1; } - return closures() && foreign_futures() && started_futures() ? 0 : 1; + return closures() && foreign_futures() && forwarding() && started_futures() ? 0 : 1; } /* Each breaks a contract, which must abort with a message naming the function. */ diff --git a/ffi/tests/api.cpp b/ffi/tests/api.cpp index aa1b518de..5b8541ccd 100644 --- a/ffi/tests/api.cpp +++ b/ffi/tests/api.cpp @@ -1527,6 +1527,34 @@ int futures() { } // Who owns each handle and bytes, beyond the transcript, each checked by an assertion. +// A payload Rust returned goes back to Rust in place: moved in, shared by a copy kept +// meanwhile, and written back through `&mut`. Its bytes stay where Rust put them, and +// drop once, with their last owner. +void forwarding() { + std::uint32_t before = api::dropped(); + api::Channel channel = api::Channel::new_(); + api::buf payload = api::token_bytes(3, 64); + const std::uint8_t *at = payload.data(); + channel.send(std::move(payload)); + payload = channel.recv().get(); + check(payload.data() == at && payload.size() == 64, "a payload moved into Rust stays in place"); + api::buf kept = payload; + check(kept.data() == at, "a copy of Rust's bytes shares them"); + channel.send(kept); + check(channel.recv().get().data() != at, "bytes lent to a call are copied"); + channel.send(std::move(kept)); + api::buf shared = channel.recv().get(); + check(shared.data() == at && payload.data() == at, "a copy moved into Rust stays in place"); + api::Packet packet = api::next_packet(api::Packet{1, std::move(shared)}); + check(packet.payload.size() == 64 && api::dropped() == before, "a record carries the payload on"); + payload = api::buf(); + packet = api::Packet{}; + check(api::dropped() == before + 1, "the payload drops once, with its last owner"); + api::buf digits = api::token_bytes('0', 2); + api::put_number(7, digits); + check(digits.str() == "007," && api::dropped() == before + 2, "Rust takes its bytes passed for &mut"); +} + int lifetimes() { // Copies of a type whose clone no call tells apart share one handle: a write through // one shows in the other, as it does between clones of an `Arc`. @@ -1649,6 +1677,8 @@ int lifetimes() { check(api::dropped() == before + 1, "bytes free their owner"); } + forwarding(); + std::printf("lifetimes ok\n"); return 0; } diff --git a/ffi/tests/api.dart b/ffi/tests/api.dart index d81077978..5025c9287 100644 --- a/ffi/tests/api.dart +++ b/ffi/tests/api.dart @@ -1190,10 +1190,55 @@ Future lent() async { pressure(); } check(api.dropped() == before + 2, 'the garbage collector frees bytes once'); + // Read after, so `big` stays reachable and its drop counts in none of forwarded's checks. + await forwarded(); check(big[65536] == 1, 'lent bytes read back while reachable'); print('bytes ok'); } +/// Bytes Rust lent pass back to Rust in place, sent and carried in a record, so a payload +/// read and written on holds Rust's one buffer, freed once with its last owner: a copy +/// anywhere would free the token with the first list. +Future forwarded() async { + final before = api.dropped(); + final channel = api.Channel(); + Uint8List? held = await relay(channel); + final canary = [api.Token(9)]; + for (var i = 0; i < 3; i++) { + pressure(); + } + canary.clear(); + await collect(() => api.dropped() > before, 'the garbage collector frees an old object'); + // Collections enough to free every list the relay left, had one held a copy. + for (var i = 0; i < 3; i++) { + pressure(); + await step(); + } + check(api.dropped() == before + 1, 'bytes forwarded hold the buffer Rust lent'); + check(held.length == 65537 && held.every((b) => b == 5), 'forwarded bytes read back'); + held = null; + await collect(() => api.dropped() >= before + 2, 'the garbage collector frees forwarded bytes'); + for (var i = 0; i < 3; i++) { + pressure(); + } + check(api.dropped() == before + 2, 'forwarded bytes free once'); + channel.close(); +} + +/// Lent bytes sent through `channel`, received, and carried on in a packet, whose lists +/// nothing else keeps. +@pragma('vm:never-inline') +Future relay(api.Channel channel) async { + send(channel); + return carry(await channel.recv()); +} + +@pragma('vm:never-inline') +void send(api.Channel channel) => channel.send(api.tokenBytes(5, 65537)); + +@pragma('vm:never-inline') +Uint8List carry(Uint8List payload) => api.nextPacket(api.Packet(seq: 1, payload: payload)).payload; + /// A view of returned bytes, whose list nothing else keeps. @pragma('vm:never-inline') Uint8List subView(int id) => Uint8List.sublistView(api.tokenBytes(id, 65537), 65527); diff --git a/ffi/tests/api.go b/ffi/tests/api.go index 5194cc030..37387e2e9 100644 --- a/ffi/tests/api.go +++ b/ffi/tests/api.go @@ -21,6 +21,7 @@ import ( "sync" "sync/atomic" "time" + "unsafe" "api/api" "api/dep" @@ -225,7 +226,7 @@ func script() { } } for _, needle := range []string{"lo", "x"} { - if found := api.Find([]byte("hello"), []byte(needle)); found != nil { + if found := api.Find(rt.NewBytes([]byte("hello")), rt.NewBytes([]byte(needle))); found != nil { fmt.Printf("find %d %d\n", found.V0, found.V1) } else { fmt.Println("find none") @@ -288,7 +289,7 @@ func script() { fmt.Printf("doubled Some((Some(%d),))\n", **doubled) } } - fmt.Printf("lent back %s boxed %s\n", list(api.LentBack(rt.NewBytes([]byte{1, 2})).Copy(), number), list(api.BoxedTuple([]byte{3}).Copy(), number)) + fmt.Printf("lent back %s boxed %s\n", list(api.LentBack(rt.NewBytes([]byte{1, 2})).Copy(), number), list(api.BoxedTuple(rt.NewBytes([]byte{3})).Copy(), number)) // A borrowed string and an owned one share a tuple struct. check(api.Title(false) == nil, "an informal title is none") fmt.Printf("title Some((%s,)) None named Some((%s,))\n", quoted(*api.Title(true)), quoted(*api.Named("x"))) @@ -403,12 +404,12 @@ func script() { fmt.Printf("window %t %t %s\n", window.Contains(4), window.Contains(6), window.GoString()) fmt.Println(api.Greet("x", api.ShapeSquare)) - fmt.Printf("len %d\n", api.Len([]byte("abc"))) + fmt.Printf("len %d\n", api.Len(rt.NewBytes([]byte("abc")))) binary := []byte{0xff, 0, 0x61} - fmt.Printf("hex %s\n", api.Hex(binary)) - fmt.Printf("timed hex %s\n", api.TimedHex(binary)) - fmt.Printf("sample hex %s\n", api.SampleHex(binary)) - fmt.Printf("queued hex %s\n", api.QueuedHex(binary)) + fmt.Printf("hex %s\n", api.Hex(rt.NewBytes(binary))) + fmt.Printf("timed hex %s\n", api.TimedHex(rt.NewBytes(binary))) + fmt.Printf("sample hex %s\n", api.SampleHex(rt.NewBytes(binary))) + fmt.Printf("queued hex %s\n", api.QueuedHex(rt.NewBytes(binary))) fmt.Printf("tick %s\n", api.TickName(api.Label{Name: "tick"})) fmt.Printf("rooted %s\n", api.RootedName(api.Label{Name: "rooted"})) fmt.Printf("stored %s\n", api.StoredText("stored")) @@ -422,14 +423,14 @@ func script() { fmt.Printf("secs %d\n", must(stamp.Secs())) fmt.Printf("later %t\n", must(api.StampFromMillis(3000).Later(stamp))) fmt.Printf("secs %d\n", must(stamp.Secs())) - message := must(api.NewMessage(stamp, []byte("body"))) + message := must(api.NewMessage(stamp, rt.NewBytes([]byte("body")))) fmt.Printf("payload %s\n", must(message.PayloadGet()).Copy()) fmt.Printf("stamp %s\n", must(message.StampGet()).GoString()) fmt.Printf("sent %t\n", must(message.Sent())) // Setters assign in place: the bytes are copied in, and a `Copy` handle copied. reset := api.StampFromMillis(4000) ok(message.StampSet(reset)) - ok(message.PayloadSet([]byte("edited"))) + ok(message.PayloadSet(rt.NewBytes([]byte("edited")))) fmt.Printf("payload %s\n", must(message.PayloadGet()).Copy()) fmt.Printf("stamp %s\n", must(message.StampGet()).GoString()) fmt.Printf("secs %d\n", must(reset.Secs())) @@ -437,14 +438,14 @@ func script() { fmt.Printf("stamp %t\n", must(message.StampGet()) != nil) // Rust frees the implementation once it drops it, releasing Go's. - fmt.Printf("kept %d\n", api.Emit(printLog{}, []byte{1, 2, 3, 4})) + fmt.Printf("kept %d\n", api.Emit(printLog{}, rt.NewBytes([]byte{1, 2, 3, 4}))) logger := api.NewLogger(printLog{}) fmt.Printf("write %t\n", must(logger.Write("hi"))) logger.Close() fmt.Println("dropped") // Rust calls it on another thread, before the call returns. threshold := uint8(2) - filtered := api.Filter([]byte{1, 2, 3, 4}, func(level uint8, line string) bool { + filtered := api.Filter(rt.NewBytes([]byte{1, 2, 3, 4}), func(level uint8, line string) bool { fmt.Printf("filter %d %s\n", level, line) return level > threshold }) @@ -457,12 +458,12 @@ func script() { go func() { got <- must(channel.Recv(watching)).Copy() }() <-watching.waiting fmt.Println("recv pending true") - channel.Send([]byte("hi")) + channel.Send(rt.NewBytes([]byte("hi"))) payload := <-got wakes := 1 fmt.Printf("woken %d\n", wakes) fmt.Printf("recv %s\n", payload) - channel.Send(nil) + channel.Send(rt.Bytes{}) if _, err := channel.RecvNonempty(ctx); err != nil { fmt.Printf("nonempty err %s\n", err) } @@ -474,32 +475,32 @@ func script() { // The send happens at the call: the future is cancelled before its first poll. cancelled, cancel := context.WithCancel(ctx) cancel() - check(errors.Is(channel.SendLater(cancelled, []byte("eager")), context.Canceled), "a cancelled call returns its context's error") + check(errors.Is(channel.SendLater(cancelled, rt.NewBytes([]byte("eager"))), context.Canceled), "a cancelled call returns its context's error") fmt.Printf("recv %s\n", must(channel.Recv(ctx)).Copy()) // An eager call whose callback cancels its context while it makes its future is never // polled: only the eager half logs. { inner, cancel := context.WithCancel(ctx) log := &cancelLog{cancel: cancel} - _, err := api.EmitLater(inner, log, []byte{1, 2}) + _, err := api.EmitLater(inner, log, rt.NewBytes([]byte{1, 2})) check(errors.Is(err, context.Canceled), "an eager call cancelled as it makes its future returns its context's error") check(log.lines.Load() == 2, "an eager call cancelled as it makes its future is never polled") } - channel.Send([]byte("sized")) + channel.Send(rt.NewBytes([]byte("sized"))) sized, size, err := channel.RecvSized(ctx) check(err == nil, "recv sized") fmt.Printf("sized %s %d\n", sized.Copy(), size) // The waker moves into Rust, which keeps it until the send wakes it. channel.Notify(func() { wakes++ }) - channel.Send([]byte("rung")) + channel.Send(rt.NewBytes([]byte("rung"))) fmt.Printf("woken %d\n", wakes) // A poll with no async twin is a future. fmt.Printf("take %s\n", must(channel.PollTake(ctx)).Copy()) - channel.Send([]byte("measured")) + channel.Send(rt.NewBytes([]byte("measured"))) fmt.Printf("measure %d\n", must(channel.PollMeasure(ctx, 2))) // A context is made of the waker, which Rust keeps until the send wakes it. channel.Listen(func() { wakes++ }) - channel.Send([]byte("heard")) + channel.Send(rt.NewBytes([]byte("heard"))) fmt.Printf("woken %d\n", wakes) // A returned waker wakes the waiter's task through Rust until the GC collects it. waiter := api.NewWaiter(func() { wakes++ }) @@ -642,8 +643,8 @@ func script() { fmt.Printf("sentence %s|\n", api.Sentence(&[]string{})) abc := "abc" fmt.Printf("either %d %d\n", api.Either(&abc), api.Either(nil)) - ab := []byte("ab") - fmt.Printf("prefixed %s %s\n", api.Prefixed([]byte("cd"), &ab).Copy(), api.Prefixed([]byte("cd"), nil).Copy()) + ab := rt.NewBytes([]byte("ab")) + fmt.Printf("prefixed %s %s\n", api.Prefixed(rt.NewBytes([]byte("cd")), &ab).Copy(), api.Prefixed(rt.NewBytes([]byte("cd")), nil).Copy()) fmt.Printf("coordinates %d %d\n", api.Coordinates(&api.Point{X: 2, Y: 3}), api.Coordinates(nil)) two, three := api.NewCounter("two", 2), api.NewCounter("three", 3) fmt.Printf("plus %d %d\n", must(two.Plus(three)), must(two.Plus(nil))) @@ -680,7 +681,7 @@ func script() { tallies := api.NewTallies() _ = must(must(tallies.First()).Bump()) fmt.Printf("tallied %d\n", must(must(tallies.First()).Bump())) - fmt.Printf("chunks %s\n", list(api.Chunks([]byte("abcde"), 2), func(b rt.Bytes) string { return list(b.Copy(), number) })) + fmt.Printf("chunks %s\n", list(api.Chunks(rt.NewBytes([]byte("abcde")), 2), func(b rt.Bytes) string { return list(b.Copy(), number) })) counters := api.Counters([]string{"x", "y"}) fmt.Printf("counters %s\n", strings.Trim(list(counters, func(c *api.Counter) string { return fmt.Sprintf("%s %d", must(c.Name()), must(c.Get())) @@ -738,11 +739,11 @@ func script() { noFirst = "Some" } fmt.Printf("first of Some(%s) %s\n", quoted(*api.FirstOf([]string{"a", "b"})), noFirst) - limits := api.ByteLimits([]byte("a"), []byte("bc")) + limits := api.ByteLimits(rt.NewBytes([]byte("a")), rt.NewBytes([]byte("bc"))) breadth := api.Breadth(api.LimitsDuration{Low: 5 * time.Millisecond, High: 2 * time.Second}) fmt.Printf("limits %s %s breadth %d\n", limits.Low.Copy(), limits.High.Copy(), breadth.Milliseconds()) parts := []rt.Bytes{rt.NewBytes([]byte("ab")), rt.NewBytes(nil), rt.NewBytes([]byte("c"))} - fmt.Printf("joined %s %d read %d\n", api.Joined(parts).Copy(), len(api.Joined(nil).Copy()), api.ReadLen([]byte("hello"))) + fmt.Printf("joined %s %d read %d\n", api.Joined(parts).Copy(), len(api.Joined(nil).Copy()), api.ReadLen(rt.NewBytes([]byte("hello")))) doc := must(serdejson.JsonParse(`{"b": [1, 2.5, null], "a": "é"}`)) fmt.Printf("json %s\n", must(api.Encode(doc))) fmt.Printf("timed text %s\n", must(api.TimedText(doc))) @@ -1037,7 +1038,7 @@ func futures() { channel := api.NewChannel() // The send lands from another goroutine's thread while the receive waits. - go channel.Send([]byte("blocked")) + go channel.Send(rt.NewBytes([]byte("blocked"))) check(string(must(channel.Recv(ctx)).Copy()) == "blocked", "recv returns the payload another goroutine sent") // A wake that lands while the future polls, before it returns pending, is kept: the @@ -1063,7 +1064,7 @@ func futures() { // Hold the call between the wake and its next poll, where it reads its context. gate := make(chan struct{}) watching.gate.Store(&gate) - channel.Send([]byte("kept")) + channel.Send(rt.NewBytes([]byte("kept"))) cancel() close(gate) check(errors.Is(<-done, context.Canceled), "a receive cancelled after its wake returns its context's error") @@ -1072,7 +1073,7 @@ func futures() { // A context done before the call takes nothing, even a payload ready at the first poll. { - channel.Send([]byte("ready")) + channel.Send(rt.NewBytes([]byte("ready"))) cancelled, cancel := context.WithCancel(ctx) cancel() _, err := channel.Recv(cancelled) @@ -1097,7 +1098,7 @@ func futures() { <-watching.waiting sender := must(other.Clone()) other.Close() - sender.Send([]byte("held")) + sender.Send(rt.NewBytes([]byte("held"))) check(<-got == "held", "a closed handle stays alive in the call waiting on it") sender.Close() } @@ -1238,9 +1239,50 @@ func owned() { collected(before+2, "the GC frees what nobody closed") runtime.GC() check(api.Dropped() == before+2, "the GC frees each once") + + before = api.Dropped() + forward() + collected(before+1, "a forwarded payload drops once nothing reaches it") + uncollected(before+1, "a forwarded payload drops once") fmt.Println("owned ok") } +// forward reads a payload Rust returned and writes it back, whole, sliced, and in a +// record: Rust takes a second owner of its own buffer each time, so the bytes stay where +// Rust put them, in a frame of its own so nothing keeps them reachable after. +// +//go:noinline +func forward() { + ctx := context.Background() + channel := api.NewChannel() + defer channel.Close() + // Long enough that Rust's buffer is lent rather than copied. + payload := api.TokenBytes(4, 1024) + at := addr(payload) + channel.Send(payload) + got := must(channel.Recv(ctx)) + check(addr(got) == at && got.Len() == 1024, "a payload sent back is Rust's own") + channel.Send(got.Slice(1, 1023)) + got = must(channel.Recv(ctx)) + check(addr(got) == at+1 && got.Len() == 1022, "a slice of a payload sent back is Rust's own") + packet := api.NextPacket(api.Packet{Seq: 1, Payload: got}) + check(addr(packet.Payload) == at+1 && packet.Payload.Len() == 1022, "a record carries a payload back as Rust's own") +} + +// addr is where b's bytes start, as an io.Writer sees them. +func addr(b rt.Bytes) uintptr { + var w addrWriter + _, _ = b.WriteTo(&w) + return uintptr(w) +} + +type addrWriter uintptr + +func (w *addrWriter) Write(p []byte) (int, error) { + *w = addrWriter(unsafe.Pointer(&p[0])) + return len(p), nil +} + // unclosed makes a token in a frame of its own, so nothing keeps it reachable. // //go:noinline @@ -1428,7 +1470,7 @@ func bytes() { // Bytes an object returns outlive its Close: no Bytes depends on an object. body := strings.Repeat("b", 2048) - message := must(api.NewMessage(api.StampFromMillis(1), []byte(body))) + message := must(api.NewMessage(api.StampFromMillis(1), rt.NewBytes([]byte(body)))) payload := must(message.PayloadGet()) message.Close() runtime.GC() @@ -1512,7 +1554,7 @@ func broken(which string) { case "text": api.Greet("\xff", api.ShapeSquare) case "callback panicked": - api.Emit(panicking{}, []byte{1}) + api.Emit(panicking{}, rt.NewBytes([]byte{1})) // A Rust Duration longer than Go's longest. case "duration": api.Extended(math.MaxInt64, 1) diff --git a/ffi/tests/api.kt b/ffi/tests/api.kt index 72538addb..094f5db59 100644 --- a/ffi/tests/api.kt +++ b/ffi/tests/api.kt @@ -113,7 +113,7 @@ fun script() { println(if (spread == null) "spread none" else "spread ${spread.first} ${spread.second} ${spread.third} ${spread.fourth}") } for (needle in listOf("lo", "x")) { - val found = find("hello".encodeToByteArray(), needle.encodeToByteArray()) + val found = find(Bytes("hello".encodeToByteArray()), Bytes(needle.encodeToByteArray())) println(if (found == null) "find none" else "find ${found.first} ${found.second}") } val statRows = rows(listOf(Point(1, 2), Point(3, 3))) @@ -149,8 +149,8 @@ fun script() { println(if (doubled == null) "doubled None" else "doubled Some((${doubled.first?.let { "Some($it)" } ?: "None"},))") } // A borrowed or boxed tuple of one is a tuple, not its element. - val lentOne = lentBack(Tuple1(byteArrayOf(1, 2))).first.toByteArray().toList() - val boxedOne = boxedTuple(byteArrayOf(3)).first.toByteArray().toList() + val lentOne = lentBack(Tuple1(Bytes(byteArrayOf(1, 2)))).first.toByteArray().toList() + val boxedOne = boxedTuple(Bytes(byteArrayOf(3))).first.toByteArray().toList() println("lent back $lentOne boxed $boxedOne") // A borrowed string and an owned one share a tuple struct. println("title Some((\"${title(true)!!.first}\",)) ${if (title(false) == null) "None" else "Some"} named Some((\"${named("x")!!.first}\",))") @@ -242,8 +242,8 @@ fun script() { println("window ${window.contains(4u)} ${window.contains(6u)} ${window.debug()}") println(greet("x", Shape.SQUARE)) - println("len ${len("abc".encodeToByteArray())}") - val binary = byteArrayOf(0xff.toByte(), 0, 'a'.code.toByte()) + println("len ${len(Bytes("abc".encodeToByteArray()))}") + val binary = Bytes(byteArrayOf(0xff.toByte(), 0, 'a'.code.toByte())) println("hex ${hex(binary)}") println("timed hex ${timedHex(binary)}") println("sample hex ${sampleHex(binary)}") @@ -261,26 +261,26 @@ fun script() { println("secs ${stamp.secs()}") println("later ${Stamp.fromMillis(3000u).later(stamp)}") println("secs ${stamp.secs()}") - val message = Message.new(stamp, "body".encodeToByteArray()) + val message = Message.new(stamp, Bytes("body".encodeToByteArray())) println("payload ${message.payloadGet.text()}") println("stamp ${message.stampGet!!.debug()}") println("sent ${message.sent()}") // Setters assign in place: the bytes are copied in, and a `Copy` handle copied. val reset = Stamp.fromMillis(4000u) message.stampSet(reset) - message.payloadSet("edited".encodeToByteArray()) + message.payloadSet(Bytes("edited".encodeToByteArray())) println("payload ${message.payloadGet.text()}") println("stamp ${message.stampGet!!.debug()}") println("secs ${reset.secs()}") message.stampSet(null) println("stamp ${message.stampGet != null}") - println("kept ${emit(PrintLog(), byteArrayOf(1, 2, 3, 4))}") + println("kept ${emit(PrintLog(), Bytes(byteArrayOf(1, 2, 3, 4)))}") Logger.new(PrintLog()).use { logger -> println("write ${logger.write("hi")}") } println("dropped") // Rust calls it on another thread, before the call returns. val threshold = 2 - val kept = filter(byteArrayOf(1, 2, 3, 4)) { level, line -> + val kept = filter(Bytes(byteArrayOf(1, 2, 3, 4))) { level, line -> println("filter $level $line") level.toInt() > threshold } @@ -296,13 +296,13 @@ fun script() { CoroutineScope(q.asCoroutineDispatcher()).launch { got = channel.recv() } q.drain() println("recv pending ${got == null}") - channel.send("hi".encodeToByteArray()) + channel.send(Bytes("hi".encodeToByteArray())) wakes += q.size println("woken $wakes") q.drain() println("recv ${got!!.text()}") runBlocking { - channel.send(byteArrayOf()) + channel.send(Bytes(byteArrayOf())) try { println("nonempty ${channel.recvNonempty().size}") } catch (e: ErrorException) { @@ -312,25 +312,25 @@ fun script() { pair().use { b2 -> println("fill " + (fill4(b4, 9u) + fill2(b2, 1u)).joinToString(" ")) } } // The send happens at the call: the future is dropped unawaited. - channel.sendLater("eager".encodeToByteArray()) + channel.sendLater(Bytes("eager".encodeToByteArray())) println("recv ${channel.recv().text()}") - channel.send("sized".encodeToByteArray()) + channel.send(Bytes("sized".encodeToByteArray())) val (payload, size) = channel.recvSized() println("sized ${payload.text()} $size") } // The waker moves into Rust, which keeps it until the send wakes it. channel.notify(waker) - channel.send("rung".encodeToByteArray()) + channel.send(Bytes("rung".encodeToByteArray())) println("woken $wakes") runBlocking { // A poll with no async twin is a future. println("take ${channel.pollTake().text()}") - channel.send("measured".encodeToByteArray()) + channel.send(Bytes("measured".encodeToByteArray())) println("measure ${channel.pollMeasure(2u)}") } // A context is made of the waker, which Rust keeps until the send wakes it. channel.listen(waker) - channel.send("heard".encodeToByteArray()) + channel.send(Bytes("heard".encodeToByteArray())) println("woken $wakes") // A returned waker wakes the waiter's task through Rust until the GC drops it. Waiter.new(waker).use { waiter -> @@ -453,8 +453,8 @@ fun script() { println("sentence ${sentence(words)} ${sentence(null)}") println("sentence ${sentence(emptyList())}|") println("either ${either("abc")} ${either(null)}") - val cd = "cd".encodeToByteArray() - println("prefixed ${prefixed(cd, "ab".encodeToByteArray()).toByteArray().decodeToString()} ${prefixed(cd, null).toByteArray().decodeToString()}") + val cd = Bytes("cd".encodeToByteArray()) + println("prefixed ${prefixed(cd, Bytes("ab".encodeToByteArray())).toByteArray().decodeToString()} ${prefixed(cd, null).toByteArray().decodeToString()}") println("coordinates ${coordinates(Point(2, 3))} ${coordinates(null)}") Counter.new("two", 2u).use { two -> Counter.new("three", 3u).use { three -> @@ -464,14 +464,14 @@ fun script() { } } check(Counter.new("five", 5u).intoSum(1u) == 6uL, "a consuming method's parameter named close is its own") - val encoded = Mut("0,".encodeToByteArray()) + val encoded = Mut(Bytes("0,".encodeToByteArray())) putNumber(1u, encoded) putNumber(23u, encoded) - println("put_number ${encoded.value.decodeToString()}") - val entry = Mut(Pair("ab".encodeToByteArray(), 1u)) + println("put_number ${encoded.value.toByteArray().decodeToString()}") + val entry = Mut(Pair(Bytes("ab".encodeToByteArray()), 1u)) val stamps = pushCounted(entry, 'c'.code.toUByte()) - println("push_counted $stamps ${entry.value.first.decodeToString()} ${entry.value.second}") - val digits = Mut("7x9".encodeToByteArray()) + println("push_counted $stamps ${entry.value.first.toByteArray().decodeToString()} ${entry.value.second}") + val digits = Mut(Bytes("7x9".encodeToByteArray())) val firstDigit = digit(digits) val skipped = try { digit(digits) @@ -502,7 +502,7 @@ fun script() { tallies.first().use { it.bump() } println("tallied ${tallies.first().use { it.bump() }}") } - println("chunks ${chunks("abcde".encodeToByteArray(), 2uL).map { it.toByteArray().toList() }}") + println("chunks ${chunks(Bytes("abcde".encodeToByteArray()), 2uL).map { it.toByteArray().toList() }}") val counters = counters(listOf("x", "y")) println("counters ${counters.joinToString(", ") { "${it.name()} ${it.get()}" }}") // Each counter moves in, so Kotlin closes it. @@ -546,11 +546,11 @@ fun script() { println("summed ${summed.x} ${summed.y} ${noneSummed.x} ${noneSummed.y}") println("longest ${longest(listOf("a", "abc"))} ${longest(emptyList())}") println("first of Some(\"${firstOf(listOf("a", "b"))}\") ${firstOf(emptyList()) ?: "None"}") - val limits = byteLimits("a".encodeToByteArray(), "bc".encodeToByteArray()) + val limits = byteLimits(Bytes("a".encodeToByteArray()), Bytes("bc".encodeToByteArray())) val waits = breadth(LimitsDuration(5.milliseconds, 2.seconds)) println("limits ${limits.low.toByteArray().decodeToString()} ${limits.high.toByteArray().decodeToString()} breadth ${waits.inWholeMilliseconds}") - val parts = joined(listOf("ab".encodeToByteArray(), ByteArray(0), "c".encodeToByteArray())).toByteArray().decodeToString() - println("joined $parts ${joined(emptyList()).toByteArray().size} read ${readLen("hello".encodeToByteArray())}") + val parts = joined(listOf(Bytes("ab".encodeToByteArray()), Bytes(ByteArray(0)), Bytes("c".encodeToByteArray()))).toByteArray().decodeToString() + println("joined $parts ${joined(emptyList()).toByteArray().size} read ${readLen(Bytes("hello".encodeToByteArray()))}") val doc = Json.parse("""{"b": [1, 2.5, null], "a": "\u00e9"}""") println("json ${encode(doc)}") println("timed text ${timedText(doc)}") @@ -753,7 +753,7 @@ fun futures() { // The send lands from another thread while the receive waits. runBlocking { - val sender = thread { channel.send("blocked".encodeToByteArray()) } + val sender = thread { channel.send(Bytes("blocked".encodeToByteArray())) } val payload = channel.recv() sender.join() check(payload.text() == "blocked", "recv returns the payload another thread sent") @@ -768,7 +768,7 @@ fun futures() { scope.launch { got = channel.recv().text() } q.drain() check(got == null, "recv waits for a send") - channel.send("then".encodeToByteArray()) + channel.send(Bytes("then".encodeToByteArray())) check(got == null, "recv never polls inside the wake") q.drain() check(got == "then", "recv resumes with the payload") @@ -779,7 +779,7 @@ fun futures() { runBlocking { var got: String? = null val job = launch(Dispatchers.Unconfined) { got = channel.recv().text() } - channel.send("unconfined".encodeToByteArray()) + channel.send(Bytes("unconfined".encodeToByteArray())) job.join() check(got == "unconfined", "an unconfined recv resumes outside the wake") } @@ -802,7 +802,7 @@ fun futures() { called = true } q.drain() - channel.send("kept".encodeToByteArray()) + channel.send(Bytes("kept".encodeToByteArray())) job.cancel() q.drain() check(!called && job.isCancelled, "a cancelled receive never resumes") @@ -818,7 +818,7 @@ fun futures() { q.drain() val sender = other.clone() other.close() - sender.send("held".encodeToByteArray()) + sender.send(Bytes("held".encodeToByteArray())) q.drain() check(got == "held", "a closed handle stays held by the call awaiting on it") sender.close() @@ -1017,7 +1017,7 @@ fun broken(which: String) { object : Log { override fun log(level: UByte, line: String): Boolean = throw IllegalStateException("a log failed") }, - byteArrayOf(1), + Bytes(byteArrayOf(1)), ) } } @@ -1025,6 +1025,23 @@ fun broken(which: String) { // What frees what: an object's close, or the GC once it is collected if never closed, and // Rust's bytes only the GC, once no Bytes reaching them is reachable. Rust's bytes up to // 1 KiB cross copied, freed at the call, and longer ones lent. +// Kotlin's text and bytes, which Rust copies or encodes once from where Kotlin keeps them: +// each reads back as Kotlin's own encoder has it, at a call's top level and in a record. +fun encoded() { + val texts = listOf( + "", "a\u0000b", "é\u4e2d\uD83D\uDE00", "\uD800x", "x\uDC00", "end\uD83D", "\uDE00\uD83D", + "a".repeat(1023) + "\uD83D\uDE00", "a".repeat(1023) + "\uD83Dz", "\u4e2d\uD83D\uDE00".repeat(1500), + ) + for (text in texts) { + val expected = text.encodeToByteArray().decodeToString() + check(greet(text, Shape.SQUARE) == "hello $expected, a Square", "text crosses as its UTF-8") + check(tickName(Label(text, null, null)) == expected, "a record's text crosses as its UTF-8") + } + val body = ByteArray(300) { it.toByte() } + val sliced = nextPacket(Packet(1u, Bytes(body).slice(7, 290))).payload + check(sliced.toByteArray().contentEquals(body.copyOfRange(7, 290)), "a slice of Kotlin's bytes in a record crosses") +} + /** A record's scalars cross apart from its other fields, each exactly, both ways. */ fun readings() { check(reading(0u) == null && readingDebug(0u) == "", "no reading crosses as none") @@ -1041,6 +1058,7 @@ fun readings() { } fun owned() { + encoded() readings() budgeted() for (n in 1023..1025) { @@ -1080,7 +1098,7 @@ fun owned() { // Bytes outlive the object that returned them and the record that held them. val body = ByteArray(2048) { 'b'.code.toByte() } - val message = Message.new(Stamp.fromMillis(1u), body) + val message = Message.new(Stamp.fromMillis(1u), Bytes(body)) val payload = message.payloadGet message.close() val carried = nextPacket(Packet(1u, Bytes(body))).payload @@ -1096,6 +1114,34 @@ fun owned() { raced() compared() + forwarded() +} + +// A payload Rust lent passed back to Rust: it crosses as Rust's own, never copied, so the +// Bytes it went through keep nothing of their own, and its one buffer drops once, after +// the last of them. +fun forwarded() { + val before = dropped() + forwardedRead(before) + collected(before + 1u, "a forwarded payload's buffer drops with the last Bytes reaching it") +} + +// In its own frame, so nothing on the caller's keeps the last Bytes reachable. +@OwnFrame +fun forwardedRead(before: UInt) { + val last = forward() + uncollected(before, "a payload forwarded through Rust stays the buffer Rust made") + check(last.size == 2046 && last.toByteArray().all { it == 6.toByte() }, "a forwarded payload reads as Rust made it") +} + +// Rust's bytes sent, received, sliced, carried in a record, and sent again. +fun forward(): Bytes = runBlocking { + Channel.new().use { channel -> + channel.send(tokenBytes(6u, 2048u)) + val carried = nextPacket(Packet(1u, channel.recv().slice(1, 2047))).payload + channel.send(carried) + channel.recv() + } } // Every read of [bytes], each of which is [value]. diff --git a/ffi/tests/api.py b/ffi/tests/api.py index d76582b4b..6a8ab1ff9 100644 --- a/ffi/tests/api.py +++ b/ffi/tests/api.py @@ -1208,9 +1208,59 @@ def owned(): del cycle gc.collect() check(api.dropped() == before + 5, "the cycle collector frees what a cycle held") + forwarding() + released_mid_call() + + +def released_mid_call(): + """Rust's bytes a call reads in place stay Rust's until it returns, though the view + passed is released while the arguments are still being read.""" + kept = [] + + def parts(): + before = api.dropped() + with api.token_bytes(9, 2048) as view: + yield view + kept.append(api.dropped() == before) + yield b"x" + + before = api.dropped() + joined = bytes(api.joined(parts())) + check(kept == [True], "a call keeps Rust's bytes it was passed a released view of") + check(joined == bytes([9]) * 2048 + b"x", "a released view's bytes read back") + check(api.dropped() == before + 1, "Rust's bytes drop once the call returns") print("owned ok") +def forwarding(): + """Rust's bytes, read and passed back, cross in place: whole, sliced, kept while passed + on, and in a record. The token's drop shows no copy stood in for them: it comes once, + only when the last view of the bytes Rust put it in goes.""" + loop = asyncio.new_event_loop() + channel = api.Channel.new() + before = api.dropped() + payload = api.token_bytes(3, 64) + channel.send(payload) + del payload + payload = loop.run_until_complete(channel.recv()) + channel.send(payload[1:63]) + del payload + payload = loop.run_until_complete(channel.recv()) + check(bytes(payload) == bytes([3]) * 62, "a slice of Rust's bytes crosses as itself") + channel.send(payload) + kept = loop.run_until_complete(channel.recv()) + del payload + packet = api.next_packet(api.Packet(1, kept)) + del kept + check(api.dropped() == before and len(packet.payload) == 62, "Rust's bytes cross back without a copy") + del packet + check(api.dropped() == before + 1, "Rust's bytes drop once, with their last view") + # Bytes of Python's own cross copied once, and a view of Rust's only read stays theirs. + packet = api.next_packet(api.Packet(1, bytearray(b"abc"))) + check(bytes(packet.payload) == b"abc", "a bytearray crosses copied") + loop.close() + + def values(): """The values Python's types bound: integers checked against their Rust type's range, durations truncated to the microsecond up to `timedelta.max`, which stands for longer, diff --git a/ffi/tests/api.swift b/ffi/tests/api.swift index ae6108a2a..b9f477b70 100644 --- a/ffi/tests/api.swift +++ b/ffi/tests/api.swift @@ -49,7 +49,7 @@ func debug(_ value: T) -> String { String(reflecting: value) } -func text(_ data: Data) -> String { +func text(_ data: Bytes) -> String { String(decoding: data, as: UTF8.self) } @@ -172,7 +172,10 @@ func block(_ body: @escaping @Sendable () async throws -> T) -> T { done.signal() } done.wait() - return try! result.value!.get() + // Moved out, since the task may release its hold on `result` only after this returns. + let value = result.value! + result.value = nil + return try! value.get() } func script() { @@ -237,7 +240,7 @@ func script() { } } for needle in ["lo", "x"] { - if let found = find(Data("hello".utf8), Data(needle.utf8)) { + if let found = find(Bytes("hello".utf8), Bytes(needle.utf8)) { print("find \(found.0) \(found.1)") } else { print("find none") @@ -283,7 +286,7 @@ func script() { case .some(let inner): print("doubled Some((\(inner.map { "Some(\($0))" } ?? "None"),))") } } - print("lent back \(Array(lentBack(Data([1, 2])))) boxed \(Array(boxedTuple(Data([3]))))") + print("lent back \(Array(lentBack(Bytes([1, 2])))) boxed \(Array(boxedTuple(Bytes([3]))))") // A borrowed string and an owned one share a tuple struct. print("title Some((\"\(title(true)!)\",)) \(title(false) == nil ? "None" : "Some") named Some((\"\(named("x")!)\",))") @@ -377,8 +380,8 @@ func script() { print("window \(window.contains(4)) \(window.contains(6)) \(debug(window))") print(greet("x", .square)) - print("len \(len(Data("abc".utf8)))") - let binary = Data([0xff, 0, 0x61]) + print("len \(len(Bytes("abc".utf8)))") + let binary = Bytes([0xff, 0, 0x61]) print("hex \(hex(binary))") print("timed hex \(timedHex(binary))") print("sample hex \(sampleHex(binary))") @@ -403,14 +406,14 @@ func script() { print("secs \(stamp.secs())") print("later \(Stamp.fromMillis(3000).later(stamp))") print("secs \(stamp.secs())") - var message = Message.new(stamp, Data("body".utf8)) + var message = Message.new(stamp, Bytes("body".utf8)) print("payload \(text(message.payloadGet))") print("stamp \(debug(message.stampGet!))") print("sent \(message.sent())") // Setters assign in place: the bytes are copied in, and a `Copy` handle copied. let reset = Stamp.fromMillis(4000) message.stampSet(reset) - message.payloadSet(Data("edited".utf8)) + message.payloadSet(Bytes("edited".utf8)) print("payload \(text(message.payloadGet))") print("stamp \(debug(message.stampGet!))") print("secs \(reset.secs())") @@ -418,7 +421,7 @@ func script() { print("stamp \(message.stampGet != nil)") // Bound first, so Swift's reference is gone before the print. - let kept = emit(PrintLog(), Data([1, 2, 3, 4])) + let kept = emit(PrintLog(), Bytes([1, 2, 3, 4])) print("kept \(kept)") do { let logger = Logger.new(PrintLog()) @@ -427,7 +430,7 @@ func script() { print("dropped") // Rust calls it on another thread, before the call returns. let threshold: UInt8 = 2 - let filtered = filter(Data([1, 2, 3, 4])) { level, line in + let filtered = filter(Bytes([1, 2, 3, 4])) { level, line in print("filter \(level) \(line)") return level > threshold } @@ -438,17 +441,17 @@ func script() { // On an executor the script drains, so it sees the receive wait and its wake queue // the resume. let q = Queue() - let got = Cell(nil) + let got = Cell(nil) q.launch { got.value = try await channel.recv() } q.drain() print("recv pending \(got.value == nil)") - channel.send(Data("hi".utf8)) + channel.send(Bytes("hi".utf8)) wakes.value += q.size print("woken \(wakes.value)") q.drain() print("recv \(text(got.value!))") block { - channel.send(Data()) + channel.send(Bytes()) do { print("nonempty \(try await channel.recvNonempty().count)") } catch { @@ -458,27 +461,27 @@ func script() { let filled = try await fill4(&four, 9) + fill2(&two, 1) print("fill \(filled.map { Swift.String($0) }.joined(separator: " "))") // The send happens at the call, before the future is first polled. - try await channel.sendLater(Data("eager".utf8)) + try await channel.sendLater(Bytes("eager".utf8)) print("recv \(text(try await channel.recv()))") - channel.send(Data("sized".utf8)) + channel.send(Bytes("sized".utf8)) let (payload, size) = try await channel.recvSized() print("sized \(text(payload)) \(size)") } // The waker moves into Rust, which keeps it until the send wakes it. channel.notify { wakes.value += 1 } - channel.send(Data("rung".utf8)) + channel.send(Bytes("rung".utf8)) print("woken \(wakes.value)") block { // A poll with no async twin is a future. print("take \(text(try await channel.pollTake()))") // A `&mut self` poll needs a binding of its own, which a closure cannot capture. var measured = Channel.new() - measured.send(Data("measured".utf8)) + measured.send(Bytes("measured".utf8)) print("measure \(try await measured.pollMeasure(2))") } // A context is made of the waker, which Rust keeps until the send wakes it. channel.listen { wakes.value += 1 } - channel.send(Data("heard".utf8)) + channel.send(Bytes("heard".utf8)) print("woken \(wakes.value)") // A returned waker wakes the waiter's task through Rust until released. do { @@ -606,7 +609,7 @@ func script() { print("sentence \(sentence(words)) \(sentence(nil))") print("sentence \(sentence([]))|") print("either \(either("abc")) \(either(nil))") - print("prefixed \(String(decoding: prefixed(Data("cd".utf8), Data("ab".utf8)), as: UTF8.self)) \(String(decoding: prefixed(Data("cd".utf8), nil), as: UTF8.self))") + print("prefixed \(String(decoding: prefixed(Bytes("cd".utf8), Bytes("ab".utf8)), as: UTF8.self)) \(String(decoding: prefixed(Bytes("cd".utf8), nil), as: UTF8.self))") print("coordinates \(coordinates(Point(x: 2, y: 3))) \(coordinates(nil))") do { // An optional borrowed handle is an overload with it and one without; one borrowed @@ -633,14 +636,14 @@ func script() { print("add_to \(two.addTo(&three)) \(two.addTo(&none)) \(three!.get())") } do { - var encoded = Data("0,".utf8) + var encoded = Bytes("0,".utf8) putNumber(1, &encoded) putNumber(23, &encoded) print("put_number \(String(decoding: encoded, as: UTF8.self))") - var entry = (Data("ab".utf8), UInt32(1)) + var entry = (Bytes("ab".utf8), UInt32(1)) let stamps = pushCounted(&entry, UInt8(ascii: "c")) print("push_counted \(stamps) \(String(decoding: entry.0, as: UTF8.self)) \(entry.1)") - var digits = Data("7x9".utf8) + var digits = Bytes("7x9".utf8) let firstDigit = try! digit(&digits) let skipped = (try? digit(&digits)) == nil let lastDigit = try! digit(&digits) @@ -671,7 +674,7 @@ func script() { _ = tallies.first().bump() print("tallied \(tallies.first().bump())") } - print("chunks \(chunks(Data("abcde".utf8), 2).map { Array($0) })") + print("chunks \(chunks(Bytes("abcde".utf8), 2).map { Array($0) })") var counters = counters(["x", "y"]) print("counters \(listed(counters) { "\($0.name()) \($0.get())" })") // Each counter moves in. @@ -698,13 +701,13 @@ func script() { print("summed \(sum.x) \(sum.y) \(noneSum.x) \(noneSum.y)") print("longest \(longest(["a", "abc"])) \(longest([]))") print("first of Some(\"\(firstOf(["a", "b"])!)\") \(firstOf([]) ?? "None")") - let limits = byteLimits(Data("a".utf8), Data("bc".utf8)) + let limits = byteLimits(Bytes("a".utf8), Bytes("bc".utf8)) let waits = breadth(LimitsDuration(low: .milliseconds(5), high: .seconds(2))) let low = String(decoding: limits.low, as: UTF8.self) let high = String(decoding: limits.high, as: UTF8.self) print("limits \(low) \(high) breadth \(waits.components.seconds * 1000 + waits.components.attoseconds / 1_000_000_000_000_000)") - let parts = String(decoding: Api.joined([Data("ab".utf8), Data(), Data("c".utf8)]), as: UTF8.self) - print("joined \(parts) \(Api.joined([]).count) read \(readLen(Data("hello".utf8)))") + let parts = String(decoding: Api.joined([Bytes("ab".utf8), Bytes(), Bytes("c".utf8)]), as: UTF8.self) + print("joined \(parts) \(Api.joined([]).count) read \(readLen(Bytes("hello".utf8)))") let doc = try! Json.parse(#"{"b": [1, 2.5, null], "a": "é"}"#) print("json \(encode(doc))") print("timed text \(timedText(doc))") @@ -817,7 +820,7 @@ func numbers(_ what: String, _ n: Numbers) { print("\(what) some \(n.some.map { "Some(\($0))" } ?? "None")") } -func printTail(_ read: () throws -> (Data, UInt64)) { +func printTail(_ read: () throws -> (Bytes, UInt64)) { do { let (bytes, at) = try read() print("tail \(String(decoding: bytes, as: UTF8.self)) \(at)") @@ -905,7 +908,7 @@ func futures() { // The send lands from another thread while the receive waits. let payload = block { - Thread.detachNewThread { channel.send(Data("blocked".utf8)) } + Thread.detachNewThread { channel.send(Bytes("blocked".utf8)) } return try await channel.recv() } check(text(payload) == "blocked", "recv returns the payload another thread sent") @@ -918,7 +921,7 @@ func futures() { q.launch { got.value = text(try await channel.recv()) } q.drain() check(got.value == nil, "recv waits for a send") - channel.send(Data("then".utf8)) + channel.send(Bytes("then".utf8)) check(got.value == nil, "recv never resumes inside the wake") q.drain() check(got.value == "then", "recv resumes with the payload") @@ -956,7 +959,7 @@ func futures() { } } q.drain() - channel.send(Data("kept".utf8)) + channel.send(Bytes("kept".utf8)) task.cancel() q.drain() check(!called.value && threw.value, "a cancelled receive throws CancellationError") @@ -967,7 +970,7 @@ func futures() { // A task cancelled before its call takes nothing, even a payload ready at the first poll. do { let threw = Cell(false) - channel.send(Data("ready".utf8)) + channel.send(Bytes("ready".utf8)) let task = q.launch { do { _ = try await channel.recv() @@ -990,7 +993,7 @@ func futures() { let threw = Cell(false) task.value = q.launch { do { - _ = try await emitLater(log, Data([1, 2])) + _ = try await emitLater(log, Bytes([1, 2])) } catch is CancellationError { threw.value = true } @@ -1028,7 +1031,7 @@ func futures() { q.launch { got.value = text(try await other.recv()) } q.drain() } - sender.send(Data("held".utf8)) + sender.send(Bytes("held".utf8)) q.drain() check(got.value == "held", "the call awaiting on an object holds it") } @@ -1051,7 +1054,7 @@ func futures() { // The closure waits, so it pends whether or not its task has run yet, and the check // waits for the throw, wherever the task resumes. do { - let (stream, _) = AsyncStream.makeStream() + let (stream, gate) = AsyncStream.makeStream() let threw = Cell(false) q.launch { do { @@ -1062,6 +1065,7 @@ func futures() { } until { q.drain(); return threw.value } check(threw.value, "Rust gives up on an async closure") + gate.finish() } do { let started = Cell(false) @@ -1156,7 +1160,7 @@ func until(_ done: () -> Bool) { // bytes lend Rust's buffer until it and every slice of it are released. func owned() { for n in 1023...1025 { - let payload = Data((0.. UnsafeRawPointer? { + bytes.withUnsafeBytes { $0.baseAddress } +} + +// A payload Rust returned, read and written back, whole and in a record: Rust takes a +// second owner of its own buffer each time, so the bytes stay where Rust put them. +func forward() { + let channel = Channel.new() + let payload = tokenBytes(4, 64) + let at = address(payload) + channel.send(payload) + let got = block { try await channel.recv() } + check(address(got) == at && got.count == 64, "a payload sent back is Rust's own") + let packet = nextPacket(Packet(seq: 1, payload: got)) + check(address(packet.payload) == at && packet.payload.count == 64, "a record carries a payload back as Rust's own") + let part = packet.payload.dropFirst(8) + channel.send(part) + let sent = block { try await channel.recv() } + check(address(sent) == at.map { $0 + 8 } && sent.count == 56, "a slice sent back is Rust's own") + check(Bytes(sent.data) == sent && sent.data.count == 56, "Rust's bytes copy into a Data") + // A few bytes `Data` holds inline, which pass back as Swift's. + channel.send(tokenBytes(6, 3)) + check(block { try await channel.recv() } == Bytes([6, 6, 6]), "a short payload sent back reads back") +} + // Each breaks a contract Swift's compiler checks, so it fails to compile, built alone with // its flag. #if MOVED diff --git a/ffi/tests/fixtures/shapes.rs b/ffi/tests/fixtures/shapes.rs index a04c48eed..5604d214e 100644 --- a/ffi/tests/fixtures/shapes.rs +++ b/ffi/tests/fixtures/shapes.rs @@ -876,10 +876,6 @@ fn is_bytes(shape: &Shape) -> bool { } } -fn bytes(shape: &Shape) -> bool { - within(shape, &is_bytes) -} - fn ipv6(shape: &Shape) -> bool { within(shape, &|s| { matches!(s, Shape::SocketAddr("SocketAddr" | "SocketAddrV6")) @@ -941,12 +937,6 @@ const KNOWN: &[Known] = &[ skips: ipv6, quest: "/quest/m1/ffi-export/lossless.md", }, - Known { - language: "kotlin", - what: "returned bytes passed back", - skips: bytes, - quest: "/quest/m1/ffi-export/owned-bufs.md", - }, Known { language: "kotlin", what: "a returned tuple of one passed back", @@ -959,12 +949,6 @@ const KNOWN: &[Known] = &[ skips: ipv6, quest: "/quest/m1/ffi-export/lossless.md", }, - Known { - language: "go", - what: "returned bytes passed back", - skips: bytes, - quest: "/quest/m1/ffi-export/owned-bufs.md", - }, Known { language: "cpp", what: "a returned collection of bytes passed back", @@ -1230,7 +1214,11 @@ fn kotlin(built: &Built, calls: &[Call]) -> String { }; format!("\tprintln(\"{} ${{{value}}}\")", call.label()) }); - let mut out = format!("import {}.*\nimport kotlinx.coroutines.runBlocking\n\n", built.name()); + // A block of unsigned numbers is an experimental unsigned array, such as `UIntArray`. + let mut out = format!( + "@file:OptIn(ExperimentalUnsignedTypes::class)\n\nimport {}.*\nimport kotlinx.coroutines.runBlocking\n\n", + built.name() + ); for (n, lines) in &shapes { let _ = writeln!(out, "suspend fun shape{n}() {{\n{lines}}}\n"); } @@ -1284,8 +1272,9 @@ fn swift(calls: &[Call]) -> String { } /// The Go driver: `one` takes the value of a call returning it alone or with an error, -/// `tupleK` gathers a returned tuple's values into the `rt.TupleK` Go takes, and `lift` -/// makes a closure of a function of either form. +/// `tupleK` gathers a returned tuple's values into the `rt.TupleK` Go takes, `lift` makes a +/// closure of a function of either form, and `bytesIn` and `bytesOptIn` adapt one taking an +/// `rt.Bytes` to the `[]byte` a callback is lent. fn go(built: &Built, calls: &[Call], source: &str, rt: &str) -> String { let mut body = String::new(); // How the generated `func` returns: its values, and whether an error follows them. @@ -1368,7 +1357,16 @@ fn go(built: &Built, calls: &[Call], source: &str, rt: &str) -> String { } else { "lift" }; - format!("s.{item}{n}({seed}, {lift}(s.Digest{n}))") + // A callback takes the bytes Rust lends it as a `[]byte`, which `Digest` takes + // as an `rt.Bytes`. + let at = source.find(&format!("\nfunc {item}{n}(")).unwrap(); + let line = source[at + 1..].lines().next().unwrap(); + let adapt = match () { + _ if line.contains("func([]byte)") => "bytesIn", + _ if line.contains("func(*[]byte)") => "bytesOptIn", + _ => "", + }; + format!("s.{item}{n}({seed}, {adapt}({lift}(s.Digest{n})))") } Call::Made(_, seed) => { let lift = if results(&format!("Make{n}")).1 { @@ -1387,16 +1385,14 @@ fn go(built: &Built, calls: &[Call], source: &str, rt: &str) -> String { let _ = writeln!(body, "\tfmt.Println(\"{}\", {printed})", call.label()); } format!( - "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\ts \"{name}/{name}\"\n{import})\n\n\ + "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\ts \"{name}/{name}\"\n\t\"{name}/rt\"\n)\n\n\ func one[T any](v T, err ...error) T {{\n\tif len(err) > 0 && err[0] != nil {{\n\t\tpanic(err[0])\n\t}}\n\treturn v\n}}\n\n\ {tuples}func lift[A, B any](f func(A) B) func(A) B {{\n\treturn f\n}}\n\n\ func liftErr[A, B any](f func(A) (B, error)) func(A) B {{\n\treturn func(a A) B {{ return one(f(a)) }}\n}}\n\n\ + func bytesIn[B any](f func(rt.Bytes) B) func([]byte) B {{\n\treturn func(v []byte) B {{ return f(rt.NewBytes(v)) }}\n}}\n\n\ + func bytesOptIn[B any](f func(*rt.Bytes) B) func(*[]byte) B {{\n\treturn func(v *[]byte) B {{\n\t\tif v == nil {{\n\t\t\treturn f(nil)\n\t\t}}\n\t\tb := rt.NewBytes(*v)\n\t\treturn f(&b)\n\t}}\n}}\n\n\ func main() {{\n\tctx := context.Background()\n\t_ = ctx\n{body}}}\n", name = built.name(), - import = match tuples.is_empty() { - true => String::new(), - false => format!("\t\"{}/rt\"\n", built.name()), - }, ) } diff --git a/quest/m1/ffi-export/README.md b/quest/m1/ffi-export/README.md index 8fcbb2a87..747c5f307 100644 --- a/quest/m1/ffi-export/README.md +++ b/quest/m1/ffi-export/README.md @@ -83,8 +83,8 @@ Decided with the maintainer on 2026-11-12, in the quest audit: then waited on a rind release the maintainer held. - ✅ Every change that breaks the C ABI or a generated API is a child here, so it lands before moq pins: the fused async start, - [owned buffers](/quest/m1/ffi-export/owned-bufs.md), and - numeric slices ([#743](https://github.com/kixel-ai/rust2/pull/743)). Work that changes no generated API is + owned buffers ([kixel-ai/rust2#744](https://github.com/kixel-ai/rust2/pull/744)), and + numeric slices ([kixel-ai/rust2#743](https://github.com/kixel-ai/rust2/pull/743)). Work that changes no generated API is [its own epic](/quest/m1/ffi-speed/README.md), which does not hold the pin. - Not planned: one walker emitting lift and lower instructions every generator implements, as wit-bindgen's `Bindgen` does. A new rule touches 19 to 38 files today, diff --git a/quest/m1/ffi-export/owned-bufs.md b/quest/m1/ffi-export/owned-bufs.md deleted file mode 100644 index c86d5992d..000000000 --- a/quest/m1/ffi-export/owned-bufs.md +++ /dev/null @@ -1,40 +0,0 @@ -# [M] Owned buffers are taken, not copied - -## Goal - -A byte payload a [rust2-ffi](/ffi/README.md) binding passes to Rust is copied at most -once, and a payload Rust lent can be passed back with no copy: a buffer that arrives with -an owner is Rust's own and is taken, and `_buf_clone` gives a binding a second -owner of a lent one. Kotlin's and Dart's writes on -[the binding benchmark](/bench/ffi/README.md) lose an allocation and a copy each, and a -forwarder's read-then-write moves no payload bytes in any language. - -## Plan - -Decided with the maintainer on 2026-11-12, in the quest audit: ✅ a child of the exporter -epic, since it changes what a buffer's `owner` means on the way in. - -Today, in `ffi/src/glue.rs`'s `lift`, a `Bytes` argument is `Bytes::copy_from_slice` of the -buffer, and a `Vec` or `String` is a `to_vec` or `to_owned`, whatever the buffer's -owner. Kotlin's glue has already copied the `ByteArray` into a `Vec` by then -(`ffi/runtime/jni.rs`'s `bytes`), and Dart's copies into a buffer the call frees, so each -payload is copied twice. The header calls a buffer passed in "a view Rust reads during -the call and never frees", so its owner is never read. - -- A buffer passed in with a non-null owner is taken: its bytes become the `Bytes`, `Vec`, - or `String` with no copy, and the caller no longer owns it. A null owner still lends, - and Rust copies. -- The JNI and Dart glue build the buffer once in Rust and hand it over. -- `_buf_clone` bumps the owner's count, so a binding keeps a lent payload while - passing it on. -- A C caller that leaves garbage in `owner` breaks; the contract hash changes, so a stale - header fails at load. - -Prove it in every language's `api` test with a payload read and written back, checked for -identity of the storage where the language can see it, delete its Kotlin and Go `KNOWN` -entries in [the shape tests](/ffi/tests/fixtures/shapes.rs), which a returned payload -passed back now compiles in, and re-measure the bench's write -and throughput rows. - -Public API: the C ABI's meaning of an input buffer's owner (breaking), `_buf_clone`, -and each binding's way to pass a lent payload on. diff --git a/quest/m1/ffi-export/ship.md b/quest/m1/ffi-export/ship.md index 050bbaaa3..bbf1ca45a 100644 --- a/quest/m1/ffi-export/ship.md +++ b/quest/m1/ffi-export/ship.md @@ -25,4 +25,3 @@ then see change. A sibling planned later adds itself to `Required` here. - [Copy types cross by value](/quest/m1/ffi-export/copy-values.md) - how a `Copy` type is held in every language - [Not Send](/quest/m1/ffi-export/not-send.md) - the types and futures that export checked by thread - [Export moq for every target](/quest/m1/ffi-export/moq-targets.md) - one binding across the targets moq ships -- [Owned buffers](/quest/m1/ffi-export/owned-bufs.md) - what a buffer's owner means on the way in diff --git a/quest/m1/ffi-speed/bytes-write.md b/quest/m1/ffi-speed/bytes-write.md new file mode 100644 index 000000000..07b25e808 --- /dev/null +++ b/quest/m1/ffi-speed/bytes-write.md @@ -0,0 +1,21 @@ +# [S] Owned buffers cost a write nothing + +## Goal + +A 100 B write on [the binding benchmark](/bench/ffi/README.md) costs Swift and Dart no more +CPU time than it did before [kixel-ai/rust2#744](https://github.com/kixel-ai/rust2/pull/744) +made Swift's bytes `Bytes` and gave Dart's calls buffers Rust takes, measured against +`7008d1d3` at the same load. + +## Plan + +#744 measured the write four times against `7008d1d3` on a quiet host (load 2 to 3): Swift +179 against 173 ns and Dart 214 against 203, in every pair, while each copies a 100 B frame +as often as before. The generated Swift differs only in `Native.take(Bytes)` against +`Native.view(Data)`, so the cost is per call, not a copy. + +- Profile both writes (`perf record` on `target/bench-export-swift` and the Dart program) + against `7008d1d3`'s, and remove what the new path adds: in Swift, likely what `Bytes` + costs to pass and unwrap; in Dart, the `take` flag's checks, `took()`, and the `_given` + and `_viewed` bookkeeping per call. +- Re-measure the two rows interleaved with `7008d1d3`, and update the benchmark's tables.