Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ffi/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,9 @@
- Every export defines `<prefix>_contract` and `<PREFIX>_CONTRACT`, so an exported item
that the naming rule names either, such as a crate root's `pub fn contract` or
`pub const CONTRACT`, clashes with them: give it another name with a `rename`.
- A field, parameter, or variant named a C++ keyword, such as moq's `namespace`, takes
a leading `_` in C as a C keyword does (`_namespace`), so the header compiles as C++.
- The C++ header's runtime names (`view`, `result`, `task`, `future`, `poll_t`, and
`detail`) and every variant's payload name (`<enum>_<variant>`, even without fields)
join the names the naming rule claims, so an exported item named one clashes: give it
another name with a `rename`.
59 changes: 52 additions & 7 deletions ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ to what the library builds with on any platform, read off cargo's resolution, wh
it was declared (the manifest, cargo's config, `--config`, or the environment), keyed by
its source's URL.
A `[replace]` from the source it replaces, which no `[patch]` can carry, is an error.
Beside the C header, `include/<prefix>.hpp` holds the same API in C++.

```toml
# ffi.toml, beside the package's Cargo.toml unless --manifest-path says otherwise.
Expand Down Expand Up @@ -143,15 +144,55 @@ refused, and the census counts why.
out of that one; a `rename` of a method by the generic type's path names it at every
instance.

## C++

`<prefix>.hpp` is header-only C++ over the C header, for C++17, 20, and 23, with
exceptions and RTTI on or off: it throws nothing and reads no type information.

- **Names** are the C names without the prefix, in a namespace of it: `moq_track_producer`
is `moq::track_producer`, and its method `moq_track_producer_append_group` is
`moq::track_producer::append_group`. A keyword takes a trailing `_` (`counter::new_`),
as does a member named as its class or one of its fields.
- **Objects** are classes owning their handle, move-only unless `Clone`, whose copy
clones. `&self` methods are `const`, and `self` methods are `&&`, so a call that moves
the value out reads `std::move(c).into_label()`. A handle taken by value is moved into,
by a parameter taken by value.
- **Records** are structs, and **enums** an `enum class`, or for one with data a
`std::variant` of a struct per variant, `<enum>_<variant>`. An enum's methods and
traits are free functions taking it first. `PartialEq` is `==`, `Display` `to_string()`,
`Debug` `debug()`, and `Default` `default_()`.
- **Bytes** are passed as a `view`, which any contiguous bytes convert to, and returned as
a `buf`, which owns Rust's buffer without a copy. `Option<T>` is `std::optional<T>`, a
tuple `std::tuple`, and `Result<T, E>` a `result<T, E>`, whose `value()` aborts on an
error.
- **Traits** are abstract classes, passed as a `std::unique_ptr` that Rust deletes.
- **Closures** lent for a call are any callable, taken by a template parameter, which
Rust calls only during the call, from any thread. An object it lends is a reference to
its class for that call: moving out of it aborts.
- **Futures** are move-only `future<...>`s, freed when destroyed, which cancels them and
loses nothing. `poll(waker)` polls once; `std::move(f).get()` blocks the thread;
`std::move(f).then(exec, callback)` polls on an executor and calls back there, returning
a `task` that cancels when destroyed, after which the callback has finished or never
runs; and from C++20, `co_await std::move(f).via(exec)` resumes the coroutine on the
executor. An executor is any callable taking a `void()` job to run once, later, on any
thread: never inline, since a wake may come from inside Rust. Each wake that lands while
the future polls polls it again, and a future has one poller at a time, since awaiting
it moves it.
- A **waker** is a shared callable, which Rust may call from any thread.
- **The contract** is checked when the program loads, before any other call: a header
and a library from different builds abort.

## IR

[`src/ir.rs`](src/ir.rs) is the API as each generator reads it, with no language decided:
records, enums, objects, callbacks, functions, and constants of every export, each by
its Rust path, its span, and its docs. A type is in the exact form Rust takes or returns it
(`&str` apart from `String`, `Box<dyn Trait>` apart from `impl Trait`), so a generator can
borrow where Rust borrows. [`src/abi.rs`](src/abi.rs) decides the C ABI from it once,
names and signatures, so [`src/c.rs`](src/c.rs) and [`src/glue.rs`](src/glue.rs) agree;
the runtime the glue includes is [`runtime/rt.rs`](runtime/rt.rs).
names and signatures, so [`src/c.rs`](src/c.rs), [`src/cpp.rs`](src/cpp.rs), and
[`src/glue.rs`](src/glue.rs) agree; the runtime the glue includes is
[`runtime/rt.rs`](runtime/rt.rs), and the one the C++ header includes
[`runtime/rt.hpp`](runtime/rt.hpp).

## Census

Expand All @@ -164,20 +205,24 @@ rest, and the refused instances to drop from `instances`. It ignores the config'
## Tests

```bash
just ffi test # export the fixtures, build them, and drive api from C
just ffi moq # export moq's crates at the pinned revision, build them, and run moq/moq.c
just ffi test # export the fixtures, build them, and drive api from C and C++
just ffi moq # export moq's crates at the pinned revision, build them, and run moq/moq.c and moq/moq.cpp
just ffi census # census each into target/census
nix develop .#bench --command just ffi bench # frame writes, through the export and moq's rs/moq-c
nix develop .#bench --command just ffi bench # frame writes and reads, through the export from C and C++, and moq's rs/moq-c
```

The `api` fixture covers each rule once, and names the `dep` fixture's types, which
export into the same library. `tests/api.c` runs the same script as the
fixture's `examples/transcript.rs`, and the two transcripts must match line for line;
then it breaks each runtime check, which must abort with its message. The `reject`
then it breaks each runtime check, which must abort with its message. `tests/api.cpp`
runs the script through the C++ header at C++17, 20, and 23, with exceptions and RTTI off
at 17 and 23, then checks its futures: a wake that lands during a poll, a cancel between a
wake and its poll that loses nothing, and a coroutine resumed or destroyed. The `reject`
fixture's expected errors, each with its notes, are `fixtures/reject/expected.txt`.
moq's crates export through `target/moq-ffi`, an empty library on each that `just ffi moq`
writes in place of moq-ffi's remainder. `moq/moq.c` publishes and subscribes to a moq-net
track in one process, polling each read, scopes an origin to moq-pattern's patterns and
hands it to a client to publish, polls its driver until the origin's last owner drops, reads an OpusHead into hang's catalog through moq-mux,
publishes a binary track the catalog lists, and reads the catalog lent to C through
`catalog::Producer::mutate`.
`catalog::Producer::mutate`. `moq/moq.cpp` does the same through the C++ header, at
C++17 without exceptions or RTTI, as OBS builds it.
8 changes: 8 additions & 0 deletions ffi/fixtures/api/examples/transcript.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,14 @@ fn main() {
println!("debug {c:?}");
let label = c.into_label(Some(p));
println!("label {label:?}");
let empty = api_lib::Note {
text: Some(String::new()),
};
println!(
"note {} {}",
api_lib::has_text(empty),
api_lib::has_text(api_lib::Note::default())
);

let (below, above) = Counter::new("s", 7).split(5);
println!(
Expand Down
26 changes: 26 additions & 0 deletions ffi/fixtures/api/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,17 @@ pub struct Label {
pub ttl: Option<Duration>,
}

/// A note whose text may be present but empty, which crosses as itself, not as none.
#[derive(Clone, Debug, Default, PartialEq)]
pub struct Note {
pub text: Option<String>,
}

/// Whether `note` has text, even empty.
pub fn has_text(note: Note) -> bool {
note.text.is_some()
}

/// A plain enumeration.
#[derive(Clone, Copy, Debug, PartialEq)]
pub enum Shape {
Expand Down Expand Up @@ -354,6 +365,21 @@ impl Channel {
}
}

/// `value`, on its second poll: the first wakes its waker before it returns pending, so
/// the wake lands before the poller waits for one.
pub async fn yielded(value: u32) -> u32 {
let mut woken = false;
std::future::poll_fn(|cx| match std::mem::replace(&mut woken, true) {
true => Poll::Ready(()),
false => {
cx.waker().wake_by_ref();
Poll::Pending
}
})
.await;
value
}

/// A time: an object, since its field is private, but `Copy`, so passing it by value
/// copies it rather than moving it out of its handle.
#[derive(Clone, Copy, Debug, PartialEq)]
Expand Down
9 changes: 7 additions & 2 deletions ffi/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ run *args:
census checkout="": (standin checkout)
cargo run --locked --quiet -- --config moq/ffi.toml --manifest-path target/moq-ffi/Cargo.toml --census target/census

# Time frame writes through the export and through moq's hand-written rs/moq-c, in the bench shell: nix develop .#bench.
# Time frame writes through the export, from C and C++, and through moq's hand-written rs/moq-c, in the bench shell: nix develop .#bench.
bench checkout="": (moq checkout)
#!/usr/bin/env bash
set -euo pipefail
Expand All @@ -50,12 +50,14 @@ bench checkout="": (moq checkout)
| jq -r 'select(.reason == "build-script-executed" and (.package_id | contains("moq-c#"))) | .out_dir')
cc -O2 -std=c11 -Wall -Wextra -Werror -I "$out/include" moq/bench/export.c "$out/target/release/libmoq.a" -lpthread -ldl -lm -o target/bench-export
cc -O2 -std=c11 -Wall -Wextra -Werror -I "$moqc/include" moq/bench/moq-c.c "$src/target/release/libmoq.a" -lpthread -ldl -lm -o target/bench-moq-c
c++ -O2 -std=c++17 -Wall -Wextra -Werror -I "$out/include" moq/bench/export.cpp "$out/target/release/libmoq.a" -lpthread -ldl -lm -o target/bench-export-cpp
for _ in 1 2 3; do
target/bench-moq-c
target/bench-export
target/bench-export-cpp
done

# Export moq's crates' C ABI, build it, and run moq/moq.c against it, from the moq revision `just ts moq` pins or a checkout.
# Export moq's crates' C ABI, build it, and run moq/moq.c and moq/moq.cpp against it, from the moq revision `just ts moq` pins or a checkout.
moq checkout="": (standin checkout)
#!/usr/bin/env bash
set -euo pipefail
Expand All @@ -66,6 +68,9 @@ moq checkout="": (standin checkout)
cargo build --quiet --release --manifest-path "$out/Cargo.toml"
cc -std=c11 -pedantic -Wall -Wextra -Werror -I "$out/include" moq/moq.c "$out/target/release/libmoq.a" -lpthread -ldl -lm -o target/moq-test
target/moq-test
# The C++ header at OBS's standard, without exceptions or RTTI.
c++ -std=c++17 -fno-exceptions -fno-rtti -pedantic -Wall -Wextra -Werror -I "$out/include" moq/moq.cpp "$out/target/release/libmoq.a" -lpthread -ldl -lm -o target/moq-cpp-test
target/moq-cpp-test

# Write target/moq-ffi, an empty library on every crate moq/ffi.toml exports, from the moq revision `just ts moq` pins or a checkout: the check builds with all of them, as moq-ffi's remainder will.
[private]
Expand Down
49 changes: 45 additions & 4 deletions ffi/moq/bench/export.c
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/* The frame write path through the exported C ABI: a timed track, groups of 100 frames of
* 100 bytes, each frame stamped. bench/moq-c.c times the same through moq's rs/moq-c. */
* 100 bytes, each frame stamped. bench/moq-c.c times the same through moq's rs/moq-c, and
* bench/export.cpp through the C++ header. Then the read path, each read a future. */

#define _POSIX_C_SOURCE 199309L

Expand Down Expand Up @@ -78,7 +79,6 @@ int main(void) {
moq_group_producer_free(group);
}
double reused = (now() - start) / (GROUPS * FRAMES);
moq_timestamp_free(at);

/* The floor: a borrow of a handle and a bool back. */
start = now();
Expand All @@ -88,8 +88,49 @@ int main(void) {
}
double call = (now() - start) / (GROUPS * FRAMES);

printf("export: %.0f ns per stamped 100 B frame, %.0f reusing the timestamp, %.1f ns per handle call%s\n", stamped, reused,
call, closed ? "!" : "");
/* Reads: each frame written, then read through a new future polled once, ready. */
moq_waker waker = {0};
moq_track_consumer *consumer = moq_track_producer_consume(track);
moq_track_consumer_subscribe_future *subscribing = moq_track_consumer_subscribe(consumer, NULL);
moq_track_subscriber *subscriber = NULL;
if (moq_track_consumer_subscribe_poll(subscribing, waker, &subscriber, &err) != MOQ_POLL_READY) {
return 4;
}
moq_track_consumer_subscribe_future_free(subscribing);
start = now();
for (int g = 0; g < GROUPS; g++) {
moq_group_producer *group = NULL;
moq_track_producer_append_group(track, &group, &err);
moq_track_subscriber_recv_group_future *receiving = moq_track_subscriber_recv_group(subscriber);
moq_group_consumer *reader = NULL;
if (moq_track_subscriber_recv_group_poll(receiving, waker, &reader, &err) != MOQ_POLL_READY || reader == NULL) {
return 5;
}
moq_track_subscriber_recv_group_future_free(receiving);
for (int f = 0; f < FRAMES; f++) {
if (!moq_group_producer_write_frame(group, at, frame, &err)) {
return 6;
}
moq_group_consumer_read_frame_future *reading = moq_group_consumer_read_frame(reader);
moq_frame_frame *read = NULL;
if (moq_group_consumer_read_frame_poll(reading, waker, &read, &err) != MOQ_POLL_READY || read == NULL) {
return 7;
}
moq_group_consumer_read_frame_future_free(reading);
moq_frame_frame_free(read);
}
moq_group_producer_finish(group, &err);
moq_group_producer_free(group);
moq_group_consumer_free(reader);
}
double read = (now() - start) / (GROUPS * FRAMES);
moq_timestamp_free(at);
moq_track_subscriber_free(subscriber);
moq_track_consumer_free(consumer);

printf("export: %.0f ns per stamped 100 B frame, %.0f reusing the timestamp, %.1f ns per handle call, %.0f ns per frame "
"written and read%s\n",
stamped, reused, call, read, closed ? "!" : "");
moq_track_producer_free(track);
moq_timescale_free(micro);
moq_broadcast_producer_free(broadcast);
Expand Down
103 changes: 103 additions & 0 deletions ffi/moq/bench/export.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
// bench/export.c's frame writes through the generated C++ header, so the two differ by
// what the C++ wrapper adds; then the read path, each read a future polled once ready.

#include <chrono>
#include <cstdio>

#include "moq.hpp"

namespace {

constexpr int groups = 10000;
constexpr int frames = 100;

// Nanoseconds per frame since `start`.
double per_frame(std::chrono::steady_clock::time_point start) {
std::chrono::duration<double, std::nano> elapsed = std::chrono::steady_clock::now() - start;
return elapsed.count() / (groups * frames);
}

} // namespace

int main() {
moq::broadcast_producer broadcast = moq::broadcast_info::new_().produce();
moq::timescale micro = moq::timescale::micro();
auto track = broadcast.create_track("bench", moq::track_info::default_().with_timescale(&micro));
if (!track) {
return 1;
}

std::uint8_t payload[100] = {0};
moq::view frame(payload, sizeof payload);
std::uint64_t micros = 0;

// Each frame stamped through a fresh timestamp handle, as a caller holding a u64 must.
auto start = std::chrono::steady_clock::now();
for (int g = 0; g < groups; g++) {
moq::group_producer group = std::move(*track->append_group());
for (int f = 0; f < frames; f++) {
moq::timestamp at = std::move(*moq::timestamp::from_micros(micros++));
if (!group.write_frame(&at, frame)) {
return 2;
}
}
(void)std::move(group).finish();
}
double stamped = per_frame(start);

// The same, reusing one timestamp handle: what the handle costs.
moq::timestamp at = std::move(*moq::timestamp::from_micros(micros));
start = std::chrono::steady_clock::now();
for (int g = 0; g < groups; g++) {
moq::group_producer group = std::move(*track->append_group());
for (int f = 0; f < frames; f++) {
if (!group.write_frame(&at, frame)) {
return 3;
}
}
(void)std::move(group).finish();
}
double reused = per_frame(start);

// The floor: a borrow of a handle and a bool back.
start = std::chrono::steady_clock::now();
int closed = 0;
for (int i = 0; i < groups * frames; i++) {
closed += track->is_closed();
}
double call = per_frame(start);

// Reads: each frame written, then read through a new future polled once, ready.
moq::waker waker([]() {});
moq::track_consumer consumer = track->consume();
auto subscribed = consumer.subscribe(std::nullopt).poll(waker);
if (!subscribed || !*subscribed) {
return 4;
}
moq::track_subscriber subscriber = std::move(**subscribed);
start = std::chrono::steady_clock::now();
for (int g = 0; g < groups; g++) {
moq::group_producer group = std::move(*track->append_group());
auto received = subscriber.recv_group().poll(waker);
if (!received || !*received || !received->value()) {
return 5;
}
moq::group_consumer reader = std::move(*received->value());
for (int f = 0; f < frames; f++) {
if (!group.write_frame(&at, frame)) {
return 6;
}
auto read = reader.read_frame().poll(waker);
if (!read || !*read || !read->value()) {
return 7;
}
}
(void)std::move(group).finish();
}
double read = per_frame(start);

std::printf(
"export (C++): %.0f ns per stamped 100 B frame, %.0f reusing the timestamp, %.1f ns per handle call, %.0f ns per frame written and read%s\n",
stamped, reused, call, read, closed ? "!" : "");
return 0;
}
Loading