Registry Metadata

[esoterra]

There are two related questions about registry metadata we need to answer:

1. What kind of metadata do we need to store for packages and/or releases? (e.g. author, license, website, repo)
2. What format should metadata be encoded in? (e.g. JSON, proto)

[esoterra]

In my opinion, the format/encoding we choose needs to

1. be able to support common types like strings, numbers, dates, etc.,
2. be able to be extended easily by adding extra fields or attributes, and
3. be easily encoded/decoded on a variety of platforms and languages.

JSON and protobuf can both meet these requirements.

• JSON advantages: easy to read, easy to write, tons of support, does not require codegen or out-of-band schema.
• Protobuf advantage: more compact, represents byte data without base64.

[lann]

One of the main reasons we're using protobuf for the signed payloads is that we are storing them as encoded by the user to sidestep any encoding canonicalization problems wrt signing.

[kriswuollett]

Even for an end-user, IMO the serialization should be stable; otherwise, if they update their toolchain, they might not be able to reproduce a use case with the same output given some content and metadata. Similarly, is there a use case to detect or search for duplicate content or metadata in the warg system? If so, stable JSON where needed is great.

Additionally, as much as I like using Protobufs, I'd think one wouldn't want to work with them if they are creating a warg client/server in WASM to be run in a place like a browser, since it would require pulling in additional dependency and complicate the build process.

[lann]

Even for an end-user, IMO the serialization should be stable; otherwise, if they update their toolchain, they might not be able to reproduce a use case with the same output given some content and metadata.

I don't understand the point here. Could you give an example?

[kriswuollett]

I do not have a specific example because I am not familiar with the contents of the signed payload that is referred to in #116 (reply in thread). But in general examples would be nearly any case where it is structured data matching a schema owned by warg, for the API, etc., on a serialized version of which a hash is applied. Content that this does not need to apply to is opaque content like compiled code, which may be pointed to by a hash itself instead of being included directly.

Just suggesting something to look into if it affects tests/logs reproducibility due to something like upgrading/changing Protobuf implementation, e.g., integration tests would break if checking for the hash in them.

[lann]

The signed payloads are part of the registry log. Based on the discussion in the other thread, it doesn't sound like we'll be storing package metadata there directly.

It's true that stable encoding is convenient, but I think at this point what we need to decide is whether a particular stable encoding should be required by the protocol and enforced by the server. Certainly any client would be free to use a stable e.g. JSON encoder.

[lann]

Where will the metadata be stored?

[esoterra]

Do you have any thoughts on the larger REST/JSON-first vs. gRPC/proto-first discussion?

[silesmo]

In my view there are two requirements which dictate where metadata can/should be stored.

1. Metadata should be verifiable, which means its hash must be included directly or transitively in the release entry.

2. Metadata contains arbitrary user-controlled data (with the potential to include PII) therefore it cannot exist directly in the log itself.

So, I think as a result metadata should be stored separately from the logs as content with its hash included in the log. This could be stored using the existing content storage mechanisms and/or in a different way.

How will this work in practice if you make an edit to a components description in the metadata. Would this impact the resolution of dependencies somehow if you have the same component included in different modules in the component that just have a diff in the metadata? Would the resolution in that case think of them as two different components?

[silesmo]

If metadata is verifiable do you mean all the defined fields are verifiable? Or all the defined and custom fields? @Kylebrown9

[lann]

Dependency information will be embedded in components. I don't know of any dependency metadata that will need to be external.

[esoterra]

There are some examples in the other thread of how different metadata is handled. In general, I also don't expect you to be able to "edit" metadata of prior releases. To my knowledge that's not possible with existing systems (e.g. cargo) anyway. If by edit you just mean to have a different value in the metadata than in the component the general approach is to avoid having the same info in two places.

The idea here is that the metadata as a whole (so including all fields) would be hashed and that hash included in the verifiable data structures.

[esoterra]

In terms of the fields we need to store, it would make sense to include some common subset of typical Cargo.toml, package.json, etc. metadata at the top level for all packages and allow for other fields to be added ad-hoc.

This suggests some questions to explore like

1. What existing systems (e.g. cargo, npm) metadata should we consider?
2. What data is common to all or most of them?
3. What data is necessary/useful to have about all packages globally when looking at an applications dependencies? e.g. licenses

[esoterra]

I'm open to something like that. If we do that, we should pick standard names for common fields like the examples you gave so people don't inconsistently use e.g. "repo"/"repository", "docs"/"documentation", ...

[Scroul]

I think a runtimes filed could be nice to have ex "runtimes": { "wasmtime": " 8.0.0" }.

I see it as a this is known to run as expected in the runtimes listed here at the versions listed here.

[esoterra]

I think if we need to mark our components with what versions of what runtimes can run them, other parts of the component model design have failed. You should be able to tell whether you can run a component entirely based on its interface/world and the version of the component model it specifies in its binary. As runtimes become able to support the component model and new versions of it those runtimes will become able to run components they didn't use to.

i.e. Runtimes should say "I support component-model preview2 and the http-proxy world" instead of packages saying "I support wasmtime"

[esoterra]

There are also going to be many host platforms e.g. compute@edge (Fastly), spin platform (Fermyon), ... that all use Wasmtime and could (but hopefully won't) have functional differences in the host-bindings/implementation they provide to components. These kinds of host platform differences are probably more likely to accidentally introduce some differences (e.g. one preserves header order the other doesn't) that cause components to run differently than the runtimes they use internally are. If we have a bunch of runtimes and some of them can't correctly run components that's a clearer/bigger issue.

[silesmo]

To summarize the above and add a few new questions, here's my thoughts on each of the above.

* name - do not duplicate in metadata, already constrained/defined by the verifiable log and map

* version - do not duplicate in metadata, already included in the package log release entry

* authors, description, documentation, homepage, code repo - all good/easy inclusions validating URLs would be nice

* license - not too familiar with SPDX but happy to use some standardized identifiers

* funding/donations - not clear on what this means, do you have examples/ideas?

* keywords/categories - sounds good

* dependencies - already encoded in component binary as imports with identifiers

* scripts/commands - I don't think we need this

When we get to adding search in the future how would this work in practice? I would have assumed the search would be based on the metadata to some degree and if information like the name isn't stored there, then we would need to fetch information from multiple sources, then stitch it back together? Wouldn't you in practice copy that information to the metadata store when you upload a package anyway?

[lann]

Cargo's approach to license metadata seems good to me: https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields

Note: crates.io requires either license or license-file to be set.

[esoterra]

In today's meeting we talked about making license optional. The main use cases being forwards compatibility and unlicensed e.g. internal software. Do you think those justify making the license field optional?

[lann]

I think it should be optional in the specification and probably required by any public-good registry.

[silesmo]

Metadata

To summarize what we have discussed above.

Place the content in a WebAssembly custom section called registry-metadata that would contain the following information in JSON:

• authors: List
• Description: Markdown blob
• License: SPDX
• Links: Key, Value
  • Where we have predefined meanings for the following keys:
  • documentation - link to documentation
  • homepage - link to projects homepage
  • repository - link to projects source repository
  • funding - link for users of the component to donate to the project
• Categories: List of strings

Do we want to restrict the categories somehow to not allow anything to be put into the category?

Do we want to use any compression algorithm on the metadata? If so which one Gzip, Brotli, zstd, LZ4?

I was planning on creating a library that can read and add the registry metadata to the wasm-tools repository similar to what has been done with the wasm-metadata crate. Is this what we want?

[lann]

Do we want to use any compression algorithm on the metadata?

[per late discussion in meeting] In general I think its better to leave this up to the transport, i.e. normal HTTP negotiation.

[lann]

Do we want to restrict the categories somehow to not allow anything to be put into the category?

I would think that this would be a per-registry policy, though (like for Links) we can certainly propose a default set.

[lann]

I was planning on creating a library that can read and add the registry metadata to the wasm-tools repository similar to what has been done with the wasm-metadata crate. Is this what we want?

I think the rough consensus from last week's meeting was that this doesn't belong in wasm-tools (yet, at least). A separate crate in the registry repo would make sense to me.

[silesmo]

Do we want to use any compression algorithm on the metadata?

[per late discussion in meeting] In general I think its better to leave this up to the transport, i.e. normal HTTP negotiation.

Yes I agree. Just wanted to validate that this was what everyone else was thinking as well.