feat(format): add generic metadata API

[lidavidm]

• Add a generic API for fetching metadata, modeled on ADO.NET's GetSchema
• Define the standard metadata collections
• Revert the current revisions to GetObjects in favor of not changing it

Related:

• format: provide way to escape wildcards in GetObjects and other functions #3997: define a way to escape search pattern arguments
• discussion: ConnectionGetObjects vs ConnectionGetTableSchema #1704 (comment): define a way to provide pagination tokens

For consideration:

• Should filters be (optionally) named arguments? (Pointer to struct, or assuming that every filter is preceded by a pointer to a name string.) The justification: we could more easily implement/define "standard" arguments to handle things like the pagination token above without having to replicate it into every single definition. If we only have positional arguments, it starts to get annoying to handle.
• Should filters be typed? (Pointer to struct containing union)
• Should filters be replaced by just leveraging SetOption(Int,...)? (This would obviate the former two points but is rather clunky, and precludes a hypothetical driver that could handle these operations concurrently)

Closes #4400.

[lidavidm]

Also CC @iconara

I've also been thinking about your request for stateless pagination, and I think it's reasonable enough to define a way to get/pass a pagination token. I think there are enough systems that could use it: Athena, BigQuery, Databricks, Snowflake, Iceberg REST catalog, etc.

[lidavidm]

And also CC @mullinsms, who kicked all this off by listing all the types of metadata we didn't support 🙂

In particular, Curt's suggestion means we can support database-specific metadata (though, I'm still of the opinion that something truly database-specific is probably better handled by the application; if the application has a very specific metadata query it wants to issue I'm not sure how useful it is to build it into the driver).

[lidavidm]

At the risk of overcomplicating things: similar to #3623, it might be nice to have a way to request that certain extra fields be included/omitted, e.g. table properties (#3995), where it may be efficient to fetch the data at the same time as the "standard" fields, but where some (many) clients also may not want the field. Maybe it could optionally take an Arrow schema as input for that.

Similarly, maybe the application wants to opt in to run-length-encoding certain response columns to save memory. (Or is that not really a concern so long as things are properly streamed/paginated?)

[lidavidm]

I think I'm leaning towards having all options/filters be set by SetOption. This is perhaps inconvenient for C/C++ users, but language-level bindings can present higher level APIs, and allows us to express better type safety. This would also be consistent with my suggestion in #4317.

Either way, the current declaration needs to clarify the lifetime of the filter argument anyways (the driver should copy arguments as it is not allowed to assume filters will be valid during the returned record reader's lifetime).

I thought about having a SetOptionStringList. This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts, and I would rather avoid trying to encode strings into a single string (via e.g. JSON). But maybe we can embrace Curt's suggestions and lean on Parquet Variant for encoding these sorts of complex-type arguments. That said, I fear I'm reinventing COM or some sort of intraprocess RPC mechanism...

[paleolimbot]

I think I'm leaning towards having all options/filters be set by SetOption.

This is kind of ugly but I see what you're getting at. GDAL handles this with const char** options where options is NULL terminated list of KEY=arbitrary value, with the downside that everything is serialized. If this is a get/set option on a connection is seems more problematic (on a statement it's more clear that options have a specific scope).

This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts

It's a hack, but some REST APIs use the unit separator (\x1f) or have a user-specified separator and just have "schemas" with potentially multiple parts.

I'm guessing you don't want to go this direction, but I can't help but notice there's a large amount of complexity associated with stuffing these concepts into Arrow arrays that are highly nested and very difficult to parse. I'll throw out that you could do something like

struct AdbcCatalogNode {
  AdbcStatus (*get_property)(struct AdbcCatalogNode* self, const char* what, struct ArrowSchema* out_schema, struct ArrowArray* out_array, AdbcError* err);
  AdbcStatus (*get_child)(struct AdbcCatalogNode* self, const char* what, AdbcError* err);
  void* private_data;
  void* private_driver;
}

AdbcStatus AdbcConnectionGetCatalogs(struct AdbcConnection* connection, const char** options, struct AdbcCatalogNode* out, AdbcError* err);

Not perfect, but maybe lets some of this complexity get pushed onto the driver instead of on the consumer since drivers might have abstractions for some of this already.