Revise Serialized Model Structure for Standardization

[Gnpd]

Review and refine the structure of the serialized model to align with possible existing serialization standards. The current format includes fields such as estimator_class, attributes, params,... and metadata like producer_version. Explore established standards (e.g., ONNX, PMML) to ensure compatibility and extensibility. Propose a revised structure that balances clarity, completeness, and interoperability.

Current format

serialized_estimator = {
    "estimator_class": model.__class__.__name__,  #str
    "attributes": serializable_attributes, #dict  loadings: [1,2,3]
    "attribute_types": attribute_types, #dict ndarray
    "attribute_dtypes": attribute_dtypes, #dict float64
    "params": self.convert_to_serializable(params), #dict
    "param_types": param_types, #dict
    "param_dtypes": param_dtypes, #dict
    "producer_version": getattr(model, "_sklearn_version", None),
    "producer_name": model.__module__.split(".")[0],
    "domain": "sklearn",
}

[paucablop]

@Gnpd I have been playing with it over the last few days, and I think the format is very nice. I am a bit uncertain about the difference between the producer_name and the domain. Is the domain the general framework (e.g., sklearn) and the producer_name any library that implements the same API as the domain (e.g., could be sklearn itself but also chemotools?

[Gnpd]

You're right, the idea is that producer_name refers to the specific library or package the model comes from (for example, sklearn for pure scikit-learn estimators, or chemotools or any other compatible library).

domain always refers to the general framework or API that the model follows, here it’s always sklearn, since we’re using the SklearnSerializer.
From the latest update, we have:

"producer_version": sklearn.__version__,  # instead of getattr(model, "_sklearn_version", None)

but not sure what is best...

I also realized that we’re not currently covering the version of the producer_name, which might be something to include later.

But this is still under discussion, so it’s good to have different perspectives before making a final decision.

[Gnpd]

Addressed in docs/format.md

Current format

serialized_estimator = {
      "estimator_class": model.__class__.__name__,  # str
      "params": self.convert_to_serializable(params),  # dict
      "param_types": param_types,  # dict
      "param_dtypes": param_dtypes,  # dict
      "producer_version": sklearn.__version__,  # str, e.g. "1.9.0"
      "producer_name": model.__module__.split(".")[0],  # str, e.g. "sklearn"
      "domain": "sklearn",  # str
      "openmodels_format_version": OPENMODELS_FORMAT_VERSION,  # int, e.g. 1
      "openmodels_version": _openmodels_version(),  # str, e.g. "0.1.0"
  }
  # ... attributes only added once the model is fitted:
  serialized_estimator = {
      **serialized_estimator,
      "attributes": serializable_attributes,  # dict  coef_: [1, 2, 3]
      "attribute_types": attribute_types,  # dict  ndarray
      "attribute_dtypes": attribute_dtypes,  # dict  float64
  }

Leaving this open for further structural suggestions.

[Gnpd]

Opened #57 as a first step on V2 format: the wire format now has an explicit version (openmodels_format_version, currently 2), and the previously scattered bookkeeping fields (producer_version, producer_name, domain, etc.) are grouped under a single root-level metadata object instead of being flat and duplicated on every nested estimator in a composite (Pipeline, VotingClassifier, ...). metadata also now supports optional provenance fields (title, description, author, license, metrics) and new autofilled ones (created_at, dependency_versions, producers - per-package versions across the whole estimator tree, not just the outermost one).

See details here: https://github.com/Gnpd/openmodels/blob/format-2/docs/format.md