Release 2.1.1: description refresh and backend-assigned entity IDs

.github/workflows/reusable-lint-build-test.yml

@@ -126,6 +126,9 @@ jobs:
           FORCE_COLOR: 0
           TERM: dumb
 
+      - name: Validate solution IDs and model
+        run: uv run --frozen datam8 validate --solution datam8-sample-solution/ORAYLISDatabricksSample.dm8s --json
+
       - name: Upload Test Artifacts
         if: always()
         uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7

docs/backend-contract.md

@@ -91,6 +91,19 @@ editor. Runtime configuration changes and process failure may require a restart.
 
 ## Entity and function invariants
 
+- Model and folder entity IDs are assigned centrally by the backend on create,
+  clone, source import and migration, uniformly in `1..9007199254740991`,
+  independently of the solution's highest ID. Implicit folders use the same
+  allocator. New IDs avoid IDs already used by either entity type.
+  Create ignores a supplied ID; the returned wrapper's `entity.id` is authoritative.
+  Updates may include the unchanged ID but cannot change it. Rename/move preserve it.
+  Loading (including lazy load/reload) and saving reject duplicate IDs within each
+  entity type and IDs outside that range. Duplicate errors identify the ID and
+  affected locators. Existing valid IDs are retained, including historical overlap
+  between model and folder IDs. Solution repositories must run
+  `datam8 validate --solution <solution.dm8s> --json` as a required merge check on
+  their merged solution state; independent branches cannot reserve each other's IDs.
+  Multiple creates use the regular entity endpoint followed by `/model/save`.
 - `PUT /entities/{locator}` accepts an optional `sourceFile` query parameter for
   Attribute Types, Data Types, Data Sources, Data Source Types, Data Products,
   Zones, Properties and Property Values. It selects the existing collection JSON
@@ -206,6 +219,11 @@ editor. Runtime configuration changes and process failure may require a restart.
   before the model is changed. Source-only refresh does not read attribute metadata.
 - Returning `None` keeps the default one-source, one-column mapping derived from
   `get_table_metadata()`.
+- Complete refresh adopts the entity description from source metadata. Identical
+  descriptions from multiple sources are accepted; missing descriptions are ignored
+  when another source supplies one. Conflicting descriptions are rejected. If all
+  external sources have no description, the existing description is cleared.
+  Entities without external sources and source-only refresh retain their description.
 - Complete refresh updates source-owned attribute metadata (data type, description,
   business-key flag and supplied properties), while preserving local expressions,
   display names, history, units, refactor names and deletion markers. Existing

docs/releases/2.1.1.md

@@ -0,0 +1,44 @@
+# DataM8 CLI 2.1.1
+
+This patch release fixes entity description refresh and centralizes model and folder
+entity ID allocation in the backend.
+
+## Entity description refresh
+
+- Complete refresh updates the entity description, including clearing a removed
+  description. Sources-only refresh and entities without external sources retain
+  their existing description. Multiple sources may supply the same description;
+  conflicting descriptions are rejected instead of selecting one arbitrarily.
+
+## Backend-assigned entity IDs
+
+- New model and folder entities receive random numeric IDs in the JavaScript-safe
+  range from one backend allocator. Create, clone, source import, migration and
+  implicit folders avoid IDs already used by either entity type.
+- Existing valid IDs remain unchanged on update, rename and move. Loading and
+  saving reject IDs outside the supported range and duplicates within each entity
+  type, with the affected locators in the error. Historical overlap between model
+  and folder IDs remains supported.
+- Create ignores supplied IDs; clients must use the returned entity ID. Multiple
+  creates continue to use the existing entity endpoint followed by one model save.
+
+## Other changes since 2.1.0
+
+- Include the dependency and schema security corrections already merged into main,
+  and avoid logging secret references and registry contents.
+- Clarify installation from prebuilt release packages.
+
+## Compatibility and release
+
+The solution schema version remains **2.0.0**. Existing plugin hooks and source
+location handling are unchanged. Use the updated backend package or submodule
+revision to receive these changes. Clients creating model or folder entities must
+retain the IDs returned by the backend. Existing solutions with duplicate or
+out-of-range IDs must be corrected before they can be loaded or saved; valid
+existing IDs are not renumbered.
+
+After merging the release PR into main, push the protected tag `v2.1.1` at the merged
+commit. The release workflow verifies the version, runs CI and publishes the tested
+wheel and source archive to GitHub Releases. It does not publish to PyPI.
+
+See the [backend contract](../backend-contract.md) and [release process](../ci.md).

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "datam8"
-version = "2.1.0"
+version = "2.1.1"
 description = "ORAYLIS DataM8 CLI"
 license = "GPL-3.0"
 requires-python = ">=3.12"

src/datam8/factory.py

@@ -221,7 +221,7 @@ def create_undefined_folders(_model: model.Model, /):
     Goes through all model entities and adds folder wrappers for every parent locator
     of an entity that does not exist yet.
 
-    Folder IDs are simply sequential increased during runtime.
+    Folder IDs use the model's central ID allocator.
 
     Parameters
     ----------
@@ -230,15 +230,12 @@ def create_undefined_folders(_model: model.Model, /):
     """
     logger.debug("Create undefined folders")
 
-    existing_folder_ids = [w.entity.id for w in _model.folders.values()]
-    next_id = (max(existing_folder_ids) if len(existing_folder_ids) != 0 else 0) + 1
-
     undefined_folders: model.EntityDict[f.Folder] = {}
 
     for wrapper in _model.get_entity_iterator():
         loc = wrapper.locator
         for ploc in loc.parents:
-            if ploc in _model.folders or ploc.entityName is None:
+            if ploc in _model.folders or ploc in undefined_folders or ploc.entityName is None:
                 continue
 
             source_file_path = pathlib.Path(
@@ -254,13 +251,11 @@ def create_undefined_folders(_model: model.Model, /):
                 source_file=source_file_path.absolute(),
                 locator=ploc,
                 entity=f.Folder(
-                    id=next_id,
+                    id=_model.get_next_entity_id(),
                     name=ploc.entityName,
                 ),
             )
 
-            next_id += 1
-
     for loc in undefined_folders:
         if loc not in _model.folders:
             _model.folders[loc] = undefined_folders[loc]

src/datam8/migration_v1.py

@@ -7,6 +7,7 @@
 from typing import Any, overload
 
 from datam8 import parser_v1, utils
+from datam8.utils.ids import generate_entity_id
 from datam8_model import attribute as a
 from datam8_model import base as b
 from datam8_model import data_product as dp
@@ -780,14 +781,14 @@ def create_new_properties(tags: Tags, output_path: pathlib.Path) -> None:
         with open(output_path / "property_values.json", "w", encoding="utf-8") as file:
             file.write(new_property_values.model_dump_json(**MODEL_DUMP_OPTIONS))
 
-    @staticmethod
     def create_properties_for_folders(
+        self,
         zones: Sequence[z.Zone],
         data_products: Sequence[dp.DataProduct],
         output_path: pathlib.Path,
     ) -> None:
         """Should only be called AFTER other models have been migrated. No new folders will be created."""
-        next_id = 1
+        used_ids = {ref.id for ref in self.model_file_references.values()}
 
         for zone in zones:
             zone_path = output_path / zone.name
@@ -801,9 +802,14 @@ def create_properties_for_folders(
 
                 product_folder = b.Folders(
                     type="folders",
-                    folders=[f.Folder(id=next_id, name=product.name, dataProduct=product.name)],
+                    folders=[
+                        f.Folder(
+                            id=generate_entity_id(used_ids),
+                            name=product.name,
+                            dataProduct=product.name,
+                        )
+                    ],
                 )
-                next_id += 1
 
                 with open(product_path / ".properties.json", "w", encoding="utf-8") as _file:
                     _file.write(product_folder.model_dump_json(**MODEL_DUMP_OPTIONS))
@@ -817,14 +823,13 @@ def create_properties_for_folders(
                         type="folders",
                         folders=[
                             f.Folder(
-                                id=next_id,
+                                id=generate_entity_id(used_ids),
                                 name=module.name,
                                 dataProduct=product.name,
                                 dataModule=module.name,
                             )
                         ],
                     )
-                    next_id += 1
 
                     with open(module_path / ".properties.json", "w", encoding="utf-8") as _file:
                         _file.write(module_folder.model_dump_json(**MODEL_DUMP_OPTIONS))

src/datam8/model/entity_wrapper.py

@@ -27,6 +27,7 @@
 from pydantic import BaseModel, ConfigDict, ValidationError
 
 from datam8 import errors, logging, utils
+from datam8.utils.ids import validate_entity_id
 from datam8_model import attribute as a
 from datam8_model import base as b
 from datam8_model import data_product as dp
@@ -458,6 +459,13 @@ def has_property(self, property_name: str, /) -> bool:
         return False
 
     def update(self, **kwargs: Any) -> None:
+        if isinstance(self.entity, (m.ModelEntity, f.Folder)) and "id" in kwargs:
+            validate_entity_id(kwargs["id"], locator=str(self.locator))
+            if kwargs["id"] != self.entity.id:
+                raise errors.Datam8ValidationError(
+                    message=f"Entity ID cannot be changed at {self.locator}",
+                    details={"id": self.entity.id, "requestedId": kwargs["id"]},
+                )
         new_entity = self.entity.model_copy(update=kwargs, deep=True)
 
         try:

src/datam8/model/model.py

@@ -26,6 +26,7 @@
 from pydantic import ConfigDict, Field, ValidationError
 
 from datam8 import config, errors, functions, logging, opts, utils
+from datam8.utils.ids import generate_entity_id, validate_entity_id
 from datam8_model import attribute as a
 from datam8_model import base as b
 from datam8_model import data_product as dp
@@ -146,21 +147,42 @@ def __init__(self, solution: s.Solution, /, **kwargs: EntityRepository[b.BaseEnt
         self._model_files: dict[Path, EntityFileRef] = {}
         """Internal dictionary to allow easy mapping of files and their entities"""
 
-        if len(self.modelEntities) == 0:
-            self.__next_model_id = 1
-        else:
-            self.__next_model_id = (
-                max(wrapper.entity.id for wrapper in self.modelEntities.values()) + 1
-            )
+        self.validate_entity_ids()
+        self._used_entity_ids = {
+            wrapper.entity.id
+            for repo in (self.modelEntities, self.folders)
+            for wrapper in repo.values()
+        }
 
     def __getitem__(self, type_: str) -> EntityRepository[b.BaseEntityType]:
         return getattr(self, type_)
 
-    def get_next_model_id(self) -> int:
-        with self.lock:
-            next = self.__next_model_id
-            self.__next_model_id += 1
-        return next
+    def get_next_entity_id(self) -> int:
+        return generate_entity_id(self._used_entity_ids)
+
+    get_next_model_id = get_next_entity_id
+
+    def validate_entity_ids(self) -> None:
+        for entity_type, repo in (
+            (b.EntityType.MODEL_ENTITIES, self.modelEntities),
+            (b.EntityType.FOLDERS, self.folders),
+        ):
+            locators_by_id: dict[int, list[str]] = {}
+            for wrapper in repo.values():
+                if wrapper.is_deleted:
+                    continue
+                entity_id = wrapper.entity.id
+                locator = str(wrapper.locator)
+                validate_entity_id(entity_id, locator=locator)
+                locators_by_id.setdefault(entity_id, []).append(locator)
+
+            duplicates = {key: value for key, value in locators_by_id.items() if len(value) > 1}
+            if duplicates:
+                raise errors.Datam8ValidationError(
+                    message=f"Duplicate IDs in {entity_type.value}: "
+                    + "; ".join(f"{key}: {', '.join(value)}" for key, value in duplicates.items()),
+                    details={"entityType": entity_type.value, "duplicates": duplicates},
+                )
 
     def init_file_references(self) -> None:
         self._model_files = {}
@@ -461,15 +483,17 @@ def add_entity[T: b.BaseEntityType](
         match content:
             case dict() as c:
                 c["name"] = _locator.entityName
-                if _type == b.EntityType.MODEL_ENTITIES:
-                    c["id"] = self.get_next_model_id()
+                if _type in (b.EntityType.MODEL_ENTITIES, b.EntityType.FOLDERS):
+                    c["id"] = self.get_next_entity_id()
                 if _type == b.EntityType.FOLDERS:
                     c["path"] = "/".join([*_locator.folders, _locator.entityName])
                 new_entity = class_from_type(_type).from_dict(content)
-            case m.ModelEntity() as me:
-                new_entity = me
-                new_entity.id = self.get_next_model_id()
+            case (m.ModelEntity() | f.Folder()) as identified:
+                new_entity = identified.model_copy(deep=True)
+                new_entity.id = self.get_next_entity_id()
                 new_entity.name = _locator.entityName
+                if isinstance(new_entity, f.Folder):
+                    new_entity.path = "/".join([*_locator.folders, _locator.entityName])
             case _ as rest:
                 new_entity = rest
 
@@ -897,6 +921,7 @@ def save(self, locator: str | None = None, /) -> None:
         """
         Saves the current state of the model or a specific entity to the corresponding json file.
         """
+        self.validate_entity_ids()
         changed_wrappers = [
             w
             for w in self.get_entities(locator or ROOT_LOCATOR)

src/datam8/operations.py

@@ -32,6 +32,7 @@
 from datam8.model.model import class_from_type
 from datam8.plugins.navigation import normalize_source_item
 from datam8_model import base as b
+from datam8_model import folder as f
 from datam8_model import model as m
 
 
@@ -380,7 +381,7 @@ def execute(self, operation: Operation) -> dict:
             assert op.new_locator is not None
             self.model.move_entities(locator, op.new_locator, force=op.force)
         result = self.persist()
-        if created is not None and isinstance(created.entity, m.ModelEntity):
+        if created is not None and isinstance(created.entity, (m.ModelEntity, f.Folder)):
             result["item"] = {"locator": str(created.locator), "id": created.entity.id}
         return result
 

src/datam8/parser_v1.py

@@ -2,6 +2,7 @@
 import json
 import pathlib
 
+from datam8.utils.ids import generate_entity_id
 from datam8_model.v1 import (
     CoreModelEntry,
     CuratedModelEntry,
@@ -24,7 +25,7 @@ def parse_solution_file(
     index = Index.Model.from_json_file(solution_file_path.parent / "index.json")
     indeces: list[IndexEntry] = []
     mapping_file_id: dict[str, ModelFileReference] = {}
-    next_id = 1
+    used_ids: set[int] = set()
 
     if index.rawIndex is not None and index.rawIndex.entry is not None:
         indeces.extend(index.rawIndex.entry)
@@ -40,9 +41,8 @@ def parse_solution_file(
 
     for idx in indeces:
         mapping_file_id[idx.locator.lower()] = ModelFileReference(
-            id=next_id, path=pathlib.Path(idx.absPath)
+            id=generate_entity_id(used_ids), path=pathlib.Path(idx.absPath)
         )
-        next_id += 1
 
     return solution, mapping_file_id
 

src/datam8/source.py

@@ -229,6 +229,7 @@ def compare_entity_with_source(
     refreshed_by_index: dict[int, list[m.ExternalModelSource]] = {}
     described_attributes: dict[str, at.Attribute] = {}
     described_properties: dict[str, p.PropertyReference] = {}
+    described_descriptions: set[str] = set()
     for existing, location, index in groups:
         if mode == "sources-only":
             refreshed = read_external_sources(
@@ -249,6 +250,8 @@ def compare_entity_with_source(
             refreshed = [
                 item for item in described.sources if isinstance(item, m.ExternalModelSource)
             ]
+            if described.description is not None:
+                described_descriptions.add(described.description)
             for attr in described.attributes:
                 previous = described_attributes.get(attr.name)
                 comparable = {"ordinalNumber", "dateAdded", "dateModified"}
@@ -295,6 +298,10 @@ def compare_entity_with_source(
             threshold_to_diff_deeper=0,
         )
 
+    if len(described_descriptions) > 1:
+        raise utils.create_error(ValueError("Conflicting entity descriptions"))
+    if groups:
+        wrapper.entity.description = next(iter(described_descriptions), None)
     if described_properties:
         entity_properties = {prop.property: prop for prop in wrapper.entity.properties or []}
         entity_properties.update(described_properties)