Harden Psi4 workflows and add genuine molToolkit UMAP support

agents/mol-toolkit/README.md

@@ -20,6 +20,41 @@ Before starting the deployment, ensure you have:
 3. Docker installed locally for image building
 4. Azure CLI or PowerShell for resource management
 
+## UMAP dependency and verification
+
+The `molenv` environment includes `umap-learn` (imported as `from umap import
+UMAP`). The Dockerfile runs `app/smoke_umap.py` during the build: it computes real
+UMAP coordinates from a ten-row fingerprint Tanimoto-distance matrix and rejects
+non-finite coordinates or an output shape other than `(10, 2)`. This is an
+embedding execution check, not just an import check or an MDS/PCA substitute.
+
+In the matching tool environment, the check can also be run without rebuilding:
+
+```bash
+python /app/smoke_umap.py
+```
+
+Success prints `UMAP_SMOKE_OK`, the installed package version, and `(10, 2)`.
+The check requires the image's Python dependencies; compiling the script alone
+does not validate UMAP execution. It does not validate scientific cluster quality.
+
+| Prompt behavior before | Prompt behavior after |
+|---|---|
+| UMAP was not in the agent's available-package list | The agent explicitly uses `umap-learn` |
+| Missing UMAP could lead to an MDS fallback | MDS, PCA, and t-SNE cannot be reported as UMAP |
+| Precomputed similarities could be confused with distances | Tanimoto similarities are validated and converted to distances before UMAP |
+
+For example, "visualize ten candidate fingerprints using UMAP" now requires a
+real two-dimensional UMAP embedding, a plot, saved coordinates, package version,
+and parameters. A failed import or embedding must be reported as a failure with
+diagnostics, not as a completed substitute visualization.
+
+These are source changes, not an update to any existing image or deployed agent.
+Obtain approval before building, publishing, retagging, or replacing an image.
+After approval, build and scan the matching image first, verify the smoke check,
+then update the deployed tool and agent together before rerunning affected tasks.
+Changing agent instructions alone cannot add the missing dependency to an image.
+
 ## Build Docker Image
 
 ### Step 1: Build and Publish Docker Image

agents/mol-toolkit/agent.yaml

@@ -8,7 +8,7 @@
   options:
     temperature: 0
     topP: 0
 instructions: |
   You are an expert in molecular analysis and computational biology and chemistry, with access to a wide range of open-source Python packages. Your goal is to assist users in achieving their goal using both pre-built functions and open-source packages available in the MolToolkit container. You also excel at combining various data sets together using the python packages available.
   When asked what you can do, do not delve into the details of the container or its contents. Instead, focus on your capabilities.
   Use DRY methodology when coding: Do not repeat yourself. If you find yourself writing similar code multiple times, refactor it into a function or module that can be reused.
@@ -23,11 +23,13 @@
   The container image has pre-built molecular chemistry functions exposed as Python modules in `/app`. You should **ALWAYS prioritize using these pre-built functions** by importing them directly from their modules to achieve user goals. Only generate custom Python scripts when absolutely necessary - when no combination of pre-built functions can accomplish the task.
 
   ***Open source Python packages available in the MolToolkit container:***
-      ase, biopython, matplotlib, MDAnalysis, numpy, openbabel, openpyxl, pandas, pyarrow, pymatgen, pymol-open-source, rdkit, scanpy, scikit-bio, scikit-learn, scipy, seaborn, sympy
+      ase, biopython, matplotlib, MDAnalysis, numpy, openbabel, openpyxl, pandas, pyarrow, pymatgen, pymol-open-source, rdkit, scanpy, scikit-bio, scikit-learn, scipy, seaborn, sympy, umap-learn
       You cannot use any other open-source packages that are not listed here as they are not available in your environment.
 
   **IMPORTANT scikit-learn API note:** In scikit-learn >= 1.4, `AgglomerativeClustering` renamed the `affinity` parameter to `metric`. Always use `metric=` (e.g., `AgglomerativeClustering(metric='precomputed', linkage='average')`). Using `affinity=` will raise a TypeError.
 
+  **UMAP requirements:** For a requested UMAP embedding, use `from umap import UMAP` from the installed `umap-learn` package. Never substitute MDS, PCA, or t-SNE and label it UMAP. For a Tanimoto similarity matrix, validate symmetry, unit diagonal, finite values, and values in [0, 1], then use `distances = 1.0 - similarities` with a zero diagonal and `metric="precomputed"`; UMAP expects distances, not similarities. For at least three candidates, set `n_components=2`, `n_neighbors=min(15, n_candidates - 1)`, `init="random"`, `random_state=42`, and `n_jobs=1`. For fewer than three candidates, report insufficient data instead of inventing coordinates. Validate that the embedding contains one finite 2D coordinate per candidate, preserve candidate ordering, and save the coordinates, requested plot, package version, and parameters in the workflow outputs. If the import or fit fails, preserve diagnostics and report failure; do not silently switch algorithms or declare UMAP complete.
+
   **CODE EXECUTION METHOD**: Import functions directly from `/app` modules (e.g., `from molecular_utils import function_name`) for maximum flexibility, seamless function composition, better error handling, and custom workflow creation. Unless the user explicitly specifies working with a single file or specific files, scripts should iterate through ALL relevant files in the input directory (path from dataHandlingContext) using functions from `app/io_utils.py` (like `find_smiles_files()` and `read_smiles_from_file()`).
 
   # MANDATORY INSTRUCTIONS:

agents/mol-toolkit/tools/mol-toolkit/Dockerfile

@@ -48,6 +48,7 @@ RUN mamba create -y -n molenv -c conda-forge \
       scipy \
       seaborn \
       sympy \
+      umap-learn \
  && mamba clean -a -y
 
 # ---- Patch base conda env (Trivy scans all site-packages dirs) ----
@@ -83,6 +84,8 @@ ENV PATH=/opt/conda/envs/molenv/bin:/opt/conda/bin:$PATH
 ENV PYTHONPATH=/app
 ENV PYTHONUNBUFFERED=1
 
+RUN mamba run -n molenv python /app/smoke_umap.py
+
 # Create common mount points / working dirs (single layer)
 RUN mkdir -p /workdir
 

agents/mol-toolkit/tools/mol-toolkit/app/smoke_umap.py

@@ -0,0 +1,33 @@
+"""Verify real UMAP execution on the ten-candidate Tanimoto-distance workflow."""
+
+from importlib.metadata import version
+
+import numpy as np
+from umap import UMAP
+
+
+def main() -> None:
+    fingerprints = np.random.RandomState(42).randint(0, 2, size=(10, 64))
+    intersections = fingerprints @ fingerprints.T
+    counts = np.diag(intersections)
+    unions = counts[:, None] + counts[None, :] - intersections
+    similarities = intersections / unions
+    distances = 1.0 - similarities
+    np.fill_diagonal(distances, 0.0)
+
+    embedding = UMAP(
+        n_components=2,
+        metric="precomputed",
+        n_neighbors=min(15, distances.shape[0] - 1),
+        init="random",
+        random_state=42,
+        n_jobs=1,
+    ).fit_transform(distances)
+
+    if embedding.shape != (10, 2) or not np.isfinite(embedding).all():
+        raise RuntimeError("UMAP must produce ten finite two-dimensional coordinates")
+    print(f"UMAP_SMOKE_OK: umap-learn={version('umap-learn')}, shape={embedding.shape}")
+
+
+if __name__ == "__main__":
+    main()

agents/psi4/README.md

@@ -27,7 +27,7 @@ Psi4 features a Python API for scriptable workflows, density fitting for computa
 
 ```bash
 # Navigate to the psi4 directory
-cd 6-solutions/tools-and-models/psi4/
+cd agents/psi4/tools/psi4/
 
 # Build the Docker image
 docker build -t psi4:latest .
@@ -50,7 +50,10 @@ psi4/
 ├── psi4-tool-definition.yaml     # Tool infrastructure definition
 ├── psi4-agent-definition.yaml    # Agent instructions and capabilities
 ├── psi4_utils.py                 # Python utilities library
-├── test_psi4_utils.py           # Unit tests
+├── run_psi4_job.py               # Isolated script launcher
+├── psi4_state_job.py             # Private fresh-process four-point worker
+├── test_psi4_reliability.py      # Mocked-Psi4 unittest suite
+├── smoke_psi4_runtime.py         # Real-Psi4 runtime smoke
 ├── README.md                     # This file
 └── example-input-files/
     ├── water.xyz                 # Water molecule
@@ -89,6 +92,9 @@ psi4/
 | Function | Description |
 |----------|-------------|
 | `compute_energy()` | Single-point energy calculation |
+| `compute_electronic_properties()` | One wavefunction for energy, orbital gap, and explicit dipole |
+| `compute_dipole_moment()` | Explicit one-electron dipole, atomic units and Debye |
+| `compute_reorganization_energies()` | Gas-phase four-point hole/electron reorganization |
 | `compute_gradient()` | Energy gradient (forces) |
 | `optimize_geometry()` | Geometry optimization |
 | `compute_frequencies()` | Vibrational frequencies |
@@ -151,14 +157,18 @@ psi4.set_options({
 
 ### Local Testing (without Psi4)
 
-```bash
-cd 6-solutions/tools-and-models/psi4/
-python -m venv .venv-psi4
-source .venv-psi4/bin/activate  # or .venv-psi4\Scripts\activate on Windows
-pip install numpy scipy matplotlib pandas pytest
-pytest test_psi4_utils.py -v
+```console
+cd agents/psi4/tools/psi4
+python -m unittest -v test_psi4_reliability
+python -m py_compile psi4_utils.py run_psi4_job.py psi4_state_job.py smoke_psi4_runtime.py test_psi4_reliability.py
 ```
 
+The test runner is stdlib `unittest`; it mocks Psi4 and uses the library's existing
+NumPy dependency. Tests include reference selection/validation, single-wavefunction
+dipole conversion, nonfinite rejection, optimization retry/exhaustion, candidate-ID
+and partial-failure preservation, four-point formulas, concurrent launcher isolation,
+nested artifact copying, and child/copy failure exit codes.
+
 ### Docker Testing
 
 ```bash
@@ -256,7 +266,139 @@ Microsoft Discovery team contact: discovery-catalog@microsoft.com
 
 ## Known Limitations
 
-No known limitations at this time. If you encounter issues, please report them via the support channel above.
+Mocked tests do not validate numerical accuracy, SCF/optimizer behavior, basis/method
+availability, or PSIO behavior in the deployed image. Python 3.14 host tests are not
+a substitute for the container's Python 3.11/Psi4 runtime. The scratch/working-directory
+changes are isolation hardening, **not a proven PSIO root-cause fix**. Reproduce the
+original failing input in the approved runtime after the smoke below.
+
+## Gas-phase electronic properties and four-point reorganization
+
+Public helper signatures:
+
+```python
+compute_dipole_moment(molecule, method='b3lyp', basis='def2-tzvp', **kwargs)
+compute_electronic_properties(molecule, method='b3lyp', basis='def2-tzvp', **kwargs)
+compute_reorganization_energies(
+    molecule, candidate_id, method='b3lyp-d3bj',
+    optimization_basis='def2-svp', energy_basis='def2-tzvp',
+    neutral_multiplicity=1, cation_multiplicity=2, anion_multiplicity=2, **kwargs
+)
+```
+
+`compute_energy`, `compute_homo_lumo_gap`, and `optimize_geometry` retain their
+signatures and select `rhf` for singlets or `uhf` for higher multiplicities on **each**
+calculation, including DFT. Explicit `reference='rhf'|'uhf'|'rohf'` remains supported.
+Closed-shell `rhf` with open-shell multiplicity is rejected before calculation.
+Do not substitute `rks`/`uks` keywords. Unconverged SCF acceptance is disabled.
+Open-shell orbital gaps retain the existing **alpha-spin** convention; they are not
+spin-combined, fundamental, or optical gaps.
+
+The combined helper returns the existing `homo_energy_hartree/eV`,
+`lumo_energy_hartree/eV`, `gap_hartree/eV`, and `total_energy_hartree/eV` keys, plus
+`reference`, `dipole_vector_au`, `dipole_vector_debye`, and `dipole_moment_debye`.
+It computes one energy/wavefunction and explicitly calls
+`psi4.oeprop(wfn, 'DIPOLE', title='DISCOVERY')`, retrieving
+`wfn.variable('DISCOVERY DIPOLE')` as a finite three-component vector.
+The conversion is $1\,e a_0 = 2.5417464518950259\,\mathrm{D}$.
+Charged-species dipoles depend on the coordinate origin. No polarizability or
+optional-property artifact is required for these helpers to succeed.
+
+API/units references verified against upstream documentation and implementation:
+
+- [oeprop signature, wavefunction argument, and title prefix](https://psicode.org/psi4manual/master/oeprop.html)
+- [Vector dipoles and atomic units](https://psicode.org/psi4manual/master/glossary_psivariables.html#psivar-SCF-DIPOLE)
+- [Atomic-unit-to-Debye conversion](https://psicode.org/psi4manual/master/autodoc_physconst.html)
+- [OEProp title-prefixed array storage on the wavefunction](https://github.com/psi4/psi4/blob/master/psi4/src/psi4/libmints/oeprop.cc)
+- [OptimizationConvergenceError last wavefunction](https://psicode.org/psi4manual/master/_modules/psi4/driver/p4util/exceptions.html#OptimizationConvergenceError)
+
+Four-point calculations always reoptimize the neutral, cation, and anion; an
+upstream partial neutral result is **never** assumed converged. Defaults represent
+a singlet neutral and doublet singly charged ions; supply the actual multiplicities
+for other spin states. Input must be neutral. Each state uses its own validated
+reference. Every optimization has at most **one** retry, from the exception's last
+wavefunction geometry, with doubled `geom_maxiter`. If no last wavefunction is
+available it fails without restarting the original geometry. Neither convergence
+tolerances nor method/basis are weakened.
+
+Let $Q_0$, $Q_+$, and $Q_-$ denote the optimized neutral, cation, and anion geometries.
+At the **same final method/energy basis**, the two four-point sums are:
+
+$$
+\lambda_h = [E_+(Q_0)-E_+(Q_+)] + [E_0(Q_+)-E_0(Q_0)]
+$$
+$$
+\lambda_e = [E_-(Q_0)-E_-(Q_-)] + [E_0(Q_-)-E_0(Q_0)].
+$$
+
+Results preserve `candidate_id`, `status`, method/bases, `energy_units='hartree'`,
+`states` (charges, multiplicities, references, convergence flags, geometries in
+angstrom), and `components`. Component keys are `E_neutral_at_neutral`,
+`E_cation_at_cation`, `E_anion_at_anion`, `E_cation_at_neutral`,
+`E_neutral_at_cation`, `E_anion_at_neutral`, and `E_neutral_at_anion`.
+Success adds `hole_reorganization_hartree/eV` and `electron_reorganization_hartree/eV`.
+On failure `status='failed'`, error details and all completed components remain;
+do not treat them as completed reorganization results. Negative values are not
+clamped or hidden. The helper persists `reorganization_result.json` in its unique
+`artifact_directory`, with per-state requests/results and Psi4/worker logs.
+
+### Invocation isolation and deployment use
+
+The Dockerfile **packages source only when an approved image is built**. Editing it
+does not update a deployed image. Do not build, push, retag, replace an image, or
+change live tool references without explicit approval.
+
+The execution command in the matching tool definition should use positional args:
+
+```console
+/opt/conda/envs/psi4env/bin/python /usr/local/lib/run_psi4_job.py /path/to/script.py /path/to/output
+```
+
+The launcher executes `[sys.executable, script_path]` without shell reparsing.
+It creates unique writable work/scratch directories under the output directory's
+parent, preflights writes, sets `PSI_SCRATCH` **before imports**, streams combined
+stdout/stderr to terminal and `logs.out`, and copies nested work artifacts even
+after failure. Scratch and symlinks are excluded. A child nonzero exit is preserved;
+copy failure makes an otherwise successful invocation fail. `launcher_failure.json`
+records failures. Invocation directories are cleaned only after copying.
+
+`quick_setup(input_dir=..., output_dir=...)` retains the supplied data-handling
+input/output paths; work resolves to invocation-specific `PSI4_WORK_DIR`, then an
+explicit `work_dir`, otherwise a unique directory. `setup_psi4` binds the Psi4
+IOManager to the selected writable scratch path. Use these settings rather than
+shared work/scratch names. Four-point state jobs launch seven fresh processes
+(three optimizations plus final diagonal energies, four cross energies) with their
+own scratch directories and filenames. Non-JSON wavefunctions remain process-local.
+
+```python
+from psi4_utils import (
+    quick_setup, setup_psi4, create_molecule, compute_electronic_properties,
+    compute_reorganization_energies, save_final_results, quick_finish,
+)
+# Supply actual paths from the task's dataHandlingContext.
+quick_setup(input_dir=input_directory, output_dir=output_directory)
+setup_psi4(memory='4 GB', nthreads=1)
+molecule = create_molecule(geometry, charge=0, multiplicity=1)
+properties = compute_electronic_properties(molecule, method='b3lyp', basis='def2-tzvp')
+reorganization = compute_reorganization_energies(molecule, candidate_id='candidate-001')
+save_final_results(
+    {'electronic_properties': properties, 'reorganization': reorganization},
+    status='completed' if reorganization['status'] == 'success' else 'failed',
+)
+quick_finish()
+```
+
+In the approved Psi4 runtime, run the real smoke using the launcher:
+
+```console
+python run_psi4_job.py smoke_psi4_runtime.py smoke-output
+```
+
+It checks a water dipole, charged doublet water SCF energies, and tiny LiH
+four-point calculations with a diffuse basis. It writes `smoke_results.json` and
+asserts convergence and finite values. **Not yet run successfully in a real Psi4
+runtime**: the local Windows Python has no Psi4 and Docker is unavailable. This is
+the remaining verification gate before relying on L4b production numerical results.
 
 ## Contributing