Migrate old datasets once to allow simplification of data loading

[gaberber]

Work is half done in the guan/jonginn branches. New classes are extracted from the original god class, but scaffolding still remain in order to maintain backwards compatibility. Large amounts of existing analysis code does completely unnecessary things if the HDF5 files are written with more complete info:

• Floquet cycle timing (how many us is each cycle) should be saved in HDF5 either under attr['derived_params'] (or similar names) or as a separate data key. Lots of helper code exists simply to retrieve this one number or reconcile it in various ways. Write a migration script, calculate this number using the config versions (might need to query jobs.db for this) for all old data files, and store them into each HDF5. This is a small addition so should be fast and otherwise non-invasive. But still, test before touching existing files to absolutely avoid data loss.
• Lots of scaffolding due to lack of a manifest that serves as a central pointer that stores metadata about a logically coherent group of dataset. Eg each spectrum sweeps {Fock state occupation} $\otimes$ {time} $\otimes$ {complex quadratures}. We can simply take a historical dataset (eg a list of job IDs), look up all the file names and sub dirs, store those as a manifest of paths into a new HDF5, along with other necessary metadata about this aggregate dataset, inside a processed_data dir (eg next to the original data). This means we can directly use MBRSpectrumExperiment.from_hdf5('manifest_filename.h5') to load the dataset and then directly analyze()/display().
• One alternative to above is, instead of storing new HDF5s as manifests, use yamls instead. The benefit is that yamls are designed for human readability and it's easy to add comments. Downside is it will only serve as a pointer to other HDF5s and won't store data inside. It will also require a new class method as the Experiment object constructor (from_yaml instead of from_h5file).
• There are 8 file-loading methods now. Delete all but from_h5file after the migration (or plus from_yaml).
  slab from_h5file
  EncodingHamiltonian… _from_expts, from_batch, from_job_files, from_job_ids
  saved_jobs load_h5, load_job, load_aggregate
  mbr_disorder_h5 load_h5 (the parallel copy)

After the migration, we should purge any historical scaffolding that existed because of these historical problems. Run it once (eg as tools/migrate_old_MBR_datasets and fully migrate to new architecture.
Ideally, we will have:

• An Experiment whose acquire measures one {time} $\otimes$ {complex quadratures} trace and analyze does quadrature reconstruction, manual phase shift, single-trace MPM extraction etc. (If this step requires also merging pairs of quadrature files from old datasets, merge and write a new HDF5 into processed data, do not overwrite in place as that will erase the job ID provenance.)
• An Experiment whose acquire batches series of time and quadrature sweeps to sweep over occupation. The analyze does eg MPM peak merging etc.
• An Experiment that sweeps disorder realizations with the corresponding acquire/analyze/display/from_h5file/save_data.
• The accompanying calibration experiments etc.

In summary we should we return to the canonical API defined by the Experiment class protocols. Loading should be either mechanically opening one hdf5 file and populating the data fields, or a simple loop over a list of file names stored in the manifest. The concept of "stage" should be eliminated. An object knows what it should do simply by its identity. No switchboards.

Along the way, give things better names (eg EncodingHamiltonian..., DarkXXX for things that are not dark modes related). Break backwards compatibility. Migrate and simplify.