Feature: Templates in configuration

[kparasch]

Description, motivation and use case
I thought harder after the community meeting to try and find a way to simplify configuration. Typically when writing the configuration files, they include devices with highly repeatable config. A "template" could be used to drastically reduce the number of lines in the configuration files.

@TeresiaOlsson @simoneliuzzo @gubaidulinvadim @GamelinAl @gupichon @JeanLucPons
I would very much like your opinion.

Proposed solution
A template in the configuration file can be used to repeat devices.

Example
The idea would be to have something like this in the configuration file:

- class: pyaml.configuration.template.Template
  template: |
    class: pyaml.magnet.hcorrector.HCorrector
    name: %template_parameter%
    model:
      class: pyaml.magnet.linear_model.LinearMagnetModel
      unit: rad
      hardware_unit: str
      calibration_factor: 1.0
      powerconverter: MAGNET/%template_parameter%/B1L
  string_to_replace: "%template_parameter%"
  parameter_list:
    - PKDK_WL_108
    - PKDK_WL_93
    - PKDK_WL_79
    - PKDK_WL_64
- class: pyaml.magnet.hcorrector.HCorrector
  name: ANOTHER
  model:
    class: pyaml.magnet.linear_model.LinearMagnetModel
    unit: rad
    hardware_unit: str
    calibration_factor: 1.0
    powerconverter: MAGNET/ANOTHER/B1L

then this could be expanded when building the storage ring.

Additional context
I have opened a draft PR #442 where I tried to implement something like this. It is definitely not perfect and there are a lot of edge cases that may cause problems.

Checklist

• I've assigned this issue to a project
• I've @-mentioned relevant people

[TeresiaOlsson]

I think it's a nice idea. I'm wondering if it might be useful to combine with jinja? Or maybe that's already part of the idea?

[kparasch]

I think it's a nice idea. I'm wondering if it might be useful to combine with jinja? Or maybe that's already part of the idea?

I have not considered jinja. I do not know anything about it actually.

[TeresiaOlsson]

I think it's a nice idea. I'm wondering if it might be useful to combine with jinja? Or maybe that's already part of the idea?

I have not considered jinja. I do not know anything about it actually.

I think it's quite nice. I only know of it because catap uses it for it's configuration. And then I discovered that the copier template also use it when I started setting that up.

I'm wondering now if it might be possible to generate the templates automatically from the JSON Schema? The JSON Schema is kind of a template. Just a very non user-friendly one... But to extract the JSON Schema is very easy and it might be possible to write a second step that from it produces user-friendly templates?

[JeanLucPons]

I agree that it is a good idea. I'm rather against Jinja which already causes me strong headache.

You should find a way to define your template and instanciate it. Not necessary link to a class.
We could imagine to have a template placeholder.
i.e.:

template:
   - name: SHI
     parameters:
       - sh_name
     code: 
      type: pyaml.magnet.cfm_magnet
      name: ${sh_name}
      mapping:
        - [B0, ${sh_name}-H]
        - [A0, ${sh_name}-V]
        - [A1, ${sh_name}-SQ]
        - [B2, ${sh_name}-S]
      model:
        type: pyaml.magnet.linear_cfm_model
        multipoles: [B0,A0,A1,B2]
        pseudo_factors: [1.0,-1.0,-1.0,-1.0]
        units: [rad,rad,m-1,m-2]
        hardware_units: ["A","A","A","A","A"]
        curves:
          - type: pyaml.magnet.csvcurve
            file: sr/magnet_models/SHI_h_strength.csv
          - type: pyaml.magnet.csvcurve
            file: sr/magnet_models/SHI_v_strength.csv
          - type: pyaml.magnet.csvcurve
            file: sr/magnet_models/SHI_sq_strength.csv
          - type: pyaml.magnet.csvcurve
            file: sr/magnet_models/SHI_sext_strength.csv
        matrix:
          type: pyaml.magnet.csvmatrix
          file: sr/magnet_models/SHI_matrix.csv
        powerconverters:
          - srmag/ps-corr-${sh_name[:4]}/${sh_name[:3]}-${sh_name[4]}-ch1/current
          - srmag/ps-corr-${sh_name[:4]}/${sh_name[:3]}-${sh_name[4]}-ch2/current
          - srmag/ps-corr-${sh_name[:4]}/${sh_name[:3]}-${sh_name[4]}-ch3/current
          - srmag/ps-corr-${sh_name[:4]}/${sh_name[:3]}-${sh_name[4]}-ch5/current
          - srmag/ps-corr-${sh_name[:4]}/${sh_name[:3]}-${sh_name[4]}-ch6/current

devices:
- SHI("SH1A-C04")

[JeanLucPons]

The syntax could be discussed. Here the ${} i propose will conflict with macros, but the idea is there.
Inside the ${ } a python code could be executed rather than a Jinja code. (Jinja code is not convenient at all)
It is rather easy to do directly in python.

[simoneliuzzo]

I like the template idea!

[TeresiaOlsson]

Maybe jinja is not the best option then but I think it is anyway worth to investigate if there is some other template standard we can use. Inventing our own syntax for something that already exists feels like it can just mean more maintenance work.

A quick search gave me these options that might be easier but I haven't tried them.

• https://docs.python.org/3/library/string.templatelib.html
• https://mustache.github.io/

[JeanLucPons]

Here the syntax is just python expect for the delimiter, compatible with yaml and our architecture.
IMHO, we don't need something very complex and adding new dependencies is more maintenance as you have to follow them and expect that you will have good support.
For instance what I really dislike in Jinja is that you need to produces a file which is no longer a yaml file neither a Jinja file and you lose code highlighting, code completion, etc, (may be it exits some plugin for vscode to handle this) and you also have to write a model for it.