Suggestions for configuration layer

[TeresiaOlsson]

Edit: for summary, see the post from @gubaidulinvadim

https://github.com/orgs/python-accelerator-middle-layer/discussions/27#discussioncomment-14699670

---

I'm attempting to go through the two architecture proposals step-by-step and come up with suggestions for how to merge.
The first step I have looked at is the configuration layer.

In addition I have added a suggestion to replace the functionality of the MML AcceleratorObject that Marco and I felt was missing when we tested.

I have made a little example to illustrate the idea. Since it will require discussion and refactoring effort I have put the example in a temporary separate repository. Sorry for the name, I had to come up with some temporary one to use for the example...

Here is the repository with the example:
https://github.com/python-accelerator-middle-layer/aml

• It's in no way a full implementation with correct keywords etc.
• It also includes no example of how it would work for external modules since I have not looked in detail at that part yet.

What it will replace

I think the idea is a combination of the functionality of ConfigService and YellowPages from the accml repository.
It's a replacement of pyAML and Instrument in the pyaml repository.

For historical reasons (and because I had no better idea) I called the class Accelerator but it could also be called registry or something else.

The MML AcceleratorObject (AO) and AcceleratorData (AD)

In MML there is the AcceleratorObject (AO) which is a key part of the architecture.
In my view this serves the functionality of a registry. It stores the configuration data and allows the user to browse/modify it as needed. It contains no business logic but only data.

MML also have AcceleratorData (AD) which complements the AO.
I don't think it's entirely clear in MML which type of configuration data should go where but I would say the AO is for static data
(such as PV/attribute names etc) and AD more for dynamic data (operational mode specific application settings etc).
MML has a weird way where information is searched for in various places in some order which we probably should not follow since it is error prone but the idea of static vs dynamic config data I think is a useful one. The example includes no dynamic config data yet since that is mostly for applications.

Both the AO and AD have a defined structure. I would say it is a data model for the configuration which helps the user a lot when figuring out how to set up the configuration for their machine.

For more details, see Appendix III in the MML manual.

In MML all "devices" and applications extract the configuration they need from the AO and AD when they need it.

Benefits

These are what I think are the benefits:

• The data models for the configuration are gathered in one place so they are easier to find instead of spread out in different modules.

• It is modular. You can create only the config objects you need and skip the rest.

• You can create the configuration in different ways: creating config objects directly, read in from different external sources etc.
  It works as long as you in the end have a config object with the structure described by the data models.

• There is no need to define a type in the configuration for parts where there should be no option to use external modules anyway.
  I think parts liketype: pyaml.pyaml and type: pyaml.instrument just clutters the yaml file and makes it more difficult to read.
  For cases where there are several different data models to choose from, pydantic also has the functionality that it sometimes can figure out itself which option to create based on the given keywords so no type keyword is required. I think we should make use of that where it works to simplify the yaml file.

The big difference

This I think is they key point to discuss.

In this suggestion the devices are not created at initialization of pyAML. I haven't understood the benefits of that? If I have a full configuration set up with many different control modes, lattice models and several operational modes it seems to me like then a lot of objects with business logic need to be created and kept in memory despite that they will not be used during that session?

To me it feels like a better approach to store the configuration data in a registry, and devices are only created at the point when the user needs them and then for the mode the user is asking for.

For example, the part quadForTuneDesign = sr.design.get_magnets("QForTune") in the ESRF example could instead of retrieving objects that already has been created also create them since the user at that point anyway says that they want the devices in the family "QForTune" for the simulator using the lattice model "design."

But maybe there is some benefit of initializing all devices from the start that I haven't understood?

[JeanLucPons]

I'm not sure to fully understand.
The may advantages of creating configuration objects "on the fly" using a depth fist factory is to have simple and complete pydantic schema without having to deal with paths. The pydantic schema does not contains useless information.

[TeresiaOlsson]

Then I think we should look at how ophyd-async does it because as far as I understand they don't have a separate step to initialize the control system but it's all done at the signal level.

This is an example of an ophyd-async device I wrote earlier this week as part of the issue to explore how that package works.

"""
Ophyd-async device for masterclock.
"""

from pydantic import BaseModel
from typing import Optional
from ophyd_async.core import StandardReadable
from ophyd_async.epics.core import epics_signal_rw

class MasterClockConfig(BaseModel):

    name: Optional[str] = 'masterclock'
    setpoint_pv: str
    readback_pv: str

class MasterClock(StandardReadable):

    def __init__(self, config: MasterClockConfig, prefix: Optional[str] = None) -> None:

        if prefix:
            self._setpoint_pv = prefix + ':' + config.setpoint_pv
            self._readback_pv = prefix + ':' + config.readback_pv
        else:
            self._setpoint_pv = config.setpoint_pv
            self._readback_pv = config.readback_pv


        with self.add_children_as_readables():
            self.frequency = epics_signal_rw(float, read_pv= self._readback_pv,
                                             write_pv= self._setpoint_pv, name='frequency')
        super().__init__(name=config.name)

    @property
    def setpoint_pv(self):
        return self._setpoint_pv

    @property
    def readback_pv(self):
        return self._readback_pv

To start using it I only need to do:

prefix = 'some_prefix_for_the_twin'
config = MasterClockConfig(readback_pv='MCLKHX251C:hwRdFreq', setpoint_pv='MCLKHX251C:freq')
masterclock = MasterClock(config, prefix=prefix)
await masterclock.connect()

and then I can read and set.

I thought we could do something similar with a factory to create a device with either epics_signals or tango_signals? Ophyd-async also has that but in a declarative way which I don't entirely like.

[JeanLucPons]

It is not really a problem if ophyd-asynch does not have a global init stuff. The init() is linked to the control system. In Tango we will use it mainly to initialize the TANGO_HOST but this has to be done prior to any call to tango_signal_rw().

[TeresiaOlsson]

Hmm... What does initialize the TANGO host include? For EPICS I just need to specify which PV prefix I want (which is just a string) and as long as the signals in the devices are set up including that prefix it will connect to the PVs if I'm on the correct network. I thought for TANGO this would just be to specify the name of the TANGO host but maybe you need some additional step then?

[JeanLucPons]

TANGO_HOST tell on which control system instance you want to connect.
Regarding your MasterClock() implementation, are you using epics_signal_rw() to connect to simulators ?
Do you have raw access to pyAT ?

[TeresiaOlsson]

That's what I thought. Then it's similar to the prefix for EPICS.

No, there is no simulator since that wasn't the purpose of my implementation. I'm just trying to learn how ophyd-async works and use it for some measurements we are doing in the machine where we are in desperate need of a replacement. Those measurements we would eventually do in pyAML instead when we have that. But to develop it without needing beamtime I can connect to our twin by choosing a different prefix.

In ophyd-async there is a mock option which I think the beamlines use to connect to simulated devices. Maybe that could be used to connect to pyAT but I haven't looked at that since the simulator is a pyAML thing and everything I need the simulator for at the moment I can currently do with MML.

[kparasch]

It's my impression that the most resource intensive aspect of the middle layer is the simulator. So I don't think you gain a lot by doing lazy evaluation since you load the lattice with the first magnet you want to access. Instead you will have to check that your simulator exists every single time you try to access a magnet.

Nevertheless, what instantiates the pyaml object is the command ml = pyaml("sr.yaml"). You can still parse the sr.yaml file to get information about the configuration and create some AO/AD which in my opinion will be very useful to have.

Finally, nothing really needs to be changed to dynamically modules. You just need to know where to look in the "sr.yaml" file.

[TeresiaOlsson]

I think that depends on how many and complex devices you have in your configuration. In our case we want to have insertion devices, main and harmonic cavities etc and a lot of other devices that we use in the real machine but we don't have in our lattice model (and likely will never have because they are difficult to model...). For us it is likely that the lattice model will be very basic compared to the configuration of the hardware devices.

[swhite2401]

@TeresiaOlsson could you please clarify you needs, with possibly some syntax examples? It is absolutely not clear what you want? What do you mean by configuration classes?

Would like something like:
quad = pyaml.magnet.Quadrupole(*args, **kwargs) to instantiate an element in pyAML directly from a tango device as a standalone element?

[TeresiaOlsson]

I think it's also a matter of implementation. At the moment there is no single object that I can create where I can get information about the full configuration because it's separated into two classes. Also if I understand correctly, Instrument only initializes live and design. How do I use the other controls and simulations I have configured?

[TeresiaOlsson]

For me created an object with or without logics make no difference. Or at least it should. Using Factory enable to create connections to device when they are needed. "With the logic" means that you instantiate an object with method. What is the problem of having this methods available or not ?

When I tested it, the difference was that it made it much easier to implement the registry functionality because the objects in it are just pydantic basemodels which automatically print out in a nice format. In my example I automatically get:

mls.controls
Out[5]: 
{'live': EPICSConfig(access_type=<EPICSType.CA: 'CA'>, pv_prefix=''),
 'twin': EPICSConfig(access_type=<EPICSType.CA: 'CA'>, pv_prefix='twin')

whereas in the corresponding example for the current implementation I get:

sr._cfg.controls
Out[4]: [<tango.pyaml.controlsystem.TangoControlSystem at 0x72af75be4370>]

We then need to implement the pretty printing to get the same functionality that we would get for free if the registry is just a collection of basemodels. But maybe there is some other way of doing it.

[JeanLucPons]

I think it's also a matter of implementation. At the moment there is no single object that I can create where I can get information about the full configuration because it's separated into two classes. Also if I understand correctly, Instrument only initializes live and design. How do I use the other controls and simulations I have configured?

By their names. live and design are typed properties that offer full completion for these special 2 names. We can add more. For free name, it is just a dynamic python attribute.

[JeanLucPons]

For me created an object with or without logics make no difference. Or at least it should. Using Factory enable to create connections to device when they are needed. "With the logic" means that you instantiate an object with method. What is the problem of having this methods available or not ?

When I tested it, the difference was that it made it much easier to implement the registry functionality because the objects in it are just pydantic basemodels which automatically print out in a nice format. In my example I automatically get:

mls.controls
Out[5]: 
{'live': EPICSConfig(access_type=<EPICSType.CA: 'CA'>, pv_prefix=''),
 'twin': EPICSConfig(access_type=<EPICSType.CA: 'CA'>, pv_prefix='twin')

whereas in the corresponding example for the current implementation I get:

sr._cfg.controls
Out[4]: [<tango.pyaml.controlsystem.TangoControlSystem at 0x72af75be4370>]

We then need to implement the pretty printing to get the same functionality that we would get for free if the registry is just a collection of basemodels. But maybe there is some other way of doing it.

This is just a matter of adding some repr()

[TeresiaOlsson]

Sure. But then someone needs to spend the time doing it. That also includes external modules. For me it feels like better spent time to make use of functionality that we get for free.

[gubaidulinvadim]

I want to summarise what I understood from the original message of Teresia. In terms of desired requirements, Teresia and colleagues want:

1. Not creating all devices at the initialisation of pyAML
2. Ability to only configure the objects necessary for a given measurement instead of a full Instrument with all devices.
3. Different ways to create configuration: yaml/json files or creating objects with Pydantic interactively
4. Putting data models in one place (folder), separate from the implementation.
5. Overridability of pyaml.pyaml object is not welcome

From my understanding:

1. Advantages and disadvantages of the two approaches should be listed in this discussion or in a separate document. Otherwise, this is too difficult to navigate the discussion. I suggest we focus only on this point first in the discussion and keep the other points after this one is discussed.
2. This is the same point as 1 in my opinion.
3. I think Jean-Luc mentioned that this is currently possible. If that is the case, we should provide an example of how to do it.
4. To me, either way is fine. It is not really a functionality question.
5. It is a question of whether we want some core functionality to be overridable or not. I think that, in practice, no one will be overriding that.

[TeresiaOlsson]

5. It is a question of whether we want some core functionality to be overridable or not. I think that, in practice, no one will be overriding that.

Overriding core part is really useful. For adding new features without breaking backward compatibility for instance. It is mandatory for magnet models, for Curve/Matrix format, for control system abstraction, etc...

For magnet models and conversions I fully agree but why do you want the users to be able to override the control system abstraction? For me that's a key part of pyAML and if there is something I want to change with how the EPICS bindings work I think it should be merged into the source code so it becomes available for all labs to use rather than me choosing a lab specific module in the configuration. Or you have a use case for a lab specific module?

For the backward compatibility, why do you think it's handled better by users changing module names in the configuration rather than choosing to install different versions of python packages?

I'm not against overriding. I just think we should think about which levels we want the users to be able to override by specifying another module in the configuration and which levels that are so fundamental that if you want to to override that you need to go into the source code and do it. Allowing all modules to be changed by configuration settings feels to me like we could end up with compatibility issues between labs and make it more difficult to write generic applications that work for everyone.

[JeanLucPons]

The ControlSystem abstraction is based on this fundamental principle of overriding.
Concerning higher level object, If I get for instance the example of Tango server. The main Device(Server) object is overridable. An we increase version, Device(Server)_2 Device(Server)_3. etc. This allow to maintain compatibility with older code. Today we are at version 5 and 6 is coming.

If you want to change the logic of the Intrusment object without breaking old working user code, you can do it easily.
You can even add a versioning system at the Facrory level.
Overriding is not only for user but especially for developer in order to ease future evolution integration

[TeresiaOlsson]

I understand that overriding is very important for the developers. I just doesn't understand why the developers need to specify that in the yaml file when they are people who has the capability to work directly with the code? I thought the purpose for the yaml file was to make the configuration easier to set up and modify for the users? If that's the purpose, it feels strange to me to require fields in the yaml file that are only supposed to be used by developers because no user are supposed to change them anyway?

[JeanLucPons]

It is like for the Tango Device(Server) top level object.
Users see the version (In fact today it is a bit hidden by a MACRO).
For pyaml, you can imagine to have:
type: pyaml.pyaml_v2
Hiding this is possible but will really complexify the code and make it more difficult to maintain.

[JeanLucPons]

I added an example that show how deal without yaml and creating only what is needed.
https://github.com/python-accelerator-middle-layer/pyaml/blob/14-new-element-identifier-method-names-uniqueness/examples/ESRF_tune_example/esrf_tune_example_no_yaml.py