Architecture

[TeresiaOlsson]

After talking to @JeanLucPons I have started this discussion about the architecture so everyone who wasn't on the last maintainers meeting also can see the slides.

These are the slides Jean-Luc showed for the current architecture:

[TeresiaOlsson]

Then we also showed the suggestion we have from HZB. This is based on the architecture developed for our twin presented at IPAC25 so that's why it doesn't include all parts we want in pyAML (like e.g. yaml) but we think the overall concept can also be applicable for pyAML and would simplify the implementation.

This is the high-level view of the different parts and how they connect together:

More details can be read in the IPAC paper but @Sulimankhail also has a peer-reviewed paper presented at a conference some weeks ago which has more details. It hasn't been published yet but it has been accepted with minor revisions so he can share the draft on request if you email him. That paper also includes an evaluation of the pros/cons of using these patterns. There is also a more detailed explanation of the liaison manager and the translation service which helped me a lot to understand it.

The example is currently in the dev/feature/arch-proposal-draft/ branch in the pyaml repository but after talking to Jean-Luc we will restructure it this week and separate into different repositories so it becomes clearer which parts are in the core and what are custom modules. The example also include ophyd and bluesky because we currently have no other way to show the function of a measurement execution engine and that is a key part of the idea but that will also be separated out. If people try it and like the concept I have talked to Jean-Luc that ideas could be to abstract it (to be able to use BLISS or Sardana instead of Bluesky) or to even that we implement our own pyAML specific one.

[TeresiaOlsson]

I agree. That's why I think we should perhaps consider implementing our own pyAML specific measurement execution engine if we think the concept in general is a good one. For me the measurement execution engine should handle the following things that we don't want to implement on a per application basis:

• Automatically adding metadata to all measurements so the user don't have to care about that.
• Handle pause and resume of measurements (both on demand of user or event based such as when injection is occurring).
• Handle nice restoring of initial settings if a user cancels a measurement.
• Features like live plotting, saving to files/database as the measurement progresses so not all data is lost if an error happens, automatic saving to elog etc.

[TeresiaOlsson]

I think we could also use it to simplify the implementation of hardware vs physics units. We could have a setup where you give a list of identifiers for the devices, a plan/flow for the measurement you want to run and if you want to use physics or hardware units. And then based on this it sets either the strength or the current without having to write that explicitly in the plan. The plan just says "change this device with this amount".

[TeresiaOlsson]

@Sulimankhail has restructured the example in the dev/feature/arch-proposal-draft branch so all the custom parts are now submodules in other repositories. Currently there is only epics for the sake of being able to run the example.

We have put a readme in the branch which explains how to run the example in examples/tune. This example still includes bluesky because we at the moment have no other option to make a runable example.

The example runs on a virtual accelerator of BESSY II built inside a container so to run it your unfortunately need apptainer to be installed. So the test is also a first step to figure out what is required to be able to use this type of virtual accelerator for unit tests.

[gubaidulinvadim]

@TeresiaOlsson, like you did for the measurement execution engine concept, could you please list other ways in which your suggestions will simplify the implementation? I also personally don't find that the implementation that we have at the moment is very complex.

[TeresiaOlsson]

@Sulimankhail can probably do it better than me and I don't fully understand all the parts of @JeanLucPons proposal so you need to tell me if I'm wrong but this is my understanding:

• Separation between data layer and business logic. The logical devices don't have to be created directly when loading the config but only at the point the user wants/needs them and then only the ones that are required. The user then for example has specified that they want a specific operational and control mode and the Liaison Manager maps things together and initialize the device with the required data and potential subdevices. In my understanding that can be done giving the same user interface in the end. It's just about how it happens under the hood. I also think this means that devices can be written more generic. For example I don't think we need a separate device for corrector, dipole, quadrupole, combined function etc. It can be a general magnet device which just different mappings. Also this mapping functionality makes it easier to map magnets to power supplies and vice versa since the mapping goes in both ways.

• The hardware to physics conversion is done by a dedicated functionality. So when the Liaison Manager has identified which components are involved, the Translation Service does the actual recalculation. This translation can also go in both directions. I think this means that you more easily can write a device which has get/set functionality but you don't have to specify if it's in hardware or physics units. This will be figured out and handled by the Liaison Manager and Translation Service at the point where the user says which control mode and units they want to use. That I think will make it much easier to write applications that are unit independent so labs can run them in either hardware or physics units depending on their needs.

[TeresiaOlsson]

I have some comments for @JeanLucPons proposal.

• If I correctly understand the latest implementation of the CombinedFunctionMagnet I can by just mapping a single component and choosing a simple unit conversion model for it get a CombinedFunctionMagnet object which behaves the same as a Magnet object. If that's the case wouldn't it be more clear for the users if the CombinedFunctionMagnet is turned into a generic magnet class and the Magnet class is turned into a class for a single multipole component? Then the user will always just create an object of the generic magnet class and it is the mapping and choice of unit conversion which detemine if it is a combined function magnet or not.

• I think we should consider to add a power supply layer. MML doesn't have this which is a problem for having magnet in series behave the same in the simulator as in the real machine. Just relying on hardware names to understand if a magnet is connected in series or not also doesn't feel very robust to me. It feels like a better option to explicitly have power supplies which are connected to magnets and when the user set a magnet, it's actually the underlying power supply that is doing it.

• For mapping to the lattice model I think we should also not rely on the FamName in a pyAT lattice since it's not always an unique identifier. My suggestion is that we introduce a standard for the lattice data model in pyAML (likely it can be the same as in pyAT but with the addition of unique identifiers).

  Then instead of directly relying on the load module in pyAT we do the following steps:

  • Load the lattice. Maybe as json since mat files will eventually go when no one has matlab anymore and pyAT can already extract as json.
  • Convert it into the format required by pyAML. At this step we create the unique identifiers if the lattice doesn't already have it. Maybe by incrementing the name as @swhite2401 suggested.
  • Then we can extract back to json and use the function in pyAT to load as json to build the lattice object, now with unique identifiers that we can use internally in pyAML for the mapping.

  I think this will make it easier to extend to other simulation codes in the future. For doing the conversion I think we can get inspiration from https://github.com/nobeam/latticejson which is using lark to parse grammar since that project had exactly this same idea of being able to convert different lattice formats to/from a standard format. @Sulimankhail is also doing something similar with lark to be able to load different lattice formats into the data models used for the lattice inside the HZB twin.

[JeanLucPons]

Figure 1 illustrates this design. The application layer in-cludes user-facing services, such as Orbit Response Measure-ment (ORM), LOCO, or Beam-Based Alignment (BBA) and many other high level user applications. The middle layer orchestrates measurement logic, update propagation, state management, and pattern execution.

With the above description from the article, it seems that we are agree 👍

[gubaidulinvadim]

@TeresiaOlsson to your third point only. The easiest solution is just to require users to prepare a lattice for pyAML properly with unique FamNames. Your suggestion is also good, but there's no harm is requiring people to prepare a lattice. I also agree than json should be used instead of mat.

[TeresiaOlsson]

@gubaidulinvadim Okay, we could do that. Maybe this problem is then connected to a plan for how to move from MML to pyAML? Do you think we should put as a requirement on pyAML that labs should be able to directly use their existing MML lattice or not? I'm not sure what is the best. Putting that requirement I guess will make it easier for labs to transition but on the other hand it might require a lot of extra work to support. I see the amount of work pyAT is putting in to be compatible with matlab AT and all the various things people might have put into their lattices... Might not be worth it and it's better to force people to update their lattices?

[gubaidulinvadim]

Some labs would not have MML or would only have a limited configuration of it. I don't think we should assume preexisting MML experience. In my view, we can require people to prepare a pyAML-compatible lattice (with unique FamNames, for example). It does kind of require a lattice in AT format (but for this most labs would have it).

[TeresiaOlsson]

Mm, I agree. Very likely it will anyway not be automatic to translate the configuration from MML so perhaps it's fine that some manual work will be required for both the config and the lattice.

Then if we say that unique FamName will be required we can perhaps also say that sliced elements need to follow the convention FamName_index and then we can use that to map elements that are sliced in the lattice model to the correct pyAML element?

[JeanLucPons]

@Sulimankhail ,

I look at your branch and I still don't understand how you want to dynamically load external modules ?
How you will handle different ways to measure the tune reponse matrix ? Familly of qd anf qf that gives a 2x2 matrix, random number of quads that gives nx2 matrix ? specific needs for tune calculation ? etc...
Could you share a bit on your development and propose how you will handle these different needs ?
Thanks

I recall that at the end we should be able to write something close to this:

# Here, arrays of quadrupoles used for tune are in the config file.
# If a tunemat file is already present in data_folder, update_optics() invert it to get tunecorrectionmat
# if both tunemat and tunecorrectionmat are already there, update_optics() do nothing concerning the tune
import pyaml
sr = pyaml('sr.yaml')
sr.live.tune.get()
[76.1585, 27.3401]
# request tune variation
sr.update_optics(model) # Computes tune response from the model and saves files tunemat and tunecorrectionmat in data_folder if needed
sr.live.tune.set([76.25, 27.44]) # Add tunecorrectionmat*dq to quads
sr.live.tune.get()
[76.2505, 27.4398]

Notes:

• For dynamic loading of a set of strength in the model, the optics data will be reseted.
• For usage of a measured response instead of a calculated one, the tune response measurement should be put in the data_folder prior the call to update_optics().
• By default, pyaml works with strenght, if the response is done in current, then sr.live.mode.set('current') should be called prior to set_tune().

[JeanLucPons]

@Sulimankhail

• We use calculated response matrix only for EBS (using fit_tune)
• Concerning dynamic loading of module, it is mandatory. We wont add all magnet models in pyaml core.
• I'm sorry but I still don't clearly understand how you want to handle various way of measuring the tune response. For instance for a non serialized family of qd and qf that should provide de 2x2 matrix ? How to use a factor instead of a step in the quad excitation ?
  It should be easy. The measurement engine should be able to run a random script that returns the data in simple manner.

I recall that pyaml should work on models without a measurement engine and without a control system.
Our views seems very different for the moment, and I really don't see how to converge...

[JeanLucPons]

Personally, i stop to work on the code until we find an agreement.

[TeresiaOlsson]

@JeanLucPons

• At HZB we use experimental response matrices . I wish we could also use a calculated one but our machine model is currently not great :( It's for the same reason we need to do it in current rather than strength.

• I think dynamic loading will also be required. @Sulimankhail: the idea is that if you have a magnet (or could also be some other device) which no one else in the world has you shouldn't have to go through the procedure of getting it merged into pyAML since no one else is interested in it anyway but instead you write it by yourself (following some standard defined by pyAML) and then pyAML can set up the config for it and use it as if it was a standard device. I guess similar to like you have HZB specific ophyd devices in bact?

• The third point I don't entirely understand. Do you mean that depending on how the quads are connected in series the dimensions of the matrix should be different? I thought this step would be connected to the power supplies? So if you send in an array of magnets, the code will figure out which power supplies they are connected to, change those power supplies and return that dimension of the matrix. But perhaps you need it to work in some other way because of your combined function magnets?

For me I think we can converge. We just need to understand our different use cases better and find a common solution that works for all labs. I now understand why you need to work in physics units at ESRF when we need to work in hardware units. I think it would help if we could go through the tune correction use case step-by-step and make a list of all required steps and which steps are same/different at different labs. And then do if for both live machine and simulator in case there are things that will differ between them.

[TeresiaOlsson]

@simoneliuzzo Just a small update, we have to support several format and if matrix is done in current, it for control system only, no ?

sr.live.tune.update_optics_dependent_properties(matrix=CSVMatrix('MeasuredTuneResp.csv'))

If you want to work like this and have global update_optics() function, it will have lots of params !

I think it would be nice if there was the possibility to have both a theoretical and experimental response matrix at the same time. In MML there is the concept of golden settings. So the update process is done in two steps:

1. You measure the response matrix. In MML this you can do on the live machine for the experimental one and on the simulator for the theoretical one. The theoretical one is then not derived analytically but just by running the exact same measurement on the simulator instead. The units are determined by what the user say in the input. So I can get the response matrix in either current or strength both for live and simulator. I can also use it in either unit later when using it for correction. MML handles the conversion to the required unit for the control mode. In MML however it is bad because it only saves the data in the specified unit. So if you measured in hardware units and later want in physics units you might need to run the measurement again or use some conversion function. I think @simoneliuzzo suggestion is that it should automatically save the data for both current and strength. The unit is then more for being able to specify how much a quad should be changed since some people know their machine in currents and some people in strengths.

2. Then you can assign the response matrix to the golden settings. In MML this is a separate function but I guess you could also call this function automatically if you want it to be done in one step. In MML however it is bad because I can only have one golden tune response matrix. It would be nicer if I could have two so I could have one theoretical and one experimental if I wanted to and then choose which one to use (maybe with some default setting too) when running the tune correction. But I can at least have one golden tune response matrix per operational mode.

[JeanLucPons]

• If you choose a family of Q that have different design strengths and you want to make a mean then you have to use a factor to get rid off the difference. (i.e. add 5% and not +constant). You can imagine tons of way of doing, this is why i suggest that it is to the responsibility of the user to write his scan as he want and to produce data. yaml core will just check if the result has the good size and provide an API that the user can use (or not) to send event to the core.
• Thanks for clarifications on how you are using experimental matrices, we should also be able to work without them.
• Yes dynamic modules are mandatory (not only for magnet): it brings only advantages even concerning maintainability. You don't have to take care about external modules in the core, you see directly in the config file if it is external or not. This is the classic way of adding external extensions. Matlab for instance does not take care about MML. It just provides a way to add extension and to load them dynamically via a shared lib, same for python.

[gubaidulinvadim]

It is not clear to me how the two proposals are related to each other. What are the advantages of one proposal over another? From what I see, the HZB version is a completely different way of doing things. I don't even see where the elements are in an HZB proposal. For example, where would a betatron tune monitor be in this scheme? You send a command to make a tune measurement / read a tune value from a device / calculate a tune from the lattice? And do a whole measurement execution engine to read a value from the control system?

[JeanLucPons]

No, the MarcroServer is the run engine, the device pool the hardware abstraction layer mainly used for motors.

[TeresiaOlsson]

I would say an ophyd device is not so different from a pyAML device. It has some set/read and a way to handle status to say when it's ready. So for pyAML the pyAML devices will just fill the same function ophyd has.

[TeresiaOlsson]

No, the MarcroServer is the run engine, the device pool the hardware abstraction layer mainly used for motors.

Ah, okay. Then it seems very different to ophyd. Or at least compared to how I have used ophyd so far.

[TeresiaOlsson]

Only reason for ophyd in the example is because bluesky compatible devices were needed to show how the measurement execution engine works. No need for ophyd in pyAML if we have our own measurement execution engine.

[TeresiaOlsson]

I agree and this is the main problem. The 2 approaches are totally divergent and 2 branches are totally unmergeable.
I my proposal, I presented only where I am in the dev and the tune monitor is not yet implemented. It will be implemented as other modules and will follow user specifications.
But before this, I want to know which branch to follow. So I stopped the dev. I don't want to work for nothing.

@JeanLucPons Are you planning to make a high-level view of your proposal? I would very much like to see it to be able to understand your thoughts on how different parts should connect together.

At the moment I don't entirely understand the data layer in your proposal. Is it correct that it isn't separated from the business logic? And I also don't understand if your proposal includes a measurement execution engine or not. And if not, what your proposal then is for automatically generating metadata and run applications in both physics and hardware units. I would have thought some kind of translation service would be required for that which makes use of the unit conversions of the devices? But maybe you have another idea of how that will work on the application level?

[JeanLucPons]

Here is the big picture:

[gubaidulinvadim]

Since we are now competing in who will do the best picture, I've redrawn the big picture of Jean-Luc in a BESSY style with some small modifications. Namely, I've highlighted the part that is included in the CORE of the Middle Layer that we need to develop. I hope this will eliminate any misunderstanding on the side of BESSY. At first sight, it does not seem that the two views on the big picture are VERY divergent. The scheme below is also aligned with the one presented by L. Nadolski during the pyAML workshop in Hamburg about MML (https://indico.desy.de/event/43233/contributions/169040/attachments/90944/122729/OperationML_nadolski.pdf)

[JeanLucPons]

Thanks Vadim. 👍
But I'm afraid it will be difficult to judge with only few rectangles that present what we all already know more or less.
The only concrete reading i can have now is that the 4/5 branches are coherent while 1/5 is totally different.
Even if the goal is the same, the 2 implementations are totally divergent.
Last but not least, we are against any dependencies like bluesky,bliss or sardana.
I can't anticipate the decision of steering committee and honestly I don't how they can judge in an objective manner.

[TeresiaOlsson]

I could set up the TODO list tomorrow after the maintainers' meeting in GitHub Projects if no one has any objection to it. I could look into implementing some DIAG devices (for RF, I rarely use something that is not MasterClock) in the core next week. And I will talk to Patrick regarding the Github actions setup (he was not available in the first half of July for pyAML, as I mentioned some time ago).

I like a lot. I can try to figure out the json schema and the loading of the configuration. I anyway wanted to test hydra to see if it's something that could help us manage our distributed yaml files or if it's useless for the type of structure we have. I can also start to write documentation for how to configure different editors and the MetaConfigurator to use the json schema so we can use it when we need to setup the config for tests on live machines.

[JeanLucPons]

I don't want to spend my time in evaluating tons of proposal with totally different implementations and in other hand I agree that I can't close the door to new proposals. Unfortunately, in software development, there are infinite ways of achieving one same goal.
Why Tango or Epics which achieve more or less same goals have totally different implementations ?

There are 3 possible immediate objectives in software development:

• Code size
• Performance
• Number of path

You can make a Pareto front in you want , each proposal should solve exactly the same problem same inputs, same outputs to make correct measurements. If you manage to find an optimum then you are likely on a good way but you may miss things that are difficult (even impossible) to measure such as possibility to extend the software, maintainability, learning curve, ...

[JeanLucPons]

Concerning deadline, Soleil II count on pyaml, we still have time, but not too much.

[gubaidulinvadim]

@TeresiaOlsson Jean-Luc is right concerning the timeline. The timeline that we all agreed on is the following:

May 2025 Close up documents (Specification and Governance)
June 2025 Start coding for V0.0.0 and implement 1 use case
Dec 2025 End coding for V0.0.0 and 1-2 use cases
Jan-Feb 2026 Soleil Workshop and (question) Hackathon

At the end of the year, we should have a prototype with working use cases. For me, this also includes somewhat complex use cases, the testing phase (on real machines and digital twins), where machine beam time and personal availability are also factors. And, for me, this also includes the first round of feedback after testing. There isn't so much time. And for SOLEIL II, we aim to make use of the pyAML. For this, we need to convince people that this is a good alternative to an MML. SOLEIL II is already in an accelerator construction phase, and the technical design phase is finished.

Also, all projects have different phases. In the case of PyAML, we've already had something close to one year of discussion, two in-person workshops, and countless weekly meetings. It is crucial to demonstrate that we can converge on working solutions, work together effectively and not devolve into infinite arguments. This does not mean that we should not have discussion but we should organise it productively. For this, I've made some minimal suggestions in the comments above on how to structure our proposals. We should make it easy to make decisions on implementation.

[TeresiaOlsson]

@gubaidulinvadim I agree with that timeline. I think it's still worth spending the time to evaluate options for the architecture. I think if we have an architecture which is modular and where it is clear for all maintainers what function the different parts have and how they fit together, it will be faster to make the implementation because we can divide into tasks and don't have to discuss every detail in the big maintainers group.

[swhite2401]

Dear all, it is very good you are having this discussion and I can see that it goes in the right direction, however, if I may I would like to suggest to keep discussion on project organization outside of github and use this repository only for the technical aspects of the projects.
It would make it easier for everyone to follow the code developments.
Cheers.

[gubaidulinvadim]

We close this discussion because now there are two separate discussion for the architecture proposals.