Ping-sort-GUI

WARNING: This Project has been archived (13 November 2025).
Active development has been migrated to the scientific software repo of MAX IV on GitLab.

The project below remains as a demo that mimics the main functionality of the full project.

It does not perform any of the real functionality mentioned in the description, but only showcases the interaction with the GUI front-end on some fabricated test data.

This README has largely kept the description of the full project for clarity.
Each change specific to the demo is mentioned as a "Demo Note" in the appropriate section.

Demo

Download the demo data from:

Download demo-data.zip

Unzip the archive into the data/ folder:

unzip demo-data.zip -d data/

Description

This project aims at providing a user-friendly desktop application with an intuitive graphical user interface (GUI) with the following functionality.

The app bins together detector output frames based on their ping values (time-delay values obtained from a MAUI oscilloscope) and forms an aggregate frame per bin, see Aggregate Frames.

This binning is done concurrently with the display of various visual constructs, see Ping Histogram and 2D plots, to allow users to easily see progess and intermediate results.

The app can be used on data being actively gathered during an experimental run. At repetition rates below ~1 kHz, the functionality of the app is nigh indistinguishable from a live post-processing tool.

Note: The terms "shot" and "frame" are used interchangeably throughout.

Demo Note

The ping histogram is filled with random numbers and the 2D plots with a single test image from a Pilatus detector.

The latter is saved four times in the scan-000000-exchange-{0-3}.h5 files from demo-data.zip. They are the same image, just rotated by 90 $^\circ$ each time.

To mimic an update, the current project simply cycles through these four files, such that an "update" is visibly discernable as a rotating frame in the 2D plots.

User Interface Overview

The application’s user interface is divided into the following main sections:

Ping Histogram

• Description: The top left figure displays a dynamic histogram of number of frames added per bin.
• Key Elements:
  • Histogram bars: Apart from displaying the bin frequency, the bars are interactable, see User Interactions Sect. 5.
  • Title: The title contains the total number of frames analyzed thus far.
  • Matplotlib: The figure embedded in the GUI is made using matplotlib.pyplot and inherents all of its functionality.

2D Plots

• Description: The bottom left contains four notebook tabs that can display a 2D plot each, depending on user interaction with the histogram (see User Interactions Sect. 5). See also Aggregate Frames.
• Key Elements:
  • Single Bin: Displays the aggregate frame from a single user-selected bin.
  • Group A: Displays the aggregate frames across multiple user-selected bins, defined as Group A.
  • Group B: Displays the aggregate frames across multiple user-selected bins, defined as Group B.
  • Diff (A-B): Displays the difference between the aggregate frames of groups A and B.
  • Title: The titles contain the total number of frames within the user-selected bin range, which is also displayed.
  • Matplotlib: The figure embedded in the GUI is made using matplotlib.pyplot and inherents all of its functionality.

Control Panel

• Description: The right half of the app is dedicated to user controls.
• Key Elements:
  • File-input: Select the file of raw data to be analyzed. Also allows for partial analysis of only a select range of shots, denoted by start and end index.
  • Bin parameters: Allows to set up n_bins, bin_size, bin_start, and bin_end. Though all can be set by the user, only one of the first two is strictly necessary. The remainder can be calculated from incoming data, see Auto-Completed Bin Parameters.
  • Run Controls: Used to Start and Pause a run. The aforementioned input parameters are locked during a run. A Clear button clears progress without saving and unlocks input widgets again. The current frame index being analyzed is displayed to show progress.
  • Group Selection: Allows to define and clear the Groups A & B, see User Interactions Sect. 5, and calculate the difference (A-B), see Aggregate Frames.
  • File-output: The user can specify an output file, to which the current data of the histogram and underlying aggregate frames will be saved, including the selected groups and difference data.
  • Update Now: Used for Update Now button. This makes a single batched update, see ZMQ Communication and Batched Progress, without saving.

Demo Note:

All elements of the user interface are still present in this demo, but are functional to a varying degree.

• File-input: All fields Input folder, Scan number, Start index, and End index are writeable, but otherwise do nothing. Similarly, the Browse button works insofar as it allows for the selection of a folder and writes it to the Input folder entry.
• Bin parameters: The bin parameters are fully functional, see [Auto-Completed Bin Parameters].
• Run Controls: Fully functional within the limited scope of this demo.
• Group Selection: Partially functional. Interaction with the Ping Histogram is possible, and is visible through changing aesthetics, but no underlying calculation or update is actually being done. See Demo Note in Description.
• File-output: Similar to File-input. The Entry is writeable and the Default button is fully functional. The Save button has been purposefully stripped of its function in this demo, despite what may be claimed by lingering pop-up text boxes.
• Update Now: Fully functional within the limited scope of this demo.

User Interactions

1. Opening the Application:

   • Upon launching the app, python src/main.py, from the root folder, ping-sort-GUI, a blank main frame is displayed.

2. Setting file-input parameters:

   • The entry Input folder allows one to fill in the absolute path of the folder in which the raw data to be analyzed is present. The Browse button can be used for this purpose as well.
   • The Scan number denotes which exact file in the input folder contains the relevant raw data.
   • The Start index and End index entries allows the user to select a range of frames within the raw data file to analyzed. The default considers the whole file (Start index=0 and End index=-1).

3. Initializing bin parameters:

   • All four bin-input entries, n_bins, bin_size, bin_start, and bin_end, can be set by the user if desired.
   • The bare minimal necessity is that either n_bins or bin_size is set. The remainder can be calculated from incoming data, see Auto-Completed Bin Parameters.
   • With the parameters filled out, the user can press the Init button, which will set the bin ranges in the Ping Histogram plot, as well as the bin widths, provided n_bins <= 100. This is not strictly necessary, but helps user verfication.

4. Starting a run:

   • After filling out all aforementioned user-input, the user can simply press the Start button to start a run. This locks the widgets that allow for user-input.
   • A run is done in batches of a hard-coded set of frames, see ZMQ Communication and Batched Progress, and the GUI updates its displays data every second.
   • The user can pause the updates with the Pause button, e.g., should they wish to examine an interesting or emergent feature. In doing so, the Start button will turn into a Continue button, to signal the progress of an ongoing run is not lost.
     Note that in pausing the run, there is no batched background update being done. The app is truly paused and in an idle state.
   • During a run the index of the shot being analyzed is displayed in the entry Processing shot (read-only).
   • A Clear button can be pressed to erase all progress done so far. It unlocks the input parameter widgets again and allows for a user to refine the parameters of a new run. Note that no prior information is saved in doing so.

5. Selecting groups:

   • At any time, the user can click on the Ping Histogram to select a single bin for inspection. The corresponding aggregate frame will be displayed in the lower left 2D plot under the tab Single Bin. Its title will display the number of frames in the selected bin and the bin range.
   • Alternatively, one can select one of the, mutually exclusive, toggles Select group A or Select group B. In doing so, a range of bins can be selected in the Ping Histogram, by clicking or dragging, and are identified as being part of the selected group.
     These bins are highlighted in orange for group A and green for group B, with overlapping bins denoted by orange with a green hatch.
     The corresponding aggregate frame, Aggregate Frames, will be displayed in the lower left 2D plot under the tabs Group A and Group B. Their titles display the number of frames in the selected group and the corresponding bin range.
   • Elements of the groups can be selected and deselected on multiple clicks and/or drags.
   • The groups can be cleared completely with the corresponding Clear group A and Clear group B buttons.
   • The Calculate A-B button will simply show the Diff (A-B) 2D plot if the run is ongoing.
     If instead, the GUI is paused, pressing Calculate A-B will indeed recalculate the difference A-B as selected by the user, but without a batch update, see ZMQ Communication and Batched Progress. I.e., in this state, the difference A-B can be recalculate at the current paused state of the run.

6. Saving output:

   • During a run, the user can specify an output file (with absolute path included) to which data can be saved.
   • The Default button sets this up to the same as the input file (absolute path included) with the suffix _binned added to the file name, as a suggestion.
   • The Save button subsequently saves the ping histogram, the associated binned aggregate frames, the groups A & B, and their respective difference.
   • Note that data is not automatically saved at the end of a run, since the GUI has no way of telling whether or not more data will be coming in. Hence, it doesn't know when a run "ends".

7. Update Now:

   • The Update Now button allows for the processing of a single batch of frames, see ZMQ Communication and Batched Progress. It has no effect if the GUI is in an active run state.

8. Example of GUI screen mid-run:

Demo Note

See Demo Note in User Interface Overview.

Algorithmic Details

Aggregate Frames

Demo Note: No actual frames are being added. The test data comprises of one single frame.

Auto-Completed Bin Parameters

Demo Note: The following logic is applied:

• At least two of the histogram parameters must be defined, of which one must be n_bins or bin_size.
• If all four are defined, a simple consistency check is run.
• If three of the four parameters are defined, the final one can be calcualted unambiguously.
• If two of the four parameters are defined, the following is used to fill out the remainder:
  • If n_bins and bin_size defined, set bin_start=0, calculate bin_end.
  • If n_bins and bin_start/end defined, bin_end/start = bin_start/end +/- 1, calculate bin_size.
  • If bin_size and bin_start/end defined, n_bins = 10 , calculate bin_end/start

ZMQ Communication and Batched Progress

The GUI consists of two largely independent and disconnected parts:

• The GUI front-end:
  • Written entirely in python, making use of tkinter.
  • Purpose is for display and UI only.
  • Performs near to no actual calculations.
• The C-core back-end:
  • Written entirely in C++.
  • Purpose is for efficient heavy-lifting calculations.
  • Can be used independently from the GUI front-end, using CLI.
• The intermediate ZMQ communication:
  • Both ends are communicating to one another over ZMQ, specifically over a REP/REQ socket at default address tcp://127.0.0.1:5555.
  • The socket is a single-thread message/reply setup: the GUI sends a message to the C-core, which recieves it and sends a reply to the GUI, which receives it and sends a reply to the C-core, which...
    Sending two messages without reply will lead to errors.
• Batched progress:
  • During a run, the GUI will send an update command every 1 sec. This value is currently hard-coded.
  • Upon receiving the command to update, the C-core processes at most 500 frames. This value is currently hard-coded.
  • The processed intermediate result (Ping histogram, group-aggreage frames, etc...) are saved to an exchange.h5 file. They are not sent over ZMQ. Only a success message is sent back to the GUI.
  • Upon receiving a successful reply, the GUI simply reads exchange.h5 file and displays the results therin.
  • A subsequent update overwrites a pre-existing exchange.h5 file.

Demo Note: The C-core is replaced in the demo by the mock_receiver.py found in the tests/ folder. It does not write or overwrite exchange files, but merely accepts messages and sends replies in the format expected by the GUI and performs the auto-completion of the bin parameters, Auto-Completed Bin Parameters.