Skip to content

Conversation

@rskew
Copy link
Contributor

@rskew rskew commented Oct 6, 2025

Motivation: For writing tests, its useful to have the stream stop once the frames have been processed, so the test code doesn't have to manually stop the stream.
When a data source uses create_frames() this happens nicely, as create_frames() stops the stream once all frames have been generated.
However, when we have a test that processes a single file, the default of use_create_frame=True causes DataSchemeFile to call create_frame() and then not stop the stream.

…es' start_stream() method to False, making the use of create_frames() the default
@geekscape
Copy link
Owner

There has been a long outstanding need to generally provide an out-of-the-box means of terminating Agents(Pipelines) processes in circumstances other than only CI tests …

  • Developer working on their Agent(Pipeline) and wanting that process to terminate once the single Stream stops
  • Same deal … but, when testing outside of … or within CI (your goal)
  • Production use-case, e.g a Highlighter Agent running a single Task and terminating once complete … equivalent to an Aiko Services Pipeline running a single Stream and terminating
    … those three cases above should be supported (easily out-of-the-box) regardless of the number of frames for that Stream … it could be one frame (File) (as in PR#49) or many frames (Files)

My preference is that the PR#49 suggestion shouldn’t conflict with an Agent(Pipeline), which is very long running and is processing lots of Tasks(Streams) each consisting of a single File(Frame) and not having the unnecessary overhead of starting and stopping a background thread for every Task(Stream) in that case. This use-case was the original motivation for the use_create_frame default of True (optimal performance).

@rskew
Copy link
Contributor Author

rskew commented Oct 6, 2025

Thanks @geekscape , sounds like I'll leave it with you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants