From 18a7b4c0a3f36e6f92940254b3a2207af93f0f30 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Thu, 5 Jun 2025 09:35:33 -0700 Subject: [PATCH 01/11] Add close-loop and camera tutorial --- tutorials/hobgoblin-closeloop.md | 3 +++ tutorials/toc.yml | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 tutorials/hobgoblin-closeloop.md diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md new file mode 100644 index 00000000..761cbe1e --- /dev/null +++ b/tutorials/hobgoblin-closeloop.md @@ -0,0 +1,3 @@ +# Close Loop Control + +The exercises below will help you become familiar with using the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) device for close-loop experiments. You will also learn how to use the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) to interface with external cameras. Before you begin, it is recommended that you review the Bonsai [Acquisition and Tracking](https://bonsai-rx.org/docs/tutorials/acquisition.html) tutorial, which covers key video concepts. \ No newline at end of file diff --git a/tutorials/toc.yml b/tutorials/toc.yml index 3d2fb9aa..bb9787a4 100644 --- a/tutorials/toc.yml +++ b/tutorials/toc.yml @@ -1,4 +1,5 @@ - name: Harp Hobgoblin - href: hobgoblin-setup.md - href: hobgoblin-acquisition.md -- href: hobgoblin-reaction.md \ No newline at end of file +- href: hobgoblin-reaction.md +- href: hobgoblin-closeloop.md From 8687aba031e6e1d9823e8c1c0e29301e281b7a4f Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Thu, 5 Jun 2025 11:32:56 -0700 Subject: [PATCH 02/11] Add close loop latency blurb and device pattern --- tutorials/hobgoblin-closeloop.md | 17 ++++- .../hobgoblin-closeloop-devicepattern.bonsai | 43 +++++++++++ .../hobgoblin-closeloop-devicepattern.svg | 73 +++++++++++++++++++ 3 files changed, 131 insertions(+), 2 deletions(-) create mode 100644 workflows/hobgoblin-closeloop-devicepattern.bonsai create mode 100644 workflows/hobgoblin-closeloop-devicepattern.svg diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index 761cbe1e..8acc5c1e 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -1,3 +1,16 @@ -# Close Loop Control +# Close-Loop Systems -The exercises below will help you become familiar with using the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) device for close-loop experiments. You will also learn how to use the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) to interface with external cameras. Before you begin, it is recommended that you review the Bonsai [Acquisition and Tracking](https://bonsai-rx.org/docs/tutorials/acquisition.html) tutorial, which covers key video concepts. \ No newline at end of file +The exercises below will help you become familiar with using the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) device for close-loop experiments. You will also learn how to use the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) to interface with external cameras. Before you begin, it is recommended that you review the Bonsai [Acquisition and Tracking](https://bonsai-rx.org/docs/tutorials/acquisition.html) tutorial, which covers key video concepts. + +## Close-loop latency +In a closed-loop experiment, we want the behaviour data to generate feedback in real-time into the external world, establishing a relationship where the output of the system depends on detected sensory input. Many behavioural experiments in neuroscience require some kind of closed-loop interaction between the subject and the experimental setup. + +One of the most important benchmarks to evaluate the performance of a closed-loop system is the latency, or the time it takes for a change in the output to be generated in response to a change in the input. The easiest way to measure the latency of a closed-loop system is to use a digital feedback test. + +In this test, we measure a binary output from the closed-loop system and feed it directly into the input sensor. We then record a series of measurements where we change the output to `HIGH` if the sensor detects `LOW`, and change it to `LOW` if the sensor detects `HIGH`. The time interval between `HIGH` and `LOW` signals will give us the total closed-loop latency of the system, also known as the round-trip time. + +Before begining, set up the `Hobgoblin` with the following `device pattern` that we learned about in the previous tutorial. + +:::workflow +![Hobgoblin Device Pattern](../workflows/hobgoblin-closeloop-devicepattern.bonsai) +::: \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-devicepattern.bonsai b/workflows/hobgoblin-closeloop-devicepattern.bonsai new file mode 100644 index 00000000..c8d269c7 --- /dev/null +++ b/workflows/hobgoblin-closeloop-devicepattern.bonsai @@ -0,0 +1,43 @@ + + + + + + Hobgoblin Commands + + + + Active + On + true + On + Disabled + false + COM7 + + + + + Hobgoblin.harp + true + false + Include + + + + + Hobgoblin Events + + + + + + + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-devicepattern.svg b/workflows/hobgoblin-closeloop-devicepattern.svg new file mode 100644 index 00000000..fbe9cdb2 --- /dev/null +++ b/workflows/hobgoblin-closeloop-devicepattern.svg @@ -0,0 +1,73 @@ + + +]> + + + + + + + + + + + + + + + Hobgoblin + + + Events + + + + + + Hobgoblin + + + DataWriter + + + + + + Hobgoblin + + + + + + + Hobgoblin + + + Commands + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From 17b1cc7e60694cad1601a032ae22ed318e2a9542 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Thu, 5 Jun 2025 15:57:45 -0700 Subject: [PATCH 03/11] Add 1st exercise on serial port close-loop Co-authored-by: bruno-f-cruz <7049351+bruno-f-cruz@users.noreply.github.com> --- tutorials/hobgoblin-closeloop.md | 71 ++++++++- workflows/hobgoblin-closeloop-latency.bonsai | 60 +++++++ workflows/hobgoblin-closeloop-latency.svg | 157 +++++++++++++++++++ 3 files changed, 283 insertions(+), 5 deletions(-) create mode 100644 workflows/hobgoblin-closeloop-latency.bonsai create mode 100644 workflows/hobgoblin-closeloop-latency.svg diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index 8acc5c1e..6c7ecba8 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -1,16 +1,77 @@ # Close-Loop Systems -The exercises below will help you become familiar with using the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) device for close-loop experiments. You will also learn how to use the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) to interface with external cameras. Before you begin, it is recommended that you review the Bonsai [Acquisition and Tracking](https://bonsai-rx.org/docs/tutorials/acquisition.html) tutorial, which covers key video concepts. +The exercises below will help you become familiar with using the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) device for close-loop experiments. You will also learn how to use the `Hobgoblin` to interface with external cameras. Before you begin, it is recommended that you review the Bonsai [Acquisition and Tracking](https://bonsai-rx.org/docs/tutorials/acquisition.html) tutorial, which covers key video concepts. + +## Prerequisites + +- Install the `Bonsai.Dsp` package from the [Bonsai package manager](https://bonsai-rx.org/docs/articles/packages.html). ## Close-loop latency In a closed-loop experiment, we want the behaviour data to generate feedback in real-time into the external world, establishing a relationship where the output of the system depends on detected sensory input. Many behavioural experiments in neuroscience require some kind of closed-loop interaction between the subject and the experimental setup. One of the most important benchmarks to evaluate the performance of a closed-loop system is the latency, or the time it takes for a change in the output to be generated in response to a change in the input. The easiest way to measure the latency of a closed-loop system is to use a digital feedback test. -In this test, we measure a binary output from the closed-loop system and feed it directly into the input sensor. We then record a series of measurements where we change the output to `HIGH` if the sensor detects `LOW`, and change it to `LOW` if the sensor detects `HIGH`. The time interval between `HIGH` and `LOW` signals will give us the total closed-loop latency of the system, also known as the round-trip time. - -Before begining, set up the `Hobgoblin` with the following `device pattern` that we learned about in the previous tutorial. +Before beginning, set up the `Hobgoblin` with the following `device pattern` that we learned about in the previous tutorial. :::workflow ![Hobgoblin Device Pattern](../workflows/hobgoblin-closeloop-devicepattern.bonsai) -::: \ No newline at end of file +::: + +- Set the `DumpRegisters` property in the `Hobgoblin` [`Device`] operator to `False`. This is to avoid triggering the command loop in the next exercise. + +![Hobgoblin LED](../images/hobgoblin-acquisition-led.svg){width=400px} + +- Connect one of the LED modules to digital output channel `GP15` on the `Hobgoblin`. + +### Exercise 1: Measuring serial port communication latency + +We will take advantage of the device's ability to echo back a timestamped message upon command execution to create a simple loop where each echo re-triggers the same command (toggling the digital output channel `HIGH` and `LOW`). The time interval between the echoes will give us the total closed-loop latency of the system, also known as the round-trip time. + +:::workflow +![Hobgoblin Closed-Loop Latency](../workflows/hobgoblin-closeloop-latency.bonsai) +::: + +- Insert a [`SubscribeSubject`] operator and configure the `Name` property to `Hobgoblin Events`. +- Insert a [`Parse`] operator and select [`TimestampedDigitalOutputTogglePayload`] from the `Register` property dropdown menu. +- Insert a [`CreateMessage`] operator, select [`DigitalOutputTogglePayload`] for the `Payload`, and `GP15` for the [`DigitalOutputToggle`] property. +- Insert a [`MulticastSubject`] operator and configure the `Name` property to `Hobgoblin Commands`. + +In order to measure the difference between the timestamps: + +- Right-click on the [`Parse`] operator, select the `Output (Bonsai.Harp.Timestamped)` > `Seconds`. +- Disconnect the `Seconds` node from the [`CreateMessage`] operator. +- Reconnect the [`Parse`] and [`CreateMessage`] operator. +- After the `Seconds` node, insert a [`Difference`] operator from the `Bonsai.Dsp` package. + +Lastly, we will use this sequence to initialize the command loop: + +- Insert a [`KeyDown`] operator and set the `Filter` property to the key `A`. +- Insert a [`Parse`] operator and select [`DigitalOutputTogglePayload`] from the `Register` property dropdown menu. Set the `DigitalOutputToggle` property to `GP15` +- Insert a [`MulticastSubject`] operator and configure the `Name` property to `Hobgoblin Commands`. +- Run the workflow, and open the visualizer for the [`Difference`] operator. +**What do you observe?** + + +[`CreateMessage`]: xref:Harp.Hobgoblin.CreateMessage +[`Device`]: xref:Harp.Hobgoblin.Device +[`Difference`]: xref:Bonsai.Dsp.Difference +[`DigitalOutputToggle`]: xref:Harp.Hobgoblin.DigitalOutputToggle +[`DigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.CreateDigitalOutputTogglePayload + + + + + + +[`KeyDown`]: xref:Bonsai.Windows.Input.KeyDown + +[`MulticastSubject`]: xref:Bonsai.Expressions.MulticastSubject +[`Parse`]: xref:Harp.Hobgoblin.Parse + + +[`SubscribeSubject`]: xref:Bonsai.Expressions.SubscribeSubject + +[`TimestampedDigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.TimestampedDigitalOutputToggle + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-latency.bonsai b/workflows/hobgoblin-closeloop-latency.bonsai new file mode 100644 index 00000000..2de60de9 --- /dev/null +++ b/workflows/hobgoblin-closeloop-latency.bonsai @@ -0,0 +1,60 @@ + + + + + + Hobgoblin Events + + + + + + Write + + GP15 + + + + Hobgoblin Commands + + + Seconds + + + + 1 + + + + + A + false + + + + Write + + GP15 + + + + Hobgoblin Commands + + + + + + + + + + + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-latency.svg b/workflows/hobgoblin-closeloop-latency.svg new file mode 100644 index 00000000..f58f518a --- /dev/null +++ b/workflows/hobgoblin-closeloop-latency.svg @@ -0,0 +1,157 @@ + + +]> + + + + + + + + + + + + + + + + + Hobgoblin + + + Commands + + + + + Difference + + + + + + + Hobgoblin + + + Commands + + + + + + Hobgoblin. + + + DigitalOutput + + + TogglePayload + + + + + + + Seconds + + + + + + Hobgoblin. + + + DigitalOutput + + + TogglePayload + + + + + Hobgoblin. + + + Timestamped + + + DigitalOutputT + + + + + + + KeyDown + + + + + + + Hobgoblin + + + Events + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From b8af8b9e190ba03777eb13884cac01542c9de49a Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Thu, 5 Jun 2025 22:21:32 -0700 Subject: [PATCH 04/11] Add 2nd exercise on measure video acquisition latency --- tutorials/hobgoblin-closeloop.md | 62 ++++++-- ...closeloop-latency-video-measurement.bonsai | 31 ++++ ...in-closeloop-latency-video-measurement.svg | 69 +++++++++ .../hobgoblin-closeloop-latency-video.bonsai | 65 +++++++++ .../hobgoblin-closeloop-latency-video.svg | 134 ++++++++++++++++++ 5 files changed, 353 insertions(+), 8 deletions(-) create mode 100644 workflows/hobgoblin-closeloop-latency-video-measurement.bonsai create mode 100644 workflows/hobgoblin-closeloop-latency-video-measurement.svg create mode 100644 workflows/hobgoblin-closeloop-latency-video.bonsai create mode 100644 workflows/hobgoblin-closeloop-latency-video.svg diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index 6c7ecba8..f7a343d5 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -4,7 +4,7 @@ The exercises below will help you become familiar with using the [Harp Hobgoblin ## Prerequisites -- Install the `Bonsai.Dsp` package from the [Bonsai package manager](https://bonsai-rx.org/docs/articles/packages.html). +- Install the `Bonsai.Dsp`, `Bonsai.Video` and `Bonsai.Vision` packages from the [Bonsai package manager](https://bonsai-rx.org/docs/articles/packages.html). ## Close-loop latency In a closed-loop experiment, we want the behaviour data to generate feedback in real-time into the external world, establishing a relationship where the output of the system depends on detected sensory input. Many behavioural experiments in neuroscience require some kind of closed-loop interaction between the subject and the experimental setup. @@ -19,10 +19,6 @@ Before beginning, set up the `Hobgoblin` with the following `device pattern` tha - Set the `DumpRegisters` property in the `Hobgoblin` [`Device`] operator to `False`. This is to avoid triggering the command loop in the next exercise. -![Hobgoblin LED](../images/hobgoblin-acquisition-led.svg){width=400px} - -- Connect one of the LED modules to digital output channel `GP15` on the `Hobgoblin`. - ### Exercise 1: Measuring serial port communication latency We will take advantage of the device's ability to echo back a timestamped message upon command execution to create a simple loop where each echo re-triggers the same command (toggling the digital output channel `HIGH` and `LOW`). The time interval between the echoes will give us the total closed-loop latency of the system, also known as the round-trip time. @@ -41,9 +37,9 @@ In order to measure the difference between the timestamps: - Right-click on the [`Parse`] operator, select the `Output (Bonsai.Harp.Timestamped)` > `Seconds`. - Disconnect the `Seconds` node from the [`CreateMessage`] operator. - Reconnect the [`Parse`] and [`CreateMessage`] operator. -- After the `Seconds` node, insert a [`Difference`] operator from the `Bonsai.Dsp` package. +- After the `Seconds` node, insert a [`Difference`] operator. -Lastly, we will use this sequence to initialize the command loop: +Lastly, we will use this sequence to toggle the digital output and initialize the command loop: - Insert a [`KeyDown`] operator and set the `Filter` property to the key `A`. - Insert a [`Parse`] operator and select [`DigitalOutputTogglePayload`] from the `Register` property dropdown menu. Set the `DigitalOutputToggle` property to `GP15` @@ -51,17 +47,66 @@ Lastly, we will use this sequence to initialize the command loop: - Run the workflow, and open the visualizer for the [`Difference`] operator. **What do you observe?** +### Exercise 2: Measuring video acquisition latency + +![Hobgoblin LED](../images/hobgoblin-acquisition-led.svg){width=400px} +- Connect a blue LED module to digital output channel `GP15` on the `Hobgoblin`. + +:::workflow +![Hobgoblin Closed-Loop Latency Video](../workflows/hobgoblin-closeloop-latency-video.bonsai) +::: + +- Insert a [`VideoCaptureDevice`] operator. +- Insert a [`Crop`] transform. +- Run the workflow and set the `RegionOfInterest` property to a small area around the LED. + +> [!Tip] +> You can use the visual editor for an easier calibration. While the workflow is running, right-click on the [`Crop`] transform and select `Show Default Editor` from the context menu or click in the small button with ellipsis that appears when you select the `RegionOfInterest` property. + +- Insert a [`Sum`] transform and select the `Val0` field from the output. + +> [!Note] +> The [`Sum`] operator adds the value of all the pixels in the image together, across all the color channels. Assuming the default BGR format, the result of summing all the pixels in the `Blue` channel of the image will be stored in `Val0`. `Val1` and `Val2` would store the `Green` and `Red` values, respectively. If you are using an LED with a color other than blue, please select the output field accordingly. + +- Insert a [`GreaterThan`] transform. +- Insert a [`BitwiseNot`] transform. +- Insert a [`CreateMessage`] operator, select [`DigitalOutputTogglePayload`] for the `Payload`, and `GP15` for the [`DigitalOutputToggle`] property. +- Insert a [`MulticastSubject`] operator and configure the `Name` property to `Hobgoblin Commands`. +- Run the workflow and use the visualizer of the [`Sum`] operator to choose an appropriate threshold for [`GreaterThan`]. You can use the [`KeyDown`] toggle snippet from the previous exercise to manually toggle the LED. +- Insert a [`DistinctUntilChanged`] operator after the [`BitwiseNot`] transform. + +> [!Note] +> The [`DistinctUntilChanged`] operator filters consecutive duplicate items from an observable sequence. In this case, we want to change the value of the LED only when the threshold output changes from `LOW` to `HIGH`, or vice-versa. This will let us measure correctly the latency between detecting a change in the input and measuring the response to that change. + +In order to measure the round-trip time between the LED toggle: + +:::workflow +![Hobgoblin Closed-Loop Latency Video Measurement](../workflows/hobgoblin-closeloop-latency-video-measurement.bonsai) +::: + +- Insert a [`SubscribeSubject`] operator and configure the `Name` property to `Hobgoblin Events`. +- Insert a [`Parse`] operator and select [`TimestampedDigitalOutputTogglePayload`] from the `Register` property dropdown menu. +- Right-click on the [`Parse`] operator, select `Output (Bonsai.Harp.Timestamped)` > `Seconds` from the context menu. +- Insert a [`Difference`] operator. +- Run the workflow and open the visualizer for the [`Difference`] operator. + +_Given the measurements obtained in Exercise 2, what would you estimate is the **input** latency for video acquisition?_ + +[`BitwiseNot`]: xref:Bonsai.Expressions.BitwiseNotBuilder [`CreateMessage`]: xref:Harp.Hobgoblin.CreateMessage +[`Crop`]: xref:Bonsai.Vision.Crop [`Device`]: xref:Harp.Hobgoblin.Device [`Difference`]: xref:Bonsai.Dsp.Difference [`DigitalOutputToggle`]: xref:Harp.Hobgoblin.DigitalOutputToggle [`DigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.CreateDigitalOutputTogglePayload +[`DistinctUntilChanged`]: xref:Bonsai.Reactive.DistinctUntilChanged +[`GreaterThan`]: xref:Bonsai.Expressions.GreaterThanBuilder [`KeyDown`]: xref:Bonsai.Windows.Input.KeyDown @@ -70,8 +115,9 @@ Lastly, we will use this sequence to initialize the command loop: [`SubscribeSubject`]: xref:Bonsai.Expressions.SubscribeSubject - +[`Sum`]: xref:Bonsai.Dsp.Sum [`TimestampedDigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.TimestampedDigitalOutputToggle +[`VideoCaptureDevice`]: xref:Bonsai.Video.VideoCaptureDevice \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-latency-video-measurement.bonsai b/workflows/hobgoblin-closeloop-latency-video-measurement.bonsai new file mode 100644 index 00000000..06c27734 --- /dev/null +++ b/workflows/hobgoblin-closeloop-latency-video-measurement.bonsai @@ -0,0 +1,31 @@ + + + + + + Hobgoblin Events + + + + + + Seconds + + + + 1 + + + + + + + + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-latency-video-measurement.svg b/workflows/hobgoblin-closeloop-latency-video-measurement.svg new file mode 100644 index 00000000..dc55fdb3 --- /dev/null +++ b/workflows/hobgoblin-closeloop-latency-video-measurement.svg @@ -0,0 +1,69 @@ + + +]> + + + + + + + + + + + Difference + + + + + + + Seconds + + + + + Hobgoblin. + + + Timestamped + + + DigitalOutputT + + + + + + + Hobgoblin + + + Events + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-latency-video.bonsai b/workflows/hobgoblin-closeloop-latency-video.bonsai new file mode 100644 index 00000000..85a61ef5 --- /dev/null +++ b/workflows/hobgoblin-closeloop-latency-video.bonsai @@ -0,0 +1,65 @@ + + + + + + + 2 + + + + + + + 0 + 0 + 0 + 0 + + + + + + + + Val0 + + + + 0 + + + + + + + + Write + + GP15 + + + + Hobgoblin Commands + + + + + + + + + + + + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-latency-video.svg b/workflows/hobgoblin-closeloop-latency-video.svg new file mode 100644 index 00000000..6d39df81 --- /dev/null +++ b/workflows/hobgoblin-closeloop-latency-video.svg @@ -0,0 +1,134 @@ + + +]> + + + + + + + + + + + + + + + + + + Hobgoblin + + + Commands + + + + + + Hobgoblin. + + + DigitalOutput + + + TogglePayload + + + + + DistinctUntil + + + Changed + + + + + + + BitwiseNot + + + + + + + GreaterThan + + + + + + + Val0 + + + + + Sum + + + + + + Crop + + + + + + + VideoCapture + + + Device + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From 7f9ba384ea714508a3fd9c2491bd7854da583f42 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Thu, 5 Jun 2025 22:30:27 -0700 Subject: [PATCH 05/11] Add bonsai.video xrefmap to docfx.json for prerequisite --- docfx.json | 1 + 1 file changed, 1 insertion(+) diff --git a/docfx.json b/docfx.json index f3632507..ab133e42 100644 --- a/docfx.json +++ b/docfx.json @@ -105,6 +105,7 @@ "xref": [ "https://bonsai-rx.org/docs/xrefmap.yml", "https://bonsai-rx.org/gui/xrefmap.yml", + "https://bonsai-rx.org/video/xrefmap.yml", "https://horizongir.github.io/opencv.net/xrefmap.yml", "https://horizongir.github.io/ZedGraph/xrefmap.yml", "https://horizongir.github.io/opentk/xrefmap.yml", From 00d4463ae1352232cb6967384e37842dbaaea1a5 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Fri, 6 Jun 2025 03:07:22 -0700 Subject: [PATCH 06/11] Add 3rd exercise introduction to pulse trains --- tutorials/hobgoblin-closeloop.md | 40 +++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index f7a343d5..fe2c7b0d 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -1,6 +1,6 @@ # Close-Loop Systems -The exercises below will help you become familiar with using the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) device for close-loop experiments. You will also learn how to use the `Hobgoblin` to interface with external cameras. Before you begin, it is recommended that you review the Bonsai [Acquisition and Tracking](https://bonsai-rx.org/docs/tutorials/acquisition.html) tutorial, which covers key video concepts. +The exercises below will help you become familiar with using the [Harp Hobgoblin](https://github.com/harp-tech/device.hobgoblin) device for close-loop experiments. You will also learn how to interface with external cameras. Before you begin, it is recommended that you review the Bonsai [Acquisition and Tracking](https://bonsai-rx.org/docs/tutorials/acquisition.html) tutorial, which covers key video concepts. ## Prerequisites @@ -50,7 +50,7 @@ Lastly, we will use this sequence to toggle the digital output and initialize th ### Exercise 2: Measuring video acquisition latency ![Hobgoblin LED](../images/hobgoblin-acquisition-led.svg){width=400px} -- Connect a blue LED module to digital output channel `GP15` on the `Hobgoblin`. +- Connect a red LED module to digital output channel `GP15` on the `Hobgoblin`. :::workflow ![Hobgoblin Closed-Loop Latency Video](../workflows/hobgoblin-closeloop-latency-video.bonsai) @@ -63,10 +63,10 @@ Lastly, we will use this sequence to toggle the digital output and initialize th > [!Tip] > You can use the visual editor for an easier calibration. While the workflow is running, right-click on the [`Crop`] transform and select `Show Default Editor` from the context menu or click in the small button with ellipsis that appears when you select the `RegionOfInterest` property. -- Insert a [`Sum`] transform and select the `Val0` field from the output. +- Insert a [`Sum`] transform and select the `Val2` field from the output. > [!Note] -> The [`Sum`] operator adds the value of all the pixels in the image together, across all the color channels. Assuming the default BGR format, the result of summing all the pixels in the `Blue` channel of the image will be stored in `Val0`. `Val1` and `Val2` would store the `Green` and `Red` values, respectively. If you are using an LED with a color other than blue, please select the output field accordingly. +> The [`Sum`] operator adds the value of all the pixels in the image together, across all the color channels. Assuming the default BGR format, the result of summing all the pixels in the `Red` channel of the image will be stored in `Val2`. `Val0` and `Val1` would store the `Blue` and `Green` values, respectively. If you are using an LED with a color other than red, please select the output field accordingly. - Insert a [`GreaterThan`] transform. - Insert a [`BitwiseNot`] transform. @@ -92,6 +92,34 @@ In order to measure the round-trip time between the LED toggle: _Given the measurements obtained in Exercise 2, what would you estimate is the **input** latency for video acquisition?_ +## Closed-Loop Control + +### Exercise 3: Introduction to pulse trains + +In neuroscience, pulse trains are commonly used to deliver precisely timed sequences of electrical states (`LOW` and `HIGH`) to control external devices, such as cameras for synchronization or lasers for optogenetic stimulation. However, due to operating system limitations, generating pulse trains by setting [`DigitalOutputSet`] and [`DigitalOutputClear`] in software like Bonsai can be prone to timing jitter (though this approach is sufficient for the stimuli we have been working with). Fortunately the `Hogoblin` provides dedicated `Registers` that can be used to start or stop hardware-programmed pulse trains. We will use such pulse trains as our close-loop stimuli for the next few exercises. + +- Connect a red LED module to digital output channel `GP15` on the `Hobgoblin`. +- Insert a [`KeyDown`] operator and set the `Filter` property to the key `A`. +- Insert a [`Parse`] operator and select [`StartPulseTrainPayload`] from the `Register` property dropdown menu. Set the `DigitalOutput` property to `GP15`. +- Set the `PulseCount` property to `0`, `PulsePeriod` to `50000` and `PulseWidth` to `5000`. These parameters correspond to a continuous 20Hz pulse train with 5 ms pulses. +- Insert a [`MulticastSubject`] operator and configure the `Name` property to `Hobgoblin Commands`. +- Insert another [`KeyDown`] operator and set the `Filter` property to the key `S`. +- Insert a [`Parse`] operator and select [`StopPulseTrainPayload`] from the `Register` property dropdown menu. Set the `StopPulseTrain` property to `GP15`. +- Run the workflow, use the `A` and `S` keys to start and stop the pulse train. +**What do you observe?** + +To better understand what each parameter controls, try the following modifications. Reset the values to the parameters above after each step. + +- Increase the `PulsePeriod` to `200000`. What is the frequency of this stimulation? How would you increase the frequency of the pulses to 40Hz? +- Increase the `PulseWidth` to `40000`. What do you observe? +- How would you deliver a 2 second pulse train? (Hint: Use `PulseCount`) + +>[!NOTE] +> **Optional** Verify the pulse train by connecting the output to a digital input pin. How would you visualize the results using what you've learned? + +>[!NOTE] +> **Optional** If you have cameras that support external hardware triggering, use what you have learned in this exercise to trigger frame capture. + [`BitwiseNot`]: xref:Bonsai.Expressions.BitwiseNotBuilder [`CreateMessage`]: xref:Harp.Hobgoblin.CreateMessage @@ -102,8 +130,8 @@ _Given the measurements obtained in Exercise 2, what would you estimate is the * [`DigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.CreateDigitalOutputTogglePayload [`DistinctUntilChanged`]: xref:Bonsai.Reactive.DistinctUntilChanged - - +[`DigitalOutputSet`]: xref:Harp.Hobgoblin.DigitalOutputSet +[`DigitalOutputClear`]: xref:Harp.Hobgoblin.DigitalOutputClear [`GreaterThan`]: xref:Bonsai.Expressions.GreaterThanBuilder From d7527efa0844c6946d58d871bcf018932cbcdb54 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Fri, 6 Jun 2025 04:25:49 -0700 Subject: [PATCH 07/11] Add missing xref for pulse train operators --- tutorials/hobgoblin-closeloop.md | 23 ++-- .../hobgoblin-closeloop-pulsetrain.bonsai | 51 ++++++++ workflows/hobgoblin-closeloop-pulsetrain.svg | 110 ++++++++++++++++++ 3 files changed, 175 insertions(+), 9 deletions(-) create mode 100644 workflows/hobgoblin-closeloop-pulsetrain.bonsai create mode 100644 workflows/hobgoblin-closeloop-pulsetrain.svg diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index fe2c7b0d..fd28bfa2 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -50,7 +50,7 @@ Lastly, we will use this sequence to toggle the digital output and initialize th ### Exercise 2: Measuring video acquisition latency ![Hobgoblin LED](../images/hobgoblin-acquisition-led.svg){width=400px} -- Connect a red LED module to digital output channel `GP15` on the `Hobgoblin`. +- Connect a LED module to digital output channel `GP15` on the `Hobgoblin`. :::workflow ![Hobgoblin Closed-Loop Latency Video](../workflows/hobgoblin-closeloop-latency-video.bonsai) @@ -96,29 +96,33 @@ _Given the measurements obtained in Exercise 2, what would you estimate is the * ### Exercise 3: Introduction to pulse trains -In neuroscience, pulse trains are commonly used to deliver precisely timed sequences of electrical states (`LOW` and `HIGH`) to control external devices, such as cameras for synchronization or lasers for optogenetic stimulation. However, due to operating system limitations, generating pulse trains by setting [`DigitalOutputSet`] and [`DigitalOutputClear`] in software like Bonsai can be prone to timing jitter (though this approach is sufficient for the stimuli we have been working with). Fortunately the `Hogoblin` provides dedicated `Registers` that can be used to start or stop hardware-programmed pulse trains. We will use such pulse trains as our close-loop stimuli for the next few exercises. +In neuroscience, pulse trains are commonly used to deliver precisely timed sequences of electrical states (`LOW` and `HIGH` - otherwise known as transistor-transitor-logic or `TTL`) to control external devices, such as cameras for synchronization or lasers for optogenetic stimulation. However, due to operating system limitations, generating pulse trains in software like Bonsai can be prone to timing jitter (though this approach is sufficient for the stimuli we have been working with). Fortunately the `Hogoblin` provides dedicated `Registers` that can be used to start or stop hardware-programmed pulse trains. + +:::workflow +![Hobgoblin Pulse Train](../workflows/hobgoblin-closeloop-pulsetrain.bonsai) +::: - Connect a red LED module to digital output channel `GP15` on the `Hobgoblin`. - Insert a [`KeyDown`] operator and set the `Filter` property to the key `A`. - Insert a [`Parse`] operator and select [`StartPulseTrainPayload`] from the `Register` property dropdown menu. Set the `DigitalOutput` property to `GP15`. -- Set the `PulseCount` property to `0`, `PulsePeriod` to `50000` and `PulseWidth` to `5000`. These parameters correspond to a continuous 20Hz pulse train with 5 ms pulses. +- Set the `PulseCount` property to `0`, `PulsePeriod` to `50000` and `PulseWidth` to `5000`. These parameters correspond to a continuous 20 Hz pulse train with 5 ms pulses. - Insert a [`MulticastSubject`] operator and configure the `Name` property to `Hobgoblin Commands`. - Insert another [`KeyDown`] operator and set the `Filter` property to the key `S`. - Insert a [`Parse`] operator and select [`StopPulseTrainPayload`] from the `Register` property dropdown menu. Set the `StopPulseTrain` property to `GP15`. -- Run the workflow, use the `A` and `S` keys to start and stop the pulse train. +- Run the workflow, use the `A` and `S` keys to start and stop the pulse train. **What do you observe?** To better understand what each parameter controls, try the following modifications. Reset the values to the parameters above after each step. -- Increase the `PulsePeriod` to `200000`. What is the frequency of this stimulation? How would you increase the frequency of the pulses to 40Hz? +- Increase the `PulsePeriod` to `200000`. What is the frequency of this stimulation? How would you increase the frequency of the pulses to 40 Hz? - Increase the `PulseWidth` to `40000`. What do you observe? - How would you deliver a 2 second pulse train? (Hint: Use `PulseCount`) >[!NOTE] -> **Optional** Verify the pulse train by connecting the output to a digital input pin. How would you visualize the results using what you've learned? +> **Optional** Verify the pulse train by connecting the output to a digital input pin. ->[!NOTE] -> **Optional** If you have cameras that support external hardware triggering, use what you have learned in this exercise to trigger frame capture. +>[!TIP] +> If your cameras support external triggering, you can use pulse trains to trigger frame capture. Recording the same pulse train on a digital input with the `Hobgoblin` allows you to have hardware timestamped images that are automatically aligned with other acquired data. [`BitwiseNot`]: xref:Bonsai.Expressions.BitwiseNotBuilder @@ -129,7 +133,6 @@ To better understand what each parameter controls, try the following modificatio [`DigitalOutputToggle`]: xref:Harp.Hobgoblin.DigitalOutputToggle [`DigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.CreateDigitalOutputTogglePayload [`DistinctUntilChanged`]: xref:Bonsai.Reactive.DistinctUntilChanged - [`DigitalOutputSet`]: xref:Harp.Hobgoblin.DigitalOutputSet [`DigitalOutputClear`]: xref:Harp.Hobgoblin.DigitalOutputClear @@ -142,6 +145,8 @@ To better understand what each parameter controls, try the following modificatio [`Parse`]: xref:Harp.Hobgoblin.Parse +[`StartPulseTrainPayload`]: xref:Harp.Hobgoblin.CreateStartPulseTrainPayload +[`StopPulseTrainPayload`]: xref:Harp.Hobgoblin.CreateStopPulseTrainPayload [`SubscribeSubject`]: xref:Bonsai.Expressions.SubscribeSubject [`Sum`]: xref:Bonsai.Dsp.Sum [`TimestampedDigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.TimestampedDigitalOutputToggle diff --git a/workflows/hobgoblin-closeloop-pulsetrain.bonsai b/workflows/hobgoblin-closeloop-pulsetrain.bonsai new file mode 100644 index 00000000..0d55f632 --- /dev/null +++ b/workflows/hobgoblin-closeloop-pulsetrain.bonsai @@ -0,0 +1,51 @@ + + + + + + + A + false + + + + Write + + GP15 + 5000 + 50000 + 0 + + + + Hobgoblin Commands + + + + S + false + + + + Write + + GP15 + + + + Hobgoblin Commands + + + + + + + + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-pulsetrain.svg b/workflows/hobgoblin-closeloop-pulsetrain.svg new file mode 100644 index 00000000..13cec1ec --- /dev/null +++ b/workflows/hobgoblin-closeloop-pulsetrain.svg @@ -0,0 +1,110 @@ + + +]> + + + + + + + + + + + + + + Hobgoblin + + + Commands + + + + + + + Hobgoblin + + + Commands + + + + + + Hobgoblin. + + + StartPulse + + + TrainPayload + + + + + + Hobgoblin. + + + StopPulse + + + TrainPayload + + + + + + + KeyDown + + + + + + + KeyDown + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From aec622a63a656d3e57b22142c0d0835ef260dcb6 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Fri, 6 Jun 2025 04:34:43 -0700 Subject: [PATCH 08/11] Change LED for video latency measurement to red --- tutorials/hobgoblin-closeloop.md | 4 ++-- workflows/hobgoblin-closeloop-latency-video.bonsai | 4 ++-- workflows/hobgoblin-closeloop-latency-video.svg | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index fd28bfa2..b02c80df 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -50,7 +50,7 @@ Lastly, we will use this sequence to toggle the digital output and initialize th ### Exercise 2: Measuring video acquisition latency ![Hobgoblin LED](../images/hobgoblin-acquisition-led.svg){width=400px} -- Connect a LED module to digital output channel `GP15` on the `Hobgoblin`. +- Connect a red LED module to digital output channel `GP15` on the `Hobgoblin`. :::workflow ![Hobgoblin Closed-Loop Latency Video](../workflows/hobgoblin-closeloop-latency-video.bonsai) @@ -102,7 +102,7 @@ In neuroscience, pulse trains are commonly used to deliver precisely timed seque ![Hobgoblin Pulse Train](../workflows/hobgoblin-closeloop-pulsetrain.bonsai) ::: -- Connect a red LED module to digital output channel `GP15` on the `Hobgoblin`. +- Connect a LED module to digital output channel `GP15` on the `Hobgoblin`. - Insert a [`KeyDown`] operator and set the `Filter` property to the key `A`. - Insert a [`Parse`] operator and select [`StartPulseTrainPayload`] from the `Register` property dropdown menu. Set the `DigitalOutput` property to `GP15`. - Set the `PulseCount` property to `0`, `PulsePeriod` to `50000` and `PulseWidth` to `5000`. These parameters correspond to a continuous 20 Hz pulse train with 5 ms pulses. diff --git a/workflows/hobgoblin-closeloop-latency-video.bonsai b/workflows/hobgoblin-closeloop-latency-video.bonsai index 85a61ef5..3eb802da 100644 --- a/workflows/hobgoblin-closeloop-latency-video.bonsai +++ b/workflows/hobgoblin-closeloop-latency-video.bonsai @@ -1,5 +1,5 @@  - - Val0 + Val2 diff --git a/workflows/hobgoblin-closeloop-latency-video.svg b/workflows/hobgoblin-closeloop-latency-video.svg index 6d39df81..eddcba43 100644 --- a/workflows/hobgoblin-closeloop-latency-video.svg +++ b/workflows/hobgoblin-closeloop-latency-video.svg @@ -63,7 +63,7 @@ - Val0 + Val2 From c1f8ee828943defc2aba021d5ae3984d7e2f4e72 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Fri, 6 Jun 2025 05:12:08 -0700 Subject: [PATCH 09/11] Add 4th exercise on close loop ROI --- tutorials/hobgoblin-closeloop.md | 27 ++++- workflows/hobgoblin-closeloop-roi.bonsai | 70 +++++++++++++ workflows/hobgoblin-closeloop-roi.svg | 128 +++++++++++++++++++++++ 3 files changed, 223 insertions(+), 2 deletions(-) create mode 100644 workflows/hobgoblin-closeloop-roi.bonsai create mode 100644 workflows/hobgoblin-closeloop-roi.svg diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index b02c80df..bf90dd7f 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -56,7 +56,7 @@ Lastly, we will use this sequence to toggle the digital output and initialize th ![Hobgoblin Closed-Loop Latency Video](../workflows/hobgoblin-closeloop-latency-video.bonsai) ::: -- Insert a [`VideoCaptureDevice`] operator. +- Insert a [`VideoCaptureDevice`] operator. Set the `Index` property to the right camera. - Insert a [`Crop`] transform. - Run the workflow and set the `RegionOfInterest` property to a small area around the LED. @@ -124,10 +124,31 @@ To better understand what each parameter controls, try the following modificatio >[!TIP] > If your cameras support external triggering, you can use pulse trains to trigger frame capture. Recording the same pulse train on a digital input with the `Hobgoblin` allows you to have hardware timestamped images that are automatically aligned with other acquired data. +### Exercise 4: Triggering a digital line based on region of interest activity + +:::workflow +![Hobgoblin Pulse Train](../workflows/hobgoblin-closeloop-roi.bonsai) +::: + +- Insert a [`VideoCaptureDevice`] operator. Set the `Index` property to the right camera. +- Insert a [`Crop`] transform. +- Run the workflow and set the `RegionOfInterest` property to specify the desired area. +- Insert a [`Grayscale`] and a [`Threshold`] transform (or the color segmentation operators). +- Insert a [`Sum`] transform, and select the `Val0` field from the output. +- Insert a [`GreaterThan`] transform and configure the `Value` property to an appropriate threshold. Remember you can use the visualizers to see what values are coming through the [`Sum`] and what the result of the [`GreaterThan`] operator is. +- Insert a [`CreateMessage`] operator, select [`DigitalOutputSetPayload`] for the `Payload`, and `GP15` for the [`DigitalOutputSet`] property. +- Insert a [`MulticastSubject`] operator and configure the `Name` property to `Hobgoblin Commands`. +- Run the workflow and verify that entering the region of interest turns on the LED. +- **Optional:** Replace the [`Crop`] transform by a [`CropPolygon`] to allow for non-rectangular regions. + +> [!Note] +> The [`CropPolygon`] operator uses the `Regions` property to define multiple, possibly non-rectangular regions. The visual editor is similar to [`Crop`], where you draw a rectangular box. However, in [`CropPolygon`] you can move the corners of the box by right-clicking _inside_ the box and dragging the cursor to the new position. You can add new points by double-clicking with the left mouse button, and delete points by double-clicking with the right mouse button. You can delete regions by pressing the `Del` key and cycle through selected regions by pressing the `Tab` key. + [`BitwiseNot`]: xref:Bonsai.Expressions.BitwiseNotBuilder [`CreateMessage`]: xref:Harp.Hobgoblin.CreateMessage [`Crop`]: xref:Bonsai.Vision.Crop +[`CropPolygon`]: xref:Bonsai.Vision.CropPolygon [`Device`]: xref:Harp.Hobgoblin.Device [`Difference`]: xref:Bonsai.Dsp.Difference [`DigitalOutputToggle`]: xref:Harp.Hobgoblin.DigitalOutputToggle @@ -136,8 +157,9 @@ To better understand what each parameter controls, try the following modificatio [`DigitalOutputSet`]: xref:Harp.Hobgoblin.DigitalOutputSet [`DigitalOutputClear`]: xref:Harp.Hobgoblin.DigitalOutputClear - +[`DigitalOutputSetPayload`]: xref:Harp.Hobgoblin.CreateDigitalOutputClearPayload [`GreaterThan`]: xref:Bonsai.Expressions.GreaterThanBuilder +[`Grayscale`]: xref:Bonsai.Vision.Grayscale [`KeyDown`]: xref:Bonsai.Windows.Input.KeyDown @@ -149,6 +171,7 @@ To better understand what each parameter controls, try the following modificatio [`StopPulseTrainPayload`]: xref:Harp.Hobgoblin.CreateStopPulseTrainPayload [`SubscribeSubject`]: xref:Bonsai.Expressions.SubscribeSubject [`Sum`]: xref:Bonsai.Dsp.Sum +[`Threshold`]: xref:Bonsai.Vision.Threshold [`TimestampedDigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.TimestampedDigitalOutputToggle diff --git a/workflows/hobgoblin-closeloop-roi.bonsai b/workflows/hobgoblin-closeloop-roi.bonsai new file mode 100644 index 00000000..28bac3ff --- /dev/null +++ b/workflows/hobgoblin-closeloop-roi.bonsai @@ -0,0 +1,70 @@ + + + + + + + 2 + + + + + + + 0 + 0 + 0 + 0 + + + + + + + + + 128 + 255 + Binary + + + + + + + Val0 + + + + 0 + + + + Write + + GP15 + + + + Hobgoblin Commands + + + + + + + + + + + + + + \ No newline at end of file diff --git a/workflows/hobgoblin-closeloop-roi.svg b/workflows/hobgoblin-closeloop-roi.svg new file mode 100644 index 00000000..5a27e1e3 --- /dev/null +++ b/workflows/hobgoblin-closeloop-roi.svg @@ -0,0 +1,128 @@ + + +]> + + + + + + + + + + + + + + + + + + Hobgoblin + + + Commands + + + + + + Hobgoblin. + + + DigitalOutput + + + SetPayload + + + + + + + GreaterThan + + + + + + + Val0 + + + + + Sum + + + + + + Threshold + + + + + + Grayscale + + + + + + Crop + + + + + + + VideoCapture + + + Device + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From c56fb2c5802df1a1e44942a989124f5237fba129 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Fri, 6 Jun 2025 05:20:06 -0700 Subject: [PATCH 10/11] Copy 5th exercise on CPP from state machine tutorial --- tutorials/hobgoblin-closeloop.md | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index bf90dd7f..4b516d6c 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -122,7 +122,7 @@ To better understand what each parameter controls, try the following modificatio > **Optional** Verify the pulse train by connecting the output to a digital input pin. >[!TIP] -> If your cameras support external triggering, you can use pulse trains to trigger frame capture. Recording the same pulse train on a digital input with the `Hobgoblin` allows you to have hardware timestamped images that are automatically aligned with other acquired data. +> If your camera supports external triggering, you can use pulse trains to trigger frame capture. Recording the same pulse train on a digital input with the `Hobgoblin` allows you to have hardware timestamped images that are automatically aligned with other acquired data. ### Exercise 4: Triggering a digital line based on region of interest activity @@ -144,8 +144,24 @@ To better understand what each parameter controls, try the following modificatio > [!Note] > The [`CropPolygon`] operator uses the `Regions` property to define multiple, possibly non-rectangular regions. The visual editor is similar to [`Crop`], where you draw a rectangular box. However, in [`CropPolygon`] you can move the corners of the box by right-clicking _inside_ the box and dragging the cursor to the new position. You can add new points by double-clicking with the left mouse button, and delete points by double-clicking with the right mouse button. You can delete regions by pressing the `Del` key and cycle through selected regions by pressing the `Tab` key. +### Exercise 5: Conditioned place preference + +Implement the following trial structure for conditioned place preference. `enter` and `leave` events should be triggered in real-time from the camera, by tracking an object moving in or out of a region of interest (ROI). `Reward` should be triggered once upon entering the ROI, and not repeat again until the object exits the ROI and the ITI has elapsed. + +```mermaid +stateDiagram-v2 + direction LR + ITI --> Ready: elapsed + Ready --> Reward: enter + Reward --> ITI: leave +``` + +> [!Tip] +> There are several ways to implement ROI activation, so feel free to explore different ideas. Consider using either [`Crop`], [`RoiActivity`], or [`ContainsPoint`] as part of different strategies to implement the `enter` and `leave` events. + [`BitwiseNot`]: xref:Bonsai.Expressions.BitwiseNotBuilder +[`ContainsPoint`]: xref:Bonsai.Vision.ContainsPoint [`CreateMessage`]: xref:Harp.Hobgoblin.CreateMessage [`Crop`]: xref:Bonsai.Vision.Crop [`CropPolygon`]: xref:Bonsai.Vision.CropPolygon @@ -156,24 +172,17 @@ To better understand what each parameter controls, try the following modificatio [`DistinctUntilChanged`]: xref:Bonsai.Reactive.DistinctUntilChanged [`DigitalOutputSet`]: xref:Harp.Hobgoblin.DigitalOutputSet [`DigitalOutputClear`]: xref:Harp.Hobgoblin.DigitalOutputClear - [`DigitalOutputSetPayload`]: xref:Harp.Hobgoblin.CreateDigitalOutputClearPayload [`GreaterThan`]: xref:Bonsai.Expressions.GreaterThanBuilder [`Grayscale`]: xref:Bonsai.Vision.Grayscale - [`KeyDown`]: xref:Bonsai.Windows.Input.KeyDown - [`MulticastSubject`]: xref:Bonsai.Expressions.MulticastSubject [`Parse`]: xref:Harp.Hobgoblin.Parse - - +[`RoiActivity`]: xref:Bonsai.Vision.RoiActivity [`StartPulseTrainPayload`]: xref:Harp.Hobgoblin.CreateStartPulseTrainPayload [`StopPulseTrainPayload`]: xref:Harp.Hobgoblin.CreateStopPulseTrainPayload [`SubscribeSubject`]: xref:Bonsai.Expressions.SubscribeSubject [`Sum`]: xref:Bonsai.Dsp.Sum [`Threshold`]: xref:Bonsai.Vision.Threshold [`TimestampedDigitalOutputTogglePayload`]: xref:Harp.Hobgoblin.TimestampedDigitalOutputToggle - - -[`VideoCaptureDevice`]: xref:Bonsai.Video.VideoCaptureDevice - \ No newline at end of file +[`VideoCaptureDevice`]: xref:Bonsai.Video.VideoCaptureDevice \ No newline at end of file From ad9b40874012ae60f12908aaa8ef3e2f331f3a70 Mon Sep 17 00:00:00 2001 From: Shawn Tan Date: Fri, 6 Jun 2025 13:07:33 -0700 Subject: [PATCH 11/11] Fix 4th exercise digital output toggle --- tutorials/hobgoblin-closeloop.md | 3 ++- workflows/hobgoblin-closeloop-roi.bonsai | 4 ++-- workflows/hobgoblin-closeloop-roi.svg | 8 ++++---- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/tutorials/hobgoblin-closeloop.md b/tutorials/hobgoblin-closeloop.md index 4b516d6c..25fbc203 100644 --- a/tutorials/hobgoblin-closeloop.md +++ b/tutorials/hobgoblin-closeloop.md @@ -136,10 +136,11 @@ To better understand what each parameter controls, try the following modificatio - Insert a [`Grayscale`] and a [`Threshold`] transform (or the color segmentation operators). - Insert a [`Sum`] transform, and select the `Val0` field from the output. - Insert a [`GreaterThan`] transform and configure the `Value` property to an appropriate threshold. Remember you can use the visualizers to see what values are coming through the [`Sum`] and what the result of the [`GreaterThan`] operator is. -- Insert a [`CreateMessage`] operator, select [`DigitalOutputSetPayload`] for the `Payload`, and `GP15` for the [`DigitalOutputSet`] property. +- Insert a [`CreateMessage`] operator, select [`DigitalOutputTogglePayload`] for the `Payload`, and `GP15` for the [`DigitalOutputToggle`] property. - Insert a [`MulticastSubject`] operator and configure the `Name` property to `Hobgoblin Commands`. - Run the workflow and verify that entering the region of interest turns on the LED. - **Optional:** Replace the [`Crop`] transform by a [`CropPolygon`] to allow for non-rectangular regions. +- **Optional:** Modify the workflow to replace the digital output toggle with a pulse train. > [!Note] > The [`CropPolygon`] operator uses the `Regions` property to define multiple, possibly non-rectangular regions. The visual editor is similar to [`Crop`], where you draw a rectangular box. However, in [`CropPolygon`] you can move the corners of the box by right-clicking _inside_ the box and dragging the cursor to the new position. You can add new points by double-clicking with the left mouse button, and delete points by double-clicking with the right mouse button. You can delete regions by pressing the `Del` key and cycle through selected regions by pressing the `Tab` key. diff --git a/workflows/hobgoblin-closeloop-roi.bonsai b/workflows/hobgoblin-closeloop-roi.bonsai index 28bac3ff..d72d17a3 100644 --- a/workflows/hobgoblin-closeloop-roi.bonsai +++ b/workflows/hobgoblin-closeloop-roi.bonsai @@ -48,8 +48,8 @@ Write - - GP15 + + GP15 diff --git a/workflows/hobgoblin-closeloop-roi.svg b/workflows/hobgoblin-closeloop-roi.svg index 5a27e1e3..50af2bf5 100644 --- a/workflows/hobgoblin-closeloop-roi.svg +++ b/workflows/hobgoblin-closeloop-roi.svg @@ -34,7 +34,7 @@ DigitalOutput - SetPayload + TogglePayload @@ -92,13 +92,13 @@ - + - + - +