Reviving pyharp #2
Description
Hello! I hope everyone is doing well. As you may know, we (@artursilva0, @MicBoucinha, @ZegCricket) would be really interested in developing the Python Harp ecosystem further and we believe there are a few advantages to this:
- Bringing more people to the Harp ecosystem in general. Several researchers use Python for their research and, currently, the Harp ecosystem is heavily dependant on Bonsai, which may be considered an high entrance barrier for some, since it's an extra thing people have to learn if they want to use Harp.
- Unifying the Python Harp ecosystem that is scattered all over the place. As far as we know, the message data structure that represents a message from the Harp protocol is implemented in 3 different projects:
- pyharp - additionally, there's a fork from the AIND, whose development is yet to be merged to the main repo
- harp-python
- microharp
We still don't have everything planned, but we have an idea of what we would like to have in the end. The following figure will serve as a guide for what will be explained after.
Ideally, we would like to create a core package with only the essentials in which the remaining packages would depend on. For now, the only thing I can think of that should be in the core is the definition of the Harp message data structure and all of its processing (message, payload type, payload, checksum, etc.). We are inclined to implement the Harp message structure based on the implementation made for the harp-python package, where Pydantic is used for data validation.
With a solid core developed, we should be able to build the remaining components on top of it: the board interfaces, the data package and the microcontroller package. We are more interested in creating the board interfaces, but we would like the core to be sufficiently robust in a way that it can be adopted for other ends.
For the device/communication package we plan to take a look at the current fork from the AIND and to incorporate the new features in the pyharp ecosystem. Obviously, we would like to ear from and be open to work with the people from the AIND (or from other institutes) directly.
Finally, the idea for the board interfaces is similar to what currently exists for .NET, which is an automatic generation based on the device.yml of each device. In a first stage, it could be done in a separate repository, but it would be interesting to include the Python interface generation in the reflex-generator at some point.
Questions to address
Now, we know that there are some questions that need to be addressed.
Maintainability
The first issue to address is maintainability. This usually means more work for people, but since we want to make the development of pyharp as smooth as possible, we wouldn't mind being the maintainers of the Python ecosystem (except for the harp-python package). It would be a way to take some load out of the shoulders of the other harp-tech members. We are not expecting to have a lot of developers that use Harp for their projects at the moment, so we are not expecting a lot of outside issues/pull-requests, but if the situation changes, this can be rethinked then.
Single package vs multiple packages
This is a matter of deciding what is the preferred solution. Personally, I would prefer to have separate packages as shown in the figure above, because in different projects I might need different parts of the pyharp ecosystem and so I might not want to install all of the package's modules and (specially) dependencies. However, it was already pointed out that one might prefer to install everything so that installing the packages one by one isn't needed. A solution to this problem that I'm not sure is 100% possible is to create "groups of packages", similar to the Bonsai Starter Pack.
Bonsai vs Python debate
This debate is probably the elephant in the room here. "Why would I use the Python packages for my projects when I can just use Bonsai whose Harp ecosystem is already out there, is stable and, more importantly, makes asynchronous data acquisition and processing really convenient and easy?" I do use Bonsai for my projects. I do think it's a great tool either for programming gigantic tasks that encompass different hardware sources (Bonsai is already worth using if I want to use a camera alone) either for prototyping and experimenting (the different Harp devices, for instance). However, its steep learning curve (and the fact it's a niche software) disencourages the adoption of both Bonsai and Harp by new users. This is not only true for a self-taught user (which is any user outside of the institutions that have other people that already use Bonsai), but it is true for people who are part of an institution whose peers already adopted Bonsai (I've seen this myself).
It would be easier for Harp to be adopted if there were stable and robust packages for Python, because whether we like it or not, popularity matters. It's far easier to make people adopt Harp if we try to meet our end-users halfway and not force people to learn Bonsai in order to use Harp. In fact, after people start using Harp, maybe it will be easier to introduce them to Bonsai.
Additionally, by expanding the support of the Harp ecosystem we are not only increasing the number of users, but there's also the possibility that we are increasing the number of applications for the Harp itself, which is currently limited to the experimental neuroscience landscape, due to the flexibility it will be given to the Harp users.
In conclusion, the Bonsai vs Python debate seems to be counter-productive, since, despite Bonsai and Python not being necessarily complementary (in the sense that an user might have to choose between one and the other), Bonsai might benefit from this as well.
We are eager to know what you think about this.