Support adding brick as :extra-deps for a Polylith workspace in development

[tengstrand]

The Polylith architecture comes with the poly tool written for Clojure that is based on tools.deps.

I've been working on issue-66 for a while in the issue-66 branch to support defining bricks as :extra-deps instead of :extra-paths, see deps.edn.

This works fine in e.g. Emacs/Cider and VSCode/Calva but not in IDEA/Cursive.
What happens is that the src directories are not recognised by Cursive and will therefore not show up as green.

Instead of this:

...we get this:

The good part is that the code is recognised by the REPL, but unfortunately not by the IDE.

The workaround is to add them as :extra-paths, but it would be much better if they could be added as :extra-deps instead:

• It would be consistent with how bricks are added to other projects (development is a special kind of project).
• Instead of defining two paths, src and resources, we can define one dependency per brick, resulting in less code.
• If a resources directory is added or removed from a brick, the change will automatically be detected by tools.deps.

Some teams will probably start to use the :extra-deps way of defining bricks in development, which unfortunately will stop people that joins the team from using Cursive. With better support in Cursive, this could be avoided.

Regards,
Joakim Tengstrand

[imrekoszo]

I did try to work around this by re-importing the entire workspace into idea and marking all local/root refs as modules. That did solve the folder mark problem and did not break the repl classpath: polyfy/polylith#127

However, as @tengstrand & co kindly checked for me, it does not solve the problem of symbol resolution in case a brick references another brick's interface. Due to how polylith is set up, these inter-brick dependencies are not listed as relative local/root deps in the bricks' own deps.edn files - this is how polylith achieves swappability of implementations. Multiple bricks might have the same interface signature (via some code duplication) and it's decided at the project (or development) level which brick to reference as a dependency to satisfy a requirement for an interface.

I created a repro case at https://github.com/imrekoszo/cursive-polylith

With the root deps.edn looking like

                  :extra-deps {org.clojure/clojure {:mvn/version "1.10.3"}
                               org.clojure/tools.deps.alpha {:mvn/version "0.12.1003"}
                               poly/user {:local/root "components/user"}
                               poly/foo {:local/root "components/foo"}}}

            :test {:extra-paths ["components/user/test"
                                 "components/foo/test"]}

And no cross-brick references, se.example.foo.interface cannot be navigated to from the following file:

(ns se.example.user.core
  (:require [se.example.foo.interface :as foo]))

[cursive-ghost]

I finally got some time to look at this, sorry for the delay. Thanks for the repro case @imrekoszo, that's very helpful.

So there are two problems here. One is that components added to the root project deps.edn are not added as modules, which means that their source roots are not registered in Cursive. A workaround for this problem is to add the modules manually as Imre did, which isn't terrible. I considered automatically adding :local/root deps as modules but there are cases where you don't want that (Cursive itself has one). I could potentially add some easier way to add them in the Deps toolwindow to facilitate that.

The second problem is that with this setup, there are two modules which share the same source root. In Imre's example, both the root deps.edn and the component ones share the test classpath, e.g. components/foo/test is shared by both the foo component and the root application. IntelliJ will not allow that, a particular source root has to be associated with just one module. Is it essential that that folder be shared between the two?

[imrekoszo]

Hey @cursive-ide, thanks for picking this up. I chatted to @tengstrand on the clojurians slack about this, here's what I gathered:

As for the first problem you mention, while a convenience feature would be nice, as long as reimporting manually works, that shouldn't be a deal-breaker.

For the second problem, it appears those test paths can actually be dropped from the top-level deps.edn - it was only added there as an attempt to make it more Cursive-friendly. I went ahead and dropped those paths from the repro case in my repo.

The problem that remains after these is that go to definition does not work between bricks:

I'm assuming the reason why this (orange arrow) doesn't currently work in Cursive is because in polylith there must be no reference to the foo brick from the user brick (red arrow).

polylith quick info: components and bases are in the category of bricks and there must be no direct deps.edn references between bricks. projects on the other hand act as the composing layer and will reference (usually multiple) bricks.

In the dev alias (which is a special, development-type 'project') that I have selected, there are still references to both of those bricks, meaning they will both be on the classpath if I run a repl with the Idea classpath so I guess the missing connection can be figured out somehow. This should ideally work transitively, so if I have an alias selected which :local/root references a project, and that project :local/root references some bricks, go to definition should work between those bricks too.

Edit: it appears the transitive case isn't needed for polylith.

[furkan3ayraktar]

Hi! Thank you for picking up this issue! I’m one of the Polylith team members and I’ll try to shed light on what is working and what is not working in Cursive for Polylith projects. For that, I’ll start by giving a short overview of what Polylith does.

Polylith has these two important concepts, component and base (they are referred bricks collectively). They are the building blocks with different purposes, bases being responsible of exposing an external API and component being small Lego-like bricks implementing the logic and exposing and internal functional API. Each base and component has their own deps.edn file, defining their source, resource, and test folders (with :test alias) as well as their external library dependencies. One of the most important thing in Polylith is, bricks do not define internal dependencies in their deps.edn, they are picked up when you put them together in a project and this enables Polylith users to interchange components with different implementations.

Polylith has a special project, which is called development project, that has a deps.edn file at the root of the project. Here we define our local development project and include all the components and bases in it so that we can reach to all code in the repository, even though they might be split into different projects in deployed versions.

What is working with Cursive:

If we have the following deps.edn structure, Cursive works:

{:aliases {:dev  {:extra-paths ["components/comp-a/src"
                                "components/comp-a/resources"
                                "components/comp-b/src"
                                "components/comp-b/resources"
                                "components/base-a/src"
                                "components/base-b/resources"]

                  :extra-deps  {some.external/dependency-of-comp-a {:mvn/version "1.0.0"}
                                some.external/dependency-of-comp-b {:mvn/version "1.0.0"}
                                some.external/dependency-of-base-a {:mvn/version "1.0.0"}
                                some.external/dependency-of-base-b {:mvn/version "1.0.0"}}}

           :test {:extra-paths ["components/comp-a/test"
                                "components/comp-b/test"
                                "components/base-a/test"
                                "components/base-b/test"]

                  :extra-deps  {some.external/test-dependency-of-comp-a {:mvn/version "1.0.0"}
                                some.external/test-dependency-of-comp-b {:mvn/version "1.0.0"}
                                some.external/test-dependency-of-base-a {:mvn/version "1.0.0"}
                                some.external/test-dependency-of-base-b {:mvn/version "1.0.0"}}}}}

This is a very simple example and you can see a realworld example here.

What does not work in Cursive and works in other IDEs

If we have the following deps.edn structure, Cursive does not work by default. Even if we import each component and base as a module, it still cannot resolve namespaces from classpath for the code navigation and other features of Cursive to work:

{:aliases {:dev  {:extra-deps  {poly/comp-a {:local/root "components/comp-a"}
                                poly/comp-b {:local/root "components/comp-b"}
                                poly/base-a {:local/root "bases/base-a"}
                                poly/base-b {:local/root "bases/base-b"}}}

           :test {:extra-paths ["components/comp-a/test"
                                "components/comp-b/test"
                                "components/base-a/test"
                                "components/base-b/test"]

                  :extra-deps  {some.external/test-dependency-of-comp-a {:mvn/version "1.0.0"}
                                some.external/test-dependency-of-comp-b {:mvn/version "1.0.0"}
                                some.external/test-dependency-of-base-a {:mvn/version "1.0.0"}
                                some.external/test-dependency-of-base-b {:mvn/version "1.0.0"}}}}}

Since :local/root syntax does not allow depending on an alias in the target deps.edn, we still need to duplicate test paths and extra dependencies for the tests from individual components and bases.

What would be ideal 😎

It would be the best If Cursive can understand that everything starting with poly/ in the extra-deps is a Polylith component or base, and resolve their src, resource, and test directories and src and test dependencies. Then, we wouldn't need to duplicate anything and have both sources and tests in our REPL. In a Polylith workspace, components and bases always have their tests under :test alias, since thats where the poly CLI looks for.

{:aliases {:dev  {:extra-deps  {poly/comp-a {:local/root "components/comp-a"}
                                poly/comp-b {:local/root "components/comp-b"}
                                poly/base-a {:local/root "bases/base-a"}
                                poly/base-b {:local/root "bases/base-b"}}}}}

Let me know if you need any further information from us. You can reach out to me or the rest of the team at #polylith channel in Clojurians Slack.

[cursive-ghost]

Thanks for the examples @imrekoszo and @furkan3ayraktar. I've looked into this further, and I think I have a better idea of how Polylith works now. I'm still not sure how best to support it, though. Thinking through what I would have to do to support this, it's something like:

1. User clicks on something, or Cursive tries to resolve it in the editor.
2. Cursive checks to see it looks like the reference is contained within a brick.
3. If it is, then Cursive should construct a resolve scope for the project, probably using a union of the development project scope (for source references across bricks) and the original brick's scope (to get the dependencies of the brick itself). Or maybe just the development project's scope for everything? I'm not sure about this.

There are a couple of assumptions here. One is that there's either: 1) a 1-1 correspondence between an IntelliJ project and a Polylith project, or: 2) that bricks are in a consistent place relative to the development project so that I can find the development project from them. I'm guessing that 1) is probably not a good assumption to make, since these projects might be contained within a larger monorepo. I guess the question is: how can Cursive determine that a deps project is a Polylith development project, and modify its resolution accordingly?

I spent some time looking at the Realworld project example, and there are still cases there where the same source root is present in multiple modules, e.g. the test roots are present in both the root development project and in the components themselves. I'm fairly sure this used to be a hard error in IntelliJ but it's not complaining now, perhaps that has changed in recent versions. I'll check that with JetBrains. If my memory is correct then there's nothing I can do about that, unfortunately.

[furkan3ayraktar]

Hi @cursive-ide! Thanks for looking into this, I’ll try to elaborate on the topics that you mentioned.

I can start with the project structure. Polylith has a strict directory structure. This is important for our own poly CLI to work properly. The structure is as following:

example              # workspace dir
├── .git             # git repository dir
├── bases            # bases dir
│   ├── base-1       # base dir
│   │   ├─ src       # base sources
│   │   ├─ resources # base resources
│   │   ├─ test      # base tests (use-case: unit tests)
│   │   └─ deps.edn  # Specifies src, resources, and test paths. Specifies only external dependencies. Always has a :test alias 
│   └── base-2
│   │   ├─ src
│   │   ├─ resources
│   │   ├─ test
│   │   └─ deps.edn
│   ...
│   
├── components       # components dir
│   ├── comp-1       # component dir
│   │   ├─ src       # component sources
│   │   ├─ resources # component resources
│   │   ├─ test      # component tests (use-case: unit tests)
│   │   └─ deps.edn  # Specifies src, resources, and test paths. Specifies only external dependencies. Always has a :test alias 
│   └── comp-2
│   │   ├─ src
│   │   ├─ resources
│   │   ├─ test
│   │   └─ deps.edn
│   ...
│   
├── development
│   └── src          # development specific code (only used during development, within REPL)
├── projects         # projects dir
│   ├── proj-1       # project dir
│   │   ├─ src       # project specific source dir (not recommended to have project specific sources)
│   │   ├─ resources # project specific resources dir (optional)
│   │   ├─ test      # project specific test dir (optional) (use-case: integration tests)
│   │   └─ deps.edn  # Specifies source paths, resource paths, internal brick dependencies (:local/root) and any extra external dependencies (optional). Optionally has a :test alias to include bricks used only during tests but not included in the deployable artifact. 
│   └── proj-2
│   │  ├─ src
│   │  ├─ resources
│   │  ├─ test
│   │  └─ deps.edn
│   ...
│
├── deps.edn         # development config file
└── workspace.edn    # workspace config file (an indication that you are at the root of a Polylith workspace)

The deps.edn at the root directory is for the development project and it is a valid project configuration for a developer to start a REPL with the correct classpath. All brick and external dependencies (if needed) under :dev alias and all test paths and external dependencies under :test alias. There could also be extra profiles defined by other aliases, this means not all components needs to be in the classpath (and REPL), but ideally all should be navigable from the IDE.

The reason we need to include test sources again in the root deps.edn is that tools.deps does not allow to refer to an alias (:test in this case) via :local/root. If we don’t do this, test paths and dependencies won’t be in the REPL. However, we need test sources and dependencies in the bricks deps.edn for the poly CLI to work with incremental tests.

I do not have any experience with Intellij IDE internals, so this might be a bad idea, however I’m going to say it anyway 😂. What I think is, the bricks should not be modules, since they are not independent projects. Cursive could treat them as places where source and test paths live, and resolve them and their dependencies from the development projects deps.edn.

I think assumption 1 is good enough, since that’s how Polylith works in other IDEs and I don’t see the benefit of having two Polylith workspaces in a single monorepo, it’s better to combine them to get the best out of it. When it comes to assumption 2, bricks are always in a consistent place relative to the development projects deps.edn. It is a flat structure under components and bases directories from the root directory of the project.

[imrekoszo]

@furkan3ayraktar I wouldn't completely rule out Colin's concerns wrt assumption 1 - workflows differ.

However, Cursive might be able to use workspace.edn to find polylith roots? Is that always present in the polylith root? Is there a schema/spec available for its contents, should Cursive want to also check that to differentiate from some other kind of workspace.edn?

[cursive-ghost]

Yes, for reference the largest lein project I'm aware of had > 90 project.clj files. Someone somewhere is going to want to put two Polylith apps in one IntelliJ project.