If a test changes, poly test treats it as if the source changed

[seancorfield]

From Slack:

Q about poly test figuring out what changed and what needs to be tested: if I change just a test in a widely-used component, I would expect that component's tests to be re-run (for each project that uses it) but I wouldn't expect it to trigger a cascade of everything that depends on that component since the component hasn't changed, only one of its tests -- but it seems that everything (that depends on that component) ends up getting re-tested... which is complete overkill? Thoughts?

@tengstrand said:

I agree, this could be improved. I remember that I realised this weakness in the calculation, but that I thought it was so rare and that it simplified the calculations, so I decided not to act on it. Maybe you could create an issue, and I can have a look at it.

In the end, the solution to this issue was to include dependencies to bases from bases in the workspace structure and in the dependency calculations and the deps command, and to support exclusion of bricks in tests for projects in workspace.edn. Earlier we supported :include but now we also support :exclude.

[tengstrand]

This issue grew in size, but after several week of work, now it's time to test it @seancorfield. The documentation is not updated yet, so here comes a summary of the changes.

• Earlier, we only calculated if we depended from :src or :test. Now we also calculate if we depend on :src or :test. So if a base only depend on another base's :test code, and only its :src changes, then we only run tests for the changed base, not the depender.
• :src and :test have been added to :changes -> :changed-bases, :changed-components, and :changed-projects in the workspace structure.
• :src and :test have been added to :projects:PROJECT:deps in different places of the workspace structure.
• Old exported workspaces can still be used and switched to (with switch-ws) and will be converted to the new format and stored under :src.

If running the deps command with no arguments, it will work as today:

When giving a project, we no longer include indirect changes by default:

They can be included by passing in :indirect:

By giving both :src and :test, it will split up :src and :test separately (s = :src, t = :test, st = :src and :test):

If only :src is given, then only the dependencies from :src are shown:

If only :test is given, then only the dependencies from :test are shown:

If a brick is given, it will look as before, but the :src and :test will be shown:

If both brick and project is given, it will split up the dependencies in :src and :test:

Indirect dependencies can be included by passin in :indirect. Direct dependencies will be shown as s and t, indirect as s+ and t+:

If :src is passed in, then only dependencies from :src is shown:

If :test is passed in, then only dependencies from :test is shown:

[seancorfield]

That looks great. I'll be at Conj/traveling this week so I'll take a look when I get back.

[tengstrand]

Good to hear! Have a nice trip and stay.

[seancorfield]

Testing this now. When I first try check, I get this (which is expected with the new logic):

  Error 107: Missing bases in the wsbilling-rebill project, for the test context: billing-member

And when I move the billing-member base dep from the :test deps on the billing-rebill base to the wsbilling-rebill project, it checks "OK". So that's good.

When I run test, it tests the wsbilling-rebill project and tests everything it depends on (since deps.edn has changed), including the billing-member base -- which I think is expected this first time.

I'll see how this works for a few days and report back.

[seancorfield]

OK, I'm stuck again. Because that runs both the billing-rebill tests and then runs the billing-member tests as part of that, the tests fail -- because the billing-member tests can't be run in the same project context as the billing-rebill tests (and they aren't, when they are a :test dep of the base using Polylith master).

So that seems to suggest the test selection is wrong here?

[tengstrand]

Okay. Can you export the workspace again and send me @seancorfield , using the latest commit in the issue-205 branch (sha d509a1f04824a8d20fb5964d67f976a19e8a9246). The issue-205 branch continue from issue-274, and contains the latest and greatest. I need you to export the workspace again, because now I also store target :src and :test.

[tengstrand]

I have created the example workspace cross-ref in the issue-205 branch and managed to reproduce the problem.

The workspace looks like this:

Only the test context of the bm base is included in the brp project.

The br base refers to bm from its test context, which can be displayed with deps project:brp :src :test:

...or with ws get:projects:brp:deps

If the :src context of br changes, only its own tests will be executed (from the brp project) which is correct:

If the :test context of br changes, only its own tests will be executed (from the brp project) which is correct:

If the :src context of bm changes, the tests for bm will be executed (from the brp project) which is not correct. The bug is that the tool dosn't take into account that only the test context of bm is included in brp:

Will have a look at this!

[tengstrand]

You can try out the issue-205 branch, sha e41b1a4264c4b568943bfb242b488b660556aab2 @seancorfield.

[seancorfield]

This still runs the billing-member base tests when only billing-rebill (base) and wsbilling-rebill (project) have changed -- which breaks things:

Projects to run tests from: wsbilling-rebill

Running tests for the wsbilling-rebill project using test runner: Polylith org.corfield.external-test-runner...
Running tests from the wsbilling-rebill project, including 93 bricks: a-b-tester, admin-purchase-data, affiliate, application-fixtures, 
application-system, appstoreconnect, billing-actions, billing-android, billing-authorizenet, billing-braintree, billing-core, 
billing-data, billing-dmz, billing-fixtures, billing-google, billing-i18n, billing-machine, billing-member-ref, billing-notifications, 
billing-payment-method, billing-protocols, billing-rebill-actions, billing-sdk, billing-shared, billing-specs, billing-upkeep, 
caching, command-member-profile, configuration, connections, covalence-delivery, covalence-disposition, currency, 
datasources, date-time, environment, error-reporter, event-publisher, exceptions, executors, feature-flags, file-system, 
geo-location, google-measurement, host-services, http-client-hato, http-client-response, i18n, ip6, jwt, legacy-membership, 
logging, mail-system, mail-template, mailer, member-activity, member-id, member-interactions, member-profile-fields, 
messaging-sdk, money, newrelic, notification-settings, notifications, oauth-resource, oauth-tokens, onesignal, 
packages-and-rates, photo-data, photo-files, profile-template, profile-updates, push-notifications, query-member-profile, 
query-shared-photo, redis, safe-coercions, secure-token, selmer, site, slack-notifications, string-utils, system-properties, 
tagging, user-attributes, uuid, value-lists, web-middleware, web-request, web-server, with-retry, 
billing-member, billing-rebill
   ^^^^ this is a base and it has not changed so its tests should not be run here
Running test setup for the wsbilling-rebill project: ws.application-fixtures.interface/pre-test

[tengstrand]

Okay, will have another look. Also, I just rebased the issue-205 branch, so the equivalent sha is now 73a47eec8cd7933a511ba8401e25f37b1364bf93 (with the same problem as you just reported).

[tengstrand]

if no one succeeds in convincing me otherwise, I intend to skip keeping track of whether one depends on :src or :test, even though I have spent a lot of time building support for it in the 274/205 branches. Normally, both :src and :test code depend on :src only. The problem you have reported here @seancorfield, can be solved by adding support for :exclude in workspace.edn, under the :test key for each project (as I wrote in our Slack conversation). This solution has added a lot of complexity and it gives very little value back.