Check Plugin compatibility with Openfire updates

[guusdk]

We frequently find ourselves in a situation where a change in Openfire breaks a plugin, without us realizing it. It then ends up annoying users, who only sometimes report it to us. We then take some time to address that issue, which adds to the time that the plugin is 'broken.

It would be preferable to know about plugins breaking after a change in Openfire ahead of the Openfire release that contains that change.

This should allow us to:

• release a version of the plugin that sets a maxServerVersion, ensuring that at least that version will not run with affected future versions of Openfire in a 'clean' manner.
• release a version of the plugin that fixes the incompatibility.

It would be helpful if this can be facilitated by automation. For setting a maxServerVersion, ideally, a PR is created automatically that applies the desired changes to the plugin's plugin.xml file (and related changes to the changelog file). Additionally, an issue should be raised automatically to request the plugin's implementation to be made compatible with the future version of Openfire.

[guusdk]

What would be a good approach to test this? I don't think we'll manage having runtime tests for all plugins, as those would involve having functional tests that are specific to each plugin (it would be great if we'd get those, but I think that's a bridge to far for now). An acceptable alternative to me is to check for compile-time compatibility. Can we somehow find a way to automatically build a plugin that is not using the version of Openfire that is declared as a dependency in the plugin's pom.xml file, but instead use the bleeding edge of Openfire?

Most plugins will define some kind of runtime dependency against a version of Openfire, like so:

<dependency>
    <groupId>org.igniterealtime.openfire</groupId>
    <artifactId>xmppserver</artifactId>
    <version>${openfire.version}</version>
    <scope>provided</scope>
</dependency>

I suspect that for many/most plugins this dependency is not explicitly stated in their pom.xml file, but is defined by virtue of the plugin defining a 'parent' to Openfire's plugins artifact:

<parent>
    <artifactId>plugins</artifactId>
    <groupId>org.igniterealtime.openfire</groupId>
    <version>4.7.4</version>
</parent>

If we can find a way to run checks on each plugin, for a different version of Openfire, then an additional question is: when do we trigger this? Running all plugin builds on every change seems excessive. Do we do a daily/weekly build? Do we do it only for tags of Openfire (we could come up with some kind of scheme where we trigger this check on alpha or beta releases of Openfire)?

[Fishbowler]

If we have an explicit property in the POM of openfire.version then we can mvn package -Dopenfire.version=4.9.0

when do we trigger this?

Builds are cheap. We're happy to have a java matrix for every run. And early feedback is good.
But would we want to test all versions? Maybe all major versions, since we don't (normally) break APIs in minor versions?

[Fishbowler]

having runtime tests for all plugins

We should get better at that though. I want to extend the start openfire action to include an optional plugin jar.

[guusdk]

If we have an explicit property in the POM of openfire.version then we can mvn package -Dopenfire.version=4.9.0

We'd probably need to do something like 4.10.0-SNAPSHOT (and be able to determine that value automatically), but yeah, that could work.

What's not clear to me is how we tie this together. How do we make each plugin-repo watch Openfire's repo?

Builds are cheap

Are they that cheap? I'm not sure if kicking off approximately 75 additional builds for every change in Openfire is sustainable. It seems wasteful, and prone to generate a lot of noise.

[Fishbowler]

I'm not sure if kicking off approximately 75 additional builds for every change in Openfire

Oh, I'd not really thought about it that way around. I'd assumed that we were talking about commits to the plugins.
Assuming we're okay with that (upping our plugin build job to do version compatibility checks, that you haven't slipped in an incompatible API for your declared range) then we could have a manually executed job in the Openfire repo that does a one-time call to all of the build jobs in all of the plugins.

[guusdk]

Well, I'd like to 'catch' plugins that break, because of a change in Openfire. Many of the plugin repos do not see activity enough for any kind of action triggered by an activity in the plugin's repo to reliably find the problem in time (eg: prior to the release of Openfire that ships the breaking change). That's why I was looking at some kind of mechanism in which all plugins are (re)tested against the latest/greatest Openfire.

then we could have a manually executed job in the Openfire repo

Interesting. How can we create such a job? Is that a GitHub Action or something? How would you trigger that? Can we make that maintainable (as in, I'd prefer not to have to modify that job each time we create a new plugin repo).