Branching (or not)

[eatdrinksleepcode]

While commenting on an outstanding pull request, David Kean mentioned:

We're trying to figure out our branching structure at the moment. Currently we have only a single branch that represents what we're shipping for the next update of .NET Core. Clearly this isn't scalable for the product, and we're going to creating branches for future work so PR's that have API changes and destablizing changes get a little more bake time. @joshfree is looking into that.

I would like to get the conversation about branching structure out into the open. And to do that, I would like to start by asking: why is a single branch not scalable for the product?

I have worked in code bases of various sizes, release schedules, and branching schemes, and my experience is that NOT branching is nearly always preferable to branching. In fact, the most common reasons given for wanting to branch (e.g. disruptive or long-running changes) are exactly the times when branching hurts the most, because it encourages divergence between different works-in-progress, pushing difficult integration decisions farther out and requiring more rework later.

Rather than a policy that handles change by branching, I would rather see a policy of trunk-based development that encourages all changes - even incomplete changes - to be merged into master on a frequent basis. This shortens and simplifies the merging process and encourages small refactorings and improvements by reducing the amount of work needed to merge them.

Obviously allowing unfinished work to be committed to master requires mechanisms in place to ensure that such work does not affect the releasability of the framework. Mechanisms such as feature flags and branch by abstraction are slightly more complicated to implement in a framework (especially one that is highly tuned for performance) than in a typical application, but not excessively so. And the advantage of keeping all of the ongoing work together is well worth the effort.

What are the community's thoughts on trunk-based development?

[AArnott]

@eatdrinksleepcode The feeling within Microsoft does seem to advocate primary development happening in the master branch. Are you actually arguing against release and stabilization branches? If so, there are very compelling reasons for this.

Release Branches

You need tags to know what you've shipped, and branches to be able to issue service updates for those previously shipped versions. Branches aren't bad. They are good. They let you track older versions of the product that you may need to service in the future.

Stabilization branches

As you approach a scheduled release, you need to start thinking hard about what changes you'll accept in a product. But you also don't want to slow down active development. You can either keep 'master' as your "stable" branch and move primary development to other branches, or you can branch off stabilization and keep active development in master. I think we agree that master should be used for active development. Stabilization branches become your release branches after they actually ship. We just create them before shipping so that we can still actively accept pull requests to master even if they can't fit (for churn/risk reasons, etc.) for the next upcoming release.

[terrajobst]

I don't think feature flags would work for a framework as you can't feature flag APIs. Reference assemblies can be used to a certain extent, but this creates a whole bunch of other complexity.

From personal experience I'd say that feature branches are working pretty well but it's certainly true that having all development happening in master is a lot simpler and avoids many problems that feature branches can have.

However, I think there is nothing wrong with using release branches. Branches are the easiest way to visualize which changes are part of a release, especially in git.

[eatdrinksleepcode]

@AArnott, I was mostly just trying to start a conversation to surface the latent assumptions that seemed to be driving @davkean's statements. Beyond that I was primarily arguing against feature branches. However:

Branches aren't bad. They are good.

Branches are blunt instruments. They can be used to achieve certain beneficial effects, but often there are more precise ways of achieving the same goal without the negative effects of constant branching. That was my point in starting this conversation.

You need tags to know what you've shipped, and branches to be able to issue service updates for those previously shipped versions.

This is the kind of assumption that I wanted to surface. Why do you need branches to issue service updates? Branches are one way of doing that, but certainly not the only way. An alternative would be to simply roll forward: release again from master rather than going backwards and releasing from a branch (again, I am assuming that appropriate controls are in place to hide work in progress). Depending on the servicing strategy (which I have not seen for .NET Core; is it available?), that may be sufficient. If not, there are other mechanisms for delivering updates that don't necessarily require branches, but they carry increasing complexity that may or may not be justified. In the end, branching from a tag may be the best way to deliver service updates, and if so I am not at all opposed to that. But the assumption seems to be that branching is the only way to do it, and I would like to ensure that the cons and the possible alternatives are being considered.

re: Stabilization Branches

You can either keep 'master' as your "stable" branch and move primary development to other branches, or you can branch off stabilization and keep active development in master.

Choosing between these options is only necessary when you have no other mechanisms for hiding work in progress. When you do have such mechanisms, you can easily continue working on master with the confidence that it will not affect the release. A stabilization branch has all the same negative effects as a feature branch: work diverges, merges become harder, regressions and necessary rework become more likely, etc.

To be perfectly clear: I am in no way advocating dogma that there is never a reason to create a release branch, or even a stabilization branch (although I think I would need a lot of convincing on the latter). What I am arguing is that branching as a blunt tactic comes with inherent negative effects, and as such other solutions for isolation should be considered; branching should be, if not the last resort, at least farther down the list.

[eatdrinksleepcode]

@terrajobst,

I don't think feature flags would work for a framework as you can't feature flag APIs.

You certainly can feature flag APIs. The implementation just can't be as naive as many application feature flag implementations are (often reading from a config file at runtime), but it is absolutely doable using compile-time flags. And in fact the usage options would end up looking very similar to those of runtime implementations (#ifdef instead of if, annotations affecting post-compilation instead of runtime dependency injection).

Feature flags give us a more precise tool for isolation than branching, so that only things that need to be isolated are, and things that do not need to be isolated don't have to be; as opposed to branching which automatically isolates everything. With such a system we would not have to resort to branching for things like long-running feature implementations or release stabilization.

[AArnott]

@eatdrinksleepcode Stabilization/release branches isn't about hiding incomplete features. It's about retaining the ability to service prior releases so that folks who are using 1.0 can get 1.1 without upgrading to 2.0. For instance, suppose 2.0 introduces new features that make the whole library go slower. Or perhaps there are breaking changes between 1.0 and 2.0. A bunch of folks are using 1.0 and a security vulnerability is discovered. You have to be able to ship a 1.1 release for those people so they can upgrade more quickly than if they had to upgrade to 2.0. That's what source control is all about. It's not just about sharing code across a team. It's also about version control.

I don't think we need to revisit the pros/cons of branching here. Millions of software projects out there exist and branching is a well accepted pattern for stabilization branches so I don't think we should spend time debating it here.

And since we expect feature work to occur in master anyway, I think we agree on that point. Feature work always happens in branches anyway by virtue of no work happening in this project at all -- it's all pull request from outside the project anyway. If there is to be long-running development of some new feature, whether it's kicked off in a feature branch or in master can be decided at that point. But my inclination is that all early work should start in a branch and merge into master once it's at least somewhat stable. There is really no reason to put it in master before then anyway.

[terrajobst]

@eatdrinksleepcode:

You certainly can feature flag APIs. [...](#ifdef instead of if, annotations affecting post-compilation instead of runtime dependency injection).

That's a fair point -- I was thinking of runtime feature flags. I'm just not sure whether that's actually better. In fact, many folks would try really hard to get rid of #if from our code tree.

@AArnott made an interesting point that we're already using branches by the fact that we're exclusively using a pull request workflow. Never thought of PRs basically being feature branches.

As far as release branches go: I don't think we've an option of not using branches. As @AArnott pointed out, we need to be able to service older releases. We can't just ship the newest version. See my comments on servicing security bugs why that's important.

[eatdrinksleepcode]

I don't think I am doing a good job of effectively communicating my point. I will take another crack at it after the holiday. Have a Merry Christmas :)

[terrajobst]

@eatdrinksleepcode to you, too! Merry Christmas :-)

[joshfree]

Hi All,
Later this week we will be creating a v1.0 release branch on GitHub. At that point the master branch will become 'forward looking' and will be open to riskier community PRs - such as new APIs - that up until now we've been postponing.

What this means for everyone working on dotnet/corefx (rules of the road) :

• dotnet/corefx master
  • The master branch will be open for 'forward looking' changes that ship in a future release (let's call it 1.1 for now).
  • The backlog of PRs for new APIs can now be reviewed and either accepted or closed against master.
  • All API and general product improvement PRs should continue to be submitted to master going forward
• dotnet/corefx v1.0
  • The v1.0 branch will coincide with our eventual 1.0 release and servicing branch
  • v1.0 will contain all community PRs from master up until the date of the fork.
  • build changes as well as initial library commits from the product team should happen in v1.0 and then flow upstream to master.

I'd also like to mention that since the dotnet/corefx GitHub project is just getting started we're trying hard to keep everything as simple as possible while still meeting the needs of the community and close partners. We are going to be flexible with the plan and open to adjustments as needed going forward.

Please let me know if you have any questions or concerns.

Cheers,
Josh

[eatdrinksleepcode]

Thanks for the update Josh, that sounds sensible.

[joshfree]

I've gone ahead and created the v1.0 branch.

• https://github.com/dotnet/corefx/commits/master
• https://github.com/dotnet/corefx/commits/v1.0

At this point I'll close out the issue. Feel free to reactivate if needed.

[eatdrinksleepcode]

The original reason for this issue (which was not actually related to release branching) has not been resolved, or even really discussed. I apologize for the delay in my response; sadly I do not get to work on .NET Core at my day job :) I will respond to the questions raised by the weekend.

(GitHub does not allow me to re-open my own issue if I didn't close it. Please re-open.)

[terrajobst]

Before we re-open, can you clarify what issue is that you're after? I'm not pushing back, by the way, I just feel that @AArnott, @joshfree and myself provided the answers to your question.