Merging sequences (like Merge keys for mappings)

[perlpunk]

In #35 a syntax to merge sequences was proposed, but the syntax isn't going to happen, while I can understand that it would be useful to have a standard way of doing this.

There are plans to add programmatic features in YAML 1.3, as @ingydotnet mentioned, but they simply don't exist yet.

The question is, can we come up with something simple, that doesn't introduce new syntax?

Here are some suggestions:

1. @ingydotnet 's suggestion was:

array1: &my_array_alias
- foo
- bar

array2:
- <: *my_array_alias
- baz

2. My alternative suggestion would be:

array1: &my_array_alias
- foo
- bar

array2:
- <<
- *my_array_alias
- baz

This approach would be very close to the merge key feature; also when thinking about the implementation (because I have just implemented this in my YAML processor).

3. Another variant would be almost equal to @ingydotnet's suggestion:

array1: &my_array_alias
- foo
- bar

array2:
- <<: *my_array_alias
- baz

I can't think of a reason not to use the same << as for merge keys.

[perlpunk]

Thinking about implementation, IMHO suggestion 1 and 3 will be much harder, because one would have to add additional handling for this on several levels.
Suggestion 2 can be more verbose, especially when you want to merge more than one sequence, but it should be comparably easy to add to an already existing merge key feature.

[DanySK]

More a question than a suggestion, as I might be overlooking something obvious. Why not just provide a "flattened entry" key?

array1: &my_array_alias
- foo
- bar

array2:
+ *my_array_alias
- baz

this loads + with the "flatten" meaning.

Possibly, this could work regardless of the list syntax (JSON style or multiline). All these entries should have the same result

flat: [a, b, c, 1, 2, 3]

classic:
- a
- b
- c
+ 
  - 1
  - 2
  - 3

mixed:
- a
- b
- c
+ [1, 2, 3]

json:
[a, b, c, + [1, 2, 3]]

[perlpunk]

@DanySK this looks nice, I agree. But it would mean a new syntax element, so more work for parsers, and another thing + which would be disallowed in plain scalars.
I think this should be avoided like explained in #35.
If we get a new syntax element (or more), then it should be something general which will be able to add more programmatic features, and the basic syntax should not get more complex.
I know this feature would be useful and I could use it myself, but I can't think of a simple solution.

[DanySK]

Thanks @perlpunk
Would overloading a currently disallowed symbol in plain scalars (e.g., |) cause retrocompatibility issues? As said, the idea came off the top of my mind, I did not consider all the consequences.

Yes, there will be some more work for parsers, and yes, I do understand the YAML specification is already large. However, let me for a moment look at this from a higher perspective: why does YAML exist? Why don't we just use JSON or TOML? I can see two main reasons:

1. readability is higher wrt JSON, and wrt TOML on complex specifications; but more importantly
2. support for reuse via anchors, enabling DRY strategies.

Under this point of view, I advocate for further, not simplifying syntax changes to be considered in case they lead to clear, non ambiguous, and YAML-style-like forms of reuse.

[bbrouwer]

What about syntax like this:

array1: &my_array_alias
- foo
- bar

array2:
-*my_array_alias
- baz

I'm not going to argue at length for this, just wanted to propose this syntax. I'm sure this suggestion requires parser and language changes, but it doesn't introduce any new reserved symbols and sort of fits in with the feel of a list by starting with a dash.

[ssbarnea]

For sure there is a real need for this feature. We only need to be careful on how we describe its behaviour because lists are sorted and can allow duplicate entries.

AFAIK, I do not see a need to allow duplicates (in fact preventing them may count as an advantage). Still the order of entries may be critical, do we want to merge at too, bottom, middle? Which duplicate entry takes priority the one from inside the default or the override?

[DanySK]

@ssbarnea switching from lists to sets is madness, you'd lose also JSON compatibility. Let's keep the discussion clean, the only needed feature is a "flatten-inside" operator. But I can't see any way of introducing it without tinkering with the grammar

[ssbarnea]

@ssbarnea switching from lists to sets is madness, you'd lose also JSON compatibility. Let's keep the discussion clean, the only needed feature is a "flatten-inside" operator. But I can't see any way of introducing it without tinkering with the grammar

I did not propose any divergence from JSON, it would be insane to do so. I only wasted to state that we need to define very well the merging logic, so implementations would not have different behaviours.

I had lots of cases where I had a default list and I wanted to new entries to it, at top of bottom. Based on experience I encountered cases where the tool loading the list would choke if it finds duplicates (list of packages to install). This is why I mentioned that a set like behaviour when doing the merge could be desirable.

[DanySK]

Don't mix the use for your your specific use cases with the general framework. Performing a set union would for instance be irrelevant for most of my uses and deleterious (making the feature useless) for others.
I think the only meaningful merging behaviour for a list is list inserting.

What'd be the output of merging a [2] into a [1, 1] list? [1, 2]? That'd be very surprising to me.

[perlpunk]

@bbrouwer

but it doesn't introduce any new reserved symbols

Well, it does introduce new syntax. The plain scalar -*foo is perfectly fine currently.

If I rewrite your example:

array2:
   -*my_array_alias

then its meaning changes to something which is valid YAML right now.
Also, this would look odd in flow style:

array2: [ -*my_array_alias ]

I still think that introducing new syntax for implementing just one specific programmatic function is wrong. And I bet many other people who actually implemented a YAML parser would agree.

It would also be totally different from merging mapping keys, which happens in the constructing level, while a new syntax element introduces a new type of parsing event.

[perlpunk]

Don't mix the use for your your specific use cases with the general framework

I think this is also a good example why introducing a specific programmatic element is not a good idea.

Merge keys are already not trivial to implement (look into pyyaml and try to figure out how to implement forbidding duplicate mapping keys while keeping the merge key behaviour).

And some might rather want a deep merge, which is not what merge keys do. Would you introduce another merge key (e.g. <<<) just to implement deep merging?

The solution is a more generic programmatic syntax which allows functions and parameters.

This is already possible with local tags, for example in AWS CloudFormation files you can use the !Join tag to concatenate list items.
The disadvantage of that is, that you cannot give the result of such a tag-function another tag.

[perlpunk]

@ssbarnea If I added a merge-list feature similar to merge keys, I would just concatenate the lists.

Anything more complicated needs something like a templating (jinja for example) or a more generic programmatic syntax.

[DanySK]

@perlpunk "merging list" it's not necessarily programmatic. The way I see it, it is purely declarative. The problem with local tags is that it's not a standard solution, hence many use cases, the majority actually (e.g.: CI configuration) won't enjoy them.
For my own software, I implement workarounds manually, but still I am convinced that an equivalent of the merge keys for lists would be useful.

Also re-reading your initial post, I believe:

array2:
- <<: *my_array_alias
- baz

looks good, and does not introduce any new syntax element.