Optimize ImmutableSortedSet<T>.SetEquals to avoid unnecessary allocations

[aw0lid]

Part of #127279

Summary

ImmutableSortedSet<T>.SetEquals always creates a new intermediate SortedSet<T> for the other collection, leading to avoidable allocations and GC pressure, especially for large datasets

Optimization Logic

• Type-Specific Fast Paths: Uses pattern matching to detect if other is an ImmutableSortedSet or SortedSet, triggering optimized logic only if their Comparer matches.
• O(1) Early Exit: Performs an immediate ReferenceEquals check and leverages ICollection to return false early if other.Count is less than this.Count.
• Sequential Lock-Step Comparison: Replaces the $O(\log n)$ per-element .Contains() check with a dual-enumerator while loop. This leverages the sorted nature of both sets to achieve $O(n)$ linear complexity.
• Zero Allocation Path: For compatible sorted sets, the comparison is performed directly on existing instances, eliminating the memory overhead of temporary collections.
• Refined Fallback: Even when a new SortedSet<T> is required for general IEnumerable types, the final comparison now uses the same efficient $O(n)$ sequential scan instead of repeated lookups.

Click to expand Benchmark Source Code

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Order;
using BenchmarkDotNet.Running;
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;

namespace ImmutableSortedSetBenchmarks
{
    [MemoryDiagnoser]
    [Orderer(SummaryOrderPolicy.FastestToSlowest)]
    [RankColumn]
    public class ImmutableSortedSetSetEqualsBenchmark_Int
    {
        private ImmutableSortedSet<int> _sourceSet = null!;
        private ImmutableSortedSet<int> _immutableSortedSetEqual = null!;
        private SortedSet<int> _bclSortedSetEqual = null!;
        private List<int> _listEqual = null!;
        private IEnumerable<int> _linqSelectEqual = null!;
        private int[] _arrayEqual = null!;
        private List<int> _listLastDiff = null!;
        private List<int> _listSmaller = null!;
        private ImmutableSortedSet<int> _immutableLarger = null!;
        private int[] _smallerArray = null!;
        private SortedSet<int> _smallerSortedSetDiffComparer = null!;

        private ImmutableSortedSet<int> _immutableSortedSetLastDiff = null!;
        private SortedSet<int> _bclSortedSetLastDiff = null!;
        private List<int> _listWithDuplicates = null!;
        private List<int> _listWithDuplicatesMatch = null!;

        private SortedSet<int> _bclSortedSetDiffComparer = null!;

        private ImmutableSortedSet<int> _immutableSortedSetSmaller = null!;
        private SortedSet<int> _bclSortedSetSmaller = null!;

        private IEnumerable<int> _lazyEnumerableLastDiff = null!;
       

        [Params(100000)]
        public int Size { get; set; }

        [GlobalSetup]
        public void Setup()
        {
            var elements = Enumerable.Range(0, Size).ToList();
            var elementsWithLastDiff = Enumerable.Range(0, Size - 1).Concat(new[] { Size + 1000 }).ToList();
            var smallerElements = Enumerable.Range(0, Size / 2).ToList();
            var duplicates = Enumerable.Repeat(1, Size).ToList();
            var smallerList = new List<int>();

            for(int i = 0; i < Size - 1; i++) smallerList.Add(i);

            var reverseComparer = new ReverseComparer<int>();

            _sourceSet = ImmutableSortedSet.CreateRange(elements);
            _immutableSortedSetEqual = ImmutableSortedSet.CreateRange(elements);
            _bclSortedSetEqual = new SortedSet<int>(elements);
            _listEqual = elements;
            _linqSelectEqual = elements.Select(x => x); 
            _arrayEqual = elements.ToArray();

            _immutableSortedSetLastDiff = ImmutableSortedSet.CreateRange(elementsWithLastDiff);
            _bclSortedSetLastDiff = new SortedSet<int>(elementsWithLastDiff);
            _listLastDiff = elementsWithLastDiff;

            _bclSortedSetDiffComparer = new SortedSet<int>(elements, reverseComparer);

            _immutableSortedSetSmaller = ImmutableSortedSet.CreateRange(smallerElements);
            _bclSortedSetSmaller = new SortedSet<int>(smallerElements);

            _lazyEnumerableLastDiff = elementsWithLastDiff.Select(x => x);
            _immutableLarger = ImmutableSortedSet.CreateRange(elements.Concat(new[] { -1 }));
            _listWithDuplicates = duplicates;
            _listWithDuplicatesMatch = elements.Concat(elements).ToList(); 
            _listSmaller = smallerList;
            _smallerArray = Enumerable.Range(0, Size - 1).ToArray();
            _smallerSortedSetDiffComparer = new SortedSet<int>(_listSmaller, reverseComparer);
        }

        #region Fast Path: Same Type and Comparer

        [Benchmark(Description = "ImmutableSortedSet (Match - Same Comparer)")]
        public bool Case_ImmutableSortedSet_Match() => _sourceSet.SetEquals(_immutableSortedSetEqual);

        [Benchmark(Description = "BCL SortedSet (Match - Same Comparer)")]
        public bool Case_BclSortedSet_Match() => _sourceSet.SetEquals(_bclSortedSetEqual);

        [Benchmark(Description = "ImmutableSortedSet (Mismatch - Same Count)")]
        public bool Case_ImmutableSortedSet_LastDiff() => _sourceSet.SetEquals(_immutableSortedSetLastDiff);

        [Benchmark(Description = "BCL SortedSet (Mismatch - Same Count)")]
        public bool Case_BclSortedSet_LastDiff() => _sourceSet.SetEquals(_bclSortedSetLastDiff);

        #endregion

        #region Early Exit: Count Mismatch

        [Benchmark(Description = "ImmutableSortedSet (Smaller Count)")]
        public bool Case_ImmutableSortedSet_SmallerCount() => _sourceSet.SetEquals(_immutableSortedSetSmaller);

        [Benchmark(Description = "BCL SortedSet (Smaller Count)")]
        public bool Case_BclSortedSet_SmallerCount() => _sourceSet.SetEquals(_bclSortedSetSmaller);

        [Benchmark(Description = "Array (Smaller Count)")]
        public bool Case_SmallerCollection_EarlyExit() => _sourceSet.SetEquals(_smallerArray);

        #endregion

        #region Fallback Path: Different Comparer

        [Benchmark(Description = "SortedSet (Different Comparer)")]
        public bool Case_SortedSet_DifferentComparer() => _sourceSet.SetEquals(_bclSortedSetDiffComparer);

        [Benchmark(Description = "SortedSet (Smaller Count - Different Comparer)")]
        public bool Case_SortedSet_SmallerCount_DiffComparer() => _sourceSet.SetEquals(_smallerSortedSetDiffComparer);

        #endregion

        #region Fallback Path: Non-Set Collections

        [Benchmark(Description = "List (Match - Fallback)")]
        public bool Case_List_Match() => _sourceSet.SetEquals(_listEqual);

        [Benchmark(Description = "LINQ (Mismatch - Lazy IEnumerable)")]
        public bool Case_LazyEnumerable_LastDiff() => _sourceSet.SetEquals(_lazyEnumerableLastDiff);

        [Benchmark(Description = "LINQ (Match - Lazy IEnumerable)")]
        public bool Case_LazyEnumerable_Match() => _sourceSet.SetEquals(_linqSelectEqual);

        [Benchmark(Description = "List (Last Diff - Fallback)")]
        public bool Case_List_LastDiff() => _sourceSet.SetEquals(_listLastDiff);

        [Benchmark(Description = "Array (Match - Fallback)")]
        public bool Case_Array_Match() => _sourceSet.SetEquals(_arrayEqual);

        [Benchmark(Description = "ImmutableSortedSet (Larger Count)")]
        public bool Case_LargerCount() => _sourceSet.SetEquals(_immutableLarger);

        #endregion

        #region Handling Duplicates

        [Benchmark(Description = "List with Duplicates (Mismatch)")]
        public bool Case_List_Duplicates_Mismatch() => _sourceSet.SetEquals(_listWithDuplicates);

        [Benchmark(Description = "List with Duplicates (Match)")]
        public bool Case_List_Duplicates_Match() => _sourceSet.SetEquals(_listWithDuplicatesMatch);

        #endregion
    }

    public class ReverseComparer<T> : IComparer<T> where T : IComparable<T>
    {
        public int Compare(T? x, T? y)
        {
            if (x == null && y == null) return 0;
            if (x == null) return 1;
            if (y == null) return -1;
            return y.CompareTo(x);
        }
    }

    public class Program
    {
        public static void Main(string[] args)
        {
            BenchmarkRunner.Run<ImmutableSortedSetSetEqualsBenchmark_Int>();
        }
    }
}

Click to expand Benchmark Results

Benchmark Results (Before Optimization)

Method	Size	Mean	Error	StdDev	Rank	Gen0	Gen1	Gen2	Allocated
'List with Duplicates (Mismatch)'	100000	1.931 ms	0.0321 ms	0.0268 ms	1	9.7656	9.7656	9.7656	390.8 KB
'BCL SortedSet (Smaller Count)'	100000	1.956 ms	0.0249 ms	0.0233 ms	1	371.0938	285.1563	-	1953.73 KB
'ImmutableSortedSet (Smaller Count)'	100000	3.516 ms	0.0413 ms	0.0387 ms	2	367.1875	304.6875	3.9063	2148.52 KB
'Array (Smaller Count)'	100000	4.910 ms	0.0860 ms	0.0718 ms	3	671.8750	609.3750	7.8125	4296.91 KB
'SortedSet (Smaller Count - Different Comparer)'	100000	6.799 ms	0.0873 ms	0.0816 ms	4	679.6875	609.3750	7.8125	4297.31 KB
'ImmutableSortedSet (Larger Count)'	100000	7.513 ms	0.1106 ms	0.0981 ms	5	671.8750	625.0000	7.8125	4297 KB
'List (Match - Fallback)'	100000	13.665 ms	0.1204 ms	0.1005 ms	6	671.8750	625.0000	-	4297.26 KB
'Array (Match - Fallback)'	100000	13.797 ms	0.0807 ms	0.0716 ms	6	656.2500	625.0000	-	4297.25 KB
'List (Last Diff - Fallback)'	100000	13.888 ms	0.1573 ms	0.1394 ms	6	671.8750	609.3750	-	4297.25 KB
'BCL SortedSet (Match - Same Comparer)'	100000	14.849 ms	0.2830 ms	0.2647 ms	7	671.8750	609.3750	-	3907.19 KB
'BCL SortedSet (Mismatch - Same Count)'	100000	15.137 ms	0.2700 ms	0.2526 ms	7	671.8750	625.0000	-	3907.19 KB
'LINQ (Mismatch - Lazy IEnumerable)'	100000	15.202 ms	0.1243 ms	0.1162 ms	7	750.0000	625.0000	15.6250	4931.04 KB
'SortedSet (Different Comparer)'	100000	15.214 ms	0.1632 ms	0.1447 ms	7	640.6250	625.0000	-	4297.65 KB
'LINQ (Match - Lazy IEnumerable)'	100000	15.360 ms	0.1323 ms	0.1238 ms	7	734.3750	640.6250	15.6250	4931.13 KB
'ImmutableSortedSet (Mismatch - Same Count)'	100000	17.122 ms	0.2080 ms	0.1946 ms	8	656.2500	593.7500	-	4297.26 KB
'ImmutableSortedSet (Match - Same Comparer)'	100000	17.256 ms	0.2191 ms	0.1829 ms	8	750.0000	593.7500	-	4297.25 KB
'List with Duplicates (Match)'	100000	31.781 ms	0.2065 ms	0.1724 ms	9	625.0000	562.5000	-	4687.9 KB

---

Benchmark Results (After Optimization)

Method	Mean	Error	StdDev	Gen0	Gen1	Gen2	Allocated
Array (Smaller Count)	1.291 ns	0.0560 ns	0.0497 ns	-	-	-	-
SortedSet (Smaller Count - Diff Comparer)	1.571 ns	0.0438 ns	0.0366 ns	-	-	-	-
BCL SortedSet (Smaller Count)	1.705 ns	0.0763 ns	0.0714 ns	-	-	-	-
ImmutableSortedSet (Smaller Count)	2.263 ns	0.0921 ns	0.2206 ns	-	-	-	-
ImmutableSortedSet (Larger Count)	5.672 ns	0.1561 ns	0.2137 ns	-	-	-	-
List with Duplicates (Mismatch)	2,052,830.5 ns	22,614.1 ns	21,153.3 ns	7.8125	7.8125	7.8125	400,171 B
BCL SortedSet (Mismatch - Same Count)	2,582,884.9 ns	22,630.9 ns	18,897.8 ns	-	-	-	312 B
BCL SortedSet (Match - Same Comparer)	2,590,534.6 ns	39,425.8 ns	34,950.0 ns	-	-	-	313 B
ImmutableSortedSet (Mismatch - Same Count)	3,598,708.9 ns	67,585.1 ns	63,219.2 ns	-	-	-	-
ImmutableSortedSet (Match - Same Comparer)	3,622,289.7 ns	58,696.5 ns	54,904.8 ns	-	-	-	-
Array (Match - Fallback)	7,693,944.7 ns	150,692.9 ns	133,585.3 ns	687.5000	617.1875	7.8125	4,400,387 B
List (Last Diff - Fallback)	7,719,572.2 ns	125,113.1 ns	117,030.9 ns	664.0625	609.3750	7.8125	4,400,387 B
List (Match - Fallback)	7,723,728.5 ns	103,719.3 ns	91,944.4 ns	671.8750	617.1875	7.8125	4,400,389 B
LINQ (Mismatch - Lazy IEnumerable)	8,954,665.2 ns	89,986.3 ns	75,142.6 ns	734.3750	640.6250	15.6250	5,049,386 B
LINQ (Match - Lazy IEnumerable)	9,121,811.5 ns	144,223.7 ns	127,850.6 ns	734.3750	640.6250	15.6250	5,049,386 B
SortedSet (Different Comparer)	9,359,662.8 ns	155,159.6 ns	178,681.9 ns	656.2500	625.0000	-	4,400,734 B
List with Duplicates (Match)	24,959,265.0 ns	160,574.3 ns	150,201.3 ns	656.2500	625.0000	-	4,800,386 B

Performance Analysis Summary (100,000 Elements)

Performance Summary (Speedup & Memory)

Case	Speed Improvement	Memory Improvement
SortedSet (Smaller - Diff Comp)	~4,327,816x	Zero Alloc
Array (Smaller)	~3,803,253x	Zero Alloc
ImmutableSortedSet (Smaller)	~1,553,689x	Zero Alloc
ImmutableSortedSet (Larger)	~1,324,576x	Zero Alloc
BCL SortedSet (Smaller)	~1,147,214x	Zero Alloc
BCL SortedSet (Mismatch - Same Count)	~5.86x	~99.99% (312 B)
BCL SortedSet (Match - Same Comparer)	~5.73x	~99.99% (313 B)
ImmutableSortedSet (Match - Same Comparer)	~4.76x	Zero Alloc
ImmutableSortedSet (Mismatch - Same Count)	~4.76x	Zero Alloc
List (Last Diff - Fallback)	~1.80x	Stable
Array (Match - Fallback)	~1.79x	Stable
List (Match - Fallback)	~1.77x	Stable
LINQ (Mismatch - Lazy IEnumerable)	~1.70x	Stable
LINQ (Match - Lazy IEnumerable)	~1.68x	Stable
SortedSet (Different Comparer)	~1.63x	Stable
List with Duplicates (Match)	~1.27x	Stable
List with Duplicates (Mismatch)	~1.00x	Stable

[aw0lid]

Just gentle ping in case this fell through the cracks.
Ready for any review.

[eiriktsarpalis]

Avoid putting private implementations ahead of public implementations. If factoring out code into a separate method is necessary, consider using a local method instead.

[aw0lid]

I have moved the private implementations below the public ones to follow the project's convention.

Regarding local methods, I opted for separate private static helpers to maintain the consistency of the class and avoid making the main method's body overly complex or deeply nested.

[eiriktsarpalis]

This appears to be repeating the same logic as when other is already a sorted set. I think you could rework the structure of your nested functions a little bit... Lose the "fast path" and "fallback" methods and have methods for comparing SortedSet and ImmutableSortedSet specifically.

[aw0lid]

Indeed, I had already noticed that duplication and streamlined the code in the latest update.

[eiriktsarpalis]

Suggested change

	//We check for < instead of != because other is not guaranteed to be a set; it could be a List<T> or another collection with duplicates.
	// We check for < instead of != because other is not guaranteed to be a set; it could be a List<T> or another collection with duplicates.

[eiriktsarpalis]

I suspect this could be simplified using a switch statement that pattern matches on other using a when clause to check the comparer. What do you think?