Skip to content

refactor: parametrize benchmark iterations for indexers, methods, and properties #645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vbreuss wants to merge 1 commit into from
Closed

refactor: parametrize benchmark iterations for indexers, methods, and properties #645

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .idea/.idea.Mockolate/.idea/.name
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 34 additions & 13 deletions Benchmarks/Mockolate.Benchmarks/CompleteIndexerBenchmarks.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@ namespace Mockolate.Benchmarks;
#pragma warning disable CA1822 // Mark members as static
/// <summary>
/// In this benchmark we check the case of an interface mock with an indexer, setup the indexer and verify
/// the getter was called once.
/// the getter was called exactly <see cref="InvocationCount" /> times.
/// </summary>
public class CompleteIndexerBenchmarks : BenchmarksBase
{
[Params(1, 10)]
public int InvocationCount { get; set; }

/// <summary>
/// <see href="https://awexpect.com/Mockolate" />
/// </summary>
Expand All @@ -26,9 +29,12 @@ public void Indexer_Mockolate()
IMyIndexerInterface sut = IMyIndexerInterface.CreateMock();
sut.Mock.Setup[It.IsAny<int>()].Returns("foo");

_ = sut[42];
for (int i = 0; i < InvocationCount; i++)
{
_ = sut[42];
}

sut.Mock.Verify[It.IsAny<int>()].Got().Once();
sut.Mock.Verify[It.IsAny<int>()].Got().Exactly(InvocationCount);
}

/// <summary>
Expand All @@ -40,9 +46,12 @@ public void Indexer_Moq()
Moq.Mock<IMyIndexerInterface> mock = new();
mock.Setup(x => x[Moq.It.IsAny<int>()]).Returns("foo");

_ = mock.Object[42];
for (int i = 0; i < InvocationCount; i++)
{
_ = mock.Object[42];
}
Comment on lines +49 to +52
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside the measurement loop, repeatedly accessing mock.Object adds overhead that’s unrelated to the indexer/property/method invocation being benchmarked. Cache mock.Object into a local variable before the loop and use that local inside the loop to keep the benchmark focused on call/verification cost.

Copilot uses AI. Check for mistakes.

mock.Verify(x => x[Moq.It.IsAny<int>()], Times.Once());
mock.Verify(x => x[Moq.It.IsAny<int>()], Times.Exactly(InvocationCount));
}

/// <summary>
Expand All @@ -54,9 +63,12 @@ public void Indexer_NSubstitute()
IMyIndexerInterface mock = Substitute.For<IMyIndexerInterface>();
mock[Arg.Any<int>()].Returns("foo");

_ = mock[42];
for (int i = 0; i < InvocationCount; i++)
{
_ = mock[42];
}

_ = mock.Received(1)[Arg.Any<int>()];
_ = mock.Received(InvocationCount)[Arg.Any<int>()];
}

/// <summary>
Expand All @@ -68,9 +80,12 @@ public void Indexer_FakeItEasy()
IMyIndexerInterface mock = A.Fake<IMyIndexerInterface>();
A.CallTo(() => mock[A<int>.Ignored]).Returns("foo");

_ = mock[42];
for (int i = 0; i < InvocationCount; i++)
{
_ = mock[42];
}

A.CallTo(() => mock[A<int>.Ignored]).MustHaveHappened(1, FakeItEasy.Times.Exactly);
A.CallTo(() => mock[A<int>.Ignored]).MustHaveHappened(InvocationCount, FakeItEasy.Times.Exactly);
}

/// <summary>
Expand All @@ -82,9 +97,12 @@ public void Indexer_Imposter()
IMyIndexerInterfaceImposter imposter = IMyIndexerInterface.Imposter();
imposter[Imposter.Abstractions.Arg<int>.Any()].Getter().Returns("foo");

_ = imposter.Instance()[42];
for (int i = 0; i < InvocationCount; i++)
{
_ = imposter.Instance()[42];
}
Comment on lines +100 to +103
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imposter.Instance() is called on every loop iteration, which can skew results if Instance() has non-trivial overhead. Store the instance in a local variable before the loop and use that variable inside the loop so the benchmark measures indexer access rather than instance retrieval.

Copilot uses AI. Check for mistakes.

imposter[Imposter.Abstractions.Arg<int>.Any()].Getter().Called(Count.Once());
imposter[Imposter.Abstractions.Arg<int>.Any()].Getter().Called(Count.Exactly(InvocationCount));
}

/* Indexers not supported on TUnit.Mocks
Expand All @@ -97,9 +115,12 @@ public void Indexer_TUnitMocks()
TUnit.Mocks.Mock<IMyIndexerInterface> mock = TUnit.Mocks.Mock.Of<IMyIndexerInterface>();
mock[Any<int>()].Returns("foo");

_ = mock.Object[42];
for (int i = 0; i < InvocationCount; i++)
{
_ = mock.Object[42];
}

mock[Any<int>()].WasCalled();
mock[Any<int>()].WasCalled(TUnit.Mocks.Times.Exactly(InvocationCount));
}
*/

Expand Down
47 changes: 34 additions & 13 deletions Benchmarks/Mockolate.Benchmarks/CompleteMethodBenchmarks.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@ namespace Mockolate.Benchmarks;
#pragma warning disable CA1822 // Mark members as static
/// <summary>
/// In this benchmark we check the simple case of an interface mock, setup a single method that gets called and
/// verified to be called once.
/// verified to be called exactly <see cref="InvocationCount" /> times.
/// </summary>
public class CompleteMethodBenchmarks : BenchmarksBase
{
[Params(1, 10)]
public int InvocationCount { get; set; }
Comment thread
vbreuss marked this conversation as resolved.

/// <summary>
/// <see href="https://awexpect.com/Mockolate" />
/// </summary>
Expand All @@ -26,9 +29,12 @@ public void Method_Mockolate()
IMyMethodInterface sut = IMyMethodInterface.CreateMock();
sut.Mock.Setup.MyFunc(It.IsAny<int>()).Returns(true);

sut.MyFunc(42);
for (int i = 0; i < InvocationCount; i++)
{
sut.MyFunc(42);
}

sut.Mock.Verify.MyFunc(It.IsAny<int>()).Once();
sut.Mock.Verify.MyFunc(It.IsAny<int>()).Exactly(InvocationCount);
}

/// <summary>
Expand All @@ -40,9 +46,12 @@ public void Method_Moq()
Moq.Mock<IMyMethodInterface> mock = new();
mock.Setup(x => x.MyFunc(Moq.It.IsAny<int>())).Returns(true);

mock.Object.MyFunc(42);
for (int i = 0; i < InvocationCount; i++)
{
mock.Object.MyFunc(42);
}
Comment on lines +49 to +52
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside the measurement loop, repeatedly accessing mock.Object adds overhead not directly related to the method invocation being benchmarked. Cache mock.Object into a local variable before the loop to avoid including Object retrieval cost in the measurement.

Copilot uses AI. Check for mistakes.

mock.Verify(x => x.MyFunc(Moq.It.IsAny<int>()), Times.Once());
mock.Verify(x => x.MyFunc(Moq.It.IsAny<int>()), Times.Exactly(InvocationCount));
}

/// <summary>
Expand All @@ -54,9 +63,12 @@ public void Method_NSubstitute()
IMyMethodInterface mock = Substitute.For<IMyMethodInterface>();
mock.MyFunc(Arg.Any<int>()).Returns(true);

mock.MyFunc(42);
for (int i = 0; i < InvocationCount; i++)
{
mock.MyFunc(42);
}

mock.Received(1).MyFunc(Arg.Any<int>());
mock.Received(InvocationCount).MyFunc(Arg.Any<int>());
}

/// <summary>
Expand All @@ -68,9 +80,12 @@ public void Method_FakeItEasy()
IMyMethodInterface mock = A.Fake<IMyMethodInterface>();
A.CallTo(() => mock.MyFunc(A<int>.Ignored)).Returns(true);

mock.MyFunc(42);
for (int i = 0; i < InvocationCount; i++)
{
mock.MyFunc(42);
}

A.CallTo(() => mock.MyFunc(A<int>.Ignored)).MustHaveHappened(1, FakeItEasy.Times.Exactly);
A.CallTo(() => mock.MyFunc(A<int>.Ignored)).MustHaveHappened(InvocationCount, FakeItEasy.Times.Exactly);
}

/// <summary>
Expand All @@ -82,9 +97,12 @@ public void Method_Imposter()
IMyMethodInterfaceImposter imposter = IMyMethodInterface.Imposter();
imposter.MyFunc(Imposter.Abstractions.Arg<int>.Any()).Returns(true);

imposter.Instance().MyFunc(42);
for (int i = 0; i < InvocationCount; i++)
{
imposter.Instance().MyFunc(42);
}
Comment on lines +100 to +103
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imposter.Instance() is invoked on each iteration of the loop. If Instance() isn’t a trivial getter, this will skew benchmark results; cache the instance once before the loop and call MyFunc on the cached instance inside the loop.

Copilot uses AI. Check for mistakes.

imposter.MyFunc(Imposter.Abstractions.Arg<int>.Any()).Called(Count.Once());
imposter.MyFunc(Imposter.Abstractions.Arg<int>.Any()).Called(Count.Exactly(InvocationCount));
}

/// <summary>
Expand All @@ -96,9 +114,12 @@ public void Method_TUnitMocks()
Mock<IMyMethodInterface> mock = TUnit.Mocks.Mock.Of<IMyMethodInterface>();
mock.MyFunc(Any<int>()).Returns(true);

mock.Object.MyFunc(42);
for (int i = 0; i < InvocationCount; i++)
{
mock.Object.MyFunc(42);
}
Comment on lines +117 to +120
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop repeatedly accesses mock.Object. To avoid measuring Object retrieval overhead (and to align with how other benchmarks in this suite access Object once), cache mock.Object to a local variable before the loop and call the method on the local inside the loop.

Copilot uses AI. Check for mistakes.

mock.MyFunc(Any<int>()).WasCalled();
mock.MyFunc(Any<int>()).WasCalled(TUnit.Mocks.Times.Exactly(InvocationCount));
}

public interface IMyMethodInterface
Expand Down
47 changes: 34 additions & 13 deletions Benchmarks/Mockolate.Benchmarks/CompletePropertyBenchmarks.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@ namespace Mockolate.Benchmarks;
#pragma warning disable CA1822 // Mark members as static
/// <summary>
/// In this benchmark we check the case of an interface mock with a property, setup the property and verify
/// the getter was called once.
/// the getter was called exactly <see cref="InvocationCount" /> times.
/// </summary>
public class CompletePropertyBenchmarks : BenchmarksBase
{
[Params(1, 10)]
public int InvocationCount { get; set; }

Comment thread
vbreuss marked this conversation as resolved.
/// <summary>
/// <see href="https://awexpect.com/Mockolate" />
/// </summary>
Expand All @@ -25,9 +28,12 @@ public void Property_Mockolate()
IMyPropertyInterface sut = IMyPropertyInterface.CreateMock();
sut.Mock.Setup.Counter.InitializeWith(42);

_ = sut.Counter;
for (int i = 0; i < InvocationCount; i++)
{
_ = sut.Counter;
}

sut.Mock.Verify.Counter.Got().Once();
sut.Mock.Verify.Counter.Got().Exactly(InvocationCount);
}

/// <summary>
Expand All @@ -39,9 +45,12 @@ public void Property_Moq()
Moq.Mock<IMyPropertyInterface> mock = new();
mock.SetupGet(x => x.Counter).Returns(42);

_ = mock.Object.Counter;
for (int i = 0; i < InvocationCount; i++)
{
_ = mock.Object.Counter;
}
Comment on lines +48 to +51
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside the measurement loop, repeatedly accessing mock.Object adds overhead not directly related to the property getter being benchmarked. Cache mock.Object to a local before the loop and read the property from the local inside the loop.

Copilot uses AI. Check for mistakes.

mock.VerifyGet(x => x.Counter, Times.Once());
mock.VerifyGet(x => x.Counter, Times.Exactly(InvocationCount));
}

/// <summary>
Expand All @@ -53,9 +62,12 @@ public void Property_NSubstitute()
IMyPropertyInterface mock = Substitute.For<IMyPropertyInterface>();
mock.Counter.Returns(42);

_ = mock.Counter;
for (int i = 0; i < InvocationCount; i++)
{
_ = mock.Counter;
}

_ = mock.Received(1).Counter;
_ = mock.Received(InvocationCount).Counter;
}

/// <summary>
Expand All @@ -67,9 +79,12 @@ public void Property_FakeItEasy()
IMyPropertyInterface mock = A.Fake<IMyPropertyInterface>();
A.CallTo(() => mock.Counter).Returns(42);

_ = mock.Counter;
for (int i = 0; i < InvocationCount; i++)
{
_ = mock.Counter;
}

A.CallTo(() => mock.Counter).MustHaveHappened(1, FakeItEasy.Times.Exactly);
A.CallTo(() => mock.Counter).MustHaveHappened(InvocationCount, FakeItEasy.Times.Exactly);
}

/// <summary>
Expand All @@ -81,9 +96,12 @@ public void Property_Imposter()
IMyPropertyInterfaceImposter imposter = IMyPropertyInterface.Imposter();
imposter.Counter.Getter().Returns(42);

_ = imposter.Instance().Counter;
for (int i = 0; i < InvocationCount; i++)
{
_ = imposter.Instance().Counter;
}
Comment on lines +99 to +102
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imposter.Instance() is called every iteration. Cache the result of imposter.Instance() once before the loop so the benchmark measures property access, not repeated instance retrieval overhead.

Copilot uses AI. Check for mistakes.

imposter.Counter.Getter().Called(Count.Once());
imposter.Counter.Getter().Called(Count.Exactly(InvocationCount));
}

/// <summary>
Expand All @@ -95,9 +113,12 @@ public void Property_TUnitMocks()
Mock<IMyPropertyInterface> mock = TUnit.Mocks.Mock.Of<IMyPropertyInterface>();
mock.Counter.Returns(42);

_ = mock.Object.Counter;
for (int i = 0; i < InvocationCount; i++)
{
_ = mock.Object.Counter;
}
Comment on lines +116 to +119
Copy link

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop repeatedly uses mock.Object, which can add overhead unrelated to the property getter itself. Cache mock.Object into a local variable before the loop and use the local inside the loop to keep benchmark results focused on invocation/verification cost.

Copilot uses AI. Check for mistakes.

mock.Counter.WasCalled();
mock.Counter.WasCalled(TUnit.Mocks.Times.Exactly(InvocationCount));
}

public interface IMyPropertyInterface
Expand Down