feat: add benchmarks for Imposter and Mockolate mocking frameworks by vbreuss · Pull Request #5295 · thomhurst/TUnit

vbreuss · 2026-03-29T13:37:44Z

Description

This pull request expands the mocking libraries benchmarked in the TUnit.Mocks.Benchmarks project with two other source-generated mocking frameworks:

It introduces new benchmarks for these frameworks across all scenarios (mock creation, setup, invocation, callbacks, verification, and combined workflows) to provide a fair comparison between the libraries.

Related Issue

Mock Benchmarks #5283

Fixes #

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update
Performance improvement
Refactoring (no functional changes)

Checklist

Required

I have read the Contributing Guidelines
If this is a new feature, I started a discussion first and received agreement
My code follows the project's code style (modern C# syntax, proper naming conventions)
I have written tests that prove my fix is effective or my feature works

TUnit-Specific Requirements

Testing

All existing tests pass (dotnet test)
I have added tests that cover my changes
I have tested both source-generated and reflection modes (if applicable)

Additional Notes

claude

Code Review

Good addition to expand the benchmark coverage to Imposter and Mockolate. The benchmarks cover the full set of scenarios (creation, setup, invocation, callbacks, verification, combined workflow), which is consistent with how the existing frameworks are benchmarked. A few issues worth addressing before merging:

Bug: Inconsistent use of `imposter.Instance()` in `VerificationBenchmarks.cs`

In CallbackBenchmarks.cs and CombinedWorkflowBenchmarks.cs, the Imposter benchmarks correctly retrieve the typed interface instance before invoking methods:

var instance = imposter.Instance();
instance.Send("user@test.com", "Hello");

But in VerificationBenchmarks.cs (both Imposter_Verify and Imposter_VerifyMultiple), methods are called directly on imposter without calling .Instance():

var calc = imposter;  // this is the imposter wrapper, not the interface instance
calc.Add(1, 2);       // are these calls actually being recorded?

If ICalculatorServiceImposter doesn't implement ICalculatorService, these calls go to a different code path than what's intended — meaning the verify assertions may be vacuously true (verifying 0 calls = 0 expected) rather than validating that real invocations were counted. The benchmarks would be measuring something different from the other frameworks.

Potential issue: `Imposter_CreateMock` returns the wrapper, not the instance

public object Imposter_CreateMock()
{
    var imposter = ICalculatorService.Imposter();
    return imposter;  // returns the ICalculatorServiceImposter wrapper
}

All other frameworks return the actual mock object (the ICalculatorService-implementing instance). The Mockolate version also returns the mock object directly since CreateMock() returns the mock implementing the interface. For a fair comparison, Imposter should probably return imposter.Instance() here as well.

Potential duplicate `[assembly: GenerateImposter]` attributes

The same type is registered with [assembly: GenerateImposter] in multiple files:

ICalculatorService appears in MockCreationBenchmarks.cs, SetupBenchmarks.cs, InvocationBenchmarks.cs, and VerificationBenchmarks.cs
IUserRepository appears in MockCreationBenchmarks.cs, SetupBenchmarks.cs, CombinedWorkflowBenchmarks.cs, and VerificationBenchmarks.cs

If GenerateImposterAttribute has AllowMultiple = true, this compiles but may cause the source generator to emit duplicate code (duplicate class definitions). If it has AllowMultiple = false, this is a compile error. These should be consolidated into a single location (e.g., a dedicated ImposterRegistrations.cs file, similar to how the existing code groups them), with one [assembly: GenerateImposter] per type.

Minor: Mixed indentation in `Directory.Packages.props`

The Imposter entry uses a tab while Mockolate uses spaces, which is inconsistent with the rest of the file.

Summary

The overall structure is solid and consistent with the existing benchmark patterns. The main concern is the correctness of the Imposter verification benchmarks — if methods are being called on the wrapper instead of the interface instance, those benchmarks don't measure what they appear to measure. Please verify the VerificationBenchmarks Imposter code path produces correct call-count tracking before merging.

claude

Follow-up Code Review

Comparing against my previous review on the earlier commit — two of the four issues have been addressed:

✅ Fixed: `Imposter_CreateMock` now returns `imposter.Instance()`

✅ Fixed: `VerificationBenchmarks` now correctly calls `imposter.Instance()` before invoking methods

Still present: Duplicate `[assembly: GenerateImposter]` attributes

The same types are still declared in multiple files:

ICalculatorService: MockCreationBenchmarks.cs, SetupBenchmarks.cs, InvocationBenchmarks.cs, VerificationBenchmarks.cs
IUserRepository: MockCreationBenchmarks.cs, SetupBenchmarks.cs, CombinedWorkflowBenchmarks.cs, VerificationBenchmarks.cs
ILogger: CallbackBenchmarks.cs, CombinedWorkflowBenchmarks.cs

This compiles only if GenerateImposterAttribute has [AttributeUsage(AllowMultiple = true)]. Even if it does compile, the source generator may emit duplicate class definitions. The fix is to consolidate all registrations into a single file (e.g. ImposterRegistrations.cs), one attribute per type — same pattern that's cleanest for InternalsVisibleTo and similar assembly attributes.

New: Wrong matcher type in `Imposter_MultipleSetups` (`SetupBenchmarks.cs`)

// SetupBenchmarks.cs — uses Moq.It (aliased as It) with Imposter
imposter.Exists(It.IsAny<int>()).Returns(true);  // ❌ Moq.It, not Imposter's Arg<T>

All other Imposter setups in this PR use Arg<int>.Any(), which is Imposter's own matcher API:

imposter.Add(Arg<int>.Any(), Arg<int>.Any()).Returns(42);  // ✅ consistent

This is likely a copy-paste oversight from the Mockolate or Moq setups. If Imposter's source-generated proxy doesn't accept Moq expression matchers, the setup will either silently fail to match (returning default values) or throw at runtime — either way producing misleading benchmark results. This should be:

imposter.Exists(Arg<int>.Any()).Returns(true);

Summary

Two meaningful fixes landed in this update — good progress. The remaining blocker is the potential mismatched matcher in Imposter_MultipleSetups, which could cause the setup to silently not match and benchmark incorrect behavior. The duplicate [assembly: GenerateImposter] attributes are a structural concern worth cleaning up but less likely to cause runtime issues if the attribute allows multiples.

claude

Review: Add benchmarks for Imposter and Mockolate

Thanks for expanding the benchmark suite — having more source-generated mocking frameworks in the comparison is genuinely useful. The benchmark code itself is correct and follows the existing patterns well. A few things worth raising:

Duplicate attributes

The attribute for the same types is declared in multiple files. For example, appears in both and , and appears in both and , etc.

Assembly-level attributes with the same type argument will cause a compile error if they truly duplicate. At minimum this is confusing — they should be consolidated into a single file (e.g., a dedicated or alongside other globals in ). The fact that the PR currently references five separate files each declaring for overlapping types is a red flag that this may not compile cleanly.

Asymmetric benchmark design in

For , the benchmark returns (the mock wrapper object itself), while for Imposter and all other frameworks the benchmark returns the actual interface implementation (, , etc.).

If the creation cost of the proxy is amortized into the wrapper or deferred to first use, returning the wrapper instead of the proxy could make Mockolate appear artificially faster in the creation benchmark. This should be returning the actual usable object for a fair comparison.

Inconsistency in setup between Imposter and Mockolate

In , all other frameworks (TUnit.Mocks, Moq, NSubstitute, FakeItEasy) store both the mock handle and the proxied object as fields — which is the correct pattern for invocation-only benchmarks where setup overhead should be excluded via . The Imposter fields are added correctly, but is typed as and the setup calls — it's worth confirming whether actually returns something assignable to or whether it returns a wrapper that implements the interface. If it's a wrapper (same issue as above), then in the benchmark may be going through additional dispatch that the others don't.

Missing in several files

The diff removes from , , , and , replacing it with . However, the , , etc. types are still used in those files. This works because of the namespace alias, but still requires either or fully qualified names. This may already work (since is fully qualified), but it's worth double-checking that no (unqualified) usages exist that would now ambiguously resolve to from the global using.

Specifically in :

Using (Moq's matcher) in an Imposter verify call may not match correctly — Imposter uses its own matcher. If Imposter's verification doesn't understand Moq's , these verifications could silently pass or fail incorrectly, producing misleading benchmark output.

Minor: in setup benchmarks

In and , is used. The existing TUnit.Mocks, Moq, NSubstitute, and FakeItEasy variants also use , so this is consistent, but it's a minor GC allocation that happens to be identical across all implementations, so it's fine from a fairness standpoint.

Summary

The most impactful issues to address before merging:

Consolidate declarations to avoid potential duplicate attribute issues.
Verify returns the proxied instance, not the wrapper, for a fair comparison.
Check that (Moq) used in Imposter verification calls actually works correctly with Imposter's argument matching — if not, use consistently.

Overall the contribution is solid and the benchmark coverage is a welcome addition to the suite.

claude

Review: Add benchmarks for Imposter and Mockolate

Thanks for expanding the benchmark suite — having more source-generated mocking frameworks in the comparison is genuinely useful. The benchmark code itself is correct and follows the existing patterns well. A few things worth raising:

Duplicate `[assembly: GenerateImposter(...)]` attributes

The GenerateImposter attribute for the same types is declared in multiple files. For example, [assembly: GenerateImposter(typeof(TUnit.Mocks.Benchmarks.ICalculatorService))] appears in both MockCreationBenchmarks.cs and SetupBenchmarks.cs, and ILogger appears in both CallbackBenchmarks.cs and CombinedWorkflowBenchmarks.cs.

Assembly-level attributes with the same type argument will cause a compile error if they truly duplicate. At minimum this is confusing — they should be consolidated into a single file (e.g., a dedicated ImposterAttributes.cs or alongside other globals in GlobalUsings.cs). Having five separate files each declaring [assembly: GenerateImposter] for overlapping types is a red flag that this may not compile cleanly.

Asymmetric benchmark design in `MockCreationBenchmarks`

For Mockolate_CreateMock, the benchmark returns sut (the mock wrapper object itself), while for Imposter and all other frameworks the benchmark returns the actual interface implementation (.Instance(), .Object, etc.).

If the creation cost of the proxy is amortized into the wrapper or deferred to first use, returning the wrapper instead of the proxy could make Mockolate appear artificially faster in the creation benchmark. The benchmark should return the actual usable proxied instance for a fair comparison.

Potential matcher mismatch in Imposter verification

In CombinedWorkflowBenchmarks.Imposter_FullWorkflow, Moq's It.IsAny<T>() matcher is used inside Imposter's Called() verification calls:

repoImposter.Save(It.IsAny<User>()).Called(Count.Once());
loggerImposter.Log(It.IsAny<string>(), It.IsAny<string>()).Called(Count.Once());

Imposter uses its own Arg<T>.Any() argument matchers (as used consistently throughout the rest of the Imposter benchmarks). If Imposter's verification doesn't understand Moq's It.IsAny<T>(), these verifications could silently pass regardless of actual invocations, producing misleading benchmark results. These should use Arg<T>.Any() for consistency.

Removing `using Moq` — potential ambiguity

The diff removes using Moq; from several files, replacing it with using It = Moq.It;. However, GlobalUsings.cs already has global using TUnit.Mocks;, which brings Mock<T> into scope as TUnit.Mocks.Mock<T>. Worth a quick scan to ensure no unqualified Moq types were inadvertently re-routed as a result of this change.

Minor: `InvocationBenchmarks` Mockolate field type

In InvocationBenchmarks, the Mockolate mock is stored as ICalculatorService? and used directly (rather than via a .Object or .Instance() accessor like the other frameworks). This is fine if CreateMock() returns a type implementing ICalculatorService directly, but it's worth confirming there isn't hidden wrapper dispatch happening during the invocation benchmarks.

Summary

Most impactful issues to address before merging:

Consolidate [assembly: GenerateImposter(...)] declarations into one file to avoid potential duplicate attribute compile errors.
Verify Mockolate_CreateMock returns the proxied instance (not the wrapper) for a fair comparison with other frameworks.
Replace It.IsAny<T>() with Arg<T>.Any() in Imposter verification calls in CombinedWorkflowBenchmarks to avoid potential silent false-pass verification.

Overall this is a welcome addition to the benchmark coverage.

claude

Follow-up Code Review

Comparing against the previous reviews — several issues have been resolved, but two concerns remain.

✅ Fixed: `VerificationBenchmarks` correctly calls `.Instance()` before invoking methods

✅ Fixed: `Imposter_CreateMock` now returns `imposter.Instance()`

✅ Fixed: `CombinedWorkflowBenchmarks.Imposter_FullWorkflow` uses `Arg<T>.Any()` consistently for Imposter verification

Still present: Duplicate `[assembly: GenerateImposter(...)]` attributes

The same types are still declared across multiple files:

ICalculatorService: MockCreationBenchmarks.cs, SetupBenchmarks.cs, InvocationBenchmarks.cs, VerificationBenchmarks.cs
IUserRepository: MockCreationBenchmarks.cs, SetupBenchmarks.cs, CombinedWorkflowBenchmarks.cs, VerificationBenchmarks.cs
ILogger: CallbackBenchmarks.cs, CombinedWorkflowBenchmarks.cs
INotificationService: CallbackBenchmarks.cs

This has been flagged in every prior review. If GenerateImposterAttribute has [AttributeUsage(AllowMultiple = true)], it compiles — but the source generator may still emit duplicate class definitions for the same interface, causing build errors. The correct architectural fix is to consolidate all of these into a single file (e.g., ImposterRegistrations.cs), exactly as you'd consolidate [assembly: InternalsVisibleTo] attributes. Each interface should appear exactly once.

Still present: Wrong matcher in `Mockolate_MultipleSetups` (`SetupBenchmarks.cs`)

The file has using It = Moq.It;, so It.IsAny<int>() is Moq.It.IsAny<int>(). All other Mockolate setups in this file and the rest of the PR use Mockolate.It.IsAny<T>() (fully qualified):

This was flagged in the previous review. If Mockolate's source-generated proxy uses runtime matcher inspection (not compile-time code generation), then passing Moq.It.IsAny<int>() into a Mockolate setup will silently fail to match — the mock returns the default value instead of true, and the benchmark measures a partially-broken configuration. Fix:

Summary

Three rounds of fixes have landed — good progress, and the overall benchmark structure is solid. The two remaining issues are both carryovers from earlier reviews:

Consolidate [assembly: GenerateImposter(...)] into one file — avoids potential duplicate code generation.
Fix Mockolate.It vs Moq.It in Mockolate_MultipleSetups — ensures the setup actually registers the return value and the benchmark measures correct behavior.

claude

Follow-up Code Review

Comparing against the previous reviews — several issues have been resolved, but two concerns remain.

✅ Fixed: `VerificationBenchmarks` correctly calls `.Instance()` before invoking methods

✅ Fixed: `Imposter_CreateMock` now returns `imposter.Instance()`

✅ Fixed: `CombinedWorkflowBenchmarks.Imposter_FullWorkflow` uses `Arg<T>.Any()` consistently for Imposter verification

Still present: Duplicate `[assembly: GenerateImposter(...)]` attributes

The same types are still declared across multiple files:

ICalculatorService: MockCreationBenchmarks.cs, SetupBenchmarks.cs, InvocationBenchmarks.cs, VerificationBenchmarks.cs
IUserRepository: MockCreationBenchmarks.cs, SetupBenchmarks.cs, CombinedWorkflowBenchmarks.cs, VerificationBenchmarks.cs
ILogger: CallbackBenchmarks.cs, CombinedWorkflowBenchmarks.cs
INotificationService: CallbackBenchmarks.cs

This has been flagged in every prior review. If GenerateImposterAttribute has [AttributeUsage(AllowMultiple = true)], it compiles — but the source generator may still emit duplicate class definitions for the same interface, causing build errors. The correct fix is to consolidate all of these into a single file (e.g., ImposterRegistrations.cs), exactly as you'd consolidate [assembly: InternalsVisibleTo] attributes. Each interface should appear exactly once.

Still present: Wrong matcher in `Mockolate_MultipleSetups` (`SetupBenchmarks.cs`)

The file has using It = Moq.It;, so unqualified It.IsAny<int>() resolves to Moq.It.IsAny<int>(). All other Mockolate setups in this PR use fully-qualified Mockolate.It.IsAny<T>(), but this line uses the alias:

sut.Mock.Setup.Exists(It.IsAny<int>()).Returns(true);  // ❌ Moq.It, not Mockolate.It

If Mockolate's runtime matcher inspection doesn't recognize Moq.It.IsAny<int>(), this setup silently fails to match — the mock returns the default value instead of true. The fix is straightforward:

sut.Mock.Setup.Exists(Mockolate.It.IsAny<int>()).Returns(true);

Summary

Three rounds of fixes have landed — good progress, and the overall benchmark structure is solid. The two remaining issues are both carryovers from earlier reviews:

Consolidate [assembly: GenerateImposter(...)] into one file — avoids potential duplicate code generation.
Fix Mockolate.It vs Moq.It in Mockolate_MultipleSetups — ensures the setup registers the return value correctly.

claude

Review: Add benchmarks for Imposter and Mockolate mocking frameworks

Overall this is a solid, clean addition. The benchmarks follow the existing patterns well and cover all the same scenarios as the other frameworks. A few observations and suggestions:

Potential benchmark validity concern: `Imposter_VerifyNever`

In VerificationBenchmarks.cs, the Imposter "Never" benchmark does not instantiate the mock (no .Instance() call) before verifying, which may not fully match the other frameworks' patterns that instantiate the mock even if they never call it:

[Benchmark(Description = "Imposter (Never)")]
public void Imposter_VerifyNever()
{
    var imposter = ICalculatorService.Imposter();
    imposter.Format(Arg<int>.Any()).Called(Count.Never());
    // No: var calc = imposter.Instance();
}

For fair comparison, check whether instantiation is required for verification and whether the other frameworks' "Never" benchmarks do more work (e.g., Moq creates the mock but also calls mock.Verify(...) without instantiating .Object). If Imposter requires instantiation to track calls, this could report misleadingly fast results.

Inconsistency: `Mockolate_CreateMock` returns the mock wrapper, not the underlying object

In MockCreationBenchmarks.cs, most benchmarks return the usable object:

// Imposter - returns the instance (the callable object)
public object Imposter_CreateMock()
{
    var imposter = ICalculatorService.Imposter();
    return imposter.Instance();
}

// Mockolate - returns the mock wrapper directly (includes Mock property etc.)
public object Mockolate_CreateMock()
{
    var sut = ICalculatorService.CreateMock();
    return sut;  // This is a wrapper type, not ICalculatorService
}

Since ICalculatorService.CreateMock() returns a type that implements ICalculatorService plus a Mock property, returning it directly may be fine — but it's worth confirming whether the returned object is the interface implementation itself or an additional wrapper, so the benchmark measures the same conceptual operation as the other frameworks.

`new List<User>()` allocations in setup benchmarks

In SetupBenchmarks.Mockolate_MultipleSetups and Imposter_MultipleSetups, new List<User>() is allocated inside the benchmark method (matching the pattern of the other frameworks). This is consistent, but worth noting since BenchmarkDotNet with [MemoryDiagnoser] will attribute those allocations to the mock framework being measured. All frameworks do this equivalently, so comparison is fair — just something to be aware of when reading results.

Minor: `ImposterRegistrations.cs` placement

The new ImposterRegistrations.cs file uses [assembly: GenerateImposter(...)] attributes which is the correct source-generator approach for Imposter. This is clean and follows AOT-compatible patterns. No issues here.

Qualified `Moq.It` references

The PR correctly qualifies the existing It.IsAny<T>() calls to Moq.It.IsAny<T>() to resolve ambiguity with the new Mockolate.It type. These are necessary fixups and look correct.

Summary

The implementation is straightforward and consistent with the existing benchmark structure. The main thing worth double-checking is the Imposter_VerifyNever scenario to ensure it's measuring a comparable operation to the other "Never" verification benchmarks. Everything else looks good.

vbreuss force-pushed the topic/add-benchmarks-against-sourcegenerated-mocking-frameworks branch from 8ca4bf2 to fdd6cca Compare March 29, 2026 13:38

vbreuss temporarily deployed to Pull Requests March 29, 2026 13:41 — with GitHub Actions Inactive

vbreuss had a problem deploying to Pull Requests March 29, 2026 13:41 — with GitHub Actions Failure

vbreuss temporarily deployed to Pull Requests March 29, 2026 13:41 — with GitHub Actions Inactive