Skip to content

Track when vstest provides Newtonsoft.Json for extensions in telemetry#15596

Merged
nohwnd merged 4 commits into
microsoft:mainfrom
nohwnd:track-newtonsoft-resolution
Apr 1, 2026
Merged

Track when vstest provides Newtonsoft.Json for extensions in telemetry#15596
nohwnd merged 4 commits into
microsoft:mainfrom
nohwnd:track-newtonsoft-resolution

Conversation

@nohwnd
Copy link
Copy Markdown
Member

@nohwnd nohwnd commented Mar 27, 2026

Problem

Extensions that depend on Newtonsoft.Json but don't ship their own copy silently use vstest's bundled copy. Before we can remove Newtonsoft.Json from vstest, we need to know how many extensions rely on this.

Solution

Silent telemetry tracking — when \AssemblyResolver\ resolves a tracked assembly (e.g. Newtonsoft.Json) from vstest's own directory for an external requester:

  1. Record it in a static \ConcurrentDictionary\ (one entry per assembly per process)
  2. Report in telemetry as \VS.TestPlatform.ProvidedAssemblyDependency.Newtonsoft.Json=\
  3. Filter out vstest's own internal assemblies (no false positives)
  4. No user-visible warnings by default — opt-in via \VSTEST_WARN_MISSING_EXTENSIONS_DEPENDENCIES=1\ feature flag

Test asset

\NewtonSoftDependencyMissing\ — references Newtonsoft.Json with <ExcludeAssets>runtime. The DLL is NOT in output, so vstest's copy is used at runtime.

Changes

  • \AssemblyResolver.cs\ — track provided dependencies, opt-in warning behind feature flag
  • \TelemetryDataConstants.cs\ — new \ProvidedAssemblyDependency\ constant
  • \FeatureFlag.cs\ — new \VSTEST_WARN_MISSING_EXTENSIONS_DEPENDENCIES\ flag
  • \TestRunRequest.cs\ — report provided dependencies in telemetry at run completion
  • \IAssemblyResolver.cs\ — add \RequestingAssembly\ to \AssemblyResolveEventArgs\

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Copilot AI review requested due to automatic review settings March 27, 2026 09:21
nohwnd
nohwnd commented Mar 27, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.Common/Utilities/AssemblyResolver.cs Outdated
nohwnd
nohwnd commented Mar 27, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.Common/Utilities/AssemblyResolver.cs Outdated
nohwnd
nohwnd commented Mar 27, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.Common/Utilities/AssemblyResolver.cs Outdated
nohwnd
nohwnd commented Mar 27, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.Common/Utilities/AssemblyResolver.cs Outdated
nohwnd
nohwnd commented Mar 27, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.CoreUtilities/FeatureFlag/FeatureFlag.cs Outdated
nohwnd
nohwnd commented Mar 27, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.Client/Execution/TestRunRequest.cs Outdated
@nohwnd
Copy link
Copy Markdown
Member Author

nohwnd commented Mar 27, 2026

/azp run

@azure-pipelines
Copy link
Copy Markdown

azure-pipelines Bot commented Mar 27, 2026

Azure Pipelines successfully started running 1 pipeline(s).

@nohwnd
Copy link
Copy Markdown
Member Author

nohwnd commented Mar 27, 2026

/azp run

@azure-pipelines
Copy link
Copy Markdown

azure-pipelines Bot commented Mar 27, 2026

Azure Pipelines successfully started running 1 pipeline(s).

nohwnd
nohwnd commented Mar 30, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.Client/Execution/TestRunRequest.cs Outdated
nohwnd
nohwnd commented Mar 30, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.CoreUtilities/FeatureFlag/FeatureFlag.cs
nohwnd
nohwnd commented Mar 30, 2026
View reviewed changes
Copy link
Copy Markdown
Member Author

@nohwnd nohwnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^^^

nohwnd
nohwnd commented Mar 30, 2026
View reviewed changes
Comment thread src/Microsoft.TestPlatform.Client/Execution/TestRunRequest.cs Outdated
nohwnd
nohwnd commented Mar 30, 2026
View reviewed changes
Copy link
Copy Markdown
Member Author

@nohwnd nohwnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sdf

@nohwnd nohwnd added 🚢 Ship it! Add to PRs where owner approves automated PR, but cannot approve because they "wrote it". and removed 🚢 Ship it! Add to PRs where owner approves automated PR, but cannot approve because they "wrote it". labels Mar 30, 2026
nohwnd and others added 4 commits March 31, 2026 09:33
@nohwnd
Track when vstest provides Newtonsoft.Json for extensions in telemetry
7d52d72 
When an extension depends on Newtonsoft.Json but does not ship its own
copy, vstest's AssemblyResolver resolves it from the test platform
directory. This silently tracks such resolutions in telemetry so we
can monitor how many extensions rely on vstest's bundled copy before
we remove it.

- Track resolved dependencies for known assemblies (Newtonsoft.Json)
  in AssemblyResolver.OnResolve via static ConcurrentDictionary
- Filter out vstest's own internal assemblies (no false positives)
- Report in telemetry as VS.TestPlatform.ProvidedAssemblyDependency.*
- Add VSTEST_WARN_MISSING_EXTENSIONS_DEPENDENCIES feature flag for
  opt-in user-visible warnings before enforcement
- Add NewtonSoftDependencyMissing test asset (ExcludeAssets=runtime)
- Pass RequestingAssembly through AssemblyResolveEventArgs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nohwnd
Address review: track all deps, separate user/ms telemetry, defer war…
9c3b178 
…ning

- Track ALL resolved assemblies (not just Newtonsoft.Json)
- Check against search directories (not just one dir)
- Separate user vs Microsoft/System deps for telemetry privacy:
  user assemblies get count only (no names), Microsoft/System get names
- Defer warning to run summary via LoggerManager (not inline)
- Rename feature flag to VSTEST_OPTIN_WARN_MISSING_EXTENSIONS_DEPENDENCIES
- Telemetry: 4 properties (User/Microsoft × assemblies/count)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nohwnd
Address review: add disable flag, localize warning message
75cdd91 
- Add VSTEST_DISABLE_WARN_MISSING_EXTENSIONS_DEPENDENCIES flag
- Disable overrides opt-in: if disable is set, no warning even with opt-in
- Localize warning message via Resources.resx (ProvidedDependenciesWarning)
- Update Designer.cs and xlf files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nohwnd
DoNotParallelize tests using runsettingsprovider.instance
35a9399
@nohwnd nohwnd force-pushed the track-newtonsoft-resolution branch from 2d42371 to 35a9399 Compare March 31, 2026 07:36
@nohwnd nohwnd added the 🚢 Ship it! Add to PRs where owner approves automated PR, but cannot approve because they "wrote it". label Mar 31, 2026
@nohwnd nohwnd enabled auto-merge (squash) April 1, 2026 12:08
@nohwnd nohwnd merged commit 1e65b89 into microsoft:main Apr 1, 2026
6 checks passed
This was referenced Apr 1, 2026
Closed
Merged
nohwnd added a commit to nohwnd/vstest that referenced this pull request Apr 10, 2026
@nohwnd
Revert Newtonsoft.Json dependency tracking (microsoft#15596)
1b2ffaa 
Reverts the assembly resolution tracking and warning mechanism for
Newtonsoft.Json that was added in microsoft#15596. Investigation showed that:

- vstest's AssemblyResolver.OnResolve never fires for Newtonsoft.Json
  because it is loaded via normal CLR probing before the event triggers
- On .NET Framework, MSTest has its own AssemblyResolver in a separate
  AppDomain that resolves NJ independently from vstest's resolver
- The ProvidedDependencies telemetry was always empty in practice
- No feasible proactive warning mechanism exists for this scenario

The [DoNotParallelize] test fixes from the same PR are preserved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nohwnd nohwnd mentioned this pull request Apr 10, 2026
Merged
nohwnd added a commit that referenced this pull request Apr 10, 2026
@nohwnd
Revert Newtonsoft.Json dependency tracking (#15596) (#15651)
a74013a 
Reverts the assembly resolution tracking and warning mechanism for
Newtonsoft.Json that was added in #15596. Investigation showed that:

- vstest's AssemblyResolver.OnResolve never fires for Newtonsoft.Json
  because it is loaded via normal CLR probing before the event triggers
- On .NET Framework, MSTest has its own AssemblyResolver in a separate
  AppDomain that resolves NJ independently from vstest's resolver
- The ProvidedDependencies telemetry was always empty in practice
- No feasible proactive warning mechanism exists for this scenario

The [DoNotParallelize] test fixes from the same PR are preserved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dotnet-maestro dotnet-maestro Bot mentioned this pull request Apr 27, 2026
Merged
nohwnd added a commit that referenced this pull request May 14, 2026
@nohwnd
Revert Newtonsoft.Json dependency tracking (#15596)
daeb29f 
Cherry-pick of a74013a from main. The assembly resolution tracking
and warning mechanism was proven useless — OnResolve never fires for
Newtonsoft.Json because it loads via normal CLR probing. This also
removes the RequestingAssembly property from PlatformAbstractions
which causes MissingMethodException when older xunit adapters call
into the resolver (#15765).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Evangelink Evangelink Evangelink approved these changes

Copilot code review Copilot Awaiting requested review from Copilot

Assignees

No one assigned

Labels

🚢 Ship it! Add to PRs where owner approves automated PR, but cannot approve because they "wrote it".

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nohwnd @Evangelink