Fix LibraryImportDiagnosticsAnalyzer and DownlevelLibraryImportDiagnosticsAnalyzer missing diagnostics when StringMarshalling is set

[Copilot]

Description

LibraryImportDiagnosticsAnalyzer and DownlevelLibraryImportDiagnosticsAnalyzer fail to report diagnostics (e.g. SYSLIB1051) for [LibraryImport] methods when StringMarshalling is set. This is a regression introduced in preview.3 when diagnostic reporting was moved from the generators to separate analyzer classes. Additionally, the generated inner [DllImport] stub omits CharSet = CharSet.Unicode when StringMarshalling.Utf16 is set, causing incorrect runtime marshalling of forwarded types (e.g. StringBuilder).

Root cause: The source generator marks its generated stub output with [GeneratedCode]. Roslyn's generated-code heuristics then also classify the user's partial method declaration as generated code. With ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None), both analyzers silently skipped all [LibraryImport] methods. Additionally, with Analyze | ReportDiagnostics enabled, RegisterSymbolAction fires for both the partial definition and partial implementation as separate IMethodSymbol instances, which could produce duplicate diagnostics — one in user source and one in generated source. The generated implementation syntax (which has a body) also caused a spurious SYSLIB1050 (InvalidAttributedMethodSignature) because GetDiagnosticIfInvalidMethodForGeneration rejects methods with a body.

Reproduction:

// SYSLIB1051 should be reported but is NOT (with StringMarshalling.Utf16)
[LibraryImport("kernel32.dll", StringMarshalling = StringMarshalling.Utf16)]
internal static partial int GetVolumeNameForVolumeMountPointW(
    string volumeMountPoint,
    [Out] StringBuilder volumeName,  // No error — silent bad codegen
    int bufferLength);

Changes

• LibraryImportDiagnosticsAnalyzer: Changed GeneratedCodeAnalysisFlags.None → Analyze | ReportDiagnostics so the analyzer runs on and reports diagnostics for methods whose partial declaration is classified as generated code. Added PartialDefinitionPart guard to skip the partial implementation part (avoiding duplicate diagnostics from RegisterSymbolAction firing for both parts). Added IsGeneratedByOurGenerator helper. When iterating DeclaringSyntaxReferences, the generated implementation syntax (body present + our [GeneratedCode]) is skipped to find the user's partial declaration. GetDiagnosticIfInvalidMethodForGeneration is skipped (via skipInvalidMethodCheck flag) when our generator has already produced an implementation — since the method must have been valid for the generator to run — but CalculateDiagnostics always runs to catch other issues. Diagnostics are filtered by SyntaxTree to only report those located in the user's (non-generated) source.
• DownlevelLibraryImportDiagnosticsAnalyzer: Applied the same GeneratedCodeAnalysisFlags.Analyze | ReportDiagnostics fix, the same PartialDefinitionPart guard, the same selective SYSLIB1050 guard, and the same SyntaxTree diagnostic location filter.
• LibraryImportGenerator.CreateTargetDllImportAsLocalStatement: Now forwards CharSet = CharSet.Unicode to the inner [DllImport] when StringMarshalling.Utf16 is set, using the shared CreateEnumExpressionSyntax helper (extracted to class-level) to ensure consistency with the forwarder stub.
• DownlevelLibraryImportGenerator.CreateTargetDllImportAsLocalStatement: Applied the same CharSet = CharSet.Unicode forwarding fix using the shared CreateEnumExpressionSyntax helper (also extracted to class-level in the downlevel generator).
• DownlevelLibraryImportGenerator.CreateForwarderDllImport: Updated to use the shared class-level CreateEnumExpressionSyntax helper.
• Tests (Diagnostics.cs): Added StringBuilderNotSupported_ReportsDiagnostic and StringBuilderNotSupported_WithStringParam_ReportsDiagnostic regression tests covering StringBuilder with and without StringMarshalling variants.
• Tests (Compiles.cs): Added ForwardedTypesWithStringMarshalling_InnerDllImportHasCharSet test verifying that the inner [DllImport] has CharSet.Unicode when StringMarshalling.Utf16 is set and no CharSet argument when other values are used. Fixed test assertion to use EndsWith to handle the global:: qualified name emitted by the generator. Fixed the DllImport attribute predicate to use EndsWith("DllImportAttribute") instead of Contains("DllImport") to avoid false matches on DefaultDllImportSearchPathsAttribute.

Testing

• Both generator projects (LibraryImportGenerator, DownlevelLibraryImportGenerator) build successfully with zero errors and warnings.
• All 709 unit tests pass (1 skipped, 0 failed), including previously-failing AddDisableRuntimeMarshallingAttributeFixerTests which validated the duplicate diagnostic fix.
• Regression tests added for the diagnostic and code-generation fixes.
• Code review and CodeQL scan passed with no issues.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.

[danmoseley]

It seems Copilot could not make a fix before timing out.

[jkoritzinsky]

@danmoseley I had copilot crank locally for quite a while and it couldn't figure it out but it did add some regression tests that should cover your scenario and pass without any product changes. Can you get me a complog of your build that didn't report the diagnostic?

Is it possible that you have RunAnalyzers set to false or that VS set it to false for one run?

[Copilot]

Pull request overview

This PR targets a regression in LibraryImportDiagnosticsAnalyzer where SYSLIB1051 is not reported for StringBuilder parameters when LibraryImportAttribute.StringMarshalling is specified, leading to silently incorrect forwarder stub generation.

Changes:

• Added analyzer unit tests asserting SYSLIB1051 (ParameterTypeNotSupported) is reported for StringBuilder parameters with StringMarshalling set (Utf16/Utf8) and without it.
• Added a reproduction-style test case for string + [Out] StringBuilder parameter combinations.

[danmoseley]

@danmoseley I had copilot crank locally for quite a while and it couldn't figure it out but it did add some regression tests that should cover your scenario and pass without any product changes. Can you get me a complog of your build that didn't report the diagnostic?

Is it possible that you have RunAnalyzers set to false or that VS set it to false for one run?

@jkoritzinsky I updated the repro (top post #126687) to include a global.json -- can you still not repro with that?

[danmoseley]

@jkoritzinsky also added the install command for preview 3. Using preview 2, it doesn't repro. I verified this by hand again, and it does repro for me with global.json pointing to 11.0.100-preview.3.26170.106

C:\temp\repro>type global.json | find /i "ver"
    "version": "11.0.100-preview.3.26170.106",

C:\temp\repro>type program.cs
using System;
using System.Runtime.InteropServices;
using System.Text;

Console.WriteLine("If you see this, SYSLIB1051 was NOT reported.");

static partial class NativeMethods
{
    [LibraryImport("kernel32.dll", StringMarshalling = StringMarshalling.Utf16)]
    internal static partial int GetVolumeNameForVolumeMountPointW(
        string volumeMountPoint,
        [Out] StringBuilder volumeName,
        int bufferLength);
}
C:\temp\repro>dotnet build
Restore complete (0.3s)
    info NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  doit net11.0 succeeded (0.1s) → bin\Debug\net11.0\doit.dll

Build succeeded in 0.9s

and

C:\temp\repro>set emitcompilergeneratedfiles=true

C:\temp\repro>dotnet build --no-incremental
Restore complete (0.3s)
    info NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  doit net11.0 succeeded (1.5s) → bin\Debug\net11.0\doit.dll

Build succeeded in 2.3s

C:\temp\repro>type obj\Debug\net11.0\generated\Microsoft.Interop.LibraryImportGenerator\Microsoft.Interop.LibraryImportGenerator\LibraryImports.g.cs
// <auto-generated/>
static unsafe partial class NativeMethods
{
    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.LibraryImportGenerator", "11.0.14.17106")]
    [global::System.Runtime.CompilerServices.SkipLocalsInitAttribute]
    internal static partial int GetVolumeNameForVolumeMountPointW(string volumeMountPoint, global::System.Text.StringBuilder volumeName, int bufferLength)
    {
        int __retVal;
        // Pin - Pin data in preparation for calling the P/Invoke.
        fixed (void* __volumeMountPoint_native = &global::System.Runtime.InteropServices.Marshalling.Utf16StringMarshaller.GetPinnableReference(volumeMountPoint))
        {
            __retVal = __PInvoke((ushort*)__volumeMountPoint_native, volumeName, bufferLength);
        }

        return __retVal;
        // Local P/Invoke
        [global::System.Runtime.InteropServices.DllImportAttribute("kernel32.dll", EntryPoint = "GetVolumeNameForVolumeMountPointW", ExactSpelling = true)]
        static extern unsafe int __PInvoke(ushort* __volumeMountPoint_native, [System.Runtime.InteropServices.OutAttribute] global::System.Text.StringBuilder volumeName, int __bufferLength_native);
    }
}

[danmoseley]

The idea is that you get an IDE error, then you either fix the error or copy the generated code into your source and remove the LibraryImportAttribute. Then the error goes away as you aren't using the generator any more.

If I understand right then there are two bugs -- the error no longer firing, and the generated code not working (as I discovered in my original PR)

the issue with the generated code pasted above (I think) is that it doesn't have CharSet = CharSet.Unicode so I guess it defaulted to CharSet.None for volumeName which treats it as Ansi on Windows. (Aside, maybe generated code should always be explicit about CharSet anyway).

[jkoritzinsky]

@copilot please update the diagnostics analyzer to skip the SYSLIB1050 diagnostic when a LibraryImport method has an implementation marked with our GeneratedCodeAttribute pointing to our generator.

[Copilot]

@copilot please update the diagnostics analyzer to skip the SYSLIB1050 diagnostic when a LibraryImport method has an implementation marked with our GeneratedCodeAttribute pointing to our generator.

Done in commit b2bacee. Both LibraryImportDiagnosticsAnalyzer and DownlevelLibraryImportDiagnosticsAnalyzer now have an IsGeneratedByOurGenerator helper that checks for [GeneratedCode] with a tool name starting with "Microsoft.Interop.LibraryImportGenerator". When iterating DeclaringSyntaxReferences, if the current syntax has a body AND the method has our generator's [GeneratedCode] attribute, we skip that syntax reference (the generated implementation) and continue to find the user's partial declaration to analyze.

[jkoritzinsky]

@copilot I want you to only skip the GetDiagnosticIfInvalidMethodForGeneration call if there's generated source by our generators. The rest of the diagnostics analysis must run always.

[Copilot]

@copilot I want you to only skip the GetDiagnosticIfInvalidMethodForGeneration call if there's generated source by our generators. The rest of the diagnostics analysis must run always.

Done in commit d4687fc. Both analyzers now pass isGeneratedByOurGenerator as a skipInvalidMethodCheck parameter to AnalyzeMethodSyntax. When true, GetDiagnosticIfInvalidMethodForGeneration is skipped (since we know the method was valid — the generator already ran on it), but CalculateDiagnostics always runs regardless.

[jkoritzinsky]

@copilot the analyzer is now returning two diagnostics, one in the generated code and one in the non-generated code. Please update the analyzer to only report diagnostics with the location in the non-generated source.

[Copilot]

@copilot the analyzer is now returning two diagnostics, one in the generated code and one in the non-generated code. Please update the analyzer to only report diagnostics with the loca...

Fixed in 351839b. The root cause was that with Analyze | ReportDiagnostics, Roslyn's RegisterSymbolAction fires for both the partial definition and partial implementation as separate IMethodSymbol instances. Added a method.PartialDefinitionPart is not null guard at the top of AnalyzeMethod in both analyzers to skip the implementation part. Also added a SyntaxTree filter when reporting diagnostics to ensure only user-source locations are reported.