IShellDispatch2.ShellExecute causes NotSupportedException: BSTR cannot be marshalled to a Variant

[jnm2]

System.NotSupportedException: 'Type 'Windows.Win32.Foundation.BSTR' cannot be marshalled to a Variant. Type library is not registered.'

https://learn.microsoft.com/en-us/windows/win32/shell/ishelldispatch2-shellexecute

Repro steps

1. NativeMethods.txt content:

ShellWindows
CSIDL_DESKTOP
IServiceProvider
ShellWindowTypeConstants
ShellWindowFindWindowOptions
IShellBrowser
SID_STopLevelBrowser
_SVGIO
IDispatch
IShellFolderViewDual
IShellDispatch2
SHOW_WINDOW_CMD

2. NativeMethods.json content (if present):

3. Any of your own code that should be shared?

using System;
using Windows.Win32.UI.Shell;
using Windows.Win32;
using System.Runtime.InteropServices;
using Windows.Win32.Foundation;
using Windows.Win32.UI.WindowsAndMessaging;

internal static class ProcessUtils
{
    public static void StartProcessWithoutElevation(string fileName, string workingDirectory)
    {
        var shellWindows = (IShellWindows)new ShellWindows();

        var serviceProvider = (IServiceProvider)shellWindows.FindWindowSW(
            PInvoke.CSIDL_DESKTOP,
            pvarLocRoot: null,
            ShellWindowTypeConstants.SWC_DESKTOP,
            phwnd: out _,
            ShellWindowFindWindowOptions.SWFO_NEEDDISPATCH);

        var shellBrowser = (IShellBrowser)serviceProvider.QueryService(PInvoke.SID_STopLevelBrowser, typeof(IShellBrowser).GUID);

        shellBrowser.QueryActiveShellView(out var shellView);

        shellView.GetItemObject((uint)_SVGIO.SVGIO_BACKGROUND, typeof(IDispatch).GUID, out var folderViewAsObject);

        var folderView = (IShellFolderViewDual)folderViewAsObject;
        var shellDispatch = (IShellDispatch2)folderView.Application;

        var fileNameAsBstr = (BSTR)Marshal.StringToBSTR(fileName);
        try
        {
            shellDispatch.ShellExecute(File: fileNameAsBstr, vArgs: null, vDir: workingDirectory, vOperation: "", SHOW_WINDOW_CMD.SW_NORMAL);
        }
        finally
        {
            Marshal.FreeBSTR(fileNameAsBstr);
        }
    }

    // Workaround for https://github.com/microsoft/CsWin32/issues/860
    [Guid("85CB6900-4D95-11CF-960C-0080C7F4EE85"), InterfaceType(ComInterfaceType.InterfaceIsIDispatch), ComImport]
    private interface IShellWindows
    {
        [return: MarshalAs(UnmanagedType.IDispatch)]
        object FindWindowSW([MarshalAs(UnmanagedType.Struct)] in object pvarLoc, [MarshalAs(UnmanagedType.Struct)] in object? pvarLocRoot, ShellWindowTypeConstants swClass, out int phwnd, ShellWindowFindWindowOptions swfwOptions);
    }

    [Guid("6D5140C1-7436-11CE-8034-00AA006009FA"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown), ComImport]
    private interface IServiceProvider
    {
        [return: MarshalAs(UnmanagedType.Interface)]
        object QueryService(in Guid guidService, in Guid riid);
    }

    // Workaround for https://github.com/microsoft/CsWin32/issues/861
    [ComImport, Guid("00020400-0000-0000-C000-000000000046"), InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
    private interface IDispatch
    {
    }
}

Context

• CsWin32 version: 0.2.188-beta
• Win32Metadata version (if explicitly set by project):
• Target Framework: net6.0-windows8
• LangVersion (if explicitly set by project): latest

[AArnott]

@JeremyKuhne I know you've done a lot of work with VARIANT. The documented parameter type for ShellExecute is BSTR, which ultimately is just a pointer. And the value passed in certainly should be a BSTR. So why is the runtime now trying to convert it to VARIANT (and failing)?

[AArnott]

When I tried to repro this, I got a different error:

An unhandled exception of type 'System.ArgumentException' occurred in mscorlib.dll
Value does not fall within the expected range.

Program.cs

using System;
using Windows.Win32.UI.Shell;
using Windows.Win32;
using System.Runtime.InteropServices;
using Windows.Win32.Foundation;
using Windows.Win32.UI.WindowsAndMessaging;
using IServiceProvider = Windows.Win32.System.Com.IServiceProvider;

ProcessUtils.StartProcessWithoutElevation("notepad.exe", @"c:\temp");

internal static class ProcessUtils
{
    public static void StartProcessWithoutElevation(string fileName, string workingDirectory)
    {
        var shellWindows = (IShellWindows)new ShellWindows();

        var serviceProvider = (IServiceProvider)shellWindows.FindWindowSW(
            PInvoke.CSIDL_DESKTOP,
            pvarLocRoot: null,
            ShellWindowTypeConstants.SWC_DESKTOP,
            phwnd: out _,
            ShellWindowFindWindowOptions.SWFO_NEEDDISPATCH);
        
        serviceProvider.QueryService(PInvoke.SID_STopLevelBrowser, typeof(IShellBrowser).GUID, out object shellBrowserObject);
        IShellBrowser shellBrowser = (IShellBrowser)shellBrowserObject;

        shellBrowser.QueryActiveShellView(out var shellView);

        shellView.GetItemObject((uint)_SVGIO.SVGIO_BACKGROUND, typeof(IDispatch).GUID, out var folderViewAsObject);

        var folderView = (IShellFolderViewDual)folderViewAsObject;
        var shellDispatch = (IShellDispatch2)folderView.Application;

        var fileNameAsBstr = (BSTR)Marshal.StringToBSTR(fileName);
        try
        {
            shellDispatch.ShellExecute(File: fileNameAsBstr, vArgs: null, vDir: workingDirectory, vOperation: "", SHOW_WINDOW_CMD.SW_NORMAL);
        }
        finally
        {
            Marshal.FreeBSTR(fileNameAsBstr);
        }
    }

    [ComImport, Guid("00020400-0000-0000-C000-000000000046"), InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
    private interface IDispatch
    {
    }
}

NativeMethods.txt:

ShellWindows
CSIDL_DESKTOP
IServiceProvider
ShellWindowTypeConstants
ShellWindowFindWindowOptions
IShellBrowser
SID_STopLevelBrowser
_SVGIO
IDispatch
IShellFolderViewDual
IShellDispatch2
SHOW_WINDOW_CMD
IServiceProvider
IShellWindows

[jnm2]

I can still repro with exactly the same original message on 0.2.188-beta, but everything is working properly on 0.3.49-beta. It was fixed by #1050. I confirmed that on 0.2.188-beta by using a copy of the generated interface changed to InterfaceIsDual.