MSIX Dynamic Dependencies - allow any process to use MSIX Framework packages

[DrusTheAxe]

Proposal: MSIX Dynamic Dependencies (aka DynamicDependencies aka DynDep)

Summary

Provide APIs to enable access to packaged content at runtime, regardless if the caller is packaged or not. This supplements the MSIX appmodel's current static dependency support (via in appxmanifest.xml) with a dynamic runtime equivalent. It also allows non-packaged processes (which have no appxmanifest.xml) to use packaged content.

Microsoft-internal task 23447728

Rationale

• Enable all apps (MSIX-packaged and not) to use framework packages
• Support dynamic selection of framework packages at runtime to complement static selection of framework packages at development/install time (via appxmanifest.xml)
  DynamicDependencies is the moral equivalent of LoadLibrary - DynamicDependencies supplements static (manifest'd) dependencies as LoadLibrary supplements EXE/DLL import references.

This aligns with Project Reunion's roadmap:

• Foundational technology enabling new Project Reunion features to be delivered via Framework packages but accessible by all applications (packaged and not-packaged)
• Enable all applications (packaged and not-packaged) to use WinUI's Framework package

Scope

Capability	Priority
APIs available to MSIX and non-MSIX applications	Must
APIs to add and remove a package dependency dynamically at runtime	Must
API to prevent removal of packages not in use by running processes (i.e. install-time pinning)	Must
APIs to enumerate package dependencies in use	Could
APIs to enumerate defined package dependencies (install-time pinned)	Could
Win32 API	Must
WinRT API	Should

Important Notes

All processes have a package graph. A process may be created with entries in its package graph; this is referred to as the 'static package graph'.

Packaged processes (i.e. a process with package identity) are created with a static package graph per their AppxManifest.xml. A process' static package graph cannot be altered, but it can be supplemented at runtime via the Dynamic Dependency API.

Processes without package identity have an no static package graph. They can modify their package graph using the Dynamic Dependency API.

MddPinPackageDependency defines a package dependency.

MddAddPackageDependency determines a package that satisfies a package dependency and updates the caller's process. This includes adding the resolved package to the process' package graph, updating the Loader to include the resolved package in the DLL Search Order, etc. The package dependency is resolved to a specific package if not already resolved.

A resolved PackageDependency is represented by MDD_PACKAGE_DEPENDENCY_CONTEXT.

Once a PackageDependency is resolved to a package all further MddAddPackageDependency calls yield the same result until the package dependency is unresolved. Resolved package dependencies are tracked by User + PackageDependencyId. This ensures multiple overlapping calls to MddAddPackageDependency yield the same result. A package dependency is unresolved when the last MDD_PACKAGE_DEPENDENCY_CONTEXT is closed (via MddRemovePackageDependency or process termination).

MddRemovePackageDependency removes the resolved PackageDependency from the calling process' package graph.

MddUnpinPackageDependency undefines a package dependency previously defined via PinPackageDependency.

PackageDependency definitions and usage are tracked and managed on a per-user basis.

PackageDependency definitions are not persisted or tracked across reboots if MddPinPackageDependency is called with MddPinPackageDependency_LifecycleHint_Process. Specify MddPinPackageDependency_LifecycleHint_FileOrPath or MddPinPackageDependency_LifecycleHint_RegistrySubkey for PinPackageDependency to persist the definition until explicitly removed via MddUnpinPackageDependency or the specified lifetime artifact is deleted.

If concurrent processes need the same package resolution for a defined criteria they should share the packageDependencyId returned by MddPinPackageDependency. Concurrent processes running as the same user calling MddAddPackageDependency with the same packageDependencyId get the same resolved package added to their package graph. This enables multiple concurrent processes needing the same package resolution get a consistent answer.

Package dependencies can only be resolved to packages registered for a user. As packages cannot be registered for LocalSystem the Dynamic Dependencies feature is not available to callers running as LocalSystem.

Win32 API

MsixDynamicDependency.hpp

enum class MddPinPackageDependency : uint32_t
{
    MddPinPackageDependency_None                               0,
    MddPinPackageDependency_DoNotVerifyDependencyResolution    0x00000001,
    MddPinPackageDependency_LifecycleHint_Process              0,
    MddPinPackageDependency_LifecycleHint_FileOrPath           0x00000002,
    MddPinPackageDependency_LifecycleHint_RegistrySubkey       0x00000004,

     // Define the package dependency for the system, accessible to all users
     // (default is the package dependency is defined for a specific user).
     // This option requires the caller has adminitrative privileges.
    MddPinPackageDependency_ScopeIsSystem                      0x00000008,
};
DEFINE_ENUM_FLAG_OPERATORS(MddPinPackageDependency)

enum class MddAddPackageDependency : uint32_t
{
    MddAddPackageDependency_None                       0,
    MddAddPackageDependency_OnlyUseFirstPackageFamily  0x00000001,
    MddAddPackageDependency_PrependIfRankCollision     0x00000002,
};
DEFINE_ENUM_FLAG_OPERATORS(MddAddPackageDependency)

#define PACKAGE_DEPENDENCY_RANK_DEFAULT 0

enum class MddPackageDependencyProcessorArchitectures : uint32_t
{
    MddPackageDependencyProcessorArchitectures_None = 0,
    MddPackageDependencyProcessorArchitectures_Neutral = 0x00000001,
    MddPackageDependencyProcessorArchitectures_X86     = 0x00000002,
    MddPackageDependencyProcessorArchitectures_X64     = 0x00000004,
    MddPackageDependencyProcessorArchitectures_Arm     = 0x00000008,
    MddPackageDependencyProcessorArchitectures_Arm64   = 0x00000010,
    MddPackageDependencyProcessorArchitectures_X86A64  = 0x00000020,
};
DEFINE_ENUM_FLAG_OPERATORS(MddPackageDependencyProcessorArchitectures)

DECLARE_HANDLE(MDD_PACKAGEDEPENDENCY_CONTEXT);

// Define a package dependency expressing a relationship between an
// application and a packaged component or application. The criteria
// for a PackageDependency (package family name, minimum version, etc)
// may match multiple packages, but ensures Deployment won't remove
// a package if it's the only one satisfying the PackageDependency.
//
// @note A package matching a PackageDependency pin can still be removed
//       as long as there's another that satisfies the PackageDependency.
//       For example, if Fwk-v1 is installed and a PackageDependency specifies
//       MinVersion=1 and then Fwk-v2 is installed, Deployment could remove
//       Fwk-v1 because Fwk-v2 will satisfy the PackageDependency. After Fwk-v1
//       is removed Deployment won't remove Fwk-v2 because it's the only package
//       satisfying the PackageDependency. Thus  Fwk-v1 and Fwk-v2 (and any other
//       package matching the PackageDependency) are 'loosely pinned' – Deployment
//       guarantees it won't remove a package if it would make a PackageDependency
//       unsatisfied.
//
// A PackageDependency specifies criteria (package family, minimum version, etc)
// and not a specific package. Deployment reserves the right to use a different
// package (e.g. higher version) to satisfy the PackageDependency if/when
// one becomes available.
//
// @param user the user scope of the package dependency. If NULL the caller's
//        user context is used. MUST be NULL if MddPinPackageDependency_ScopeIsSystem
//        is specified
// @param lifetimeArtifact MUST be NULL if MddPinPackageDependency_LifecycleHint_Process (default)
// @param packageDependencyId allocated via LocalAlloc; use LocalFree to deallocate
//
// @note MddPinPackageDependency() fails if the PackageDependency cannot be resolved to a specific
//       package. This package resolution check is skipped if
//       MddPinPackageDependency_DoNotVerifyDependencyResolution is specified. This is useful
//       for installers running as user contexts other than the target user (e.g. installers
//       running as LocalSystem).
STDAPI MddPinPackageDependency(
    PSID user,
    _In_ PCWSTR packageFamilyName,
    PACKAGE_VERSION minVersion,
    MddPackageDependencyProcessorArchitectures packageDependencyProcessorArchitectures,
    PCWSTR lifetimeArtifact,
    MddPinPackageDependency flags,
    _Outptr_result_maybenull_ PWSTR* packageDependencyId);

// Undefine a package dependency. Removing a pin on a PackageDependency is typically done at uninstall-time.
// This implicitly occurs if the package dependency's 'lifetime artifact' (specified via MddPinPackageDependency)
// is deleted. Packages that are not referenced by other packages and have no pins are elegible to be removed.
//
// @warn MddUnpinPackageDependency() requires the caller have administrative privileges
//       if the package dependency was pinned with MddPinPackageDependency_ScopeIsSystem.
STDAPI MddUnpinPackageDependency(
    _In_ PCWSTR packageDependencyId);

// Resolve a previously-pinned PackageDependency to a specific package and
// add it to the invoking process' package graph. Once the dependency has
// been added other code-loading methods (LoadLibrary, CoCreateInstance, etc)
// can find the binaries in the resolved package.
//
// Package resoution is specific to a user and can return different values
// for different users on a system.
//
// Each successful MddAddPackageDependency() adds the resolve packaged to the
// calling process' package graph, even if already present. There is no
// duplicate 'detection' or 'filtering' applied by the API (multiple
// references from a package is not harmful). Once resolution is complete
// the package stays resolved for that user until the last reference across
// all processes for that user is removed via MddRemovePackageDependency (or
// process termination).
//
// MddAddPackageDependency() adds the resolved package to the caller's package graph,
// per the rank specified. A process' package graph is a list of packages sorted by
// rank in ascending order (-infinity…0…+infinity). If package(s) are present in the
// package graph with the same rank as the call to MddAddPackageDependency the resolved
// package is (by default) added after others of the same rank. To add a package
// before others o the same rank, specify MddAddPackageDependency_PrependIfRankCollision.
//
// Every MddAddPackageDependency should be balanced by a MddRemovePackageDependency
// to remove the entry from the package graph. If the process terminates all package
// references are removed, but any pins stay behind.
//
// MddAddPackageDependency adds the resulting package to the process' package
// graph, per the rank and options/flags parameters. The process' package
// graph is used to search for DLLs (per Dynamic-Link Library Search Order),
// WinRT objects and other resources; the caller can now load DLLs, activate
// WinRT objects and use other resources from the framework package until
// MddRemovePackageDependency is called. The packageDependencyId parameter
// must match a package dependency defined for the calling user or the
// system (i.e. pinned with MddPinPackageDependency_ScopeIsSystem) else
// an error is returned.
//
// @param packageDependencyContext valid until passed to MddRemovePackageDependency()
// @param packageFullName allocated via LocalAlloc; use LocalFree to deallocate
STDAPI MddAddPackageDependency(
    _In_ PCWSTR packageDependencyId,
    INT32 rank,
    MddMddAddPackageDependency flags,
    _Out_ MDD_PACKAGEDEPENDENCY_CONTEXT* packageDependencyContext,
    _Outptr_opt_result_maybenull_ PWSTR* packageFullName);

// Remove a resolved PackageDependency from the current process' package graph
// (i.e. undo MddAddPackageDependency). Used at runtime (i.e. the moral equivalent
// of FreeLibrary).
//
// @note This does not unload loaded resources (DLLs etc). After removing
//        a package dependency any files loaded from the package can continue
//        to be used; future file resolution will fail to see the removed
//        package dependency.
STDAPI MddRemovePackageDependency(
    _In_ MDD_PACKAGEDEPENDENCY_CONTEXT packageDependencyContext);

// Return the package full name that would be used if the
// PackageDependency were to be resolved. Does not add the
// package to the process graph.
//
// @param packageFullName allocated via LocalAlloc; use LocalFree to deallocate
STDAPI MddGetResolvedPackageFullNameForPackageDependency(
    _In_ PCWSTR packageDependencyId,
    _Outptr_result_maybenull_ PWSTR* packageFullName);

// Return TRUE if packageDependencyId1 and packageDependencyID2
// are associated with the same resolved package.
STDAPI_(BOOL) MddArePackageDependencyIdsEquivalent(
    _In_ PCWSTR packageDependencyId1,
    _In_ PCWSTR packageDependencyId2);

// Return TRUE if PackageDependency pins would produce the same package
// when resolved e.g. whether they share the same packageFamilyName,
// minVersion, and packageDependencyProcessorArchitectures values.
STDAPI_(BOOL) MddArePackageDependencyContextsEquivalent(
    _In_ const PACKAGEDEPENDENCY_CONTEXT* packageDependencyContext1,
    _In_ const PACKAGEDEPENDENCY_CONTEXT* packageDependencyContext2);

NOTE: All APIs prefixed with Mdd/MDD for MSIX Dynamic Dependencies.

WinRI API

namespace Microsoft.ApplicationModel
{
enum PackageDependencyProcessorArchitectures
{
    None    = 0,
    Neutral = 0x00000001,
    X86     = 0x00000002,
    X64     = 0x00000004,
    Arm     = 0x00000008,
    Arm64   = 0x00000010,
    X86A64  = 0x00000020,
};

runtimeclass PinPackageDependencyOptions
{
    PinPackageDependencyOptions();

    PackageDependencyProcessorArchitectures Architectures;
    boolean DoNotVerifyDependencyResolution;
    String lifetimeArtifactFileOrPath;
    String lifetimeArtifactRegistrySubkey;
}

runtimeclass AddPackageDependencyOptions
{
    AddPackageDependencyOptions();

    int Rank;
    boolean OnlyUseFirstPackageFamily;
    boolean PrependIfRankCollision;
}

runtimeclass PackageDependency
{
    PackageDependency(String id);

    String Id { get; }

    static boolean AreEquivalent(String packageDependencyId1, String packageDependencyId2);

    boolean AreEquivalent(PackageDependency otherPackageDependency);

    static PackageDependency Pin(
        String packageFamilyName,
        PackageVersion minVersion,
        PinPackageDependencyOptions options);

    static PackageDependency PinForUser(
        Windows.System.User user,
        String packageFamilyName,
        PackageVersion minVersion,
        PinPackageDependencyOptions options);

    static PackageDependency PinForSystem(
        String packageFamilyName,
        PackageVersion minVersion,
        PinPackageDependencyOptions options);

    void Unpin();

    PackageDependencyContext Add();
    PackageDependencyContext Add(AddPackageDependencyOptions options);

    void Remove();
}

struct PackageDependencyContextId
{
    UInt64 Id;
};

runtimeclass PackageDependencyContext : ICloseable
{
    PackageDependencyContext(PackageDependencyContextId);   //Parameter is MDD_PACKAGEDEPENDENCY_CONTEXT

    PackageDependencyContextId Context { get; } //Return value is MDD_PACKAGEDEPENDENCY_CONTEXT
    String PackageFullName { get; }

    static boolean AreEquivalent(PackageDependencyContextId packageDependencyContextId1, PackageDependencyContextId packageDependencyContextId2);

    boolean AreEquivalent(PackageDependencyContext otherPackageDependencyContext);

    void Remove();
}
}

Open Questions

Q: Package dependencies are only resolved for Framework packages. Should other package types (Main, Resource, Optional) be supported?

Q: A package dependency's critieria includes user, package family, minimum version, and processor architecture. Are there other qualifiers we should be consider?

Q:WinRT: How should 'lifetimeArtifact' and MddPinPackageDependency_LifecycleHint* be expressed in the WinRT API? Some ideas:

• String file; String regkey; if both null then it's Process
• String lifetimeArtifact; boolean isFile; boolean isReg; boolean isProcess; and only 1 can be true
• ILifetimeArtifact property with multiple implementations e.g. FileLifetimeArtifact = { string file; } vs RegistryLifetimeArtifact = { HKEY root; string subkey; } vs Process=null
• ?

[jonwis]

Looks pretty reasonable. A few issues:

• Don't use declspec(dllimport/export) ... use STDAPI and use a .def file & import library for the "flat C" flavor of the API
• Use enum class Foo : uint32_t when declaring enumerations (also lets you drop the Foo_Bar prefix on everything.)
• The ...NounAreEquivalent should be ...PluralnounAreEquivalent
• PinPackageDependencyOptions needs a default ctor so it can be constructed
• Should the ...Options types be struct typed since we're being a little looser with versioning here?
• For the context type, use a "strongly typed cookie" - the pattern is struct ThingContextId { UInt64 Id; } and then there's methods like static ThingContext GetFromContextId(ThingContextId), etc. ... it's not clear the public surface needs to convert to-from a HANDLE here anyhow, can that be removed?
• Feel free to remove the static AreEquivalent method and keep a bool Equals member method instead.
• The PackageDependencyContext should probably implement IClosable - see internal Patterns document for more on this (coming to the public soon!)

[DrusTheAxe]

Thanks Jon!

Approved. Updated the proposal

• declspec/STDAPI
• Pluralnoun NOTE:It's ArePluralnounEquivalent
• PinPackageDependencyOptions ctor
• ICloseable
• context strongly-typed-cookie

Rejected:

• Equals
  ** The comparisons are for functional equivalency. PackageDependency objects are equivalent if the subset of underlying data match (e.g. same pkgfamilyname but different lifetime artifact). Equals implies same, but that would involve more data than we're checking here. The purpose is "Do these 2 definitions define the same criteria for package resolution?" Hence, 'Equivalent' not 'Equals'. 'PackageResolutionCriteriaAreEqual' is more accurate but a mouthful I'm open to suggestions
  ** Ids are strings thus trivial for callers. We could add Equals methods as { return id1.CompareNoCase(id2) } if folks feel that's useful. This is independent of equivalency methods

Open issues:

• ...Options = struct => Good question. Do our API guidelines cover this yet? Would struct restrict us to foundation data types? We need boolean, integers, strings and enums. I could see a use down the road for lists; those aren't compatible with structs, right? If we make a runtimeclass FooOptions2 { struct Foo Foo; ...new v2 properties... } where v1=struct v2=runtimeclass that inconsistency isn't a big deal (if/when that day comes)?

[marb2000]

Q: Will this work downlevel?

[DrusTheAxe]

Q: Will this work downlevel?

Yes! The target is down to whatever releases Project Reunion supports.

A design proposal is coming shortly. That'll explain 'how' we can make the magic happen. Please stay tuned

[stevenbrix]

How exactly does this work from an end user perspective? As an app developer, do I have to call these APIs in my main method? Or will there be something in the SxS manifest that declares the package dependency?

[wjk]

Question about this design. I presume the AppX packages that would be referenced using this API would be framework packages. That’s perfectly OK, until we start implementing APIs with broker processes that might need to be run in full trust. Per that linked doc, framework packages cannot declare either applications (making it awfully hard to run the broker as a separate app for boundary purposes) or capabilities (meaning we are limited to AppContainer-only APIs, since we can’t specify runFullTrust). As I understand it, only framework packages can be listed as dependencies in AppX manifests, and modification/optional packages must specify the exact PFN that they expect to be loaded into, so those won‘t work either. Could someone please clarify this? Thanks!

[DrusTheAxe]

@stevenbrix Yes, these are APIs apps would call at (un)install and runtime. The basic flow is

• Call Pin @ install-time
• Call Unpin @ uninstall-time
• Call Add in your app before using the framework's content. main() is a common answer but could be later if you choose
• Call Remove in your app when done with a framework. Or don't call at all and it's effectively done at process termination

For a rough comparison to an MSIX (packaged) application...

• Pin is akin to declaring a in appxmanifest.xml
• Add is akin to...
• ...the Deployment stack resolving that to a specific package
• ...updating the process' package graph so the specific framework's content can be found by LoadLibrary, ActivateInstance, etc
• Remove has no direct analogue in an MSIX application
• Unpin is akin to uninstalling a package for a user e.g. packageManager->RemovePackageAsync()

A packaged app has a package graph from process creation until process termination. In fact, all process have a package graph, it's just empty/null for non-packaged processes. Either way, a process' package graph is set at process creation (be it a list of packages or empty/null) and is is constant until process termination. Dynamic Dependencies supplements that static package graph with the ability to dynamically alter a process' package graph at runtime.

Unfortunately SxS is no help as Windows lacks the hooks to setup the necessary information at process creation. But let's assume for a moment we could change Windows Would that be helpful? When would you find it preferable to use the SxS manifest vs the API? Would it still be interesting if it was limited compared to the API e.g. if MddRemovePackageDependency() couldn't remove such package dependencies? Would this SxS option be attractive if it required a future version of Windows?

[DrusTheAxe]

@wjk Yes, your understanding is correct.

The Dynamic Dependency API targets Framework packages. That's both a pressing need and a natural fit for how framework packages work -- used by other packages, don't provide their own process/runtime identity etc.

Can you elaborate on what you'd like to accomplish (that frameworks don't suffice)?