ILLink: TypeMapAttribute entry incorrectly trimmed when target type's MarkInstantiated is poisoned by TypeMapAssociationAttribute

[rolfbjarne]

Description

When a type X is both:

1. The proxy target (arg[1]) of a TypeMapAssociationAttribute<Group>
2. The trimTarget (arg[2]) of a TypeMapAttribute<Group>

...the TypeMapAttribute entry is incorrectly trimmed.

Root Cause

TypeMapHandler.MarkTypeMapAttribute calls Annotations.MarkInstantiated(targetTypeDef) directly when marking a TypeMapAssociationAttribute. When the proxy association for type X is marked first (because its source type is instantiated before X), MarkInstantiated(X) is called.

Later, when newobj X is encountered, MarkStep.MarkRequirementsForInstantiatedTypes(X) has an early return:

protected virtual void MarkRequirementsForInstantiatedTypes(TypeDefinition type)
{
    if (Annotations.IsInstantiated(type))
        return;  // Early return — ProcessType never called!
    ...
    _typeMapHandler.ProcessType(type);
}

Since IsInstantiated(X) is already true (set by the proxy association), ProcessType(X) is never called. This means X's own TypeMapAttribute entries remain in _unmarkedExternalTypeMapEntries and are silently trimmed.

Conditions

The bug requires:

• A type used as both proxy target and trim target in the same TypeMap universe
• Both external and proxy maps for that universe are used
• The proxy association's source type is instantiated before the trim target type (order-dependent)

In practice this affects types with generic variants mapping to the same ObjC class (e.g., NSOrderedSet/NSOrderedSet<T>, NSArray/NSArray<T>).

Proposed Fix

In MarkTypeMapAttribute, after calling MarkInstantiated, also call _typeMapHandler.ProcessType(targetTypeDef) so that external TypeMap entries are processed regardless of instantiation order. Alternatively, route through MarkRequirementsForInstantiatedTypes but remove or adjust the early return guard so ProcessType is always called.

Note: PR #127452 fixes a related but different bug (associations dropped when target is only instantiated via TypeMap processing). That fix alone does not resolve this issue because MarkRequirementsForInstantiatedTypes still has the IsInstantiated early return that prevents ProcessType from running.

Reproduction

A failing test has been written (see below). The test adds a type ProxyTargetIsAlsoTrimTarget that is both the proxy target of a TypeMapAssociation and the trimTarget of a TypeMap entry. The source type is instantiated before the trim target to trigger the ordering-dependent bug.

// Assembly-level attributes:
[assembly: TypeMap<UsedTypeMap>("ProxyTargetIsAlsoTrimTarget",
    typeof(ProxyTargetIsAlsoTrimTargetTarget), typeof(ProxyTargetIsAlsoTrimTarget))]
[assembly: TypeMapAssociation<UsedTypeMap>(
    typeof(ProxyTargetIsAlsoTrimTargetSource), typeof(ProxyTargetIsAlsoTrimTarget))]

// In Main — source instantiated BEFORE trim target:
_ = new ProxyTargetIsAlsoTrimTargetSource();
Console.WriteLine(new ProxyTargetIsAlsoTrimTarget());

Expected: Both the TypeMapAttribute and TypeMapAssociationAttribute are kept.
Actual: The TypeMapAttribute is trimmed and ProxyTargetIsAlsoTrimTargetTarget is removed.

Also available as a ready-to-be-cherry-picked commit: rolfbjarne@9827813

/cc @AaronRobinsonMSFT @jtschuster

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/illink
See info in area-owners.md if you want to be subscribed.

[rolfbjarne]

CC @simonrozsival for this one too, related to the previous one, but not quite the same.