Add the ability to generate manifest info for an endpoint in diagnostics

[PhilBastian]

alternative to #7406

This is the first step for Ability to export resources manifest required for endpoints deployment

To start with, the ASB transport and the SqlPersistence have had manifest generation implementations added.

• ASB spike
• SQL persistence spike
• ServiceControl Metrics spike
• Heartbeats spike

sample output with:

• Receive only endpoint
• ASB transport
• 2 events mapped from two topics. "SubscribedEventToTopicsMap": {"Shared.EventOne": [ "event-one", "event-bla" ],"Shared.EventTwo": [ "event-bla" ]
• audit, heartbeat and SC metrics enabled
• uniquely addressable
• SQL persistence
• outbox enabled

  "Manifest-ASBSettings": {
    "EntityMaximumSize": "5GB",
    "PrefetchCount": "default",
    "PrefetchMultiplier": "10",
    "EnablePartitioning": "false"
  },
  "Manifest-AuditQueue": "audit",
  "Manifest-ErrorQueue": "error",
  "Manifest-InputQueues": [
    "samples.azureservicebus.options.subscriber",
    "samples.azureservicebus.options.subscriber-bla"
  ],
  "Manifest-MessageTypes": [
    {
      "Name": "EventOne",
      "FullName": "Shared.EventOne",
      "IsMessage": true,
      "IsEvent": true,
      "IsCommand": false,
      "Schema": [
        {
          "Name": "Content",
          "Type": "String"
        },
        {
          "Name": "PublishedOnUtc",
          "Type": "DateTime"
        }
      ]
    },
    {
      "Name": "EventTwo",
      "FullName": "Shared.EventTwo",
      "IsMessage": true,
      "IsEvent": true,
      "IsCommand": false,
      "Schema": [
        {
          "Name": "Content",
          "Type": "String"
        },
        {
          "Name": "PublishedOnUtc",
          "Type": "DateTime"
        }
      ]
    }
  ],
  "Manifest-Persistence": {
    "Dialect": "MsSqlServer",
    "Prefix": "Samples.AzureServiceBus.Options.Subscriber",
    "Outbox": {
      "TableName": "[dbo].[Samples.AzureServiceBus.Options.Subscriber_OutboxData]",
      "PrimaryKey": "MessageId",
      "Indexes": [
        {
          "Name": "Index_DispatchedAt",
          "Columns": "DispatchedAt"
        },
        {
          "Name": "Index_Dispatched",
          "Columns": "Dispatched"
        }
      ],
      "TableColumns": [
        {
          "Type": "string",
          "Length": "200",
          "Name": "MessageId",
          "Mandatory": true
        },
        {
          "Type": "boolean",
          "Default": false,
          "Name": "Dispatched",
          "Mandatory": true
        },
        {
          "Name": "DispatchedAt",
          "Type": "datetime",
          "Mandatory": false
        },
        {
          "Type": "string",
          "Length": "23",
          "Name": "PersistenceVersion",
          "Mandatory": true
        },
        {
          "Type": "string",
          "Length": "max",
          "Name": "Operations",
          "Mandatory": true
        }
      ]
    },
    "Sagas": [],
    "Subscription": null,
    "StorageTypes": [
      "Outbox",
      "Sagas",
      "Subscriptions"
    ]
  },
  "Manifest-SendingQueues": [
    "audit",
    "error"
  ],
  "Manifest-Subscriptions": [
    {
      "TopicName": "event-one",
      "MessageTypes": [
        "Shared.EventOne"
      ]
    },
    {
      "TopicName": "event-bla",
      "MessageTypes": [
        "Shared.EventOne",
        "Shared.EventTwo"
      ]
    }
  ],
  "Manifest-UniqueAddressDiscriminator": "bla",

[andreasohlund]

Does this add value? We are iterating over the known message types, so this would always be true?

[PhilBastian]

it was pre-existing in AutoSubscribe, I just moved and refactored the code

[danielmarbach]

Would it be possible to introduce the concept of a "category" for a section? Essentially, currently we are using "category-" as a convention to say these are all manifest related sections. Today the feature diagnostic code does a distinct on the section names and warns for duplicated entries. If we add an optional "category" (or whatever else makes sense) we could augment the writing code to move stuff of the same category into a subsection. That would allow us to also leverage this, for example, in the persistence or transport to have a category "Persistence" or "Transport" and then have all the related sections grouped under there. This means it is no longer strictly necessary to have unique names like "NServiceBus.Persistence.Sql.Outbox" but we could standardize on having a persistence category with sections like Outbox, Sagas, Subscriptions (per storage type) or in mixed scenarios keep "Persistence" as a category and have "SqlOutbox", "SqlSagas" and "RavenSubscriptions" (just making up examples for now for the discussion).

[PhilBastian]

I was thinking along the same lines, but since the diagnostics order the output alphabetically the "Manifest-" achieved the same thing without having to force a new standard on every diagnostic call

[danielmarbach]

This might not necessarily be a problem we have to solve now, but it is worth discussing. More reflection done at startup will increase the startup time. The other PR made the manifest writing opt-in. Diagnostics are currently something that is always written (which has pros and cons). That being said, even the original spike also did the "compute intensive" part as part of the startup, independent of whether the manifest was enabled. So the concern I raise here is the same.

[PhilBastian]

fair call, but that would require re-adding in an EnableManifest api, either experimental or otherwise. This isn't the only place that reflection is used as part of startup, but I conceded the point that it will be adding to the already significant burden.

As an aside, wouldn't the assembly scan be the slowest part?

[PhilBastian]

as discussed, pass Func<object> rather than object for expensive operations, and enable these to be filtered out by the diagnostics writer should startup performance be an issue

[PhilBastian]

design document for the manifest implementation