[Bug] ADO required custom fields not fully handled in backlog add/map-fields

[djm81]

Describe the Bug

When using specfact backlog add with the Azure DevOps adapter, work item creation fails for projects that enforce required custom fields and picklist constraints.

The CLI currently needs stronger end-to-end handling for:

• Required custom fields discovered per work item type.
• Picklist value validation before submit.
• Mapping flow from map-fields into backlog add payload and ADO create request.

To Reproduce

Steps to reproduce the behavior:

# 1) Configure mappings for ADO fields (including custom required fields)
specfact backlog map-fields --ado-org [ORG] --ado-project [PROJECT] --provider ado --ado-framework [FRAMEWORK]

# 2) Try to create a story in a sprint/iteration with required custom fields
specfact backlog add \
  --adapter ado \
  --project-id [ORG]/[PROJECT] \
  --type story \
  --title "[TITLE]" \
  --body "[DESCRIPTION]" \
  --acceptance-criteria "[AC]" \
  --sprint "[ITERATION_PATH]" \
  --custom-field [CUSTOM_FIELD_1]=[VALUE_1] \
  --custom-field [CUSTOM_FIELD_2]=[VALUE_2] \
  --non-interactive

Example (anonymized skeleton):

specfact backlog add \
  --adapter ado \
  --project-id ORG/PROJECT \
  --type story \
  --title "Migration preparation story" \
  --body "Scaffolded body text" \
  --acceptance-criteria "Scaffolded acceptance criteria" \
  --sprint "PROJECT\\YYYY\\Sprint XX" \
  --custom-field value_area=Business \
  --custom-field definition_of_ready_is_met=false \
  --custom-field finops_category=[VALID_CATEGORY] \
  --custom-field finops_subcategory=[VALID_SUBCATEGORY] \
  --non-interactive

Expected Behavior

• map-fields should detect required fields for the selected work item type and block saving incomplete mappings.
• backlog add should support passing mapped custom fields.
• CLI should validate picklist values client-side and return clear suggestions for valid options.
• Item should be created successfully when required fields and values are valid.

Actual Behavior

Creation fails with ADO validation errors such as:

• Missing required custom field(s).
• Invalid picklist value for custom field.
• Invalid type for custom field value.

Environment

• OS: [e.g., Windows / Linux / macOS]
• Python Version: [e.g., 3.11.x / 3.12.x / 3.13.x]
• SpecFact CLI Version: [run specfact --version]
• Installation Method: [e.g., pip, uvx, from source]

Command Output

Include the full command output (with --debug if applicable):

Paste sanitized command output here

Codebase Context (for brownfield issues)

If this bug occurs in an existing enterprise project setup:

• Project Type: [e.g., Python automation repo]
• Codebase Size: [e.g., ~N files]
• Python Version in Target Codebase: [e.g., 3.10 / 3.11 / 3.12]

Additional Context

Add any additional context:

• Custom process template includes required custom fields.
• Custom fields include picklists with restricted values.
• Mappings are stored in .specfact/templates/backlog/field_mappings/ado_custom.yaml.
• Provider settings are stored in .specfact/backlog-config.yaml.

Applied Code Diff (sanitized)

The following diff excerpts summarize the concrete fix that was implemented locally for validation and reproduction.

1) map-fields: detect required fields by work item type and enforce mapping completeness

--- a/specfact_cli/modules/backlog/src/commands.py
+++ b/specfact_cli/modules/backlog/src/commands.py
@@
-    # canonical fields (static)
+    # canonical fields (extended with required custom fields per selected work item type)
@@
+    def _fetch_ado_work_item_types() -> list[str]:
+        # GET .../_apis/wit/workitemtypes
+
+    def _fetch_required_fields_for_work_item_type(work_item_type_name: str) -> list[dict[str, Any]]:
+        # GET .../_apis/wit/workitemtypes/{type}/fields
+        # collect fields where alwaysRequired == true
@@
+    # Prompt user to select work item type for validation context
+    selected_work_item_type = ...
+    required_fields_for_selected_type = _fetch_required_fields_for_work_item_type(selected_work_item_type)
@@
+    # Add required custom fields to interactive mapping targets
+    canonical_fields["finops_category"] = "FinOps Category (required custom)"
+    canonical_fields["finops_subcategory"] = "FinOps Subcategory (required custom)"
@@
+    # Block save if required fields remain unmapped
+    if missing_required_fields:
+        raise typer.Exit(1)
@@
+    # Persist required fields metadata in provider settings
+    settings["required_fields_by_work_item_type"] = {
+        selected_work_item_type: ["..."]
+    }

2) backlog add: support repeatable custom required fields via CLI

--- a/specfact_cli/modules/backlog-core/src/backlog_core/commands/add.py
+++ b/specfact_cli/modules/backlog-core/src/backlog_core/commands/add.py
@@
+    custom_field: list[str] = typer.Option(
+        [],
+        "--custom-field",
+        help="Custom field value as key=value (repeatable)",
+    )
@@
+    def _coerce_custom_field_value(raw_value: str) -> Any:
+        # true/false -> bool
+        # numeric strings -> int/float
+        # else keep string
@@
+    # Parse --custom-field key=value entries and merge into payload.provider_fields
+    payload["provider_fields"].update(custom_fields_payload)

3) ADO adapter: map custom provider fields + validate picklist values before PATCH

--- a/specfact_cli/adapters/ado.py
+++ b/specfact_cli/adapters/ado.py
@@
+    def _get_work_item_type_field_metadata(self, org, project, work_item_type):
+        # GET .../_apis/wit/workitemtypes/{type}/fields
+        # return referenceName -> {name, allowed_values}
@@
+    # In create_issue():
+    # - resolve provider_fields canonical names via AdoFieldMapper
+    # - append mapped /fields/<ReferenceName> operations to patch_document
@@
+    # Preflight validation:
+    # if field has allowed_values and provided value is not in list -> raise ValueError
@@
+    # Improved error/debug output includes exact ADO validation message
+    # (e.g., TF401320 / invalid picklist value / invalid field type)

4) Local test mapping file update (example)

--- a/.specfact/templates/backlog/field_mappings/ado_custom.yaml
+++ b/.specfact/templates/backlog/field_mappings/ado_custom.yaml
@@
 field_mappings:
  System.Description: description
  Microsoft.VSTS.Common.AcceptanceCriteria: acceptance_criteria
  Microsoft.VSTS.Scheduling.StoryPoints: story_points
  Microsoft.VSTS.Common.BusinessValue: business_value
  Microsoft.VSTS.Common.Priority: priority
+  Microsoft.VSTS.Common.ValueArea: value_area
+  Custom.DefinitionofReadyismet: definition_of_ready_is_met
+  Custom.FinOpsCategory: finops_category
+  Custom.FinOpsSubcategory: finops_subcategory
  System.WorkItemType: work_item_type

5) Repro validation outcome

• Before fix: create failed with missing required custom fields / invalid field type / invalid picklist value.
• After fix: CLI validates mapping completeness and field values, then create succeeds when valid custom values are provided.

Anonymization Checklist

Before posting externally, ensure these are replaced/removed:

• [ORG], [PROJECT], [ITERATION_PATH]
• Work item IDs, URLs, tenant GUIDs, PAT/token fragments
• Internal team names and proprietary backlog content
• Any command output containing sensitive data