VAULT-32657 prevent duplicate keys in HCL files

[bosouza]

Description

Turns out the simple breaking change approach proposed in this PR is not acceptable, we'll instead need to use the deprecation process to ensure users have time to adapt without disrupting updates, especially regarding automated policy definitions that would all break with this change if using duplicate attributes. Also, the work here was not completed

As discussed in hashicorp/hcl#35, HCL has allowed duplicate keys where the last one "wins", which has been flagged as a security risk. A fix preventing duplicate keys in HCL was added in hashicorp/hcl#704, and additional methods for preserving the old behavior came in hashicorp/hcl#707.

This PR upgrades all HCL dependencies to the fixed version, which effectively introduces a breaking change where all HCL files accepted by Vault that might have previously worked fine containing duplicate keys will now fail to be parsed, erroring out the operation. This is a comprehensive list of affected features:

• HCL configs accepted by vault CLI commands:
  • $HOME/.vault HCL file with token_helper field
  • -config HCL file passed to vault operator migrate command
  • -config HCL file passed to vault agent command
  • -config HCL file passed to vault proxy command
  • -config HCL file passed to vault operator diagnose command
  • -config HCL file passed to vault server command
    • users trying to upgrade their Vault clusters with duplicate keys in their config will fail to upgrade, this should be mentioned in the upgrade guide.
• HCL policy definitions:
  • HCL password policy provided to /sys/policies/password/:name cannot have duplicate fields anymore
  • Generating a password from a password policy with duplicate HCL attributes is still allowed for backwards compatibility reasons (/sys/policies/password/:name/generate)
  • TODO: ACL policies are also affected

TODO only if you're a HashiCorp employee

• Backport Labels: If this fix needs to be backported, use the appropriate backport/ label that matches the desired release branch. Note that in the CE repo, the latest release branch will look like backport/x.x.x, but older release branches will be backport/ent/x.x.x+ent.
  • LTS: If this fixes a critical security vulnerability or severity 1 bug, it will also need to be backported to the current LTS versions of Vault. To ensure this, use all available enterprise labels.
• ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature
  of a public function, even if that change is in a CE file, double check that
  applying the patch for this PR to the ENT repo and running tests doesn't
  break any tests. Sometimes ENT only tests rely on public functions in CE
  files.
• Jira: If this change has an associated Jira, it's referenced either
  in the PR description, commit message, or branch name.
• RFC: If this change has an associated RFC, please link it in the description.
• ENT PR: If this change has an associated ENT PR, please link it in the
  description. Also, make sure the changelog is in this PR, not in your ENT PR.

[github-actions]

CI Results: failed ❌
Failures:

Test Type	Package	Test	Logs
standard	command/server	TestConfigRaftRetryJoin	view test results
standard	command/server	TestConfigRaftRetryJoin/attr	view test results
standard	command/server	TestConfigRaftRetryJoin/mixed	view test results
standard	command/server	TestLoadConfigFile	view test results
standard	command/server	TestLoadConfigFileWithLeaseMetricTelemetry	view test results
standard	command/server	TestLoadConfigFile_topLevel	view test results
standard	command/server	TestMetricFilterConfigs	view test results
standard	command/server	TestMetricFilterConfigs/validate_metric_filter_configs	view test results

[digital-content-events]

📄 Content Checks

Updated: Mon, 24 Mar 2025 15:27:21 GMT

Found 4 error(s)

content/docs/upgrading/upgrade-to-1.20.x.mdx

Error parsing frontmatter: YAMLParseError: Implicit keys need to be on a single line at line 4, column 1:

description: |-
Deprecations, important or breaking changes, and remediation recommendations
^

Position	Description	Rule
1:1-1:1	Document does not have a page_title key in its frontmatter. Add a page_title key at the top of the document.	ensure-valid-frontmatter
1:1-1:1	Document does not have a description key in its frontmatter. Add a description key at the top of the document.	ensure-valid-frontmatter
1:1-1:1	This file is not present in the nav data file at data/docs-nav-data.json. Either add a path that maps to this file in the nav data or remove the file. If you want the page to exist but not be linked in the navigation, add a hidden property to the associated nav node.	no-unlinked-pages