[BUG] Cannot provide multiple outputs of the same type

[japtain-cack]

Your setup

Formula commit hash / release tag

master

Versions reports (master & minion)

sudo salt --versions-report
Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.7.1
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.13.1
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: amzn 2
        locale: utf-8
       machine: x86_64
       release: 4.14.330-250.540.amzn2.x86_64
        system: Linux
       version: Amazon Linux 2

Pillar / config used

{%- set provider_map = {"sh": "singlehop", "ln": "linode", "am": "amazon"} %}
{%- set region_map = {"az": "arizona", "il": "illinois", "tx": "texas", "ga": "georgia", "sy": "sydney", "or": "oregon", "oh": "ohio"} %}
{%- set env_map = {"example.com": "production", "example-dev.com": "development"} %}
{%- set parsed_hostname = grains["id"] | regex_search('(^\w{2})(\w{2})(.+?)\.(.*)$') %}
{%- set provider, location, role, domain = parsed_hostname %}

telegraf:
  config: "/etc/telegraf/telegraf.d/example.generic.conf"
  conf:
    global_tags:
      role: "{{ role }}"
      env: "{{ env_map.get(domain, 'unknown') }}"
      location: "{{ region_map.get(location, 'unknown') }}"
      provider: "{{ provider_map.get(provider, 'unknown') }}"
    agent:
      metric_batch_size: 500
      metric_buffer_limit: 10000
      hostname: ""
      collection_jitter: "2s"
      interval: "10s"
      logfile: '/var/log/telegraf/telegraf.log'
      quiet: true
      debug: false
      round_interval: true
      flush_interval: "15s"
      flush_jitter: "14s"
  outputs:
    influxdb:
      username: "{{ INFLUXUSER }}"
      timeout: "5s"
      database: "telegraf"
      precision: "s"
      urls:
        - "http://{{ INFLUXHOSTEAST }}:{{ INFLUXPORT }}"
      password: "{{ INFLUXPW }}"
      namepass:
        - "kernel"
        - "processes"
        - "mem"
        - "netstat"
        - "system"
        - "swap"
        - "linux_sysctl_fs"
        - "net"
        - "disk"
        - "cpu"
        - "systemd_units"
        - "influxdb"
    influxdb:
      username: "{{ INFLUXUSER }}"
      timeout: "5s"
      database: "telegraf"
      precision: "s"
      urls:
        - "http://{{ INFLUXHOSTWEST }}:{{ INFLUXPORT }}"
      password: "{{ INFLUXPW }}"
      namepass:
        - "kernel"
        - "processes"
        - "mem"
        - "netstat"
        - "system"
        - "swap"
        - "linux_sysctl_fs"
        - "net"
        - "disk"
        - "cpu"
        - "systemd_units"
        - "influxdb"
  inputs:
    kernel: {}
    processes: {}
    mem: {}
    netstat: {}
    system: {}
    swap: {}
    linux_sysctl_fs: {}
    net:
      interfaces:
        - "lo"
        - "ens5"
    disk:
      ignore_fs:
        - "tmpfs"
        - "devtmpfs"
        - "overlay"
        - "aufs"
    cpu:
      collect_cpu_time: false
      totalcpu: true
      report_active: false
      percpu: true
    systemd_units:
      interval: "5m"

---

Bug details

Describe the bug

Cannot specify multiple influxdb outputs.

Steps to reproduce the bug

Define multiple influxdb output endpoints.

Expected behaviour

As per the documentation provided by influxdb and toml, to ship metrics to more than one instance, you must create multiple [[outputs.influxdb]] entries.

Attempts to fix the bug

Converting YAML to TOML, which seems to be happening based on this template, doesn't seem to capture the full functionality and configuration possibilities. In YAML, you cannot have duplicate keys in a map.

Additional context

[KaneTW]

Just ran into this as well. Very annoying.

[japtain-cack]

If outputs were a list of maps, this wouldn't be an issue. However, because the template responsible for generating the TOML file is using salt['slsutil.serialize']('toml', telegraf) to directly convert a YAML structure to TOML, this would require re-designing a lot of components.