Missing "spec" in the batch ingestion spec created by the web console

[jihoonson]

Affected Version

0.17.0 branch

Description

The below is an ingestion spec created by the web console. dataSchema, ioConfig, and tuningConfig should be in spec which is missing. But if you submit this ingestion spec, the spec is properly inserted and the task succeeds.

{
  "type": "index_parallel",
  "ioConfig": {
    "type": "index_parallel",
    "inputSource": {
      "type": "http",
      "uris": [
        "https://static.imply.io/data/wikipedia.json.gz"
      ]
    },
    "inputFormat": {
      "type": "json"
    }
  },
  "tuningConfig": {
    "type": "index_parallel",
    "partitionsSpec": {
      "type": "dynamic"
    }
  },
  "dataSchema": {
    "dataSource": "wikipedia",
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "DAY",
      "queryGranularity": "HOUR",
      "rollup": true
    },
    "timestampSpec": {
      "column": "timestamp",
      "format": "iso"
    },
    "dimensionsSpec": {
      "dimensions": [
        "channel",
        "cityName",
        "comment",
        "countryIsoCode",
        "countryName",
        "diffUrl",
        "flags",
        "isAnonymous",
        "isMinor",
        "isNew",
        "isRobot",
        "isUnpatrolled",
        "namespace",
        "page",
        "regionIsoCode",
        "regionName",
        "user"
      ]
    },
    "metricsSpec": [
      {
        "name": "count",
        "type": "count"
      },
      {
        "name": "sum_added",
        "type": "longSum",
        "fieldName": "added"
      },
      {
        "name": "sum_commentLength",
        "type": "longSum",
        "fieldName": "commentLength"
      },
      {
        "name": "sum_deleted",
        "type": "longSum",
        "fieldName": "deleted"
      },
      {
        "name": "sum_delta",
        "type": "longSum",
        "fieldName": "delta"
      },
      {
        "name": "sum_deltaBucket",
        "type": "longSum",
        "fieldName": "deltaBucket"
      }
    ]
  }
}

[suneet-s]

Interesting... When I used the console, I had no idea that the web console was changing the spec on me. From a UX perspective, I think it would be good that we let the user know what changes are being made and why so that they learn the recommended way or choose to use a legacy way because of a known bug. Should I create a separate issue for that?