How to properly add multiple field mappings for a single field?

[ajroetker]

I'm trying to add a <field> mapping that's text and another <field>.keyword mapping at the same field.

I have a document that successfully gets mapped:

 Name: key, Value: iphone-15-pro-max
Name: key.keyword, Value: iphone-15-pro-max

using this mapping:

"default_mapping": {
           "enabled": true,
           "dynamic": true,
           "properties": {
             "key": {
               "enabled": true,
               "dynamic": true,
               "fields": [
                 {
                   "type": "text",
                   "index": true,
                   "include_term_vectors": true
                 },
                 {
                   "name": "key.keyword",
                   "type": "text",
                   "analyzer": "keyword",
                   "index": true,
                   "include_term_vectors": true
                 }
               ]
             },

but when I try to search:

		query := bleve.NewMatchQuery("iphone-15-pro-max")
		query.SetField("key.keyword")

it fails to find the doc. If I only use keyword as the field mapping with no key.keyword path suffix for indexing and search it works, so there's definitely something with the multiple field mappings I'm missing? I can share more code example or details as well.

Any insights would be helpful, thank you!

[abhinavdangeti]

@ajroetker We've a design decision limitation with that field name that has .s within them.

While we allow the name that uses . within them, during search while resolving the field name - . is used as a separator for document hierarchy. So in your case we will look for a child mapping with the name key and then a child field name keyword within that child mapping. Since that's not discovered, we fail the search.

Separately, I'm not quite sure I follow your use case here - is it fair to assume this is your document format ..

[
    {"Name": "key", "Value": "iphone-15-pro-max"},
    {"Name": "key.keyword", "Value": "iphone-15-pro-max"}
]

.. in which case you should be indexing the field name Name instead. If you could clarify your exact document structure - we may be able to make an alternative recommendation for you.

[ajroetker]

Thanks @abhinavdangeti for responding so quickly!

	// Index test documents
	doc1 := map[string]any{
		"name":  "Smartphone Apple iPhone",
		"key":   "iphone-15-pro-max",
		"_type": "product",
	}

Is the document I'm using. What I'm trying to do is use bleve like elasticsearch where I index a field key as a text field AND a keyword field at .keyword for doing exact matching. The thing is I've got an autosuggest type using ngram, I create also a subfield name._2gram (like es) and can do match queries on that just fine.

func TestBleveMatchQueryDirect(t *testing.T) {
	// Create schema with search_as_you_type field
	schema := &TableSchema{
		DefaultType: "product",
		DocumentSchemas: map[string]DocumentSchema{
			"product": {
				Schema: map[string]any{
					"type": "object",
					"properties": map[string]any{
						"key": map[string]any{
							"type":           "string",
							"x-my-types": []string{"text", "keyword"},
						},
						"name": map[string]any{
							"type":           "string",
							"x-my-types": []string{"search_as_you_type"},
						},
					},
				},
			},
		},
	}

	// Create index mapping from schema
	indexMapping := NewIndexMapFromSchema(schema)

	// Create in-memory bleve index
	index, err := bleve.NewMemOnly(indexMapping)
	require.NoError(t, err)
	defer index.Close()

	// Index test documents
	doc1 := map[string]any{
		"name":  "Smartphone Apple iPhone",
		"key":   "iphone-15-pro-max",
		"_type": "product",
	}
	err = index.Index("doc1", doc1)
	require.NoError(t, err)

	mappingBytes, err := json.MarshalIndent(index.Mapping(), "", "  ")
	require.NoError(t, err)
	t.Logf("Index Mapping:\n%s", string(mappingBytes))

	doc := document.NewDocument("")
	require.NoError(t, index.Mapping().MapDocument(doc, doc1))
	t.Logf("Document Fields:")
	for _, f := range doc.Fields {
		t.Logf(" - Name: %s, Value: %s", f.Name(), f.Value())
	}

	t.Run("match query for 'Smartphone Apple iPhone' on name field", func(t *testing.T) {
		query := bleve.NewMatchQuery("Smartphone Apple iPhone")
		query.SetField("name")
		searchReq := bleve.NewSearchRequest(query)
		searchReq.Size = 10

		result, err := index.Search(searchReq)
		require.NoError(t, err)
		assert.Len(t, result.Hits, 1, "Should find 1 doc")
		if len(result.Hits) > 0 {
			assert.Equal(t, "doc1", result.Hits[0].ID)
		}
	})
	t.Run("match query for 'Smartphone Apple iPhone' on name field ngram", func(t *testing.T) {
		query := bleve.NewMatchQuery("sm")
		query.SetField("name._2gram")
		searchReq := bleve.NewSearchRequest(query)
		searchReq.Size = 10

		result, err := index.Search(searchReq)
		require.NoError(t, err)
		assert.Len(t, result.Hits, 1, "Should find 1 doc")
		if len(result.Hits) > 0 {
			assert.Equal(t, "doc1", result.Hits[0].ID)
		}
	})
	t.Run("match query for 'iphone-15-pro-max' on key field", func(t *testing.T) {
		query := bleve.NewQueryStringQuery("key.keyword:iphone-15-pro-max")
		searchReq := bleve.NewSearchRequest(query)
		searchReq.Size = 10

		result, err := index.Search(searchReq)
		require.NoError(t, err)
		assert.Len(t, result.Hits, 1, "Should find 1 doc")
		if len(result.Hits) > 0 {
			assert.Equal(t, "doc1", result.Hits[0].ID)
			// assert.NotEqual(t, "doc1", result.Hits[0].ID)
		}
	})
}

Only the last test fails, so I'm not sure why what I'm doing with edge ngrams doesn't work for keyword?
PST: this logs

--- FAIL: TestBleveMatchQueryDirect (0.00s)
    bleve_test.go:388: Index Mapping:
        {
          "types": {
            "_all": {
              "enabled": false,
              "dynamic": true
            },
            "product": {
              "enabled": true,
              "dynamic": true,
              "properties": {
                "key": {
                  "enabled": true,
                  "dynamic": true,
                  "fields": [
                    {
                      "type": "text",
                      "index": true,
                      "include_term_vectors": true
                    },
                    {
                      "name": "key.keyword",
                      "type": "text",
                      "analyzer": "keyword",
                      "index": true,
                      "include_term_vectors": true
                    }
                  ]
                },
                "name": {
                  "enabled": true,
                  "dynamic": true,
                  "fields": [
                    {
                      "type": "text",
                      "index": true,
                      "include_term_vectors": true
                    },
                    {
                      "name": "name._2gram",
                      "type": "text",
                      "analyzer": "search_as_you_type_analyzer",
                      "index": true,
                      "include_term_vectors": true
                    }
                  ]
                }
              }
            }
          },
          "default_mapping": {
            "enabled": true,
            "dynamic": true,
            "properties": {
              "key": {
                "enabled": true,
                "dynamic": true,
                "fields": [
                  {
                    "type": "text",
                    "index": true,
                    "include_term_vectors": true
                  },
                  {
                    "name": "key.keyword",
                    "type": "text",
                    "analyzer": "keyword",
                    "index": true,
                    "include_term_vectors": true
                  }
                ]
              },
              "name": {
                "enabled": true,
                "dynamic": true,
                "fields": [
                  {
                    "type": "text",
                    "index": true,
                    "include_term_vectors": true
                  },
                  {
                    "name": "name._2gram",
                    "type": "text",
                    "analyzer": "search_as_you_type_analyzer",
                    "index": true,
                    "include_term_vectors": true
                  }
                ]
              }
            }
          },
          "type_field": "_type",
          "default_type": "_default",
          "default_analyzer": "standard",
          "default_datetime_parser": "dateTimeOptional",
          "scoring_model": "bm25",
          "default_field": "_all",
          "store_dynamic": false,
          "index_dynamic": true,
          "docvalues_dynamic": false,
          "analysis": {
            "token_filters": {
              "edge_ngram_filter": {
                "max": 4,
                "min": 2,
                "type": "edge_ngram"
              }
            },
            "analyzers": {
              "search_as_you_type_analyzer": {
                "char_filters": [
                  "zero_width_spaces"
                ],
                "token_filters": [
                  "to_lower",
                  "stop_en",
                  "edge_ngram_filter"
                ],
                "tokenizer": "unicode",
                "type": "custom"
              }
            }
          }
        }
    bleve_test.go:392: Document Fields:
    bleve_test.go:394:  - Name: key, Value: iphone-15-pro-max
    bleve_test.go:394:  - Name: key.keyword, Value: iphone-15-pro-max
    bleve_test.go:394:  - Name: _type, Value: product
    bleve_test.go:394:  - Name: name, Value: Smartphone Apple iPhone
    bleve_test.go:394:  - Name: name._2gram, Value: Smartphone Apple iPhone
    ```

[abhinavdangeti]

Got it, I think this makes more sense now.

I haven't tried reproducing your test case, but would you rename your field aliases to not use ., and retry the same with the new names ..
For example ..

• key.keyword -> key_keyword
• name._2gram -> name__2gram

[ajroetker]

Interesting @abhinavdangeti the key_keyword works nows!!, but the name__2gram and name._2gram worked in both cases, which is a bit confusing.

[abhinavdangeti]

The type mapping being dynamic and the data you're using is very likely the cause for this .. which brings us to one more recommendation for you here - you should disable the "dynamic"ness for the default_mapping which would only index the specific fields. This would avoid any false positives and also more importantly make your index lean and performant for your use case.