From 2890692a855cea3d1c2293926db4c1ded5b19555 Mon Sep 17 00:00:00 2001 From: John Trengrove Date: Mon, 2 Feb 2026 13:33:57 +1000 Subject: [PATCH 1/2] HFresh docs --- .../howto/go/docs/manage-data.classes_test.go | 2 +- .../code/howto/manage-data.collections.py | 1 + .../code/howto/manage-data.collections.ts | 1 + .../concepts/indexing/vector-index.md | 19 ++++++++++++ .../config-refs/indexing/vector-index.mdx | 31 ++++++++++++++++--- .../manage-collections/vector-config.mdx | 2 +- 6 files changed, 49 insertions(+), 7 deletions(-) diff --git a/_includes/code/howto/go/docs/manage-data.classes_test.go b/_includes/code/howto/go/docs/manage-data.classes_test.go index 3e23dccd7..94eb18532 100644 --- a/_includes/code/howto/go/docs/manage-data.classes_test.go +++ b/_includes/code/howto/go/docs/manage-data.classes_test.go @@ -239,7 +239,7 @@ func Test_ManageDataClasses(t *testing.T) { }, }, Vectorizer: "text2vec-openai", - VectorIndexType: "hnsw", // Or "flat", "dynamic" + VectorIndexType: "hnsw", // Or "flat", "dynamic", "hfresh" } // END SetVectorIndexType diff --git a/_includes/code/howto/manage-data.collections.py b/_includes/code/howto/manage-data.collections.py index 51ede8c31..d8f8be3f7 100644 --- a/_includes/code/howto/manage-data.collections.py +++ b/_includes/code/howto/manage-data.collections.py @@ -159,6 +159,7 @@ vector_index_config=Configure.VectorIndex.hnsw(), # Use the HNSW index # vector_index_config=Configure.VectorIndex.flat(), # Use the FLAT index # vector_index_config=Configure.VectorIndex.dynamic(), # Use the DYNAMIC index + # vector_index_config=Configure.VectorIndex.hfresh(), # Use the HFRESH index # highlight-end ), properties=[ diff --git a/_includes/code/howto/manage-data.collections.ts b/_includes/code/howto/manage-data.collections.ts index 3581b69f0..21a9bc14b 100644 --- a/_includes/code/howto/manage-data.collections.ts +++ b/_includes/code/howto/manage-data.collections.ts @@ -226,6 +226,7 @@ await client.collections.create({ vectorIndexConfig: configure.vectorIndex.hnsw(), // Use HNSW // vectorIndexConfig: configure.vectorIndex.flat(), // Use Flat // vectorIndexConfig: configure.vectorIndex.dynamic(), // Use Dynamic + // vectorIndexConfig: configure.vectorIndex.hfresh(), // Use HFresh // highlight-end }), properties: [ diff --git a/docs/weaviate/concepts/indexing/vector-index.md b/docs/weaviate/concepts/indexing/vector-index.md index ba39e80f8..9435cd855 100644 --- a/docs/weaviate/concepts/indexing/vector-index.md +++ b/docs/weaviate/concepts/indexing/vector-index.md @@ -15,6 +15,7 @@ Weaviate supports these vector index types: * [flat index](#flat-index): a simple, lightweight index that is designed for small datasets. * [HNSW index](#hierarchical-navigable-small-world-hnsw-index): a more complex index that is slower to build, but it scales well to large datasets as queries have a logarithmic time complexity. * [dynamic index](#dynamic-index): allows you to automatically switch from a flat index to an HNSW index as object count scales +* [HFresh index](#hfresh-index): a cluster-based index that uses HNSW for the centroid index, providing memory efficiency for large datasets. This page explains what vector indexes are, and what purpose they serve in the Weaviate vector database. @@ -261,6 +262,23 @@ Currently, this is only a one-way upgrade from a flat to an HNSW index, it does This is particularly useful in a multi-tenant setup where building an HNSW index per tenant would introduce extra overhead. With a dynamic index, as individual tenants grow their index will switch from flat to HNSW, while smaller tenants' indexes remain flat. +## HFresh index + +:::info Experimental feature +HFresh indexing is an experimental feature. +::: + +The **HFresh index** is a cluster-based vector index that uses HNSW for the centroid index. It is based on the SPFresh algorithm, which organizes vectors into posting lists (clusters) for efficient retrieval. + +HFresh works by: +1. Partitioning vectors into clusters, each with a centroid vector +2. Using an HNSW index to efficiently search the centroids +3. Searching only the most relevant posting lists (clusters) for a query + +This approach can provide memory efficiency benefits for large datasets while maintaining good search performance. The key trade-off is between memory usage and search recall, controlled by parameters like `searchProbe` (number of posting lists to search) and `replicas` (number of posting lists each vector is added to). + +For configuration details, see the [HFresh index parameters](../../config-refs/indexing/vector-index.mdx#hfresh-index-parameters). + ## Vector cache considerations For optimal search and import performance, previously imported vectors need to be in memory. A disk lookup for a vector is orders of magnitudes slower than memory lookup, so the disk cache should be used sparingly. However, Weaviate can limit the number of vectors in memory. By default, this limit is set to one trillion (`1e12`) objects when a new collection is created. @@ -286,6 +304,7 @@ When choosing an index type, use the following as a guide: - **Flat index**: Use for small collections with a known size. - **HNSW index**: Use for large collections with a known size. - **Dynamic index**: Use for collections with an unknown size or collections that may grow over time. +- **HFresh index**: Use for very large collections where memory efficiency is a priority. Note that the vector index type parameter only specifies how the vectors of data objects are *indexed*. The index is used for data retrieval and similarity search. diff --git a/docs/weaviate/config-refs/indexing/vector-index.mdx b/docs/weaviate/config-refs/indexing/vector-index.mdx index 96e693380..39e16c260 100644 --- a/docs/weaviate/config-refs/indexing/vector-index.mdx +++ b/docs/weaviate/config-refs/indexing/vector-index.mdx @@ -4,11 +4,12 @@ description: Reference for vector index types and parameters in Weaviate. --- **[Vector indexes](../../concepts/indexing/vector-index.md)** facilitate efficient, vector-first data storage and retrieval. -There are three supported vector index types: +There are four supported vector index types: - **[HNSW index](#hnsw-index)** - **[Flat index](#flat-index)** - **[Dynamic index](#dynamic-index)** +- **[HFresh index](#hfresh-index)** ## Index configuration parameters @@ -18,16 +19,18 @@ Available starting in `v1.25`. Dynamic indexing is an experimental feature. Use Use these parameters to configure the index type and their properties. They can be set in the [collection configuration](../../manage-collections/vector-config.mdx#set-vector-index-type). -| Parameter | Type | Default | Details | -| :------------------ | :----- | :------ | :------------------------------------------------------------------- | -| `vectorIndexType` | string | `hnsw` | Optional. The index type - can be `hnsw`, `flat` or `dynamic`. | -| `vectorIndexConfig` | object | - | Optional. Set parameters that are specific to the vector index type. | +| Parameter | Type | Default | Details | +| :------------------ | :----- | :------ | :----------------------------------------------------------------------------- | +| `vectorIndexType` | string | `hnsw` | Optional. The index type - can be `hnsw`, `flat`, `dynamic`, or `hfresh`. | +| `vectorIndexConfig` | object | - | Optional. Set parameters that are specific to the vector index type. |
How to select the index type Generally, the `hnsw` index type is recommended for most use cases. The `flat` index type is recommended for use cases where the data the number of objects per index is low, such as in multi-tenancy cases. You can also opt for the `dynamic` index which will initially configure a `flat` index and once the object count exceeds a specified threshold it will automatically convert to an `hnsw` index. +The `hfresh` index is a cluster-based index that uses HNSW for the centroid index. It can provide memory efficiency benefits for large datasets while maintaining good search performance. + See [this section](../../concepts/indexing/vector-index.md#which-vector-index-is-right-for-me) for more information about the different index types and how to choose between them.
@@ -167,6 +170,24 @@ The goal of `dynamic` indexing is to shorten latencies during query time at the | `flat` | object | default Flat | [Flat index configuration](#flat-index) to be used. | | `threshold` | integer | 10000 | Threshold object count at which `flat` to `hnsw` conversion happens | +## HFresh index + +:::info Experimental feature +HFresh indexing is an experimental feature. +::: + +HFresh is a cluster-based vector index that uses HNSW for the centroid index. It is based on the SPFresh algorithm but uses HNSW for the centroid index, providing a balance between memory efficiency and search performance for large datasets. + +### HFresh index parameters + +| Parameter | Type | Default | Details | +| :------------------- | :------ | :-------- | :--------------------------------------------------------------------------------------------------------- | +| `distance` | string | `cosine` | Distance metric. The metric that measures the distance between two arbitrary vectors. | +| `maxPostingSizeKb` | integer | dynamic | Maximum allowed size in KB for a posting list. By default, this is computed dynamically. | +| `replicas` | integer | `4` | Number of posting lists in which a vector will be added. | +| `rngFactor` | integer | `10` | Factor used when adding a vector to a posting list. If too close to a selected posting list, it is skipped. | +| `searchProbe` | integer | `64` | Number of posting lists to search during a query. | + ## Quantization parameters ### RQ parameters diff --git a/docs/weaviate/manage-collections/vector-config.mdx b/docs/weaviate/manage-collections/vector-config.mdx index 9da03b95a..44ca1db2c 100644 --- a/docs/weaviate/manage-collections/vector-config.mdx +++ b/docs/weaviate/manage-collections/vector-config.mdx @@ -314,7 +314,7 @@ Multi-vector embeddings use up more memory than single vector embeddings. You ca ## Set vector index type -The vector index type can be set for each collection at creation time, between `hnsw`, `flat` and `dynamic` index types. +The vector index type can be set for each collection at creation time, between `hnsw`, `flat`, `dynamic`, and `hfresh` index types. From cfed1a58aaeef07a36c5edf3de6a2fe3280f9a3d Mon Sep 17 00:00:00 2001 From: John Trengrove Date: Tue, 3 Feb 2026 14:28:29 +1000 Subject: [PATCH 2/2] Update RQ config options --- .../weaviate/config-refs/indexing/vector-index.mdx | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/weaviate/config-refs/indexing/vector-index.mdx b/docs/weaviate/config-refs/indexing/vector-index.mdx index 39e16c260..a55684360 100644 --- a/docs/weaviate/config-refs/indexing/vector-index.mdx +++ b/docs/weaviate/config-refs/indexing/vector-index.mdx @@ -183,10 +183,20 @@ HFresh is a cluster-based vector index that uses HNSW for the centroid index. It | Parameter | Type | Default | Details | | :------------------- | :------ | :-------- | :--------------------------------------------------------------------------------------------------------- | | `distance` | string | `cosine` | Distance metric. The metric that measures the distance between two arbitrary vectors. | -| `maxPostingSizeKb` | integer | dynamic | Maximum allowed size in KB for a posting list. By default, this is computed dynamically. | +| `maxPostingSizeKB` | integer | `48` | Maximum allowed size in KB for a posting list. Weaviate uses this value along with the vector size to calculate the maximum posting size. | | `replicas` | integer | `4` | Number of posting lists in which a vector will be added. | -| `rngFactor` | integer | `10` | Factor used when adding a vector to a posting list. If too close to a selected posting list, it is skipped. | | `searchProbe` | integer | `64` | Number of posting lists to search during a query. | +| `rq` | object | -- | Rotational quantization (RQ) compression configuration. RQ is enabled by default with `bits: 1`. See [RQ parameters for HFresh](#rq-parameters-for-hfresh). | + +### RQ parameters for HFresh + +HFresh uses rotational quantization (RQ) compression by default. The following parameters are available under `rq`: + +| Parameter | Type | Default | Details | +| :------------- | :------ | :------ | :------------------------------------------------------------------------------------------ | +| `enabled` | boolean | `true` | Enable RQ compression. RQ is mandatory and cannot be disabled for HFresh. | +| `bits` | integer | `1` | Number of bits used for quantization of postings. Currently only 1 bit supported. | +| `rescoreLimit` | integer | `350` | The minimum number of candidates to fetch before rescoring. Behaves like `rescoreLimit` in other index types. | ## Quantization parameters