Skip to content

SOLR-18267: Add scalar quantized flat vector index support with no HNSW - #4698

Open
adamjq wants to merge 1 commit into
apache:mainfrom
adamjq:SOLR-18267-add-scalar-quantized-flat-vector-index
Open

SOLR-18267: Add scalar quantized flat vector index support with no HNSW#4698
adamjq wants to merge 1 commit into
apache:mainfrom
adamjq:SOLR-18267-add-scalar-quantized-flat-vector-index

Conversation

@adamjq

@adamjq adamjq commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-18267

Description

This PR follows up from #4492, which added a new knnAlgorithm=flat option to DenseVectorField. It adds the same flat index support to ScalarQuantizedDenseVectorField so that scalar-quantized fields can also be indexed without building HNSW graphs.

The new behavior introduced is:

  • ScalarQuantizedDenseVectorField now supports knnAlgorithm=flat. Searching works through the {!knn}, {!knn_text_to_vector}, and {!vectorSimilarity} query parsers, and the vectorSimilarity() function query.
  • ScalarQuantizedDenseVectorField with knnAlgorithm=flat and vectorEncoding=BYTE is rejected explicitly

Note: knnAlgorithm=flat is not supported for BinaryQuantizedDenseVectorField

Solution

This PR adds support for ScalarQuantizedDenseVectorField with knnAlgorithm=flat by delegating to Lucene104ScalarQuantizedVectorsFormat.

This implementation differs from #4492 in two main ways:

  1. The reader in Lucene104ScalarQuantizedVectorsFormat implements brute-force KNN search, meaning the knn, knn_text_to_vector and vectorSimilarity query parsers are supported by default
  2. Lucene104ScalarQuantizedVectorsFormat is registered in SPI, so no wrapper class is needed

The table below shows the different segment files generated between the flat vs HNSW configurations:

Ext Contents Scalar Quantized (Flat) Scalar Quantized (HNSW)
.vec Raw float vectors Yes Yes
.vemf Raw-float metadata Yes Yes
.veq Quantized vectors Yes Yes
.vemq Quantized metadata Yes Yes
.vem HNSW graph metadata No Yes
.vex HNSW graph No Yes

AI Disclosure: Claude was used to assist with this PR. All code has been reviewed and tested by me.

Tests

Unit tests for ScalarQuantizedDenseVectorField.

Checklist

Please review the following and check all that apply:

  • I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
  • I have created a Jira issue and added the issue ID to my pull request title.
  • I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
  • I have developed this patch against the main branch.
  • I have run ./gradlew check.
  • I have added tests for my changes.
  • I have added documentation for the Reference Guide
  • I have added a changelog entry for my change

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests cat:search cat:schema labels Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cat:schema cat:search documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant