77 Commits

Author SHA1 Message Date
Andrey Vasnetsov
a6b99fd583
distributed optimization info (#8120)
* [AI] initial implementation

* [AI] review fixes

* [manaul] some small fix

* [AI] fix issue about serializing optional fields

* upd schema

* [manual] review fixes

* [manual] only ask updatable replicas about optimizations
2026-02-16 10:17:28 +01:00
Luis Cossío
bc6246df5a
[relevance feedback] add rest and grpc interfaces (#7399)
* add rest and grpc interfaces

Also handle inference for this query

* follow refactor from base branch

* renaming from Ms Cooper

* get started on validation testing

* more validations

* test equivalence with query when less than 2 feedback elements

* rename feedback query to relevance_feedback query

* fix extraction of context pairs

* rename feedback vector to example

* make coderabbit happier

* upd test

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-02-13 09:55:37 +01:00
AMIR
8eec2046d7
Add list shard keys API (#7615) 2025-12-03 10:20:40 +01:00
xzfc
21329ca529
Add ACORN-1 search (#7414)
* GraphLayersBase::try_for_each_link

* Add FilteredScorer::score_points_unfiltered

* Add GraphLayersBase::search_on_level_acorn()

* Add SearchParams::acorn API parameter

* Integrate ACORN to HNSW search

* Add doc

* review fixes

* Misc fixes

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-11-14 12:30:12 +01:00
xzfc
6a2830db1c
Use fs-err (#7319) 2025-11-14 12:26:51 +01:00
Arnaud Gourlay
57831a355c
[strict-mode] Add max number of payload index count (#7222)
* [strict-mode] Add max number of payload index count

* improve docs and validation for StrictModeConfig

* fix from coderabbit

* test blocking access to payload indexes

* polish test
2025-09-29 11:29:50 +02:00
Arnaud Gourlay
e4d61c768c
Fix missing gRPC API validations for PointsSelector (#7221) 2025-09-29 11:28:41 +02:00
Arnaud Gourlay
338b683064
Fix missing API validation for MinShould (#7217) 2025-09-29 11:28:28 +02:00
Luis Cossío
e379d78c3c
Custom RRF k parameter (#7065)
* allow custom K parameter for RRF

* generate grpc docs and openapi

* use tagged type approach for parametrized fusions

* use params approach in grpc

* simplify api structure

* upd schema

* nits

* rest: parameterized rrf as query variant

* consistency in doc comments

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2025-08-26 13:18:02 +02:00
Andrey Vasnetsov
9c8ef5574f
Update if (#7006)
* implement condition parameter for upsert operation

* fmt

* update api schema

* implement conditional update for update-vectors op

* rename to update_filter for consistency

* add tests

* explicilty ignore strict mode for conditional updates

* rabbit review fix

* Also assert vector element before, confirm we don't normalize

* Simplify filter creation using new helper

* Rename merge_with_ids to with_point_ids

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-08-26 13:16:18 +02:00
Kumar Shivendu
2b6256327b
Allow retaining more closed segments (#6976)
* Retain more closed segments

* Add back rocksdb

* Update WalOptions/WalConfig across the code

* Use NonZeroUsize

* Expose via APIs

* Update gRPC docs

* Fix stoarge compat test

* recompile openapi.json with rocksdb

* default wal retain closed fn

* update openapi.json

* Use qdrant/wal latest commit and remove from config.yaml
2025-08-11 13:17:47 +02:00
Luis Cossío
e8daf9c79f [MMR] expose as query variant in rest and grpc (#6797)
* expose mmr in rest and grpc

* fix referenced vectors in mmr queries

* rename to MmrInput in grpc for consistency

* mmr is now an optional parameter of nearest query

* fix score_threshold at local shard

* fix conversion of rest -> collection

* fix and refactor planned query

* nits

* rename `lambda` to `diversity` in interface

* handle score_threshold within mmr calculation

* add openapi test

* fix candidate limit at collection level

* candidate_limit -> candidates_limit

* finish rename to candidates_limit

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-07-17 13:56:46 +02:00
Tim Visée
44be97644f Add validation for legacy collection names (#6752)
* Add function for validating legacy collection names

* Apply legacy collection name validation everywhere

* Add more validation to gRPC endpoints

* Adjust tests
2025-07-17 13:24:51 +02:00
Arnaud Gourlay
b9771dabdb Fix missing API validations (#6499) 2025-05-22 23:01:09 +02:00
Arnaud Gourlay
eaaa4e4ee1 Validate IntegerIndexParams (#6494)
* Validate IntegerIndexParams

* focus
2025-05-22 23:00:52 +02:00
Arnaud Gourlay
92d324c0c7 Fix missing gRPC query validations (#6485) 2025-05-22 22:59:02 +02:00
Andrey Vasnetsov
f1884161cc Faster wal-delta transfer (#6458)
* implement internal batch-update api

* use batch update in wal-delta

* fix ci
2025-05-22 22:53:56 +02:00
Arnaud Gourlay
820c9e17d1
Remove validation on CollectionInfo response type (#6107) 2025-03-21 11:41:26 +01:00
Tim Visée
e35914b154
Fix deleted threshold validation in gRPC, should be in range [0.0, 1.0] (#5810) 2025-01-16 10:51:47 +01:00
Tim Visée
3decc3ca20
Fix collection creation validation, disallow replication factor 0 (#5643)
* Validate default collection config in settings

* Validate shard number, replication/write consistency factor in gRPC

* Cleanup

* Add more validation rules

* Also validate strict mode config in gRPC CollectionConfig

* Update OpenAPI specification
2025-01-08 14:08:25 +01:00
Tim Visée
93b57186de
Simplify gRPC validation (#5650) 2025-01-08 14:03:59 +01:00
Arnaud Gourlay
993121b3ea
Rate limit requests per minute (#5597)
* Rate limit requests per minute

* rename to remove time unit for API
2025-01-08 14:01:28 +01:00
Arnaud Gourlay
ba57d63200
gRPC API for Distance Matrix (#5011)
* gRPC API for Distance Matrix

* generate API docs

* apply correct limit to sample

* add JWT tests

* pluralitiy

* fix COO naming

* drops COO as it trips codespell and is redundant

* clarify docs
2024-09-17 15:26:02 +02:00
Jojii
3377530c62
[Strict-Mode] Basic implementation (#4887)
* add CollectionRequestVerification

* add to api

* rebase

* improve implementation

* implement strict mode for SearchRequest+Batch

* improve code + fix Clippy

* improve error handling

* restructure StrictModeVerification trait

* generate docs

* check `enabled` option

* review remarks

* rename StrictModeConfigDiff in grpc

* use missing payload detection from issue api

* performance improvement

* decouple extractor from issues (#4945)

* some review remarks

* don't default to empty functions in StrictModeVerification trait

* update openapi

* filter_limit => query_limit

* replace discovery_max_context_size and recommend_max_examples with max_input_examples

* review remarks

* review fix: include possible index types into error message

* review remarks

---------

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-08-29 15:24:11 +02:00
Luis Cossío
5f5fd25d64 Add grpc endpoint for Facet (#4933)
* add grpc endpoint

* gen grpc docs
2024-08-27 01:13:12 +02:00
Arnaud Gourlay
29126883bc Update Validator 0.18 (#4894)
* Update Validator 0.18

* fix new test error message

* clearer geo error message

* fix error message

* fix unit tests

* Put spaces back

---------

Co-authored-by: timvisee <tim@visee.me>
2024-08-27 01:04:34 +02:00
Luis Cossío
12c5d6b6b6 Support timeout in Facets (#4792)
* nits in segments_searcher

* implement timeout into segment faceting

* Add timeout to internal service api

* refactor iterator_ext, and add test
2024-08-09 18:17:26 +02:00
Luis Cossío
88fa434f53 Facets in internal service (#4790)
* add `facet` to shard trait

* Add internal service

* gen grpc docs
2024-08-09 18:17:00 +02:00
Arnaud Gourlay
19d878d277 universal-query: gRPC batch API (#4513)
* universal-query: gRPC batch API

* regen gRPC docs

* enable jwt access test
2024-06-21 23:43:16 +02:00
Arnaud Gourlay
922fdefc6b universal-query: gRPC query API (#4495)
* universal-query: gRPC query API

* misc fixes

* more review

* add custom validation on QueryPoints.limit

* fix jwt_access test for gRPC query
2024-06-21 23:41:18 +02:00
Luis Cossío
a740f7944a universal-query: Introduce new RawVector and VectorInput messages (#4209)
* Introduce `RawVector` and `VectorInput` messages

* gen grpc docs

* remove extra `optional`

* gen grpc docs
2024-05-26 12:31:02 +02:00
Arnaud Gourlay
99ed158841
Add missing validation for gRPC UpsertPoints (#4074) 2024-04-22 10:34:38 +02:00
Tim Visée
8d1fd9419e
Add peer metadata to consensus, tracking Qdrant versions (#3702)
* Add peer metadata state to consensus, update on sync local state

* Minor formatting improvements

* Use Qdrant version from constant

* Initialize empty metadata if not present on disk

* Refactor old parameter name

* Logging improvements based on review feedback

* Assert that Qdrant version matches crate version

* Do not use a custom (de)serializer for peer metadata

* Use semver directly for handling Qdrant versions

* Rename is_outdated to is_different_version

* Assert Qdrant version in build.rs

* Disable updating peer metadata until Qdrant 1.9

We do this, because if a node is running 1.8, there may still be nodes
running 1.7. Those nodes do not support this operation.

* clippy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-03-05 17:16:33 +01:00
Kumar Shivendu
ad6b4b2532
fix: Include git commit id while building docker image if possible (#3640)
* feat: Print commit id in dev container build workflow

* fix: Keep .git while building Qdrant binary in docker

* fix: Remove redundant printing of git commit id

* fix: Copy only git files first

* fix: Docker build should have commit id if present

* refactor: Use fewer lines of code

* feat: Use git commit id from build arg

* refactor: Use cleaner code

* Fix string reference issue

* fix: Dont print info log if commit is not set

* fix: Include git commit id in release containers

---------

Co-authored-by: timvisee <tim@visee.me>
2024-03-05 17:09:58 +01:00
Tim Visée
bb8dcb15c8
Add gRPC API to set shard cutoff point (#3661)
* Add functions to propagate updating cutoff point from collection level

* Add gRPC endpoint to set cutoff point

* Lock highest and cutoff clock maps separately
2024-03-05 17:09:49 +01:00
Ivan Pleshkov
4397b28530
Validate sparse indices in grpc search (#3225)
* validate sparse indices in grpc search

* add test

* validate into vector conversion
2024-03-05 17:06:04 +01:00
Tim Visée
f08df0b22e
Add endpoint to request recovery point for remote shard (#3510)
* Add initial gRPC call for requesting WAL recovery point for remote shard

* Add remote shard method to request WAL recovery point

* Add recovery point type in gRPC, use it in recovery point functions

* Add function to extend recovery point with missing clocks from clock map

* Add new gRPC type for recovery point clocks

* Remove atomic loading, because we use regular integers now
2024-03-05 17:04:41 +01:00
xzfc
2f76603ddf DateTime payload index (#3395)
* Datetime payload index

* Introduce IndexMapItem

* Drop FieldIndex::DatetimeIndex

* Rename OpenAPI struct names

* Switch to microseconds

* Validate and serialize grpc timestamps

* Add tests with different timezones

* minor review fixes

* Revert "Drop FieldIndex::DatetimeIndex"

This reverts commit d55f251afdbb418ef732a3e6799b92f924fc3035.

* Revert "Introduce IndexMapItem"

This reverts commit c5255f6b1aafa2b9552bac5d1811f9e826eb8d61.

* fix: back to microseconds after reverts

* extract range conversion from boxed checker fn

* add log to deps

* don't run macro doctest

* no_run -> ignore

* remove prost-types in favor of prost-wkt-types

* better assertion on test_payload_indexing.py

* propagate unparsable datetime

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2024-03-05 16:57:08 +01:00
Gabriel
3392407f6a API for checking collection exists (#3472)
* saving progress

* tests assertions are working

* fixed lint

* fixed typo, existance -> existence

* added timing on grpc

* added timing on grpc

* fixed double time
2024-03-05 16:55:27 +01:00
Andrey Vasnetsov
e59280f1bb Revert "fix: Include git commit id while building docker image if possible" (#3468) 2024-03-05 16:52:57 +01:00
Kumar Shivendu
de5d120197 fix: Include git commit id while building docker image if possible (#3419)
* feat: Print commit id in dev container build workflow

* fix: Keep .git while building Qdrant binary in docker

* fix: Remove redundant printing of git commit id

* fix: Copy only git files first

* fix: Docker build should have commit id if present

* refactor: Use fewer lines of code
2024-03-05 16:52:51 +01:00
xzfc
a11c6f6a5b add optional checksum to snapshot recovery request (#3381)
* add optional checksum to snapshot recovery request

* Update openapi yaml files

* Add pytests for snapshots restoration
2024-03-05 16:52:38 +01:00
Kumar Shivendu
0fe19cfb60 feat: Expose git commit id in the health check endpoint (#3346)
* feat: Expose git commit id in the health check endpoint

* fix: CI errors

* test: Add test for health check api

* feat: Add / endpoint to openapi schema

* Make git commit hash optional

* ci: Enable debugging setup-protoc action

* Install later protobuf compiler through GitHub Action

* Disable debug mode for setup-protoc job

* refactor: Use commit instead of commit_id

* fix: Use commit instead of commit_id gRPC docs

* test: Update ping API test

* refactor: Rename ping api to root api

---------

Co-authored-by: timvisee <tim@visee.me>
2024-03-05 16:46:04 +01:00
dependabot[bot]
11ebb5d8a7
Bump mio from 0.8.9 to 0.8.11 (#3771)
* Bump mio from 0.8.9 to 0.8.11

Bumps [mio](https://github.com/tokio-rs/mio) from 0.8.9 to 0.8.11.
- [Release notes](https://github.com/tokio-rs/mio/releases)
- [Changelog](https://github.com/tokio-rs/mio/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/mio/compare/v0.8.9...v0.8.11)

---
updated-dependencies:
- dependency-name: mio
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Resolve clippy warnings

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
2024-03-05 10:38:41 +01:00
Ivan Pleshkov
47690244bc Sparse vector grpc definition (#3070)
* Sparse vector grpc definition

remove obsolete use

update grpc docs

update openapi

more convertion fixes

update grpc docs

* validations

* review renamings

* add todos
2023-12-06 17:11:06 +01:00
Andrey Vasnetsov
75f2c8b07b remove overvalidation (#3034) 2023-12-06 17:05:55 +01:00
Luis Cossío
91eb3f36e2 Discovery API (#2861)
* create and connect discovery http and grpc interfaces

* add openapi tests

* fix bad rebase

* Add better descriptions

* remove numpy from openapi tests

* fix rebase artifact

* remove already addressed TODO

* add more tests

* 🤡🔫 (cfg batch handler)

* add timeout query param for discover requests

* More gRPC validation

* make fields pydantic_openapi_generator_v3 friendly

* `context_pairs` -> `context` with struct for pairs

* discovery api is only discovery or context,
move struct description to fields

---------

Co-authored-by: timvisee <tim@visee.me>
2023-12-06 16:57:14 +01:00
Luis Cossío
87524275d4 Expose timeout query param for search requests (#2748)
* add timeout query param for search requests

* enable timeout for recommend requests

* Add query timeout for group by requests

* update openapi models

* Don't decrease timeout after recommend preprocessing

* Add openapi test

* code review

* add timeout to individual group by requests, non-decreasing

* handle timeout for discover

* Update timeout field tag in SearchBatchPoints
message
2023-12-06 16:52:49 +01:00
Tim Visée
b1eb5a2a68 Improve shard snapshot transfer replica set state synchronization (#2884)
* Add `WaitForShardState` gRPC call definition

* Implement logic for `WaitForShardState` call

* In next shard snapshot transfer stage, wait for remote to reach `Partial`

* In shard snapshot transfer, synchronize consensus as last step

We don't have to synchronize consensus right away. Instead we just
confirm that the remote shard has reached `Partial` state. Then we
transform the queue proxy shard into the forward proxy.

Right before we finalize the transfer we do want to synchronize
consensus. First make sure the shard has reached `Partial` state in our
local replica set. Then synchronize all other nodes to make sure they
reach at least the same consensus state.

* Reformat internal collection service definition
2023-12-06 16:49:49 +01:00
Roman Titov
a3cc037afb Add shard snapshot gRPC API (#2825)
* Fix paste-bugs in `snapshot_service.proto`

* Add shard snapshot gRCP API definition

* Add validation to shard snapshot gRPC API definition

* Implement conversions between gRPC and `collection` types

* Extract shard snapshot API implementation into common sub-module

* Implement shard snapshot gRPC API

* Generate gRPC docs

* Refactor `ShardSnapshots` gRPC service to be internal API only

* fixup! Refactor `ShardSnapshots` gRPC service to be internal API only

Move `ShardSnapshotRecoverResponse` to `shard_snapshots_service.proto`

* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only

Update `api/src/grpc/qdrant.rs`

* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only

Update gRPC docs

* Switch `ShardSnapshots` gRPC service to use `validate_and_log` instead of `validate`
2023-12-06 16:44:59 +01:00