134 Commits

Author SHA1 Message Date
Andrey Vasnetsov
7f3d0eef01
Fix condition deserialization issue (#8455)
* [AI] fix cbor deserialization error

* fmt

* Address review feedback

- Use let-chains for cleaner Condition deserialization logic
- Add WAL-level roundtrip test for delete-by-filter with UUID has_id

Made-with: Cursor

---------

Co-authored-by: Cursor Agent <agent@cursor.com>
2026-03-26 19:03:47 +01:00
qdrant-cloud-bot
685466df60
Add #[must_use] to RAII guard and task handle types (#8499)
* Add `#[must_use]` to RAII guard and task handle types

These types rely on being held for a scope — dropping them immediately is
almost always a bug:

- `StoppingGuard`: sets `is_stopped` flag on drop
- `UpdateGuard`: decrements update counter on drop
- `UpdatesGuard`: releases mutex preventing concurrent updates on drop
- `ClockGuard`: marks clock as inactive on drop
- `IsAliveGuard`: releases liveness lock on drop
- `ScopeTrackerGuard`: decrements scope counter on drop
- `CancellableAsyncTaskHandle`: detaches task on drop
- `StoppableTaskHandle`: may abort task on drop (AbortOnDropHandle)

Made-with: Cursor

* Remove redundant function-level #[must_use] now covered by type

With ScopeTrackerGuard marked #[must_use] at the type level, the
function-level attributes on measure_scope(), measure(),
track_create_snapshot_request(), and count_snapshot_creation() are
redundant and trigger clippy::double_must_use.

Made-with: Cursor

---------

Co-authored-by: Cursor Agent <agent@cursor.com>
2026-03-26 18:48:41 +01:00
Ivan Pleshkov
8534e8388b
Deferred updates by filter (#8485)
* deferred updates by filter

* Don't use points.retain()

---------

Co-authored-by: jojii <jojii@gmx.net>
2026-03-26 18:43:39 +01:00
Ivan Boldyrev
e9695fef2f
Further split segment traits (#8434)
* Refactor: `SearchSegmentEntry` for search ops

* Move `has_deferrend_points_method`

to `SearchSegmentEntry`

* Reorg imports

* Renamings per review

* fixup! Renamings per review

* `StorageSegmentEntry` trait

It contains methods dealing with storage and syncronization

* Move index methods from `ReadSegmentEntry`

to `SegmentEntry`

* Add `_concurrent` suffix to some methods

* move field index methods into NonAppendableSegmentEntry

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-03-26 18:43:32 +01:00
Andrey Vasnetsov
a7fca00f5b
remove incorrect assertion (#8481) 2026-03-26 18:35:17 +01:00
Andrey Vasnetsov
6924ac787f
account for deferred points in proxies (#8477) 2026-03-26 18:34:30 +01:00
Ivan Pleshkov
5e82e2ea8c
Delete points by filter with deferred (#8389)
* delete_points_by_filter with deferred

* versions check

* tests

* Check for deferred points in first loop

* fix after rebase

* refactor

* fix tests

* fix build after rebase

* refactor

* fix test

* refactor

* fix after rebase

* refactor

---------

Co-authored-by: timvisee <tim@visee.me>
2026-03-26 18:34:05 +01:00
Andrey Vasnetsov
0ae7dc5ef0
propagate WAL errors (#8460)
* [AI] propagate WAL errors

* review fixes
2026-03-26 18:33:04 +01:00
Ivan Pleshkov
d73558ed16
Chunked points deduplication (#8464)
* batched points deduplication

* chunks
2026-03-26 18:30:06 +01:00
xzfc
4d560bd88f
Touch-up UIO traits (#8457)
* UIO traits: require Sized

I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.

* Move `AccessPattern` to the `common` package

Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern

* UIO traits: replace `bool` with `AccessPattern`
2026-03-26 18:28:33 +01:00
Ivan Boldyrev
0b26b035b5
Explicit point_mappings guard (#8261)
Instead of `IdTracker` to have `iter_*` methods, it now has a `point_mappings()` method, returning a guard to `PointMapping` value.  It prepares for moving the point_mappings under a lock to allow parallel searches and deletions.

Also, an incorrect unsafe in `Segment::iter_points` is replaced by a safe version with `self_cell`.
2026-03-26 18:26:03 +01:00
Luis Cossío
9bcef50544
Use UniversalIo for MmapBitSlice (#8339)
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`

Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.

Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read

Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage

* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper

Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).

Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark

Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
  file creation + sizing + open (eliminates duplicated size math
  across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
  instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
  on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker

* use bitvec's approach for offset

* less api

* misc improvements

* refactor write method

* move to segment crate

* handle creation of file outside of StoredBitSlice logic

* fix codespell

* document bitwise calculations

* coalesce more updates into a single write, batch all writes

* use native `extend_from_bitslice` instead of iterating.

* clarity refactor

* clippyyy

* use `u64` as BitStore everywhere

* assume iterator is sorted

* only use chunk_by for generating runs

* fix update wrapper flusher

* review fixes

* larger set bits batch test

* remove reintroduced file

* oops, fix new test

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2026-03-26 18:25:42 +01:00
Jojii
39028a9031
Add deferred point count to UpdateQueueInfo (#8414)
* Add deferred point count to UpdateQueueInfo

* Hide if prevent_unoptimized is false

* Clippy

* Openapi

* Iterate over appendable segments only

* Only calculate deferred point count if prevent_unoptimized is true

* Rebase fixes

* Codespell
2026-03-26 18:24:55 +01:00
Jojii
7b196be231
Correct calculation of deferred point counts (#8366)
* Don't account for deferred points in some places

# Conflicts:
#	lib/collection/src/shards/local_shard/scroll.rs

* Add in QueryContext

* Cover more places

* Coderabbit review remarks

* Properly count amount of deleted deferred points (#8386)

* Properly count amount of deleted deferred points

* Prevent double-counting of the same point

* Remove hints to estimations

* Properly handle counts in ProxySegment

* Add tests and fix deleted point count issue

* Adjusts tests + fix issues

* Separte fields for deferred points in telemetry (SegmentInfo)

* Remove deferred_points_count()

* openapi

* Fix test by manually calculating visible points

* Adjust ProxySegment test to revertion of SegmentInfo

* Throw error if collection was not found in telemetry (e2e Test)

* Update lib/segment/src/segment/segment_ops.rs

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* Make new fields in API optional

* Don't take range if no deferred point exist

* Review remarks

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2026-03-26 18:24:36 +01:00
xzfc
09c96ead01
Propagate OperationResult [2/4]: SegmentEntry (#8445) 2026-03-26 18:24:24 +01:00
qdrant-cloud-bot
de9f918b2b
Replace has_deferred_points with deferred_points_count in SegmentEntry (#8391)
* Replace `has_deferred_points` with `deferred_points_count` in SegmentEntry

Change the `SegmentEntry` trait method from returning a boolean to
returning the actual count of deferred points (`usize`). This gives
callers more information without needing a separate method, while
existing boolean checks simply compare against zero.

Made-with: Cursor

* Fix clippy collapsible_if warning in deferred_points_count

Made-with: Cursor

---------

Co-authored-by: Cursor Agent <agent@cursor.com>
2026-03-26 18:15:53 +01:00
qdrant-cloud-bot
c661d9f586
Propagate deferred_internal_id into SegmentInfo and telemetry (#8382)
Add `deferred_internal_id` field to `SegmentInfo` so it is exposed
through segment telemetry. The field is optional and skipped from
serialization when `None`. It is excluded from anonymization via
`#[anonymize(false)]`.

Made-with: Cursor

Co-authored-by: Cursor Agent <agent@cursor.com>
2026-03-26 17:57:53 +01:00
xzfc
da4c15a5fc
DiscoveryQuery -> DiscoverQuery (#8378) 2026-03-26 17:55:57 +01:00
Jojii
24aeed6bbe
Filter deferred points: facets (#8313)
* Filter deferred: facets

* Add test (facet)

* Review remark

* Review remarks

* Fix unique_values returning values even if occurring only in deferred
points

* Clippy

* Rename Filter=>Exclude
2026-03-26 17:51:43 +01:00
Jojii
987db54135
Disable deferred point filtering in resharding (#8310)
* Add ignore_deferred flag for internal scroll API

* Ignore deferred points in ProxyShard::update

* Use enum instead of bool as type in function parameters

* Use consistent parameter name

* fmt

* Apply suggestions from code review

Co-authored-by: Tim Visée <tim+github@visee.me>

* More explicit naming

* Add DeferredBehavior to `count()` and disable filtering in stream_records

* Don't filter `retrieve()` everywhere

* Use consistent filter behavior in read operations

* Add TODO for ignored parameter in remote_shard

* Rebase fixes

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2026-03-26 17:51:34 +01:00
Ivan Pleshkov
4ac2a29723
find_duplicated_points with deferred points (#8343)
* find_duplicated_points with deferred points

* remove trace logs

* fix ci

* add tests

* fix ci after rebase

* are you happy clippy

* refactor

* less changes

* review remarks
2026-03-26 17:51:03 +01:00
Ivan Pleshkov
1ae65ed1a5
deletions with deferred points (#8354) 2026-03-26 17:50:45 +01:00
Arnaud Gourlay
cb6ed23192
Change backpressure policy for deferred points (#8239)
* Change backpressure policy for deferred points

* back pressure waits for deferred points

* Remove prevent_unoptimized_threshold_kb

* tests

* are you happy fmt

* better optimization wait

* fix after rebase

* send signal if optimizer is over the budget

* Make wait_for_deferred_points_ready cancel safe

* fix test

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
2026-03-26 17:50:29 +01:00
Andrey Vasnetsov
df8f846b9d
Deduplicate points after optimization (#8222)
* [manunal] points deduplication after optimization

* take a list of deferred point ids

* test

* review remarks

* review remarks

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
2026-03-26 17:47:21 +01:00
Ivan Pleshkov
3ff0927629
Use predefined deferred ID (#8329)
* Adjust points selection for deferred points update

* adjust proxy segment implementation

* simplify

* use simpler proxy impl

* stick to Entry API

* renaming to stay closer to the original

* two passes and simpler impl.

* fmt

* fmt

* use predefined deferred internal id

* calculate deferred point id

* move deferred check to the entry

* fix after rebase

* fmt

* fix tests

* review remarks

* fix tests

* codespell fix

* are you happy clippy

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-03-26 17:40:56 +01:00
qdrant-cloud-bot
4d23f32681
Introduce EdgeShardConfig for edge shard (#8322)
* Introduce EdgeShardConfig for edge shard

- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
  - Segment config (vector_data, sparse_vector_data, payload_storage_type)
  - Global hnsw_config and per-vector HNSW in segment config
  - Optimizer params: deleted_threshold, vacuum_min_vector_number,
    default_segment_number, max_segment_size, indexing_threshold,
    prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
    max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
  falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
  (update and persist)
- Python and examples use load_with_segment_config with existing config API

Made-with: Cursor

* Refactor EdgeShardConfig: user-facing params only, config module

- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
  - on_disk_payload (bool) instead of payload_storage_type
  - vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
    no per-vector quantization; global quantization_config only
  - sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
  storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
  to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config

Made-with: Cursor

* Move optimizer threshold helpers to shard crate

- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
  get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers

Made-with: Cursor

* Use destructuring in config conversions to avoid missing new fields

- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.

Made-with: Cursor

* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic

- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment

Made-with: Cursor

* refactor(edge): use EdgeShardConfig directly, drop segment_config

- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams

Made-with: Cursor

* [manual] review changes

* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams

- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)

Made-with: Cursor

* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig

- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
  PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
  EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt

Made-with: Cursor

* [manual] review changes

* [manual] review changes

* [manual] fix test

* Address CodeRabbit review comments for PR 8322 (#8324)

* Address CodeRabbit review comments for PR 8322

- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization

Made-with: Cursor

* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer

- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
  - Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
  - set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
  - max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count

Made-with: Cursor

* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration

---------

Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>

* update docstring and pyi

* fmt

* fmt

* clipy

---------

Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2026-03-26 17:40:51 +01:00
Arnaud Gourlay
eaa2ede153
Do not delete source when CoW deferred points (#8290) 2026-03-26 17:40:28 +01:00
Arnaud Gourlay
e75c41c1b9
Adjust points selection for deferred points update (#8279)
* Adjust points selection for deferred points update

* adjust proxy segment implementation

* clean

* simplify

* use simpler proxy impl

* cleanup

* stick to Entry API

* renaming to stay closer to the original

* renaming to stay closer to the original

* nits

* two passes and simpler impl.

* fmt

* fmt

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
2026-03-26 17:40:24 +01:00
Daniel Boros
5d74df25e5
feat/edge segment opt (#8224)
* feat: add edge shard optimize

* feat: refactor edge optimize logic

* chore: remove unused &self

* feat: add more tests

* fix: linter

* fix: missing threshold prop

* fix: local nightly version

* fix: linter

* fix: linter issues

* fix: use of explicit from

* feat: add some notes

* fix: feature_flags call once

* [manual] review refactor

* fix:
- default_segment_number -> move shard
- rename: default_hnsw_config -> hnsw_config
- infer existing hnsw_config

* feat: add optimize to python

* feat: add python optimize example

* feat: add hnsw config load tests

* fix: linter

* feat: make unified build config

* fix: linter

* fix: openapi definition'

* fix: review comments

* fix: remove mut self & reset_temp_segments_dir

* fix: linter

* review: rename for simpler public name + use explicit strucuture deconstruction

* clipy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-03-26 17:39:02 +01:00
qdrant-cloud-bot
07f8533b53
fix: codespell 2.4.2 - pre-selected -> preselected, pre-select -> preselect (#8303)
* fix: codespell 2.4.2 - pre-selected -> preselected, pre-select -> preselect

Fixes CI failure with codespell 2.4.2 which flags hyphenated forms.
Updated in: types.rs, query/mod.rs, qdrant.rs, points.proto

Made-with: Cursor

* chore: regenerate OpenAPI spec (tools/generate_openapi_models.sh)

Updates oversampling description to use preselected spelling.

Made-with: Cursor

---------

Co-authored-by: Cursor Agent <agent@cursor.com>
2026-03-26 17:38:55 +01:00
xzfc
e23cc13d0a
Remove shard dependency on api (#8284)
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`

* Move `OrderByInterface` from `api` to `segment`

Reason: it's used in `edge` which shouldn't depend on `api`.

* Make `shard` -> `api` dependency optional

* Remove `api` from the amalgamation

* Don't install protoc in edge Actions
2026-03-26 17:37:48 +01:00
Tim Visée
6ccd7c8e64
Improve point deduplication loop (#5590)
* Replace deduplication binary heap with kmerge

* Rework deduplication finding, chunk point IDs and only keep highest

* Simplify point group iterator

* Add benchmark

* Fix tests, reverse version

* Reformat
2026-03-26 17:30:56 +01:00
Ivan Pleshkov
cf07d4ef10
Deferred threshold integration (#8246)
* Deferred threshold integration

* update deferred id

* apply update_deferred_internal_id

* fix segment inspector

* use avaliable bytes count

* renamings

* review remarks

* review remarks

* move has_deferred_points

* remove todo

* update comments
2026-03-26 17:26:21 +01:00
generall
b25fc897ff
new formatter rules 2026-03-26 17:26:10 +01:00
dependabot[bot]
a7739dbf2d
build(deps): bump rand_distr from 0.5.1 to 0.6.0 (#8148)
* build(deps): bump rand_distr from 0.5.1 to 0.6.0

Bumps [rand_distr](https://github.com/rust-random/rand_distr) from 0.5.1 to 0.6.0.
- [Release notes](https://github.com/rust-random/rand_distr/releases)
- [Changelog](https://github.com/rust-random/rand_distr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand_distr/compare/0.5.1...0.6.0)

---
updated-dependencies:
- dependency-name: rand_distr
  dependency-version: 0.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Migrate main code base to rand 0.10

* Migrate tests

* Migrate benches

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
2026-03-26 17:09:49 +01:00
xzfc
8fb7adea98
Move github.com/qdrant/wal into lib/wal (#8185)
* Copy github.com/qdrant/wal into lib/wal

Commit: c07fb56ebc8120ebe4e3c602d31ce98f356f4676 2026-02-18

* Clean cruft

* Integrate lib/wal into workspace

* Cargo fmt

* Fix clippy warnings

* Adhere to our conventions

* Fix codespell warnings
2026-03-26 16:53:55 +01:00
Daniel Boros
0be340cdd2
refactor/opt (#8065)
* refactor: opt system

* feat: move tracker seperately

* feat: migrate opt logic to shards

* feat: add OptStrategy trait

* fix: missing drop

* fix: linter error

* fix: clippy errors

* fix: linter error

* fix: missing fns

* fix: linter

* fix: missing import

* fix: missing import

* fix: schema_gen imports

* chore: remove comment

* feat: add missing comments

* feat: restore optimize_segment_propagate_changes fn

* chore: move fs4 to workspace deps

* fix: compiler error

* chore: add import fixes

* feat: move optimzers to shard

* fix: clippy errors

* chore: restore tests and adjust them

* fix: linter issues

* fix: linter issue

* fix: remove CollectionResult where call is infallible

* fix: comment
2026-03-26 16:51:37 +01:00
Arnaud Gourlay
0d9db0ea05
Optimize process search results (#8163) 2026-02-17 17:42:11 +01:00
xzfc
66d0c36009
Merge io and memory into common (#8155)
* Unify parking_lot/arc_lock feature

* Move lib/common/{io,memory}/* -> lib/common/common/*

- Mmap-related items are grouped into `common::mmap` sub-module:
  - `memory/src/chunked_utils.rs`      -> `common/src/mmap/chunked.rs`
  - `memory/src/madvise.rs`            -> `common/src/mmap/advice.rs`
  - `memory/src/mmap_ops.rs`           -> `common/src/mmap/ops.rs`
  - `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
  - `memory/src/mmap_type.rs`          -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
  - `common/src/fs.rs`          -> `common/src/fs/sync.rs`
  - `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
  - `io/src/move_files.rs`      -> `common/src/fs/move.rs`
  - `io/src/safe_delete.rs`     -> `common/src/fs/safe_delete.rs`
  - `memory/src/checkfs.rs`     -> `common/src/fs/check.rs`
  - `memory/src/fadvise.rs`     -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
  - `io/src/storage_version.rs` -> `common/src/storage_version.rs`

The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.

* Replace uses of `io` and `memory` with new paths in `common`

Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.

* Remove `io` and `memory` crates
2026-02-17 11:01:14 +01:00
Ivan Pleshkov
ba5231b48b
Dont lock WAL while serialization (#8093)
* dont lock WAL while serialization

* review remarks

* remove error log, its already logged in update worker
2026-02-13 09:55:46 +01:00
Andrey Vasnetsov
037600e8af
Weighted rrf (#8063)
* weighted rrf implementation

* test

* fmt

* fix edge

* validate number of sources and number of weights

* do not partial match

* upd schema

* review fixes

* update formula

* remove calcualtions from tests

* update comment, because AI have OCD

* fmt
2026-02-10 00:04:33 +01:00
Jojii
65d6b7d016
Don't lock SegmentHolder for the entire duration of read operations (#8056)
* Don't lock SegmentHolder during read operations

* Add comment clarifying eager segment allocation

* review fixes

* Remove TODO

* Shorter locking of segment holder in calculate_local_shard_stats

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-02-10 00:03:52 +01:00
xzfc
b1e3e421b0
Chore: promote dependencies to workspace level (#8061)
* Promote `env_logger` to workspace dependency

* Promote `anyhow` to workspace dependency

* Promote `rmp-serde` to workspace dependency

* Promote `tinyvec` to workspace dependency

* Promote `async-trait` to workspace dependency

* Promote `url` to workspace dependency

* Promote `self_cell` to workspace dependency

* Promote `cc` to workspace dependency

* Promote `bitpacking` to workspace dependency
2026-02-10 00:02:45 +01:00
Ivan Boldyrev
47da1f6661
Split SegmentEntry into appendable and non-appendable part (#8047)
* Split `SegmentEntry`

Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.

* Rename to `NonAppendableSegmentEntry`

It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion.  Move point deletion to the trait too.

* Fix docstring

* use NonAppendableSegmentEntry where possible

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2026-02-10 00:02:32 +01:00
Arnaud Gourlay
0d87284c5c
WAL replay honors applied_seq (#8008)
* WAL replay honors applied_seq

fixes

* add integration test

* restore test after rebase

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
2026-02-09 23:58:00 +01:00
Daniel Boros
7422d20f19
feat/edge-facet-api (#8045)
* feat: add sync facet api

* feat: add facet-tests

* fix: formatting

* chore: remove exact typing

* feat: reduce code duplication

* fix: add missing type def

* fix: serde default value
2026-02-09 23:57:47 +01:00
Andrey Vasnetsov
157aa6596a
faster segment holder locks (#8007)
* non-locking retrieve

* optimization & write exclusive lock via upgradable_read

* faster segment holder locks debug (#8024)

* fix deadlock

* move LockedSegmentHolder into a dedicated file

* reorder locks for snapshot

* fmt

* upd comments

* Update lib/collection/src/collection_manager/optimizers/segment_optimizer.rs

Co-authored-by: Ivan Boldyrev <ivan.boldyrev@qdrant.com>

* Add UpdatesGuard type wrapper for updates_guard (#8031)

* Initial plan

* Add UpdatesGuard newtype wrapper for updates_guard

Co-authored-by: generall <1935623+generall@users.noreply.github.com>

* Fix linter issue - remove trailing space in doc comment

Co-authored-by: generall <1935623+generall@users.noreply.github.com>

* Remove unnecessary new() method, use tuple constructor directly

Co-authored-by: generall <1935623+generall@users.noreply.github.com>

* fix clippy

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: generall <1935623+generall@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>

* Remove aliassed lifetime

Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>

---------

Co-authored-by: Ivan Boldyrev <ivan.boldyrev@qdrant.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: generall <1935623+generall@users.noreply.github.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
2026-02-09 23:56:21 +01:00
Ivan Pleshkov
7758792ced
Drop wal from api (#8000)
* Drop WAL from API

Flatten if statement

another approach

review remarks

switch flag back guard

unit test

* review remarks

* review remarks

* test missing records which are not truncated and not applied

* Drop wal from api: don't use applied seq number (#8030)

* Don't rely on applied seq num

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* remove unused

* Repurpose existing plunger, unset first_skipped in operation branch

* more complicated tests with empty wal

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>

* review remarks

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-09 23:56:09 +01:00
Andrey Vasnetsov
d91e8db5f9
Streaming snapshot unpacking (#8025)
* download tar

* compute sha256 for stream download

* wip: propagate unpacking into down to the logic, todo: validation

* unpacked snapshot validation

* Minor tweaks

* Fix typo

* validation during unpack

* cancellation token

* update docstring

* remove redundant dep

* Rearrange unpack functions

- Rename `safe_unpack.rs` into `tar_unpack.rs` so it would be listed
  near `tar_ext.rs` in IDEs.
- Replace calls like `ar = open_snapshot_archive(…); safe_unpack(ar, …);`
  with a single call to `tar_unpack_file(…)`.
- Put calls to `Archive::new(); Archive::set_overwrite(false);` inside
  `tar_unpack_reader` (was `safe_unpack`). So, now it is the only place
  that does `set_overwrite`.

* Let clippy complain if tar::Archive::unpack used

* Mock snapshot download URL

Instead of downloading from storage.googleapis.com every time the test
runs, put small snapshot file to the repo.

The snapshot file is created using this command:

    curl -s \
      https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot \
    | tar \
      --delete segments/4ea958d8-0b64-4312-9a53-0cd857e93535.tar \
      --delete segments/65ac6276-8cca-4f5c-b767-9722190cee8b.tar \
      > lib/storage/src/content_manager/snapshots/test-shard.snapshot

File contents:

    $ tar tf lib/storage/src/content_manager/snapshots/test-shard.snapshot
    wal/
    wal/closed-255
    newest_clocks.json
    replica_state.json
    shard_config.json
    
    $ du -sh lib/storage/src/content_manager/snapshots/test-shard.snapshot
    12K	lib/storage/src/content_manager/snapshots/test-shard.snapshot
    
    $ sha256sum < lib/storage/src/content_manager/snapshots/test-shard.snapshot       
    5d94eac5c1ede3994a28bc406120046c37370d5d45b489a0d2252531b4e3e1f2  -

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
2026-02-09 23:53:16 +01:00
Arnaud Gourlay
72dd7c8cbe
Persist applied seq at regular interval (#7976)
* Persist applied seq at regular interval

* Error handling

* fix brain fart

* don't reset update_count

* handles explicitly failure to load

* init from WAL instead of zero

* fixes

* try to regenerate file on corruption

* fix

* trust the WAL if higher watermark most likely from a transfer

* use last WAL index & handles file shard lifecycle

* last WAL index is inclusive

* include applied_seq in partial snapshot

* snapshot applied_seq file

* fix

* remove outdated comment

* Describe AppliedSeq structure, it is used for persisting

---------

Co-authored-by: timvisee <tim@visee.me>
2026-02-09 23:38:44 +01:00