* feat: add search_max_batchsize to strict mode config
* added test case for search_max_batchsize
* Changes for fixing CI issue dure openapi
* Modify check_strict_mode_batch
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Remove UniversalRead::is_empty
Reason: Less methods to override in wrappers. And I don't think this
method makes much sense.
* Move common::universal_io::{ => wrappers}::read_only
* Update ReadOnly wrapper
Implement missing methods, and follow the code style of the upcoming
TypedStrorage wrapper.
* Add TypedStorage
* Use TypedStorage
* Add reminder comments
* Clarify TypedStorage use-case
Two bugs introduced in 9a14ed7:
1. `HashSet::insert` returns `true` when the value is *new*, but the
code assigned it to `already_seen` and skipped on `true` — inverting
the deduplication so every unique geohash was skipped during removal.
2. `filter()` returns `OperationResult<Option<Box<dyn Iterator>>>`;
the new tests called `.unwrap()` once (unwrapping the Result) but
missed the second `.unwrap()` for the Option, causing a compile error.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add reproducing test for spurious geo index warning on duplicate geo values
When a point has multiple geo values that produce the same max-precision
geohash (e.g. duplicate coordinates in a multi-value geo field),
`InMemoryGeoMapIndex::remove_point` logs a spurious warning:
"Geo index error: no points for hash X was found".
The root cause is an asymmetry between `point_to_values` (stores all
values including duplicates) and `points_map` (uses a HashSet, so
duplicates are collapsed). During removal the loop processes each value
individually — the first iteration removes the `points_map` entry, and
subsequent iterations for the same hash can't find it.
This commit adds:
- A `debug_assert!` in `remove_point` (matching the existing pattern in
`decrement_hash_value_counts`) to make the issue detectable in tests.
- `test_remove_point_with_duplicate_geo_values` — reproduces the bug by
adding a point with `[BERLIN, BERLIN]` then removing it.
- `test_frequent_add_remove_geo_points` — exercises repeated add/remove
cycles to cover the user-reported scenario.
Made-with: Cursor
* fix: redulplicate hash IDs, replace warn with debug assertion
* fmt
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Snapshot endpoints used raw `web::Path<String>` / `web::Path<(String, ...)>`
instead of validated path types, allowing invalid collection names (including
NUL bytes) to bypass `validate_collection_name` checks and reach the
filesystem, causing 500 errors.
Replace all raw path extractors with `valid::Path<CollectionPath>` and new
validated structs (`CollectionSnapshotPath`, `CollectionShardPath`,
`CollectionShardSnapshotPath`) that enforce legacy collection-name validation,
consistent with the rest of the API.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Don't insert deferred points into sparse index
# Conflicts:
# lib/segment/src/segment/tests.rs
# lib/segment/src/segment_constructor/segment_constructor_base.rs
# Conflicts:
# lib/segment/src/segment_constructor/segment_constructor_base.rs
* Clippy
* Assert consistency of deferred_internal_id var
* Return before SparseVector conversion in case of deferred point
* Use debug_assert instead
* Add `#[must_use]` to RAII guard and task handle types
These types rely on being held for a scope — dropping them immediately is
almost always a bug:
- `StoppingGuard`: sets `is_stopped` flag on drop
- `UpdateGuard`: decrements update counter on drop
- `UpdatesGuard`: releases mutex preventing concurrent updates on drop
- `ClockGuard`: marks clock as inactive on drop
- `IsAliveGuard`: releases liveness lock on drop
- `ScopeTrackerGuard`: decrements scope counter on drop
- `CancellableAsyncTaskHandle`: detaches task on drop
- `StoppableTaskHandle`: may abort task on drop (AbortOnDropHandle)
Made-with: Cursor
* Remove redundant function-level #[must_use] now covered by type
With ScopeTrackerGuard marked #[must_use] at the type level, the
function-level attributes on measure_scope(), measure(),
track_create_snapshot_request(), and count_snapshot_creation() are
redundant and trigger clippy::double_must_use.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add REST API for reading audit logs across the cluster
Introduces a new `GET /audit/logs` endpoint that retrieves audit log
entries from all peers in the cluster. The API supports filtering by
time range (time_from/time_to), dynamic key=value field filters, and
a built-in limit parameter to prevent reading too many logs at once.
- Add `audit_reader` module in storage crate for efficient file-based
log retrieval, selecting only files whose date range overlaps the
query window
- Add `GetAuditLog` internal gRPC RPC for cross-peer log retrieval
- Add `GET /audit/logs` REST endpoint restricted to management access
- Aggregate and sort results from all peers by timestamp (newest first)
Made-with: Cursor
* [AI] Update filter_files_by_time_range make it aware of the log rotation granulatity (either hourly or daily)
* [AI] For AuditLogParams, try to use DateTime types natively into serde, instead of manual parsing
* [AI] Refactor API into separate file, propagate timeout, use spawn-blocking
* [AI] introduce cancellation token
* [AI] move timestamp to constant
* small manual fixes
* review fixes part 1
* review: switch to POST instead of GET
* [AI] review: sorting update
* [AI] use strict typing
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Enable async_scorer in integration tests
Enable `async_scorer=true` for all openapi and shell-based integration
tests by setting the environment variable in `integration-tests.sh`.
This exercises the io_uring vector storage path (on Linux) across all
vector configurations tested there, with graceful fallback if io_uring
is unavailable on the CI runner.
Made-with: Cursor
* Scope async_scorer to test_points_search consensus test only
Revert the blanket async_scorer=true from integration-tests.sh to keep
the normal mmap path covered. Instead, enable it via extra_env in just
the test_points_search consensus test, so we exercise the io_uring path
without losing coverage of the non-io_uring path on Linux.
Made-with: Cursor
* Parametrize test_points_search for sync/async scorer modes
Use pytest.mark.parametrize to run test_points_search twice: once with
the default scorer (sync_scorer) and once with async_scorer=true. This
exercises both the normal mmap and io_uring paths in a single test.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Test all shard transfer methods for deferred points
* WAL delta carries deferred points
* add point count assertions
- snapshot: asserts target_visible == source_visible — deferred state preserved exactly
- stream_records: asserts target_visible >= source_visible — points may not be deferred on target
* Adjust debug logline's format of
ShardTransferRestart to match with ShardTransfer
* Convert into shardtransfer
* Deconstruct ShardTransferRestart struct
* [ai] Allow peer to bootstrap with existing URI if peer has no shards
Prompt:
I have a Qdrant cluster in distributed mode with some nodes registered
in consensus. If I bootstrap a new node with an URL that is already used
it is rejected and an error is returned. I would like to change this
behavior and allow this to happen. This would effectively replace a peer
because internally we'd drop the existing peer first, and then we'd add
the new peer so we can reuse the same peer URL. We must still reject
bootstrapping with the same URL if the peer that used the URL before us
still has any shards on it.
* [ai] Add test to assert new behavior, can rejoin if empty
Prompt:
Add two tests to assert the new behavior.
The first test should:
- create a cluster
- create a collection
- bootstrap a new peer
- kill and delete the local data for this peer without removing it from consensus
- bootstrap a new peer but reuse the URI of the peer we just killed to rejoin
- bootstrapping is expected to succeed
The second test should:
- create a cluster
- create a collection
- bootstrap a new peer but reuse the URI of the last node
- bootstrapping is expected to fail because the node being replaced has data on it
* [ai] Attempt to fix new tests
* Reformat
* [ai] Fix deadlock when rejoining with same peer URI
* [ai] Add test to ensure existing peer stops consensus on replace
* delete_points_by_filter with deferred
* versions check
* tests
* Check for deferred points in first loop
* fix after rebase
* refactor
* fix tests
* fix build after rebase
* refactor
* fix test
* refactor
* fix after rebase
* refactor
---------
Co-authored-by: timvisee <tim@visee.me>