92 lines
2.6 KiB
Python
Raw Permalink Normal View History

# Common helper items for the examples.
# See lib/edge/publish/examples/src/lib.rs for the equivalent Rust helpers.
import os
import shutil
import uuid
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
from pathlib import Path
Introduce EdgeShardConfig for edge shard (#8322) * Introduce EdgeShardConfig for edge shard - Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs - Segment config (vector_data, sparse_vector_data, payload_storage_type) - Global hnsw_config and per-vector HNSW in segment config - Optimizer params: deleted_threshold, vacuum_min_vector_number, default_segment_number, max_segment_size, indexing_threshold, prevent_unoptimized (excludes memmap_threshold, flush_interval_sec, max_optimization_threads) - Persist/load as edge_config.json in shard path - EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>, falls back to file or infer from segments; compatibility checked on load - load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig) - optimize() uses EdgeShardConfig for hnsw and optimizer thresholds - Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config() (update and persist) - Python and examples use load_with_segment_config with existing config API Made-with: Cursor * Refactor EdgeShardConfig: user-facing params only, config module - Replace SegmentConfig inside EdgeShardConfig with user-facing fields: - on_disk_payload (bool) instead of payload_storage_type - vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector, no per-vector quantization; global quantization_config only - sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk - EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in to_segment_config() - Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs - from_segment_config(&SegmentConfig) fills all inferrable params - to_segment_config() builds SegmentConfig for segments and optimize() - load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config Made-with: Cursor * Move optimizer threshold helpers to shard crate - Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb, get_deferred_points_threshold_bytes in shard::optimizers::config - Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these - Single place for threshold logic; collection and edge use shard helpers Made-with: Cursor * Use destructuring in config conversions to avoid missing new fields - EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config - EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config - EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config Adding new fields to source structs will now cause compile errors until conversions are updated. Made-with: Cursor * refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic - PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap) - VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap) - QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable) - collection: use from_on_disk_payload in non-rocksdb branch - edge shard/vectors: use new helpers; remove duplicated conditionals - shard optimizers: use from_on_disk and for_appendable_segment Made-with: Cursor * refactor(edge): use EdgeShardConfig directly, drop segment_config - Add plain_segment_config() for create_appendable_segment (no HNSW) - Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers - Add vector_data_config(name) for query/MMR - build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig - create_appendable_segment: use plain_segment_config() - search/query: use config().vectors and vector_data_config() instead of segment_config() - Remove segment_config() from EdgeShardConfig and EdgeShard - Add to_plain_vector_data_config on EdgeVectorParams Made-with: Cursor * [manual] review changes * refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams - PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.) - PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types - PyEdgeOptimizersConfig for optional optimizer settings - EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate - cargo fmt + clippy (remove map_identity) Made-with: Cursor * refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig - Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType, PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings - Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs - Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors, EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig - Update examples (common.py, repr.py) to use new config API - Run cargo fmt Made-with: Cursor * [manual] review changes * [manual] review changes * [manual] fix test * Address CodeRabbit review comments for PR 8322 (#8324) * Address CodeRabbit review comments for PR 8322 - Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API - HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings - EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors - EdgeShardConfig::load: use try_exists(), propagate I/O errors - from_segment_config: infer hnsw_config from per-vector HNSW when all agree - EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors - Segment compat: prefix vector name in error messages; resolve None datatype to Float32 - max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides - SegmentOptimizerConfig::new: build plain and optimizer maps in single pass - config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into() - vectors.rs: doc updates for per-vector quantization Made-with: Cursor * Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer - Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this') - Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist - set_vector_hnsw_config: clone then mutate then write (fallible setter) - max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl') - max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count Made-with: Cursor * [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: generall <andrey@vasnetsov.com> * update docstring and pyi * fmt * fmt * clipy --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: generall <andrey@vasnetsov.com>
2026-03-10 00:10:04 +01:00
from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
EdgeVectorParams,
Point,
UpdateOperation,
)
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
DATA_DIR = Path(__file__).parent.parent.parent / "data"
TMP_DIR = DATA_DIR / "tmp"
refactor(edge): split EdgeShard load into new() and load() (#8326) * refactor(edge): split EdgeShard load into new() and load() - new(path, config): create edge shard only when path has no existing segments - load(path, config?): load from existing files; config optional (load from edge_config.json or infer from segments) - Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config, load_segments, ensure_appendable_segment - Python: EdgeShard.create_new(path, config); __init__ still calls load() - Examples use new() for creation and load(path, None) for reopen - Error instead of panic on config mismatch; explicit load/create in Python - fix: use OptimizerThresholds.deferred_internal_id for dev compatibility Made-with: Cursor * rollback threshold changes * fix(edge): address dancixx review comments - Persist inferred config in load() when edge_config.json does not exist (SaveOnDisk::new only saves when file exists) - Python: add #[new] delegating to load() for backward compatibility - qdrant_edge.pyi: Optional return types for deleted_threshold, vacuum_min_vector_number, default_segment_number; add __init__ doc Made-with: Cursor * Revert "fix(edge): address dancixx review comments" This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c. * fix: optional returns * fix: save config it is not exist * fix: save data on disk always * fix: python examples * chore: remove edge.close() * fix: wal lock * fix: rename of EdgeShardConfig -> EdgeConfig --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Daniel Boros <dancixx@gmail.com>
2026-03-18 15:25:01 +01:00
def load_new_shard():
refactor(edge): split EdgeShard load into new() and load() (#8326) * refactor(edge): split EdgeShard load into new() and load() - new(path, config): create edge shard only when path has no existing segments - load(path, config?): load from existing files; config optional (load from edge_config.json or infer from segments) - Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config, load_segments, ensure_appendable_segment - Python: EdgeShard.create_new(path, config); __init__ still calls load() - Examples use new() for creation and load(path, None) for reopen - Error instead of panic on config mismatch; explicit load/create in Python - fix: use OptimizerThresholds.deferred_internal_id for dev compatibility Made-with: Cursor * rollback threshold changes * fix(edge): address dancixx review comments - Persist inferred config in load() when edge_config.json does not exist (SaveOnDisk::new only saves when file exists) - Python: add #[new] delegating to load() for backward compatibility - qdrant_edge.pyi: Optional return types for deleted_threshold, vacuum_min_vector_number, default_segment_number; add __init__ doc Made-with: Cursor * Revert "fix(edge): address dancixx review comments" This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c. * fix: optional returns * fix: save config it is not exist * fix: save data on disk always * fix: python examples * chore: remove edge.close() * fix: wal lock * fix: rename of EdgeShardConfig -> EdgeConfig --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Daniel Boros <dancixx@gmail.com>
2026-03-18 15:25:01 +01:00
"""Create a new edge shard (clears DATA_DIRECTORY first)."""
print("---- Load shard ----")
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
# Clear and recreate tmp directory
if os.path.exists(TMP_DIR):
shutil.rmtree(TMP_DIR)
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
os.makedirs(TMP_DIR)
config = EdgeConfig(
Introduce EdgeShardConfig for edge shard (#8322) * Introduce EdgeShardConfig for edge shard - Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs - Segment config (vector_data, sparse_vector_data, payload_storage_type) - Global hnsw_config and per-vector HNSW in segment config - Optimizer params: deleted_threshold, vacuum_min_vector_number, default_segment_number, max_segment_size, indexing_threshold, prevent_unoptimized (excludes memmap_threshold, flush_interval_sec, max_optimization_threads) - Persist/load as edge_config.json in shard path - EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>, falls back to file or infer from segments; compatibility checked on load - load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig) - optimize() uses EdgeShardConfig for hnsw and optimizer thresholds - Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config() (update and persist) - Python and examples use load_with_segment_config with existing config API Made-with: Cursor * Refactor EdgeShardConfig: user-facing params only, config module - Replace SegmentConfig inside EdgeShardConfig with user-facing fields: - on_disk_payload (bool) instead of payload_storage_type - vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector, no per-vector quantization; global quantization_config only - sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk - EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in to_segment_config() - Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs - from_segment_config(&SegmentConfig) fills all inferrable params - to_segment_config() builds SegmentConfig for segments and optimize() - load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config Made-with: Cursor * Move optimizer threshold helpers to shard crate - Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb, get_deferred_points_threshold_bytes in shard::optimizers::config - Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these - Single place for threshold logic; collection and edge use shard helpers Made-with: Cursor * Use destructuring in config conversions to avoid missing new fields - EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config - EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config - EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config Adding new fields to source structs will now cause compile errors until conversions are updated. Made-with: Cursor * refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic - PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap) - VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap) - QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable) - collection: use from_on_disk_payload in non-rocksdb branch - edge shard/vectors: use new helpers; remove duplicated conditionals - shard optimizers: use from_on_disk and for_appendable_segment Made-with: Cursor * refactor(edge): use EdgeShardConfig directly, drop segment_config - Add plain_segment_config() for create_appendable_segment (no HNSW) - Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers - Add vector_data_config(name) for query/MMR - build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig - create_appendable_segment: use plain_segment_config() - search/query: use config().vectors and vector_data_config() instead of segment_config() - Remove segment_config() from EdgeShardConfig and EdgeShard - Add to_plain_vector_data_config on EdgeVectorParams Made-with: Cursor * [manual] review changes * refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams - PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.) - PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types - PyEdgeOptimizersConfig for optional optimizer settings - EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate - cargo fmt + clippy (remove map_identity) Made-with: Cursor * refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig - Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType, PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings - Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs - Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors, EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig - Update examples (common.py, repr.py) to use new config API - Run cargo fmt Made-with: Cursor * [manual] review changes * [manual] review changes * [manual] fix test * Address CodeRabbit review comments for PR 8322 (#8324) * Address CodeRabbit review comments for PR 8322 - Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API - HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings - EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors - EdgeShardConfig::load: use try_exists(), propagate I/O errors - from_segment_config: infer hnsw_config from per-vector HNSW when all agree - EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors - Segment compat: prefix vector name in error messages; resolve None datatype to Float32 - max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides - SegmentOptimizerConfig::new: build plain and optimizer maps in single pass - config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into() - vectors.rs: doc updates for per-vector quantization Made-with: Cursor * Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer - Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this') - Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist - set_vector_hnsw_config: clone then mutate then write (fallible setter) - max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl') - max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count Made-with: Cursor * [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: generall <andrey@vasnetsov.com> * update docstring and pyi * fmt * fmt * clipy --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: generall <andrey@vasnetsov.com>
2026-03-10 00:10:04 +01:00
vectors=EdgeVectorParams(size=4, distance=Distance.Dot),
)
refactor(edge): split EdgeShard load into new() and load() (#8326) * refactor(edge): split EdgeShard load into new() and load() - new(path, config): create edge shard only when path has no existing segments - load(path, config?): load from existing files; config optional (load from edge_config.json or infer from segments) - Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config, load_segments, ensure_appendable_segment - Python: EdgeShard.create_new(path, config); __init__ still calls load() - Examples use new() for creation and load(path, None) for reopen - Error instead of panic on config mismatch; explicit load/create in Python - fix: use OptimizerThresholds.deferred_internal_id for dev compatibility Made-with: Cursor * rollback threshold changes * fix(edge): address dancixx review comments - Persist inferred config in load() when edge_config.json does not exist (SaveOnDisk::new only saves when file exists) - Python: add #[new] delegating to load() for backward compatibility - qdrant_edge.pyi: Optional return types for deleted_threshold, vacuum_min_vector_number, default_segment_number; add __init__ doc Made-with: Cursor * Revert "fix(edge): address dancixx review comments" This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c. * fix: optional returns * fix: save config it is not exist * fix: save data on disk always * fix: python examples * chore: remove edge.close() * fix: wal lock * fix: rename of EdgeShardConfig -> EdgeConfig --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Daniel Boros <dancixx@gmail.com>
2026-03-18 15:25:01 +01:00
return EdgeShard.create(TMP_DIR, config)
def fill_dummy_data(shard: EdgeShard):
refactor(edge): split EdgeShard load into new() and load() (#8326) * refactor(edge): split EdgeShard load into new() and load() - new(path, config): create edge shard only when path has no existing segments - load(path, config?): load from existing files; config optional (load from edge_config.json or infer from segments) - Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config, load_segments, ensure_appendable_segment - Python: EdgeShard.create_new(path, config); __init__ still calls load() - Examples use new() for creation and load(path, None) for reopen - Error instead of panic on config mismatch; explicit load/create in Python - fix: use OptimizerThresholds.deferred_internal_id for dev compatibility Made-with: Cursor * rollback threshold changes * fix(edge): address dancixx review comments - Persist inferred config in load() when edge_config.json does not exist (SaveOnDisk::new only saves when file exists) - Python: add #[new] delegating to load() for backward compatibility - qdrant_edge.pyi: Optional return types for deleted_threshold, vacuum_min_vector_number, default_segment_number; add __init__ doc Made-with: Cursor * Revert "fix(edge): address dancixx review comments" This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c. * fix: optional returns * fix: save config it is not exist * fix: save data on disk always * fix: python examples * chore: remove edge.close() * fix: wal lock * fix: rename of EdgeShardConfig -> EdgeConfig --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Daniel Boros <dancixx@gmail.com>
2026-03-18 15:25:01 +01:00
shard.update(
UpdateOperation.upsert_points(
[
Point(
1,
[0.05, 0.61, 0.76, 0.74],
{"color": "red", "city": ["Moscow", "Berlin"]},
),
Point(2, [0.19, 0.81, 0.75, 0.11], {"color": "red", "city": "Mexico"}),
Point(
3,
[0.36, 0.55, 0.47, 0.94],
{"color": "blue", "city": ["Berlin", "Barcelona"]},
),
Point(
4,
[0.12, 0.34, 0.56, 0.78],
{"color": "green", "city": "Lisbon", "rating": 4.5},
),
Point(
5,
[0.88, 0.12, 0.33, 0.44],
{"color": "yellow", "city": ["Paris"], "active": True},
),
Point(
6,
[0.21, 0.22, 0.23, 0.24],
{"color": "blue", "city": "Tokyo", "tags": ["night", "food"]},
),
Point(
7,
[0.99, 0.01, 0.50, 0.50],
{"color": "red", "city": ["New York", "Boston"], "visits": 7},
),
Point(
8,
[0.10, 0.20, 0.30, 0.40],
{"color": "blue", "city": "Seoul", "meta": {"source": "import"}},
),
Point(
9,
[0.45, 0.55, 0.65, 0.75],
{"color": "green", "city": ["Berlin"], "score": 0.92},
),
Point(
10,
[0.01, 0.02, 0.03, 0.04],
{"color": "yellow", "city": None, "featured": False},
),
]
)
)