qdrant/lib/edge/python/examples/restore-snapshot.py

68 lines
2.0 KiB
Python
Raw Permalink Normal View History

#!/usr/bin/env python3
# See lib/edge/publish/examples/src/bin/restore-snapshot.rs for the equivalent Rust example.
import os
import requests
import shutil
import json
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
from common import DATA_DIR
from qdrant_edge import *
SNAPSHOT_URL = 'https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot'
# Obtained via download_partial.py and manually inserting a new point with ID 100500
PARTIAL_SNAPSHOT_URL = 'https://storage.googleapis.com/qdrant-benchmark-snapshots/partial.snapshot'
# Download the snapshot file into data directory
def download_snapshot(url: str, dest_folder: str):
if not os.path.exists(dest_folder):
os.makedirs(dest_folder)
local_filename = os.path.join(dest_folder, url.split('/')[-1])
if os.path.exists(local_filename):
print(f"Snapshot already exists at: {local_filename}")
return local_filename
with requests.get(url, stream=True, timeout = 10) as r:
r.raise_for_status()
with open(local_filename, 'wb') as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
return local_filename
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
snapshot_path = download_snapshot(SNAPSHOT_URL, DATA_DIR)
print(f"Snapshot downloaded to: {snapshot_path}")
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
recovered_path = os.path.join(DATA_DIR, "restored_shard")
print(f"Restoring shard from snapshot to: {recovered_path}")
if os.path.exists(recovered_path):
print("Removing existing recovered shard directory...")
shutil.rmtree(recovered_path)
EdgeShard.unpack_snapshot(snapshot_path, recovered_path)
refactor(edge): split EdgeShard load into new() and load() (#8326) * refactor(edge): split EdgeShard load into new() and load() - new(path, config): create edge shard only when path has no existing segments - load(path, config?): load from existing files; config optional (load from edge_config.json or infer from segments) - Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config, load_segments, ensure_appendable_segment - Python: EdgeShard.create_new(path, config); __init__ still calls load() - Examples use new() for creation and load(path, None) for reopen - Error instead of panic on config mismatch; explicit load/create in Python - fix: use OptimizerThresholds.deferred_internal_id for dev compatibility Made-with: Cursor * rollback threshold changes * fix(edge): address dancixx review comments - Persist inferred config in load() when edge_config.json does not exist (SaveOnDisk::new only saves when file exists) - Python: add #[new] delegating to load() for backward compatibility - qdrant_edge.pyi: Optional return types for deleted_threshold, vacuum_min_vector_number, default_segment_number; add __init__ doc Made-with: Cursor * Revert "fix(edge): address dancixx review comments" This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c. * fix: optional returns * fix: save config it is not exist * fix: save data on disk always * fix: python examples * chore: remove edge.close() * fix: wal lock * fix: rename of EdgeShardConfig -> EdgeConfig --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Daniel Boros <dancixx@gmail.com>
2026-03-18 15:25:01 +01:00
shard = EdgeShard.load(recovered_path)
points = shard.retrieve(point_ids=[1, 2, 3], with_vector=False, with_payload=True)
for point in points:
print(point)
print("Manifest of restored shard:", json.dumps(shard.snapshot_manifest(), indent=2))
Test edge examples on CI (#8318) * lib/edge/publish/cargo: improve target directory discovery Don't hardcode workspace root, ask cargo instead. * Fix warning in qdrant-edge amalgamation `cargo check` in `lib/edge/publish` produces the following warning: warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending` * lib/edge/publish/examples: fix compiler warnings * refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs To match python examples that have the same constant in `common.py`. * Use `lib/edge/data` dir for both Python and Rust examples Before this commit, `lib/edge/publish/data` and `lib/edge/python/data` were used. I'd like Python and Rust to use the same prepared resources in the future. * ❗Add lib/edge/Justfile with recipes to build/run examples I constantly keep forgetting how to compile and run edge stuff. Also, this would be used in CI in subsequent commits. * ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes Also now we can rename `creates-io-readme.md` to `README.md` like reasonable people. * ❗doc: Add qdrant-edge-py README.md Since the previous commit removed qdrant-edge-py README.md, we need to add something in its place instead. The new README is adapted from Rust edge README. And the previous README wasn't good anyway, see [1], it mentions maturin which is irrelevant for users. [1]: https://pypi.org/project/qdrant-edge-py/0.5.0/ * GHA: Add "Setup Qdrant" composite action We need running Qdrant instance to test edge examples on CI. We can't use docker for this because we want to run tests on Windows and macOS. OTOH, this action downloads binaries for all platforms provided in https://github.com/qdrant/qdrant/releases/tag/v1.17.0 * ❗GHA: Use single worflow for python and rust edge examples Merge workflows to reuse build cache. Also, modernize it by using `uv`, and recently introduced `setup-qdrant` and Justfile. Also, make it run on three platforms (only on merges). * ❗GHA: edge-{py,rust}-release: also run examples before publishing Use similar approach as in the previous commit. But note that this time we build/test also on ARM Linux and Windows.
2026-03-11 17:13:46 +00:00
partial_snapshot_path = download_snapshot(PARTIAL_SNAPSHOT_URL, DATA_DIR)
shard.update_from_snapshot(partial_snapshot_path)
points = shard.retrieve(point_ids=[100500], with_vector=False, with_payload=True)
for point in points:
print(point)
info = shard.info()
print(info)