2026-08-17 13:40:09 +03:00

1.5 KiB

Partitioning

lawsynth_connectors.partitioning plans deterministic row and time partitions so large sources can be read in reproducible, evenly sized slices. Planning is pure: the same inputs always yield the same partitions, independent of any driver.

Row partitions

from lawsynth_connectors.partitioning import plan_row_partitions

plan_row_partitions(total_rows=250, target_rows=100)
# (RowPartition(0, 0, 100), RowPartition(1, 100, 200), RowPartition(2, 200, 250))

Each RowPartition is a half-open [start, stop) range with a positive size; the final partition absorbs the remainder. total_rows may be zero (yielding no partitions); target_rows must be positive.

Time partitions

from datetime import datetime, timedelta, timezone
from lawsynth_connectors.partitioning import plan_time_partitions

plan_time_partitions(
    datetime(2024, 1, 1, tzinfo=timezone.utc),
    datetime(2024, 1, 3, tzinfo=timezone.utc),
    timedelta(days=1),
)

Both bounds must be timezone-aware; they are normalized to UTC before slicing. Each TimePartition is a half-open [start, stop) interval, and the last one is clamped to the requested end. An empty or reversed range, a non-positive interval, or a naive datetime raises ValueError.

Why it matters

Partitions give connectors and downstream jobs a stable unit of parallelism and retry. Because boundaries are deterministic, a re-read of the same partition index produces the same rows or time window, which keeps fingerprints and provenance consistent across runs.