16 KiB
Python Port Specification: Taxonomy Validation and Export
Overview
This specification describes how to port the Zig-based crypto-ecosystems taxonomy validation and export system to Python, using the Python standard library wherever possible.
Architecture
The system consists of three main components:
- CLI Command Handler - Parses and executes user commands
- Taxonomy Core - Manages ecosystem/repository relationships and migration processing
- Utilities - Helper functions for timestamp validation and shell-like lexing
1. CLI Command Handler (commands.py)
Purpose
Parse command-line arguments and dispatch to appropriate handlers.
Commands
validate
Validates all migrations in a directory and prints statistics.
Options:
-r, --root DIR: Directory containing migration files (default:./migrations)-h, --help: Show help message
Output:
┃ N Migrations
┃ N Ecosystems
┃ N Repos
┃ N Tags
export
Exports the taxonomy to a JSON Lines file.
Options:
-r, --root DIR: Directory containing migration files (default:./migrations)-e, --ecosystem STR: Export only a specific ecosystem-m, --max-date STR: Maximum date to run migrations until (YYYY-MM-DD format)<output>: Output file path (required, positional argument)
Output Format: JSON Lines with each line containing:
{
"eco_name": "Bitcoin",
"branch": ["Lightning", "Lightspark"],
"repo_url": "https://github.com/...",
"tags": ["#protocol", "#sdk"]
}
Implementation Details
Standard Library Modules:
argparse: Command-line argument parsingsys: Exit codes and stderr outputos.path: Path operations
Error Handling:
MissingOutputPath: When export command lacks output fileInvalidCommand: Unknown commandInvalidOption: Invalid option providedDuplicateOption: Option specified multiple times
Argument Parsing:
- Use
argparse.ArgumentParserwith subparsers for each command - Validate that options aren't duplicated (track with seen flags)
- For export command, ensure output path is provided and is the last argument
- Validate that option values don't start with
-
2. Taxonomy Core (taxonomy.py)
Data Structures
Use dictionaries and sets to represent the taxonomy:
class Taxonomy:
def __init__(self):
self.eco_auto_id: int = 0
self.repo_auto_id: int = 0
self.tag_auto_id: int = 0
self.migration_count: int = 0
# String to ID mappings
self.eco_ids: dict[str, int] = {}
self.repo_ids: dict[str, int] = {}
self.tag_ids: dict[str, int] = {}
# ID to string mappings
self.tag_id_to_name: dict[int, str] = {}
self.repo_id_to_url: dict[int, str] = {}
self.eco_id_to_name: dict[int, str] = {}
# Relationship mappings
self.eco_to_repos: dict[int, set[int]] = {}
self.repo_to_eco: dict[int, int] = {}
self.parent_to_children: dict[int, set[int]] = {}
self.child_to_parents: dict[int, set[int]] = {}
self.eco_repo_to_tags: dict[tuple[int, int], set[int]] = {}
# Error tracking
self.errors: list[dict] = []
Migration Commands
The system processes migration files with the following commands:
-
ecoadd
<ecosystem_name>- Adds a new ecosystem to the taxonomy
-
repadd
<ecosystem_name><repo_url>[<tag1>] [<tag2>] ...- Adds a repository to an ecosystem with optional tags
-
ecocon
<parent_ecosystem><child_ecosystem>- Connects a child ecosystem to a parent (creates hierarchy)
-
ecodis
<parent_ecosystem><child_ecosystem>- Disconnects a child ecosystem from a parent
-
ecorom
<ecosystem_name>- Removes an ecosystem (preserves child ecosystems as orphans)
-
repmov
<old_repo_url><new_repo_url>- Renames/moves a repository URL
-
ecomov
<old_ecosystem_name><new_ecosystem_name>- Renames an ecosystem
-
reprem
<ecosystem_name><repo_url>- Removes a repository from an ecosystem
Load Process
def load(self, root_dir: str, max_date: str | None = None) -> None:
"""
Load and process migration files from root_dir.
Steps:
1. List all files in directory
2. Filter files with valid timestamps (YYYY-MM-DDTHHMMSS format)
3. If max_date provided, filter files before that date
4. Sort files by timestamp (first 19 characters)
5. Process each file line by line
6. Track errors during processing
7. If errors exist, print them and raise ValidationFailed
"""
Standard Library Modules:
os: Directory listing withos.listdir()os.path: Path joining withos.path.join()re: Pattern matching for timestamp validation (optional, can use string slicing)
File Processing:
- Read entire file into memory
- Split by newlines
- Skip comment lines (lines starting with
#after whitespace) - Skip lines shorter than 6 characters
- Parse command (first 6 characters) and remainder
- Execute appropriate command handler
- Catch exceptions and append to errors list with line number and path
Export Process
def export_json(self, output_file: str, ecosystem: str | None = None) -> None:
"""
Export taxonomy to JSON Lines format.
Steps:
1. Collect ecosystems to export (all or specific one)
2. Sort ecosystems alphabetically (case-insensitive)
3. For each ecosystem, emit repositories and recursively emit sub-ecosystems
4. Write to file using buffered I/O
"""
Standard Library Modules:
json: JSON serialization withjson.dumps()io: Buffered file writing for performance
Output Format:
- One JSON object per line (JSON Lines format)
- Each object represents a repository in an ecosystem
branchfield shows the hierarchy path from top ecosystem to current level- Repositories and sub-ecosystems sorted case-insensitively
Recursive Emission:
- Emit all repositories at current ecosystem level
- For each sub-ecosystem:
- Append sub-ecosystem name to branch list
- Recursively emit that sub-ecosystem
- Pop from branch list
Statistics
def stats(self) -> dict:
"""
Return statistics about the taxonomy.
Returns:
{
'migration_count': int,
'eco_count': int,
'repo_count': int,
'tag_count': int
}
"""
3. Utilities
Timestamp Validation (timestamp.py)
def has_valid_timestamp(filename: str) -> bool:
"""
Validate that filename starts with YYYY-MM-DDTHHMMSS format.
Validation rules:
- Minimum length of 17 characters
- Pattern: YYYY-MM-DDTHHMMSS where Y/M/D/H/M/S are digits
- Hyphens at positions 4 and 7
- 'T' at position 10
- Month: 1-12
- Day: 1-31 (adjusted for month and leap years)
- Hour: 0-23
- Minute: 0-59
- Second: 0-59
Returns:
True if filename has valid timestamp prefix, False otherwise
"""
Standard Library Modules:
datetime: Usedatetime.date(year, month, day)to validate date components- String slicing for parsing components
str.isdigit()for character validation
Leap Year Handling:
def is_leap_year(year: int) -> bool:
return (year % 4 == 0 and year % 100 != 0) or (year % 400 == 0)
Shell-like Lexer (shlex_parser.py)
def split(line: str) -> list[str]:
"""
Split a line into tokens, handling quotes and escapes.
Rules:
- Whitespace separates tokens
- Single quotes (') and double quotes (") group tokens
- Backslash (\) escapes the next character
- Quotes are removed from tokens
- Escaped characters have backslash removed
Raises:
ValueError: If quotes are unterminated
Returns:
List of parsed tokens
"""
Standard Library Modules:
shlex: Python's built-in shell lexer module- Use
shlex.split()for basic functionality - Configure with
posix=Truefor proper escaping
- Use
Alternative Implementation:
If shlex.split() doesn't match exact Zig behavior, implement manually:
- Iterate character by character
- Track state: in_quote, escape_next, current_token
- Build tokens character by character
- Handle quote pairs and escape sequences
4. Error Handling
Error Structure
@dataclass
class TaxonomyError:
message: str
line_num: int
path: str
Error Types
All errors should be implemented as custom exceptions:
class TaxonomyException(Exception):
pass
class ValidationFailed(TaxonomyException):
pass
class InvalidEcosystem(TaxonomyException):
pass
class InvalidParentEcosystem(TaxonomyException):
pass
class InvalidChildEcosystem(TaxonomyException):
pass
# ... etc for all error types
Error Reporting
When errors are detected during validation:
- Collect all errors during processing
- Print errors to stderr in format:
{path}:{line_num}: error.{error_name} - Raise
ValidationFailedexception - Exit with non-zero exit code
5. Testing Strategy
Unit Tests
Use unittest module from standard library:
import unittest
import tempfile
import os
class TestTaxonomy(unittest.TestCase):
def setUp(self):
self.temp_dir = tempfile.mkdtemp()
def tearDown(self):
# Clean up temp directory
pass
def test_load_single_ecosystem(self):
# Create test migration file
# Load taxonomy
# Assert stats
def test_export_json(self):
# Load test data
# Export to temp file
# Read and verify JSON content
# ... more tests
Test Coverage:
- Timestamp validation (valid/invalid dates, leap years)
- Shell lexer (quotes, escapes, edge cases)
- Each migration command (ecoadd, repadd, etc.)
- Error handling (invalid commands, duplicate options)
- JSON export format (hierarchy, sorting, tags)
- Date filtering during load
- Ecosystem relationships (parent/child, disconnect)
6. Implementation Notes
Performance Considerations
- File I/O: Use buffered I/O for export (
io.BufferedWriter) - Memory: Load migration files into memory (files are typically small)
- Sorting: Use
sorted()with case-insensitive key for ecosystem/repo ordering - Sets: Use
setfor repo/ecosystem relationships (O(1) lookups)
Python Standard Library Equivalents
| Zig Feature | Python Equivalent |
|---|---|
ArrayList |
list |
HashMap |
dict |
AutoHashMap |
dict |
std.mem.sort |
sorted() |
std.fs.cwd().openDir() |
os.listdir() |
std.json.stringify() |
json.dumps() |
std.mem.eql |
== operator |
std.mem.startsWith |
str.startswith() |
std.ascii.isWhitespace |
str.isspace() |
std.heap.page_allocator |
Automatic (GC) |
Color Output
For colored terminal output (magenta ┃ character):
MAGENTA = '\x1b[35m'
RESET = '\x1b[0m'
print(f"{MAGENTA}┃{RESET} {count:<6} Migrations")
Case-Insensitive Sorting
sorted(items, key=str.casefold) # Proper Unicode case-folding
# OR
sorted(items, key=str.lower) # Simple lowercase
7. Entry Point
# main.py
import sys
from commands import main
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
sys.exit(130)
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)
8. Project Structure
crypto-ecosystems/
├── migrations/ # Migration files (input)
├── src/ # Zig source (reference)
├── python/ # Python implementation
│ ├── __init__.py
│ ├── main.py # Entry point
│ ├── commands.py # CLI handler
│ ├── taxonomy.py # Core taxonomy logic
│ ├── timestamp.py # Timestamp validation
│ └── shlex_parser.py # Shell lexer (if needed)
└── tests/ # Test fixtures
└── python/ # Python tests
├── __init__.py
└── test_taxonomy.py
9. Dependencies
Required: None (Python 3.10+ standard library only)
Optional:
pytest: For more advanced testing features (if preferred overunittest)mypy: For type checking
10. Compatibility Notes
String Handling
- Python strings are Unicode by default (Zig uses UTF-8 bytes)
- Use UTF-8 encoding when reading/writing files:
open(file, 'r', encoding='utf-8')
Integer IDs
- Use
inttype (arbitrary precision in Python) - No overflow concerns like in Zig's
u32
Dictionary Ordering
- Python 3.7+ dicts maintain insertion order
- Use this for predictable iteration when needed
11. Migration Commands Reference
Command Format
Each line in a migration file follows the format:
<command> <arguments>
Commands are 6 characters followed by space-separated arguments. Arguments may be quoted if they contain spaces.
Argument Quoting
- Use double quotes
"or single quotes'for arguments with spaces - Use backslash
\to escape quotes within quotes - Use backslash
\to escape spaces without quotes
Examples:
ecoadd Bitcoin
ecoadd "Magic Eden Wallet"
repadd Bitcoin https://github.com/bitcoin/bitcoin
repadd Lightning https://github.com/example/repo "#protocol" "#layer2"
ecocon Bitcoin Lightning
12. Validation Rules
Migration File Validation
- Files must start with valid timestamp:
YYYY-MM-DDTHHMMSS - Timestamps must be in chronological order (enforced by sorting)
- Commands must be exactly 6 characters
- All referenced ecosystems must exist before use
- Repository URLs must be unique within the taxonomy
Command-Specific Validation
ecoadd: Ecosystem name requiredrepadd: Ecosystem must exist, repo URL requiredecocon: Both parent and child ecosystems must existecodis: Parent must have child relationshipecorom: Ecosystem must existrepmov: Source repo must existecomov: Source ecosystem must exist, destination must not existreprem: Ecosystem and repo must exist, repo must be in ecosystem
13. JSON Export Format Details
Hierarchical Export
The export flattens the hierarchy by repeating repositories at each level with appropriate branch values:
Example:
Bitcoin (top-level)
├── bitcoin/bitcoin repo
└── Lightning (sub-ecosystem)
├── lightningnetwork/lnd repo
└── Lightspark (sub-sub-ecosystem)
└── lightsparkdev/lightspark-rs repo
Exports as:
{"eco_name":"Bitcoin","branch":[],"repo_url":"https://github.com/bitcoin/bitcoin","tags":["#protocol"]}
{"eco_name":"Bitcoin","branch":["Lightning"],"repo_url":"https://github.com/lightningnetwork/lnd","tags":["#protocol"]}
{"eco_name":"Bitcoin","branch":["Lightning","Lightspark"],"repo_url":"https://github.com/lightsparkdev/lightspark-rs","tags":["#sdk"]}
{"eco_name":"Lightning","branch":[],"repo_url":"https://github.com/lightningnetwork/lnd","tags":["#protocol"]}
{"eco_name":"Lightning","branch":["Lightspark"],"repo_url":"https://github.com/lightsparkdev/lightspark-rs","tags":["#sdk"]}
{"eco_name":"Lightspark","branch":[],"repo_url":"https://github.com/lightsparkdev/lightspark-rs","tags":["#sdk"]}
Note: Each ecosystem appears with all its sub-ecosystems' content when exported at top level.
14. Implementation Priority
- Phase 1: Core data structures and taxonomy class
- Phase 2: Timestamp and shlex utilities
- Phase 3: Migration command handlers
- Phase 4: Load process and validation
- Phase 5: Export functionality
- Phase 6: CLI command handler
- Phase 7: Testing and validation against Zig implementation
15. Success Criteria
The Python implementation should:
- Parse the same command-line arguments as the Zig version
- Load and validate migration files identically
- Produce byte-for-byte identical JSON output
- Report the same validation errors
- Display the same statistics format
- Use only Python standard library (no external dependencies)
- Pass all unit tests equivalent to Zig test suite