# Python Port Specification: Taxonomy Validation and Export ## Overview This specification describes how to port the Zig-based crypto-ecosystems taxonomy validation and export system to Python, using the Python standard library wherever possible. ## Architecture The system consists of three main components: 1. **CLI Command Handler** - Parses and executes user commands 2. **Taxonomy Core** - Manages ecosystem/repository relationships and migration processing 3. **Utilities** - Helper functions for timestamp validation and shell-like lexing ## 1. CLI Command Handler (commands.py) ### Purpose Parse command-line arguments and dispatch to appropriate handlers. ### Commands #### validate Validates all migrations in a directory and prints statistics. **Options:** - `-r, --root DIR`: Directory containing migration files (default: `./migrations`) - `-h, --help`: Show help message **Output:** ``` ┃ N Migrations ┃ N Ecosystems ┃ N Repos ┃ N Tags ``` #### export Exports the taxonomy to a JSON Lines file. **Options:** - `-r, --root DIR`: Directory containing migration files (default: `./migrations`) - `-e, --ecosystem STR`: Export only a specific ecosystem - `-m, --max-date STR`: Maximum date to run migrations until (YYYY-MM-DD format) - ``: Output file path (required, positional argument) **Output Format:** JSON Lines with each line containing: ```json { "eco_name": "Bitcoin", "branch": ["Lightning", "Lightspark"], "repo_url": "https://github.com/...", "tags": ["#protocol", "#sdk"] } ``` ### Implementation Details **Standard Library Modules:** - `argparse`: Command-line argument parsing - `sys`: Exit codes and stderr output - `os.path`: Path operations **Error Handling:** - `MissingOutputPath`: When export command lacks output file - `InvalidCommand`: Unknown command - `InvalidOption`: Invalid option provided - `DuplicateOption`: Option specified multiple times **Argument Parsing:** - Use `argparse.ArgumentParser` with subparsers for each command - Validate that options aren't duplicated (track with seen flags) - For export command, ensure output path is provided and is the last argument - Validate that option values don't start with `-` ## 2. Taxonomy Core (taxonomy.py) ### Data Structures Use dictionaries and sets to represent the taxonomy: ```python class Taxonomy: def __init__(self): self.eco_auto_id: int = 0 self.repo_auto_id: int = 0 self.tag_auto_id: int = 0 self.migration_count: int = 0 # String to ID mappings self.eco_ids: dict[str, int] = {} self.repo_ids: dict[str, int] = {} self.tag_ids: dict[str, int] = {} # ID to string mappings self.tag_id_to_name: dict[int, str] = {} self.repo_id_to_url: dict[int, str] = {} self.eco_id_to_name: dict[int, str] = {} # Relationship mappings self.eco_to_repos: dict[int, set[int]] = {} self.repo_to_eco: dict[int, int] = {} self.parent_to_children: dict[int, set[int]] = {} self.child_to_parents: dict[int, set[int]] = {} self.eco_repo_to_tags: dict[tuple[int, int], set[int]] = {} # Error tracking self.errors: list[dict] = [] ``` ### Migration Commands The system processes migration files with the following commands: 1. **ecoadd** `` - Adds a new ecosystem to the taxonomy 2. **repadd** `` `` `[] [] ...` - Adds a repository to an ecosystem with optional tags 3. **ecocon** `` `` - Connects a child ecosystem to a parent (creates hierarchy) 4. **ecodis** `` `` - Disconnects a child ecosystem from a parent 5. **ecorom** `` - Removes an ecosystem (preserves child ecosystems as orphans) 6. **repmov** `` `` - Renames/moves a repository URL 7. **ecomov** `` `` - Renames an ecosystem 8. **reprem** `` `` - Removes a repository from an ecosystem ### Load Process ```python def load(self, root_dir: str, max_date: str | None = None) -> None: """ Load and process migration files from root_dir. Steps: 1. List all files in directory 2. Filter files with valid timestamps (YYYY-MM-DDTHHMMSS format) 3. If max_date provided, filter files before that date 4. Sort files by timestamp (first 19 characters) 5. Process each file line by line 6. Track errors during processing 7. If errors exist, print them and raise ValidationFailed """ ``` **Standard Library Modules:** - `os`: Directory listing with `os.listdir()` - `os.path`: Path joining with `os.path.join()` - `re`: Pattern matching for timestamp validation (optional, can use string slicing) **File Processing:** - Read entire file into memory - Split by newlines - Skip comment lines (lines starting with `#` after whitespace) - Skip lines shorter than 6 characters - Parse command (first 6 characters) and remainder - Execute appropriate command handler - Catch exceptions and append to errors list with line number and path ### Export Process ```python def export_json(self, output_file: str, ecosystem: str | None = None) -> None: """ Export taxonomy to JSON Lines format. Steps: 1. Collect ecosystems to export (all or specific one) 2. Sort ecosystems alphabetically (case-insensitive) 3. For each ecosystem, emit repositories and recursively emit sub-ecosystems 4. Write to file using buffered I/O """ ``` **Standard Library Modules:** - `json`: JSON serialization with `json.dumps()` - `io`: Buffered file writing for performance **Output Format:** - One JSON object per line (JSON Lines format) - Each object represents a repository in an ecosystem - `branch` field shows the hierarchy path from top ecosystem to current level - Repositories and sub-ecosystems sorted case-insensitively **Recursive Emission:** - Emit all repositories at current ecosystem level - For each sub-ecosystem: - Append sub-ecosystem name to branch list - Recursively emit that sub-ecosystem - Pop from branch list ### Statistics ```python def stats(self) -> dict: """ Return statistics about the taxonomy. Returns: { 'migration_count': int, 'eco_count': int, 'repo_count': int, 'tag_count': int } """ ``` ## 3. Utilities ### Timestamp Validation (timestamp.py) ```python def has_valid_timestamp(filename: str) -> bool: """ Validate that filename starts with YYYY-MM-DDTHHMMSS format. Validation rules: - Minimum length of 17 characters - Pattern: YYYY-MM-DDTHHMMSS where Y/M/D/H/M/S are digits - Hyphens at positions 4 and 7 - 'T' at position 10 - Month: 1-12 - Day: 1-31 (adjusted for month and leap years) - Hour: 0-23 - Minute: 0-59 - Second: 0-59 Returns: True if filename has valid timestamp prefix, False otherwise """ ``` **Standard Library Modules:** - `datetime`: Use `datetime.date(year, month, day)` to validate date components - String slicing for parsing components - `str.isdigit()` for character validation **Leap Year Handling:** ```python def is_leap_year(year: int) -> bool: return (year % 4 == 0 and year % 100 != 0) or (year % 400 == 0) ``` ### Shell-like Lexer (shlex_parser.py) ```python def split(line: str) -> list[str]: """ Split a line into tokens, handling quotes and escapes. Rules: - Whitespace separates tokens - Single quotes (') and double quotes (") group tokens - Backslash (\) escapes the next character - Quotes are removed from tokens - Escaped characters have backslash removed Raises: ValueError: If quotes are unterminated Returns: List of parsed tokens """ ``` **Standard Library Modules:** - `shlex`: Python's built-in shell lexer module - Use `shlex.split()` for basic functionality - Configure with `posix=True` for proper escaping **Alternative Implementation:** If `shlex.split()` doesn't match exact Zig behavior, implement manually: - Iterate character by character - Track state: in_quote, escape_next, current_token - Build tokens character by character - Handle quote pairs and escape sequences ## 4. Error Handling ### Error Structure ```python @dataclass class TaxonomyError: message: str line_num: int path: str ``` ### Error Types All errors should be implemented as custom exceptions: ```python class TaxonomyException(Exception): pass class ValidationFailed(TaxonomyException): pass class InvalidEcosystem(TaxonomyException): pass class InvalidParentEcosystem(TaxonomyException): pass class InvalidChildEcosystem(TaxonomyException): pass # ... etc for all error types ``` ### Error Reporting When errors are detected during validation: 1. Collect all errors during processing 2. Print errors to stderr in format: `{path}:{line_num}: error.{error_name}` 3. Raise `ValidationFailed` exception 4. Exit with non-zero exit code ## 5. Testing Strategy ### Unit Tests Use `unittest` module from standard library: ```python import unittest import tempfile import os class TestTaxonomy(unittest.TestCase): def setUp(self): self.temp_dir = tempfile.mkdtemp() def tearDown(self): # Clean up temp directory pass def test_load_single_ecosystem(self): # Create test migration file # Load taxonomy # Assert stats def test_export_json(self): # Load test data # Export to temp file # Read and verify JSON content # ... more tests ``` **Test Coverage:** - Timestamp validation (valid/invalid dates, leap years) - Shell lexer (quotes, escapes, edge cases) - Each migration command (ecoadd, repadd, etc.) - Error handling (invalid commands, duplicate options) - JSON export format (hierarchy, sorting, tags) - Date filtering during load - Ecosystem relationships (parent/child, disconnect) ## 6. Implementation Notes ### Performance Considerations 1. **File I/O**: Use buffered I/O for export (`io.BufferedWriter`) 2. **Memory**: Load migration files into memory (files are typically small) 3. **Sorting**: Use `sorted()` with case-insensitive key for ecosystem/repo ordering 4. **Sets**: Use `set` for repo/ecosystem relationships (O(1) lookups) ### Python Standard Library Equivalents | Zig Feature | Python Equivalent | |-------------|-------------------| | `ArrayList` | `list` | | `HashMap` | `dict` | | `AutoHashMap` | `dict` | | `std.mem.sort` | `sorted()` | | `std.fs.cwd().openDir()` | `os.listdir()` | | `std.json.stringify()` | `json.dumps()` | | `std.mem.eql` | `==` operator | | `std.mem.startsWith` | `str.startswith()` | | `std.ascii.isWhitespace` | `str.isspace()` | | `std.heap.page_allocator` | Automatic (GC) | ### Color Output For colored terminal output (magenta `┃` character): ```python MAGENTA = '\x1b[35m' RESET = '\x1b[0m' print(f"{MAGENTA}┃{RESET} {count:<6} Migrations") ``` ### Case-Insensitive Sorting ```python sorted(items, key=str.casefold) # Proper Unicode case-folding # OR sorted(items, key=str.lower) # Simple lowercase ``` ## 7. Entry Point ```python # main.py import sys from commands import main if __name__ == '__main__': try: main() except KeyboardInterrupt: sys.exit(130) except Exception as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1) ``` ## 8. Project Structure ``` crypto-ecosystems/ ├── migrations/ # Migration files (input) ├── src/ # Zig source (reference) ├── python/ # Python implementation │ ├── __init__.py │ ├── main.py # Entry point │ ├── commands.py # CLI handler │ ├── taxonomy.py # Core taxonomy logic │ ├── timestamp.py # Timestamp validation │ └── shlex_parser.py # Shell lexer (if needed) └── tests/ # Test fixtures └── python/ # Python tests ├── __init__.py └── test_taxonomy.py ``` ## 9. Dependencies **Required:** None (Python 3.10+ standard library only) **Optional:** - `pytest`: For more advanced testing features (if preferred over `unittest`) - `mypy`: For type checking ## 10. Compatibility Notes ### String Handling - Python strings are Unicode by default (Zig uses UTF-8 bytes) - Use UTF-8 encoding when reading/writing files: `open(file, 'r', encoding='utf-8')` ### Integer IDs - Use `int` type (arbitrary precision in Python) - No overflow concerns like in Zig's `u32` ### Dictionary Ordering - Python 3.7+ dicts maintain insertion order - Use this for predictable iteration when needed ## 11. Migration Commands Reference ### Command Format Each line in a migration file follows the format: ``` ``` Commands are 6 characters followed by space-separated arguments. Arguments may be quoted if they contain spaces. ### Argument Quoting - Use double quotes `"` or single quotes `'` for arguments with spaces - Use backslash `\` to escape quotes within quotes - Use backslash `\` to escape spaces without quotes Examples: ``` ecoadd Bitcoin ecoadd "Magic Eden Wallet" repadd Bitcoin https://github.com/bitcoin/bitcoin repadd Lightning https://github.com/example/repo "#protocol" "#layer2" ecocon Bitcoin Lightning ``` ## 12. Validation Rules ### Migration File Validation - Files must start with valid timestamp: `YYYY-MM-DDTHHMMSS` - Timestamps must be in chronological order (enforced by sorting) - Commands must be exactly 6 characters - All referenced ecosystems must exist before use - Repository URLs must be unique within the taxonomy ### Command-Specific Validation - `ecoadd`: Ecosystem name required - `repadd`: Ecosystem must exist, repo URL required - `ecocon`: Both parent and child ecosystems must exist - `ecodis`: Parent must have child relationship - `ecorom`: Ecosystem must exist - `repmov`: Source repo must exist - `ecomov`: Source ecosystem must exist, destination must not exist - `reprem`: Ecosystem and repo must exist, repo must be in ecosystem ## 13. JSON Export Format Details ### Hierarchical Export The export flattens the hierarchy by repeating repositories at each level with appropriate `branch` values: Example: ``` Bitcoin (top-level) ├── bitcoin/bitcoin repo └── Lightning (sub-ecosystem) ├── lightningnetwork/lnd repo └── Lightspark (sub-sub-ecosystem) └── lightsparkdev/lightspark-rs repo ``` Exports as: ```json {"eco_name":"Bitcoin","branch":[],"repo_url":"https://github.com/bitcoin/bitcoin","tags":["#protocol"]} {"eco_name":"Bitcoin","branch":["Lightning"],"repo_url":"https://github.com/lightningnetwork/lnd","tags":["#protocol"]} {"eco_name":"Bitcoin","branch":["Lightning","Lightspark"],"repo_url":"https://github.com/lightsparkdev/lightspark-rs","tags":["#sdk"]} {"eco_name":"Lightning","branch":[],"repo_url":"https://github.com/lightningnetwork/lnd","tags":["#protocol"]} {"eco_name":"Lightning","branch":["Lightspark"],"repo_url":"https://github.com/lightsparkdev/lightspark-rs","tags":["#sdk"]} {"eco_name":"Lightspark","branch":[],"repo_url":"https://github.com/lightsparkdev/lightspark-rs","tags":["#sdk"]} ``` Note: Each ecosystem appears with all its sub-ecosystems' content when exported at top level. ## 14. Implementation Priority 1. **Phase 1**: Core data structures and taxonomy class 2. **Phase 2**: Timestamp and shlex utilities 3. **Phase 3**: Migration command handlers 4. **Phase 4**: Load process and validation 5. **Phase 5**: Export functionality 6. **Phase 6**: CLI command handler 7. **Phase 7**: Testing and validation against Zig implementation ## 15. Success Criteria The Python implementation should: 1. Parse the same command-line arguments as the Zig version 2. Load and validate migration files identically 3. Produce byte-for-byte identical JSON output 4. Report the same validation errors 5. Display the same statistics format 6. Use only Python standard library (no external dependencies) 7. Pass all unit tests equivalent to Zig test suite