# Crypto Ecosystems Taxonomy - Python Implementation A Python implementation of the crypto-ecosystems taxonomy validation and export system, originally written in Zig. This implementation uses only the Python standard library (Python 3.10+). ## Features - **Validate**: Validate all migration files and display statistics - **Export**: Export the taxonomy to JSON Lines format - **Date Filtering**: Export taxonomy state at specific past dates - **Single Ecosystem Export**: Export only a specific ecosystem - **Zero Dependencies**: Uses only Python standard library ## Installation No installation required! Just use Python 3.10 or newer. ## Usage ### As a Python Module ```bash # Validate migrations python -m python validate # Validate migrations from specific directory python -m python validate -r /path/to/migrations # Export all ecosystems python -m python export output.json # Export with custom migrations directory python -m python export -r /path/to/migrations output.json # Export single ecosystem python -m python export -e Bitcoin bitcoin.json # Export with date filter (state as of that date) python -m python export -m 2015-01-01 historical.json # Show help python -m python --help # Show version python -m python version ``` ### Running Tests ```bash # Run all tests python -m unittest discover tests/python -v # Run specific test module python -m unittest tests.python.test_taxonomy -v python -m unittest tests.python.test_timestamp -v python -m unittest tests.python.test_shlex_parser -v ``` ## Commands ### validate Validates all migrations in a directory and prints statistics. **Options:** - `-r, --root DIR`: Directory containing migration files (default: `./migrations`) - `-h, --help`: Show help message **Example Output:** ``` ┃ 451 Migrations ┃ 7362 Ecosystems ┃ 557789 Repos ┃ 662 Tags ``` ### export Exports the taxonomy to a JSON Lines file. **Options:** - `-r, --root DIR`: Directory containing migration files (default: `./migrations`) - `-e, --ecosystem STR`: Export only a specific ecosystem - `-m, --max-date STR`: Maximum date to run migrations until (YYYY-MM-DD format) - ``: Output file path (required) **JSON Lines Format:** Each line is a JSON object representing a repository in an ecosystem: ```json {"eco_name":"Bitcoin","branch":[],"repo_url":"https://github.com/bitcoin/bitcoin","tags":["#protocol"]} {"eco_name":"Bitcoin","branch":["Lightning"],"repo_url":"https://github.com/lightningnetwork/lnd","tags":["#protocol"]} ``` The `branch` field shows the hierarchy path from the top-level ecosystem to the current location. ## Architecture ### Core Components 1. **timestamp.py** - Validates migration file timestamps (YYYY-MM-DDTHHMMSS format) 2. **shlex_parser.py** - Parses shell-like quoted arguments 3. **taxonomy.py** - Core taxonomy management with all data structures and operations 4. **commands.py** - CLI argument parsing and command execution 5. **__main__.py** - Module entry point ### Migration Commands The system processes migration files with these commands: - `ecoadd ` - Add an ecosystem - `repadd [tags...]` - Add a repository to an ecosystem - `ecocon ` - Connect ecosystems (create hierarchy) - `ecodis ` - Disconnect ecosystems - `ecorem ` - Remove an ecosystem - `repmov ` - Rename/move a repository - `ecomov ` - Rename an ecosystem - `reprem ` - Remove a repository from an ecosystem ### Data Structures The Taxonomy class uses Python dictionaries and sets for efficient operations: - String-to-ID mappings for ecosystems, repos, and tags - ID-to-string reverse mappings - Ecosystem-to-repos relationships - Parent-to-children hierarchy - Ecosystem-repo-to-tags associations ## Performance The Python implementation efficiently handles large datasets: - **Full validation**: 451 migrations, 7362 ecosystems, 557K+ repos in ~1 second - **Full export**: 4.5M+ JSON lines exported successfully - **Memory efficient**: Loads migration files incrementally ## Testing Comprehensive test suite with 27 tests covering: - Timestamp validation (valid/invalid dates, leap years, time components) - Shell lexer (quotes, escapes, edge cases) - All migration commands - Hierarchical ecosystems - Date filtering - JSON export format - Error handling All tests use the same test fixtures as the Zig implementation. ## Compatibility - **Python Version**: 3.10+ (uses modern type hints) - **Platform**: Cross-platform (Linux, macOS, Windows) - **Dependencies**: None (standard library only) - **Output**: Byte-compatible with Zig implementation ## Development ### Project Structure ``` python/ ├── __init__.py # Package initialization ├── __main__.py # Module entry point ├── commands.py # CLI handler ├── taxonomy.py # Core taxonomy logic ├── timestamp.py # Timestamp validation ├── shlex_parser.py # Shell lexer └── README.md # This file tests/python/ ├── __init__.py ├── test_timestamp.py # Timestamp tests ├── test_shlex_parser.py # Lexer tests └── test_taxonomy.py # Taxonomy tests ``` ### Code Style The code follows Python best practices: - Type hints for all functions - Comprehensive docstrings - PEP 8 style guide - Clear error messages - Efficient algorithms ## Differences from Zig Implementation The Python implementation maintains functional compatibility while adapting to Python idioms: 1. **Memory Management**: Automatic garbage collection instead of manual allocation 2. **Data Structures**: Python dicts/sets instead of HashMap/AutoHashMap 3. **Error Handling**: Exception-based instead of error unions 4. **Type System**: Dynamic typing with type hints instead of compile-time types 5. **String Handling**: Unicode by default instead of UTF-8 bytes ## License Same as the parent project. ## Version 2.0 - Matches the Zig implementation version.