Crypto Ecosystems Taxonomy - Python Implementation
A Python implementation of the crypto-ecosystems taxonomy validation and export system, originally written in Zig. This implementation uses only the Python standard library (Python 3.10+).
Features
- Validate: Validate all migration files and display statistics
- Export: Export the taxonomy to JSON Lines format
- Date Filtering: Export taxonomy state at specific past dates
- Single Ecosystem Export: Export only a specific ecosystem
- Zero Dependencies: Uses only Python standard library
Installation
No installation required! Just use Python 3.10 or newer.
Usage
As a Python Module
# Validate migrations
python -m python validate
# Validate migrations from specific directory
python -m python validate -r /path/to/migrations
# Export all ecosystems
python -m python export output.json
# Export with custom migrations directory
python -m python export -r /path/to/migrations output.json
# Export single ecosystem
python -m python export -e Bitcoin bitcoin.json
# Export with date filter (state as of that date)
python -m python export -m 2015-01-01 historical.json
# Show help
python -m python --help
# Show version
python -m python version
Running Tests
# Run all tests
python -m unittest discover tests/python -v
# Run specific test module
python -m unittest tests.python.test_taxonomy -v
python -m unittest tests.python.test_timestamp -v
python -m unittest tests.python.test_shlex_parser -v
Commands
validate
Validates all migrations in a directory and prints statistics.
Options:
-r, --root DIR: Directory containing migration files (default:./migrations)-h, --help: Show help message
Example Output:
┃ 451 Migrations
┃ 7362 Ecosystems
┃ 557789 Repos
┃ 662 Tags
export
Exports the taxonomy to a JSON Lines file.
Options:
-r, --root DIR: Directory containing migration files (default:./migrations)-e, --ecosystem STR: Export only a specific ecosystem-m, --max-date STR: Maximum date to run migrations until (YYYY-MM-DD format)<output>: Output file path (required)
JSON Lines Format:
Each line is a JSON object representing a repository in an ecosystem:
{"eco_name":"Bitcoin","branch":[],"repo_url":"https://github.com/bitcoin/bitcoin","tags":["#protocol"]}
{"eco_name":"Bitcoin","branch":["Lightning"],"repo_url":"https://github.com/lightningnetwork/lnd","tags":["#protocol"]}
The branch field shows the hierarchy path from the top-level ecosystem to the current location.
Architecture
Core Components
- timestamp.py - Validates migration file timestamps (YYYY-MM-DDTHHMMSS format)
- shlex_parser.py - Parses shell-like quoted arguments
- taxonomy.py - Core taxonomy management with all data structures and operations
- commands.py - CLI argument parsing and command execution
- main.py - Module entry point
Migration Commands
The system processes migration files with these commands:
ecoadd <name>- Add an ecosystemrepadd <eco> <url> [tags...]- Add a repository to an ecosystemecocon <parent> <child>- Connect ecosystems (create hierarchy)ecodis <parent> <child>- Disconnect ecosystemsecorem <name>- Remove an ecosystemrepmov <old_url> <new_url>- Rename/move a repositoryecomov <old_name> <new_name>- Rename an ecosystemreprem <eco> <url>- Remove a repository from an ecosystem
Data Structures
The Taxonomy class uses Python dictionaries and sets for efficient operations:
- String-to-ID mappings for ecosystems, repos, and tags
- ID-to-string reverse mappings
- Ecosystem-to-repos relationships
- Parent-to-children hierarchy
- Ecosystem-repo-to-tags associations
Performance
The Python implementation efficiently handles large datasets:
- Full validation: 451 migrations, 7362 ecosystems, 557K+ repos in ~1 second
- Full export: 4.5M+ JSON lines exported successfully
- Memory efficient: Loads migration files incrementally
Testing
Comprehensive test suite with 27 tests covering:
- Timestamp validation (valid/invalid dates, leap years, time components)
- Shell lexer (quotes, escapes, edge cases)
- All migration commands
- Hierarchical ecosystems
- Date filtering
- JSON export format
- Error handling
All tests use the same test fixtures as the Zig implementation.
Compatibility
- Python Version: 3.10+ (uses modern type hints)
- Platform: Cross-platform (Linux, macOS, Windows)
- Dependencies: None (standard library only)
- Output: Byte-compatible with Zig implementation
Development
Project Structure
python/
├── __init__.py # Package initialization
├── __main__.py # Module entry point
├── commands.py # CLI handler
├── taxonomy.py # Core taxonomy logic
├── timestamp.py # Timestamp validation
├── shlex_parser.py # Shell lexer
└── README.md # This file
tests/python/
├── __init__.py
├── test_timestamp.py # Timestamp tests
├── test_shlex_parser.py # Lexer tests
└── test_taxonomy.py # Taxonomy tests
Code Style
The code follows Python best practices:
- Type hints for all functions
- Comprehensive docstrings
- PEP 8 style guide
- Clear error messages
- Efficient algorithms
Differences from Zig Implementation
The Python implementation maintains functional compatibility while adapting to Python idioms:
- Memory Management: Automatic garbage collection instead of manual allocation
- Data Structures: Python dicts/sets instead of HashMap/AutoHashMap
- Error Handling: Exception-based instead of error unions
- Type System: Dynamic typing with type hints instead of compile-time types
- String Handling: Unicode by default instead of UTF-8 bytes
License
Same as the parent project.
Version
2.0 - Matches the Zig implementation version.