2025-11-18 21:15:05 -08:00
..
2025-11-18 21:15:05 -08:00
2025-11-18 21:15:05 -08:00
2025-11-18 21:15:05 -08:00
2025-11-18 21:15:05 -08:00
2025-11-18 21:15:05 -08:00
2025-11-18 21:15:05 -08:00
2025-11-18 21:15:05 -08:00

Crypto Ecosystems Taxonomy - Python Implementation

A Python implementation of the crypto-ecosystems taxonomy validation and export system, originally written in Zig. This implementation uses only the Python standard library (Python 3.10+).

Features

  • Validate: Validate all migration files and display statistics
  • Export: Export the taxonomy to JSON Lines format
  • Date Filtering: Export taxonomy state at specific past dates
  • Single Ecosystem Export: Export only a specific ecosystem
  • Zero Dependencies: Uses only Python standard library

Installation

No installation required! Just use Python 3.10 or newer.

Usage

As a Python Module

# Validate migrations
python -m python validate

# Validate migrations from specific directory
python -m python validate -r /path/to/migrations

# Export all ecosystems
python -m python export output.json

# Export with custom migrations directory
python -m python export -r /path/to/migrations output.json

# Export single ecosystem
python -m python export -e Bitcoin bitcoin.json

# Export with date filter (state as of that date)
python -m python export -m 2015-01-01 historical.json

# Show help
python -m python --help

# Show version
python -m python version

Running Tests

# Run all tests
python -m unittest discover tests/python -v

# Run specific test module
python -m unittest tests.python.test_taxonomy -v
python -m unittest tests.python.test_timestamp -v
python -m unittest tests.python.test_shlex_parser -v

Commands

validate

Validates all migrations in a directory and prints statistics.

Options:

  • -r, --root DIR: Directory containing migration files (default: ./migrations)
  • -h, --help: Show help message

Example Output:

┃ 451    Migrations
┃ 7362   Ecosystems
┃ 557789 Repos
┃ 662    Tags

export

Exports the taxonomy to a JSON Lines file.

Options:

  • -r, --root DIR: Directory containing migration files (default: ./migrations)
  • -e, --ecosystem STR: Export only a specific ecosystem
  • -m, --max-date STR: Maximum date to run migrations until (YYYY-MM-DD format)
  • <output>: Output file path (required)

JSON Lines Format:

Each line is a JSON object representing a repository in an ecosystem:

{"eco_name":"Bitcoin","branch":[],"repo_url":"https://github.com/bitcoin/bitcoin","tags":["#protocol"]}
{"eco_name":"Bitcoin","branch":["Lightning"],"repo_url":"https://github.com/lightningnetwork/lnd","tags":["#protocol"]}

The branch field shows the hierarchy path from the top-level ecosystem to the current location.

Architecture

Core Components

  1. timestamp.py - Validates migration file timestamps (YYYY-MM-DDTHHMMSS format)
  2. shlex_parser.py - Parses shell-like quoted arguments
  3. taxonomy.py - Core taxonomy management with all data structures and operations
  4. commands.py - CLI argument parsing and command execution
  5. main.py - Module entry point

Migration Commands

The system processes migration files with these commands:

  • ecoadd <name> - Add an ecosystem
  • repadd <eco> <url> [tags...] - Add a repository to an ecosystem
  • ecocon <parent> <child> - Connect ecosystems (create hierarchy)
  • ecodis <parent> <child> - Disconnect ecosystems
  • ecorem <name> - Remove an ecosystem
  • repmov <old_url> <new_url> - Rename/move a repository
  • ecomov <old_name> <new_name> - Rename an ecosystem
  • reprem <eco> <url> - Remove a repository from an ecosystem

Data Structures

The Taxonomy class uses Python dictionaries and sets for efficient operations:

  • String-to-ID mappings for ecosystems, repos, and tags
  • ID-to-string reverse mappings
  • Ecosystem-to-repos relationships
  • Parent-to-children hierarchy
  • Ecosystem-repo-to-tags associations

Performance

The Python implementation efficiently handles large datasets:

  • Full validation: 451 migrations, 7362 ecosystems, 557K+ repos in ~1 second
  • Full export: 4.5M+ JSON lines exported successfully
  • Memory efficient: Loads migration files incrementally

Testing

Comprehensive test suite with 27 tests covering:

  • Timestamp validation (valid/invalid dates, leap years, time components)
  • Shell lexer (quotes, escapes, edge cases)
  • All migration commands
  • Hierarchical ecosystems
  • Date filtering
  • JSON export format
  • Error handling

All tests use the same test fixtures as the Zig implementation.

Compatibility

  • Python Version: 3.10+ (uses modern type hints)
  • Platform: Cross-platform (Linux, macOS, Windows)
  • Dependencies: None (standard library only)
  • Output: Byte-compatible with Zig implementation

Development

Project Structure

python/
├── __init__.py          # Package initialization
├── __main__.py          # Module entry point
├── commands.py          # CLI handler
├── taxonomy.py          # Core taxonomy logic
├── timestamp.py         # Timestamp validation
├── shlex_parser.py      # Shell lexer
└── README.md           # This file

tests/python/
├── __init__.py
├── test_timestamp.py    # Timestamp tests
├── test_shlex_parser.py # Lexer tests
└── test_taxonomy.py     # Taxonomy tests

Code Style

The code follows Python best practices:

  • Type hints for all functions
  • Comprehensive docstrings
  • PEP 8 style guide
  • Clear error messages
  • Efficient algorithms

Differences from Zig Implementation

The Python implementation maintains functional compatibility while adapting to Python idioms:

  1. Memory Management: Automatic garbage collection instead of manual allocation
  2. Data Structures: Python dicts/sets instead of HashMap/AutoHashMap
  3. Error Handling: Exception-based instead of error unions
  4. Type System: Dynamic typing with type hints instead of compile-time types
  5. String Handling: Unicode by default instead of UTF-8 bytes

License

Same as the parent project.

Version

2.0 - Matches the Zig implementation version.