52 lines
3.1 KiB
Markdown
52 lines
3.1 KiB
Markdown
# CatOpt-Flow: Category-Theoretic Compositional Optimizer
|
||
|
||
CatOpt-Flow is a production-grade, open-source platform for defining and solving per-job optimization problems in multi-tenant ML training pipelines across heterogeneous accelerators.
|
||
|
||
Key abstractions (category-theory inspired)
|
||
- Objects: local training tasks representing per-job optimization problems.
|
||
- Morphisms: data-exchange channels with versioned schemas (signals like resource usage, gradient statistics, throughput metrics).
|
||
- Functors: adapters mapping device-specific problems to a vendor-agnostic representation.
|
||
- Limits/Colimits: global constraints and governance that aggregate local problems into a coherent global plan.
|
||
- Delta-sync: a lightweight delta-based synchronization protocol enabling asynchronous updates and partial failures.
|
||
- Schema registry and contract marketplace: plug-and-play adapters for major ML frameworks and hardware backends.
|
||
- Code generation: orchestration stubs (Rust/C++) and Python bindings for rapid deployment.
|
||
|
||
What you get
|
||
- A pragmatic, test-driven architecture suitable for large-scale, multi-tenant ML workloads.
|
||
- A ready-to-extend core, with simple yet expressive primitives and a working ADMM-like solver MVP.
|
||
- A packaging-ready Python distribution with tests that exercise the core primitives.
|
||
|
||
Getting started
|
||
- This is a Python project. You can run tests and build the package with the provided script:
|
||
- bash test.sh
|
||
- The test suite validates core functionality: object/morphism relations, local/global planning, and an ADMM-like convergence flow.
|
||
- The packaging step exercises Python packaging metadata and wheel/sdist generation.
|
||
|
||
What’s inside
|
||
- catopt_flow_category_theoretic_compositi/core.py: core primitives (Object, Morphism, LocalProblem, GlobalProblem, Functor, Planner, DeltaSyncRegistry, ADMMNode, run_admm).
|
||
- tests/test_core.py: unit tests for core primitives.
|
||
- A minimal, production-ready packaging layout with pyproject.toml and a README hook.
|
||
|
||
Development and contribution
|
||
- See AGENTS.md for architectural guidelines and contribution rules.
|
||
- All changes should be reflected in tests and documented in this README.
|
||
|
||
Roadmap (high level)
|
||
- Phase 0: protocol skeleton with 2 starter adapters per platform and delta-sync.
|
||
- Phase 1: governance ledger with DID-based identities.
|
||
- Phase 2: cross-domain demo with a simulated satellite domain.
|
||
- Phase 3: hardware-in-the-loop validation.
|
||
- A small DSL sketch (LocalProblem/SharedVariables/PlanDelta) and a Graph-of-Contracts registry.
|
||
|
||
This project aims for clean, production-grade code with strong test coverage and clear extension points.
|
||
|
||
Adapters (Starter)
|
||
- This repo now includes two starter adapters to bootstrap cross-framework optimization flows:
|
||
- PyTorchAdapter and TensorFlowAdapter
|
||
- Path: catopt_flow_category_theoretic_compositi/adapters
|
||
- Each adapter implements a minimal adapt(local_problems) -> GlobalProblem using the core Planner.
|
||
- Example usage:
|
||
- from catopt_flow_category_theoretic_compositi.adapters import PyTorchAdapter
|
||
- adapter = PyTorchAdapter(budget_gpu=2.0)
|
||
- global_plan = adapter.adapt(local_problems)
|