Open Source · Apache 2.0 · Privacy by Design
MoE Sovereign orchestrates freely available language models into a powerful Mixture-of-Experts system — distributed across consumer and legacy hardware, fully self-hosted, zero cloud dependency.
The Concept
Instead of a single massive model on an expensive GPU, many specialized models are coordinated — each running on the hardware best suited for it.
Modern Large Language Models like GPT-4 or Claude require five- to six-figure investments in GPU hardware alone for self-hosting — and create a permanent cloud dependency with corresponding privacy risks. For businesses, research institutions, and privacy-conscious users, that is simply not an option.
MoE Sovereign distributes inference across a cluster of more affordable nodes. Each request is analyzed by an intelligent planner, routed to the appropriate expert models, and the results are synthesized by a merger model. The outcome: quality comparable to cloud LLMs — with full data control at a fraction of the cost.
The API is fully OpenAI-compatible, so existing tools like Open WebUI, Claude Code, or any OpenAI SDK integration work without modification.
All data stays on your infrastructure. Not a single API call leaves your network — Privacy by Design from the ground up.
Instead of one expensive high-end GPU: many affordable consumer GPUs and repurposed enterprise hardware like Tesla M10 and K80 cards.
New GPU nodes are simply added to the configuration. Load distribution and VRAM management happen automatically.
Fully open source under Apache 2.0. No vendor lock-in, no hidden costs, no proprietary stack.
Capabilities
MoE Sovereign ships with all components needed for a production-ready AI infrastructure.
Drop-in replacement for OpenAI endpoints. Every tool that works
with /v1/chat/completions works immediately.
Specialized models for math, code, medicine, law, translation, data analysis and more — dynamically routed via LangGraph.
20 deterministic tools for math, date arithmetic, unit conversion, cryptography and German law — no hallucinations.
Neo4j-based knowledge database with automatic extraction and 2-hop traversal for deep contextual knowledge.
Semantic similarity search (ChromaDB) + Redis plan cache + GraphRAG cache + performance scores for minimal latency.
SearXNG integration for real-time research without tracking. Results are incorporated before the response is generated.
SQLite-based user and API key management with token budgets, permissions, and full audit log via Kafka.
5 pre-built dashboards for operations, model performance, cache hit rates, tool calls and hardware metrics.
FastAPI + Jinja2 dashboard for live configuration of experts, models, system prompts and users — no container restart needed.
Apache Kafka in KRaft mode for audit logs, async GraphRAG learning and feedback loop — no Zookeeper overhead.
Tier-2 experts for image, screenshot and document analysis.
Base64 images are passed directly in /v1/chat/completions.
17 built-in skills (PDF, DOCX, XLSX, code review, MCP builder and more) plus synchronization with upstream skill libraries.
Technical Architecture
LangGraph orchestrates a deterministic state graph that routes every request through up to eleven processing nodes.
┌─────────────────────────────────────────────────────────────────────┐
│ MoE Sovereign Pipeline │
└─────────────────────────────────────────────────────────────────────┘
User Request (OpenAI-compatible via /v1/chat/completions)
│
▼
┌─────────────────┐
│ Cache Lookup │──── HIT (cosine < 0.15) ────────────────────┐
│ (ChromaDB L1) │ │
└────────┬────────┘ │
│ MISS │
▼ │
┌─────────────────┐ │
│ Planner │ Judge LLM decomposes request into 1–4 tasks │
│ (magistral) │ → Plan as JSON │
└────────┬────────┘ │
│ │
▼ (parallel execution) │
┌────────────────────────────────────────────────────┐ │
│ │ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ Expert │ │ Math │ │ MCP │ │ │
│ │ Worker │ │ Node │ │ Tools │ │ │
│ │ (up to 5)│ │ (SymPy) │ │ (20 det.)│ │ │
│ └──────────┘ └──────────┘ └──────────┘ │ │
│ ┌──────────┐ ┌──────────┐ │ │
│ │ Research │ │ GraphRAG │ │ │
│ │ (SearXNG)│ │ (Neo4j) │ │ │
│ └──────────┘ └──────────┘ │ │
│ │ │
└────────────────────────┬───────────────────────────┘ │
│ │
▼ │
┌─────────────────┐ │
│ Research Fallb. │ Low confidence → additional web research │
└────────┬────────┘ │
│ │
▼ │
┌─────────────────┐ │
│ Thinking Node │ Chain-of-Thought (conditional, on demand) │
└────────┬────────┘ │
│ │
▼ │
┌─────────────────┐ ◄──────────────────────────────────────────┘
│ Merger │ Judge LLM synthesizes all partial answers
│ (magistral) │ → Fast-path if 1 expert + high confidence
└────────┬────────┘
│
▼
┌─────────────────┐
│ Critic │ Fact-check (medical & legal responses only)
└────────┬────────┘
│
▼
Streaming Response → ChromaDB cache → Redis metadata → Kafka event
| Service | Technology | Port | Role |
|---|---|---|---|
| Orchestrator | FastAPI + LangGraph | 8002 |
OpenAI-compatible API endpoint and pipeline controller |
| Admin UI | FastAPI + Jinja2 | 8088 |
Web dashboard for configuration and monitoring |
| MCP Server | FastMCP (Python) | 8003 |
20 deterministic precision tools |
| ChromaDB | ChromaDB | 8001 |
Vector database for semantic caching |
| Redis | Redis Stack | 6379 |
Plan cache, performance scores, LangGraph checkpoints |
| Neo4j | Neo4j 5 Community | 7474 / 7687 |
Knowledge graph for GraphRAG |
| Kafka | Apache Kafka (KRaft) | 9092 |
Event streaming, audit log, feedback loop |
| Prometheus | Prometheus | 9090 |
Metrics collection (API, GPU, containers, host) |
| Grafana | Grafana | 3001 |
5 pre-provisioned dashboards |
| SearXNG | SearXNG | 8888 |
Privacy-respecting meta search engine |
The system distinguishes two model tiers to dynamically balance speed and quality:
| Tier | Parameters | VRAM (4-bit) | Use case | Escalation trigger |
|---|---|---|---|---|
| T1 | ≤ 20 B | 8–16 GB | Fast first opinion, handles most requests | Confidence < 0.65 |
| T2 | > 20 B | 16–40 GB | Complex reasoning, low-confidence escalations | Final answer |
ChromaDB vector search
Cosine distance < 0.15 → direct hit, full pipeline skipped
Redis: planner LLM output
Saves ~1,600 tokens per cache hit
Redis: Neo4j context queries
Avoids redundant graph traversals
Redis: model ratings per category
Laplace smoothing for expert routing
Expert System
Every expert is optimized for its domain — with tailored system prompts, model selection and tier strategy.
| Category | Tier | Example Models | Domain | Special Feature |
|---|---|---|---|---|
| general | T2 | gemma3:27b, qwen3.5:35b |
General knowledge, definitions, explanations | — |
| math | T1+T2 | phi4:14b → qwq:32b |
Calculations, equations, statistics | + MCP + SymPy |
| technical_support | T1+T2 | deepseek-coder-v2:16b → devstral:24b |
IT, DevOps, Docker, networking, Linux | — |
| code_reviewer | T2 | devstral:24b, qwen3-coder:30b |
Code review, security audits, refactoring | — |
| creative_writer | T2 | gemma3:27b, qwen3.5:35b |
Content creation, marketing, storytelling | — |
| medical_consult | T1+T2 | phi4:14b → gemma3:27b |
Medical information, symptoms, health topics | Critic node |
| legal_advisor | T2 | magistral:24b, command-r:35b |
German law: BGB, StGB, HGB and more | Critic + MCP |
| translation | T2 | translategemma:27b, qwen3.5:35b |
Professional translation, multiple languages | — |
| data_analyst | T1 | phi4:14b |
Statistics, Pandas, data analysis, SQL | + MCP stats |
| science | T2 | gemma3:27b |
Chemistry, biology, physics, research | — |
| reasoning | T1+T2 | phi4:14b → deepseek-r1:32b |
Complex logic, strategy, multi-step analysis | Thinking node |
| vision | T2 | Multimodal T2 models | Image, screenshot, document analysis | Base64 input |
Every expert returns a confidence score alongside its answer. This determines whether the result is used directly or escalated to a more capable Tier-2 model:
The model field controls the desired output style:
| Model ID | Mode | Description |
|---|---|---|
moe-orchestrator | Standard | Full answers with explanations |
moe-orchestrator-code | Code | Code output only, no prose |
moe-orchestrator-concise | Concise | Maximum 120 words, no padding |
moe-orchestrator-research | Research | Deep analysis with citations |
moe-orchestrator-report | Report | Structured report with sections |
moe-orchestrator-agent | Agent | Tool-use optimized for agents |
moe-orchestrator-plan | Plan | Task planning with step list |
MCP Precision Tools
LLMs hallucinate on calculations, date arithmetic and legal paragraphs. MCP Precision Tools replaces these with exact, verifiable computations.
calculate – Safe arithmetic evaluationsolve_equation – SymPy equation solverprime_factorize – Prime factorizationgcd_lcm – Greatest common divisor / LCMroman_numeral – Arabic ↔ Roman numeralsdate_diff – Difference between datesdate_add – Add/subtract from a dateday_of_week – Calculate day of weekunit_convert – km, miles, kg, lb, °C, °F …statistics_calc – Mean, median, stdev, percentileshash_text – MD5, SHA-256, SHA-512base64_codec – Base64 encode/decodesubnet_calc – CIDR analysis, netmask, broadcastregex_extract – Apply regular expressionstext_analyze – Word count, characters, sentencesjson_query – JSONPath extractionlegal_search_laws – Search across statuteslegal_get_law_overview – Law overviewlegal_get_paragraph – Fetch specific paragraphslegal_fulltext_search – Full-text (BGB, StGB …)Infrastructure
The motivation: consumer GPUs and decommissioned enterprise cards running stable under full load. Distributed inference does not require cutting-edge hardware.
Motivation: Privacy by design, independence, and putting hardware others called scrap to work on a cutting-edge problem.
API & Integration
MoE Sovereign behaves like the OpenAI API. Any existing integration — Open WebUI, Claude Code, LangChain, LlamaIndex — works without code changes.
curl -X POST http://<YOUR-SERVER>:8002/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR-API-KEY>" \
-d '{
"model": "moe-orchestrator",
"messages": [
{"role": "user", "content": "Explain the difference between TCP and UDP"}
],
"stream": false
}'curl -X POST http://<YOUR-SERVER>:8002/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR-API-KEY>" \
-d '{
"model": "moe-orchestrator-code",
"messages": [{"role": "user", "content": "Write a Python Fibonacci function"}],
"stream": true
}'from openai import OpenAI
client = OpenAI(
base_url="http://<YOUR-SERVER>:8002/v1",
api_key="<YOUR-API-KEY>"
)
response = client.chat.completions.create(
model="moe-orchestrator-research",
messages=[{"role": "user", "content": "Analyze the trade-offs of Kubernetes"}]
)
print(response.choices[0].message.content)curl -X POST http://<YOUR-SERVER>:8002/v1/feedback \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR-API-KEY>" \
-d '{
"response_id": "chatcmpl-abc123",
"rating": 5,
"comment": "Very precise answer"
}'Feedback is processed via Kafka and feeds into expert scoring with Laplace smoothing — the system learns which models perform best for each category over time.
Full API reference, authentication, budget management and more are covered in the documentation. Read the full documentation ↗
Project Status
The project is in active development. Hardware and infrastructure are stable; the focus is now on fine-tuning expert configuration.
All GPU nodes assembled, Ollama cluster configured, services (Redis, Neo4j, ChromaDB, Kafka, Prometheus, Grafana) operational. VRAM management optimized, inference running stable.
Identifying best models per task type, optimizing system prompts, building expert templates in the MoE Portal, testing MCP tool integration end-to-end.
Publication on GitHub under Apache 2.0 license once all core features are established and thoroughly tested. Full documentation available via MkDocs at docs.moe-admin.de.
License: Apache 2.0 · Language: Python + FastAPI + LangGraph · Minimum hardware: 1 GPU with ≥ 8 GB VRAM