Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
104 lines (81 loc) · 6.17 KB

ARCHITECTURE.md

File metadata and controls

104 lines (81 loc) · 6.17 KB
Raw

NeuralBlocks — Architecture

[English] | 简体中文

1. Overview

A visual desktop app for building neural networks by drag-and-drop (CNN / RNN / Transformer building blocks), auto-generating trainable PyTorch code, training locally (CPU / CUDA / DirectML), with a built-in DeepSeek (OpenAI-compatible, function-calling) AI assistant.

Audience: AI students familiar with CNNs, learning RNN/Transformer. Node templates carry clear Chinese descriptions and shape hints.

2. Three-layer architecture

┌────────────────────────────────────────────────────────────┐
│ Frontend canvas (Electron renderer)                        │
│  React + Vite + TypeScript + React Flow + Recharts         │
│  [template library] [React Flow] [inspector] [training]    │
└───────────────┬────────────────────────────────────────────┘
                │  DSL (JSON) — single source of truth
┌───────────────▼────────────────────────────────────────────┐
│ Python engine (FastAPI subprocess)                         │
│  DSL validation · shape inference · codegen · training     │
│  hardware · environment · AI agent loop · data flow        │
└────────────────────────────────────────────────────────────┘

DSL is the single source of truth. All three sides (canvas, Python engine, AI assistant) operate on the DSL object {name, input, layers[], training, io?, data?} — never on canvas objects.

  • layers is an ordered array (linear pipeline), not a graph
  • input.shape excludes the batch dim: CNN [C,H,W], sequence [seq, feat], FC [D]
  • Multi-input uses input.inputs ({entryName: [shape]})

Three-layer boundary (must not be broken)

  1. Frontend does not generate PyTorch or do authoritative shape math (only instant hints)
  2. Python engine does not care about the canvas UI — eats DSL, emits DSL/results/errors
  3. AI assistant operates DSL only through tools, never touches frontend state directly

3. Node templates

55 templates = 48 network layers + 7 data-processing blocks. Metadata lives authoritatively in shared/templates.json (name / description / params, param types int|float|bool|select|list). Both src/dsl/templates.ts and dsl.py::load_templates load from it.

Adding a layer = change three places:

  1. shared/templates.json — register the template (param schema)
  2. shape_infer.py::_infer_layer — shape inference rule
  3. codegen.py — _gen_init_line (module creation) and _forward_code (forward call)

Template families & colors

FAMILY_COLORS in src/dsl/templates.ts (shared by palette + canvas nodes) uses blue/purple/cyan/brown — avoiding AI-change highlight (green/orange/red) and shape-error red.

4. Processes & communication

  • Electron main spawns python -m app.main (cwd=python-backend); backend prints PORT=<random> to stdout
  • Frontend talks to http://127.0.0.1:<port> via HTTP + SSE
  • Key endpoints:
    • /api/graph — DSL read/write (backend holds a graph_state)
    • /api/training/start + /api/training/events (SSE) — training & progress
    • /api/ai/chat — AI agent loop (stream:true → SSE token stream)
    • /api/env/setup — environment setup (background thread + status polling)
    • /api/datasets/search — Hugging Face fuzzy search
  • Subprocess output decoded utf-8, errors=replace (GBK-safe on Windows)
  • Structured JSON lines prefixed @@NS_EVENT@@ / @@NS_RESULT@@ / @@NS_IO@@ are parsed & forwarded by the trainer

5. Python engine modules

Module Responsibility
main.py FastAPI entry, routes, global graph_state, AI tool context
dsl.py DSL structure/param validation + defaults + 11 optimizer schemas
shape_infer.py Per-layer shape inference & topology validation (authoritative)
codegen.py DSL → train/inference PyTorch scripts
trainer.py Training subprocess mgmt + SSE broadcast + safe stop + inference
ai_assistant.py DeepSeek agent loop (15 tools + self-heal + fallback reply)
data_pipeline.py Dataset platform: data-flow DSL validation + source codegen + dataop transforms
env_manager.py Env create/rebuild (conda first, venv fallback, Tsinghua mirror, hardware-based torch)
hardware.py CUDA / DirectML / CPU detection (detect(python) can probe a target interpreter)

6. Key design decisions (lessons learned)

  • flowToDsl never re-sorts layers by x-position — snake layout reverses even rows, x-sorting scrambles the order
  • Node animation is one-shot per new node (.ns-node-enter), never global on .react-flow__node
  • fitView is not a controlled prop — call rf.fitView() once on mount
  • Edge handles are redirected by node relative position; hover never changes stroke-width (hitbox jitter)
  • Unconnected nodes auto-gray (inactive) — BFS from input nodes; highlights/errors take priority
  • AI output layout: each turn's thinking/tool cards follow that turn's reply (pendingActs + flushActs)
  • AI multi-session + history + persistence: localStorage neuralblocks_* keys
  • Environment setup: progress 0-100 + rolling logs; WinError-5 retry; full China mirrors; SSL self-check
  • Hardware detection probes the training interpreter (_resolve_python()) so env-CUDA-torch isn't masked by system-CPU-torch
  • HF downloads route through hf-mirror (HF_ENDPOINT); pip/torch use Tsinghua/Aliyun mirrors

7. Layout

  • Left/right panes are full-height columns (top:48px → window bottom); bottom bar sits between them
  • Layout persisted to localStorage (neuralblocks_layout); last-saved model auto-restored on launch
  • App menu bar: File / Edit / View / Window / Help
  • Bottom uses VSCode-style tabs (Training / Inference)