Skip to content
View JohnScheuer's full-sized avatar

Block or report JohnScheuer

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
JohnScheuer/README.md

Joao Felipe De Souza

Systems engineer focused on LLM inference infrastructure, GPU memory systems, and serving optimization.

I build simulators and tools to study the latency, memory, throughput, routing, and quantization tradeoffs inside LLM serving systems — and validate them against real GPU measurements.

LinkedIn

LLM Inference Stack — 15-Project Series

Project Focus Key Finding
kv-cache-compaction-lab KV-cache page compaction ThresholdCompaction dominates; 11 free-compaction points
prefix-cache-sim Prefix sharing with RadixTree LFU dominates under Zipf; multi-turn hit rate 60%+
llm-inference-scheduler Continuous batching scheduler ChunkedPrefill eliminates starvation; FCFS collapses under load
tensor-memory-allocator GPU tensor memory allocation Free-list beats buddy/slab for continuous size distributions
llm-serving-sim End-to-end LLM serving ChunkedPrefill + LFU: 41% lower TTFT p95, 94% prefix hit rate
speculative-decoding-sim Speculative decoding 6.06x max speedup; breakeven at cost_ratio = 0.25
moe-router-sim MoE routing and load balancing ExpertChoice best balance; NoisyTopK best practical tradeoff
admission-control-sim Admission control under overload Tight token budget maximizes goodput; proactive beats reactive
kv-cache-disaggregation-sim Prefill/decode disaggregation Disagg wins at arrival>=20 AND prompt>=1024; 29% TTFT gain
speculative-decoding-validation Real GPU validation Median 1.14x speedup with KV cache; simulation confirmed
quantization-impact-analyzer Weight quantization sensitivity INT8-g32: 1.8x compression, +0.13 PPL; group-wise reduces INT4 error 99%
latency-breakdown-simulator Where each millisecond goes Compute = 99.8%; prefix cache saves 17% TTFT; disagg adds 8-20% overhead
request-lifecycle-tracker Per-request event tracing 24 event types; full lifecycle from arrival to memory release
real-model-profiler Real GPU cost measurement Prefill: 30-70 us/tok; decode memory-bound at 5300-10800 us/tok single-req

All projects: C++20 or Python, quantitative results, open source.

Core Systems Insight

Optimizing one component in isolation is not enough.

  • The scheduler that minimizes TTFT hits OOM first under memory pressure
  • The prefix cache that saves compute also consumes memory
  • The allocator that reduces fragmentation can increase lookup cost
  • Speculative decoding can hurt throughput if the draft model is too expensive
  • The MoE router that achieves perfect balance sacrifices expert specialization
  • Admission control that accepts everything destroys goodput under overload
  • Disaggregation that eliminates interference pays KV transfer cost instead
  • Quantization that maximizes compression destroys model quality
  • The breakdown shows: compute dominates, overhead is real but small
  • Real measurements show: simulation defaults are correct for server-level batching

End-to-end systems thinking matters more than any single optimization.

Stack

  • C++20 -- allocators, schedulers, caches, routers, simulators, tracers
  • Python + PyTorch -- real model profiling, validation, quantization, sweeps
  • CMake + Ninja -- build system
  • RTX 2070 (8GB) -- GPU for real measurements

Currently Exploring

  • Waterfall/Gantt chart visualization from lifecycle traces
  • Concurrency model for realistic queue buildup under load
  • Adaptive K selection for speculative decoding based on real acceptance rates

Pinned Loading

  1. mini-llm-inference-engine mini-llm-inference-engine Public

    A pure Transformer inference engine written in C++ from scratch, with no dependencies on PyTorch, TensorFlow, or any other ML framework. It implements the entire modern LLM architecture (RoPE, RMSN…

    C++ 9 1

  2. hardware-aware-llm-runtime hardware-aware-llm-runtime Public

    Hardware-calibrated LLM inference performance model using Roofline theory and analytical batch optimization.

    C++

  3. llm-runtime-simulator llm-runtime-simulator Public

    Systems-level simulation of LLM serving runtime including paged KV cache, priority scheduling, SLA modeling, and adaptive QoS control.

    C++

  4. llm-serving-scheduler-engine llm-serving-scheduler-engine Public

    Discrete-event simulation engine modeling LLM serving under stochastic load: dynamic batching, SLA-aware admission control, and horizontally scalable multi-GPU (M/M/k) architecture.

    C++

  5. sm75-tensorcore-microkernel sm75-tensorcore-microkernel Public

    Instruction-level Tensor Core micro-kernel engineering on SM75 (RTX 2070), including PTX manual emission, shared-memory staging, and hardware-aware auto-tuning.

    Cuda

  6. llm-serving-trace-replay llm-serving-trace-replay Public

    Trace-driven discrete-event LLM serving simulator with SLA-aware GPU cluster sizing and prefill/decode resource optimization.

    Python