PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices

Yangyijian Liu, Jun Li, Wu-Jun Li | School of Computer Science, Nanjing University, China | arXiv:2504.03664v2 (cs.DC), 13 Jun 2025 | Preprint, under review

Problem

Large language models have high memory and computation demand, making them hard to deploy on consumer devices (PCs, laptops) that use GPUs with limited memory — a 7B–30B model needs 15GB to tens of GB just for weights, and the KV-cache grows with sequence length and batch size. Quantization (4x reduction) is insufficient for a 30B model on an 8GB GPU, and sparsification causes accuracy loss, so offloading weights/KV-cache to CPU memory or NVMe disk is necessary. But existing offloading frameworks have two issues: insufficient inference concurrency (for OPT-30B with CPU-offloading, over 90% of inference time is spent on data transfer and GPU computation is only ~5%, leaving the GPU idle) and underutilization of disk bandwidth (frameworks like FlexGen primarily rely on CPU-offloading, requiring up to 200GB of CPU memory, and do not exploit NVMe SSD bandwidth).

Core Insight

A fine-grained, task-level-synchronized offloading pipeline backed by a compact three-thread pool — combined with an NVMe-aware data transfer suite and dequantization-free INT4 compute kernels — converts the GPU-idle, transfer-bound offloading regime into a high-concurrency one, raising GPU utilization from below 40% to over 90% on a 6GB consumer GPU.

Method

PIPO has three key components: (1) a pipeline + thread pool, (2) a transfer suite

• compute kernel, and (3) automatic configuration.

• Offloading architecture: weights live in GPU memory, CPU memory, or NVMe disk per capacity; KV-cache lives in CPU memory and is loaded to GPU before the MHA layer and saved back after. Only one or a few layers reside on the GPU at a time. Inference is split into four task types: computation, weight loading, KV-cache loading, and KV-cache saving.
• Pipeline design: a compact thread pool of size three (one per data-transfer type), with computation handled by the main thread outside the pool. Threads are not statically assigned, allowing dynamic load balancing. Algorithm 1 schedules per layer: call data loading for the next layer, prepare input, synchronize the current layer's loads, compute, then store KV-cache (if MHA). KV-cache saving completion is advanced one layer early so the cache is ready before the next token-generation loop needs it.
• Two pipeline modes: a performance-optimized pipeline (preloads next-layer weight + KV-cache, two layers in GPU memory at once; decoding LLaMA3.1-8B stays under 2GB) and a memory-efficient pipeline (only a single layer's weights + KV-cache; one KV-pair on GPU at a time during prefill).
• Data transfer suite (replaces PyTorch/NumPy methods): blockwise transfer (pipelined disk->CPU->GPU), multi-thread parallel transfer (chunked, CPU threads signal GPU threads), and data merging (merge a layer's separate weight tensors into one tensor / single I/O request).
• Computation optimization: custom handwritten compute kernels do matrix-vector multiply directly on 4-bit quantized weights, avoiding dequantization, helping most at small batch sizes.
• Automatic configuration: given model, batch size, length, precision, GPU/CPU memory, and GPU/SSD bandwidth, Eq. 1 decides weight placement (GPU / CPU / Disk) and pipeline mode (performance-optimized if peak memory M < M_GPU, else memory-efficient); INT4 compute kernel is enabled for batch sizes < 16.

Experimental Setup

Headline Quantitative Results

End-to-end throughput vs. FlexGen:

• OPT-1.3B-FP16: average 2.03x improvement.
• Up to 3.10x higher throughput in disk-offloading for larger models (OPT-6.7B ~13GB, LLaMA3.1-8B >16GB).
• Across OPT and LLaMA3.1 in INT4: 1.97x average, 3.04x peak improvement; PIPO wins in all cases.
• Selected (FlexGen / PIPO, tokens/s): OPT-6.7B-INT4 C-8 (3.99 / 12.95), D-32 (10.60 / 21.65); LLaMA3.1-8B-FP16 D-8 (1.61 / 4.49), D-16 (OOM / 7.95).

GPU utilization: raised from below 40% to over 90% — 36%->97% (LLaMA3.1-8B INT4 bs=16) and 37%->93% (OPT-30B INT4 bs=12).

Transfer speed: 26% improvement in disk-to-GPU transfer speed (suite beats FlexGen's PyTorch implementation once data size exceeds 8MB).

Ablation (INT4 OPT-13B, bs=4, disk): pipeline scheduling alone = 1.97x; +transfer suite = 2.41x; +compute kernel = 2.66x over FlexGen — pipeline scheduling is the largest single contributor.

Latency (LLaMA3.1-8B, bs=1, disk): 42.5% TTFT reduction; e.g., at context 512, TTFT 2.120s->1.218s and decode latency 2.074s->0.837s.

Memory footprint: vs. a non-offload implementation, PIPO reduces VRAM by 66.4% at only 11.2% performance degradation; disk-offloading cuts DRAM usage by up to 10GB while matching FlexGen throughput.

MoE on a 6GB/16GB laptop: 12.482 tokens/s on Mixtral 8x7B; DeepSeek-R1 671B completed at 0.13 tokens/s (disk-offloading) — no prior work deployed these on such limited VRAM/DRAM.

High-end (RTX4090) partial offloading: LLaMA3.1-70B-INT4 (40% GPU/60% CPU) 5.454->7.718 tokens/s; OPT-30B-FP16 (45% GPU/55% CPU) 6.870->8.665 tokens/s.

Limitations

• Single-GPU consumer device is the primary target; multi-GPU (DP/TP/PP/EP) is discussed conceptually in Appendix D but not benchmarked end-to-end.
• MoE expert-prediction before the gate operator remains an open challenge, not solved.
• The INT4 dequantization-free compute kernel benefits only small batch sizes (bs < 16).
• The 32MB block size is tuned on one device (RTX3060 Thinkbook); the optimum may differ on other storage/PCIe configurations.
• Quantization is limited to INT4 weights and KV-cache; the accuracy impact is not quantified.
• PCIe/disk bandwidth is the fundamental ceiling; PIPO improves utilization and scheduling but cannot exceed the hardware transfer envelope.

Open Problems Called Out

1. Predicting/scheduling MoE experts before the gate operator to better overlap expert weight loading with computation.
2. Deeper integration of DP/TP/PP/EP parallelism with pipelined offloading for multi-GPU consumer and server settings.
3. Generalizing automatic block-size and configuration selection across a wider range of consumer hardware.