PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices — Detailed Summary

Yangyijian Liu, Jun Li, Wu-Jun Li | School of Computer Science, Nanjing University, China | arXiv:2504.03664v2 (cs.DC), 13 Jun 2025 | Preprint, under review

Per-section summary organized by the paper's headings. Each section includes paragraph-level bullet points and exact quantitative results where the paper provides them.

Abstract

• Large language models (LLMs) have high memory and computation demand, making them hard to deploy on consumer devices that typically have limited GPU memory.
• Offloading can mitigate the memory constraint but often suffers from low GPU utilization, which yields low inference efficiency.
• The paper proposes a framework called pipelined offloading (PIPO) for efficient inference on consumer devices.
• PIPO designs a fine-grained offloading pipeline, complemented with optimized data transfer and computation, to achieve high concurrency and efficient scheduling.
• Headline empirical claim: compared with the state-of-the-art baseline, PIPO raises GPU utilization from below 40% to over 90% and achieves up to 3.1x higher throughput, running on a laptop with an RTX3060 GPU of 6GB memory.

1. Introduction

Background and motivation:

• LLMs show strong capability in writing, conversation, and code generation.
• Demand for privacy preservation and reduced deployment cost makes inference of LLMs on local devices (typically consumer devices) increasingly important.

Memory wall on consumer devices:

• High memory and computation demand makes LLMs hard to deploy on PCs, which use GPUs with limited memory.
• Example: 7B or 30B models require 15GB to tens of GB of memory for model weights; a 30B model can fit on an RTX3090 with 24GB but not on other RTX30 series cards with 6–12GB.
• The Key-Value (KV) cache, essential for optimizing attention computation, grows proportionally with sequence length and batch size, further worsening the memory insufficiency problem.

Three memory-reduction techniques:

• To fit LLMs into limited GPU memory, three techniques are widely used: quantization, sparsification, and offloading.
• Quantization uses lower precision to compress the model.
• Sparsification improves efficiency by pruning/removing weights and KV-cache.
• Offloading transfers weights and KV-cache from GPU to CPU memory or disk to overcome the GPU memory limit.

Why offloading is necessary:

• Quantization decreases model size by 4x with 4-bit precision (vs. 16-bit original), but this is still insufficient for a 30B model on a GPU with 8GB or less.
• Sparsification can decrease model size by up to 80% but suffers non-negligible accuracy degradation.
• Hence offloading is a necessary strategy, especially for relatively large LLMs on consumer devices.

Offloading mechanics and prior art:

• Offloading uses multi-level memory to schedule excessive memory demands.
• llama.cpp distributes model layers between CPU and GPU, leveraging both for inference; however, this hybrid approach is hindered by the CPU's limited compute ability on consumer devices, resulting in high inference latency.
• FlexGen focuses on GPU-centric computation, using a three-level memory hierarchy (GPU memory, CPU memory, disk); during inference, activation, weight, and KV-cache are scheduled to be transferred between these levels, overlapping with the GPU computation.

Two issues in existing offloading frameworks:

• Insufficient Inference Concurrency: LLMs are often memory-bound due to frequent low-arithmetic-intensity matrix multiplications, and offloading exacerbates the high data-transfer latency. Even with overlap techniques in FlexGen, its coarse-grained control and low concurrency cause substantial inefficiency. For OPT-30B with CPU-offloading, over 90% of inference time is spent on data transfer, with GPU computation accounting for only about 5%. This insufficient concurrency causes significant GPU idle time (low GPU utilization) and severely limits throughput.
• Underutilization of Disks' Bandwidth: Existing frameworks like FlexGen support disk-offloading but primarily rely on CPU-offloading, requiring up to 200GB of CPU memory to offload the entire model — far exceeding typical consumer device memory and neglecting the potential of disks. NVMe M.2 SSDs offer significantly higher bandwidth than HDDs, providing a more practical offloading solution, but existing methods do not effectively utilize SSD bandwidth and pay insufficient attention to disk-based optimizations.

Main contributions:

• PIPO designs a fine-grained offloading pipeline with high concurrency that optimally balances memory usage and inference efficiency, supporting larger models on limited GPU memory.
• PIPO emphasizes the crucial role of NVMe SSDs in offloading on consumer devices and designs a comprehensive suite to optimize data transfer speed between GPU memory, CPU memory, and disk.
• Experimental results: PIPO increases GPU utilization from below 40% to over 90% and achieves up to 3.1x higher throughput on a laptop with an RTX3060 GPU of 6GB memory.

2. Related Work

• Offloading-based inference frameworks have gained attention for deploying LLMs on local devices.
• DeepSpeed and HuggingFace Accelerate are among the first systems to integrate offloading into inference, directly adopting offloading techniques from training without in-depth study of inference-phase characteristics.
• FlexGen proposes an efficient offloading strategy via a search-based algorithm, optimizing data placement and overlapping computation with I/O.
• PowerInfer and LLM in a flash use sparsification by keeping hot neurons on the GPU while offloading the rest to the CPU; during inference, they dynamically fetch necessary neurons from storage. This improves inference speed but still suffers accuracy degradation and high CPU memory overhead.
• These offloading-based frameworks fall short of a thorough analysis of the associated performance overhead; FlexGen attempts to enhance efficiency through overlapping but still falls short of fully leveraging concurrency in the inference phase.
• An alternative is to distribute computation across GPU and CPU. llama.cpp partitions the model at the Transformer layer level, storing some weights on the GPU and offloading the rest to the CPU. The CPU processes its assigned layers first, then transfers results to the GPU for further computation. While this reduces PCIe data transfer, it results in substantial GPU idle time and slows down inference.

Quantization from an algorithmic perspective:

• Quantization is widely adopted to reduce memory footprint and accelerate computation.
• For weight-only quantization, recent research achieves extremely low-bit precision, down to 3 bits and even below 2 bits.
• However, such aggressive quantization faces significant challenges including deployment complexity and potential degradation in inference speed.
• In contrast, simultaneously quantizing weights, activations, and KV-cache to relatively low-bit precisions (e.g., W8A8, W4A4, W4A8KV4) balances memory efficiency and inference throughput.
• PIPO supports quantizing both weights and KV-cache to INT4 to accommodate limited consumer-device memory and enhance inference throughput.

3. Pipelined Offloading

• PIPO is an offloading inference framework with high throughput designed for consumer devices with limited GPU and CPU memory.
• Per Figure 1, the architecture consists of three key components: pipeline and thread pool, transfer suite and compute kernel optimizations, and automatic configuration.

                    model info        hardware spec / system load
  +-------+        -----------> +----------------+ <----------------- +-----------+
  |  LLM  |                     |   Automatic    |                    | Hardware  |
  | layers|                     | Configuration  |                    |  Disk     |
  +-------+                     +-------+--------+                    | (NVMe M.2)|
                                        |                            |  CPU/DRAM |
  +-------------------------------------v---------------------+      |  GPU/VRAM |
  | Task Queue --> Task Divide                                |      +-----------+
  |  [comp][wload][kvload][kvsave] ...                        |
  | +-----------+   +-------------------------------------+   |   +-------------+
  | | ThreadPool|   | Pipeline:                           |   |   | Transfer    |
  | | Thread 0  |   | load weight | load cache | ...      |   |   | Suite       |
  | | Thread 1  |   | InputEmbed->MHA->MLP->MHA-> ... ->  |   |   +-------------+
  | | Thread 2  |   | save cache       MLP->OutputEmbed   |   |   | Compute     |
  | | Thread 3  |   +-------------------------------------+   |   | Kernel      |
  | +-----------+                                             |   +-------------+
  +-----------------------------------------------------------+

• Pipeline and Thread Pool: a fine-grained inference pipeline with an efficient thread pool maximizes concurrency, enhancing hardware utilization and inference throughput.
• Transfer Suite and Compute Kernel Optimizations: a comprehensive suite optimizes data transfer between disk, CPU memory, and GPU memory; custom quantized compute kernels minimize computation latency.
• Automatic Configuration: PIPO automatically determines the most efficient offloading and pipeline strategies and resource allocation before execution, based on hardware specifications, model parameters, and system load.

Operational overview:

• PIPO starts by analyzing key input parameters: model details (model size, batch size, precision), hardware specifications, and current system workload.
• It then automatically configures the offloading and pipeline strategy and applies optimization techniques (data transfer and computation) to maximize performance.
• During inference, the workload is divided into four distinct task types — computation, weight loading, KV-cache loading, and KV-cache saving — sequentially organized into a task queue.
• A thread pool retrieves tasks from the queue and executes them in parallel, while the main thread coordinates synchronization to ensure fine-grained control and minimize unnecessary delays.
• The remaining subsections cover: offloading architecture, pipeline design, data transfer suite, computation optimization, and automatic configuration.

3.1 Offloading Architecture

3.1.1 Offloading Strategies

• PIPO stores the model's weights in GPU memory, CPU memory, or NVMe disk depending on available capacity.
• The KV-cache is stored in CPU memory; the GPU loads the required cache before computing the MHA layer and stores newly generated cache back to CPU memory after computation.
• After computing each layer, the outputs are retained in GPU memory as inputs for the subsequent layer, while the memory occupied by the weight and cache is released to accommodate the next layer.
• This strategy significantly reduces GPU memory usage by storing only one or a few layers (in the case of preloading) on the GPU at a time.
• SOTA models like LLaMA3.1 contain 62 and 162 layers in the 8B and 70B models respectively (treating MHA and MLP as separate layers); by loading only a small fraction of the model at once, PIPO drastically reduces memory usage, enabling most GPU memory to support long context lengths or large batch sizes.
• Alternative considered and rejected: storing as many weights as possible in GPU memory and loading the remainder from CPU/disk. Benefit: eliminates latency of loading weights already on the GPU. Drawbacks: (i) latency reduction is minimal since only a small portion of weights fit on GPU, and (ii) even with a larger portion stored, available GPU memory only accommodates small batch sizes, severely limiting inference throughput.

3.1.2 Task Design

• For higher performance and resource management, PIPO adapts and modifies existing task division approaches (from FlexGen), dividing inference into four task types: computation, weight loading, KV-cache loading, and KV-cache saving.
• Computation: encompasses MHA and MLP layers plus input/output embedding layers at the beginning/end of the model. Due to the sequential nature, all computation tasks must execute one by one.
• Weight Loading: transfers model weights from CPU memory or disk to GPU memory. Since weight loading is often the bottleneck (PCIe bandwidth limits), it must execute concurrently with computation. As weight loading latency is typically longer than computation, loading weights too early increases memory usage without improving performance. To address this, weight loading for each layer begins only after the previous loading task completes, overlapping with the computation of the previous layer.
• KV-cache Loading: transfers necessary KV-cache from CPU to GPU memory for token generation; scheduled to overlap with computation from the preceding layer. Since KV-cache is only used in MHA layers, its execution can be optimized by advancing it one layer ahead to overlap with computation from the previous MHA layer, further enhancing pipeline concurrency.
• KV-cache Saving: stores newly generated KV-pairs back to CPU memory for future token generation. Unlike loading tasks, it provides greater flexibility in synchronization. It is scheduled immediately after MHA computation, but its completion is only ensured when the saved cache is required for the current layer in the next token generation cycle.
• PIPO decouples preprocessing and postprocessing of weights and KV-cache from computation and assigns them to loading tasks, as their sequential dependencies with data transfer prevent concurrent execution.

3.2 Pipeline Design

• A well-designed pipeline is essential for high concurrency during inference while minimizing resource overhead.
• The pipelined inference is backed by a thread pool, which provides the foundation for concurrent execution and efficient task coordination.
• This section details thread pool configuration, pipeline scheduling, and the performance-memory tradeoff.

Algorithm 1 — Pipeline Scheduling:

for i in generation_length do
  for j in num_layer do
    CallLoadData(i, j)         # Preload weight and cache for subsequent layers
    PrepareInput(i, j)         # Prepare hidden and mask
    SynchronizeLoadTask(i, j)  # Synchronize current layer's data loading tasks
    Compute(i, j)              # Current layer's computation
    if layer[j] == MHA then
      CallStoreCache(i, j)     # Store KV-cache before loaded
    end if
  end for
end for

3.2.1 Thread Pool Configuration

• During inference, the thread pool retrieves tasks from the queue and assigns them to individual threads for execution.
• Keeping the thread pool compact and tidy is essential to maintain simplicity and avoid unnecessary overhead.
• Design principles: except for KV-cache saving, only one instance of each operation type can execute at a time due to the sequential nature of LLM inference. KV-cache saving supports multiple requests and has relatively delayed synchronization; it requires minimal compute and bandwidth and is assigned lower priority in the scheduling hierarchy.
• To reduce idle time from waiting for other threads, the main thread not only schedules tasks within the pipeline but also directly handles computation.
• Consequently, PIPO configures the thread pool size to three, corresponding to the data transfer types, while computation is handled outside the pool by the main thread.
• Threads within the pool are not statically assigned to specific tasks; since execution time varies across layers, this flexible scheduling lets threads dynamically handle incoming tasks, minimizing idle time.

3.2.2 Pipeline Scheduling

• Per Figure 2, PIPO's pipeline design significantly enhances pipelining efficiency and reduces idle time (bubbles) compared to FlexGen's overlapping approach. (In Figure 2, 'C' = 'Call' to instruct the device to execute a task, 'S' = 'Synchronize' to ensure completion of a single task / device sync in FlexGen.)
• By implementing task-level synchronization, PIPO achieves precise control over task execution, minimizing unnecessary delays between tasks. It also adjusts synchronization order and timing for each task, unlocking greater efficiency and improving GPU utilization.
• Pipeline scheduling process (per Algorithm 1): for a specific layer, PIPO first initiates loading of weights and KV-cache (if the next layer involves an MHA op) for the next layer. Simultaneously, it prepares input data for the current layer and synchronizes the data transfer tasks for the current layer to ensure necessary data is available for computation. Computation is executed on the main thread, with other threads concurrently handling data transfer tasks in the queue. After computation completes, PIPO first launches the KV-cache saving task (if the current layer is MHA) and stores the computation output.
• A key aspect: ensuring KV-cache saving finishes before the same layer's KV-cache loading in the next token generation loop. If saving tasks are delayed, the required cache cannot be correctly loaded. To address this, PIPO advances the completion check for KV-cache saving one layer earlier, ensuring the required cache is ready when needed for subsequent loading and computation.

3.2.3 Performance-Memory Tradeoff

• PIPO's pipeline achieves high performance but incurs certain memory overhead.
• For higher throughput, PIPO preloads the weight and KV-cache for the next layer, storing data of two layers in GPU memory simultaneously.
• PIPO also synchronizes KV-cache saving tasks to complete before the same layer in the next loop, which could temporarily store all generated KV-pairs on the GPU in extreme cases.
• During decoding, the amount of KV-pairs equals the number of MHA layers — far smaller than the KV-cache loaded for a single layer (equal to the input length). When decoding LLaMA3.1-8B, GPU memory usage remains under 2GB, feasible on almost all consumer GPUs.
• The prefill stage has significantly higher memory demand: it processes the entire input sequence and generates all KV-cache for the sentence at once, leading to substantial memory usage. To mitigate, PIPO reduces task concurrency but lowers memory usage, ensuring the model fits within available GPU memory. By synchronizing the cache-saving task before launching the next, only a single KV-pair occupies GPU memory at a time.
• At the lowest memory usage, PIPO only requires the weights and KV-cache of a single model layer to perform inference.
• Conclusion: PIPO offers two pipeline options — a performance-optimized pipeline and a memory-efficient pipeline — automatically chosen based on available resources (detailed in Section 3.5).

3.3 Data Transfer Suite

• To address data transfer challenges introduced by offloading, PIPO incorporates a specialized data transfer suite, replacing standard PyTorch and NumPy methods.

• These challenges arise from the need to transfer large data volumes across memory hierarchies, particularly from disk to GPU memory. The suite optimizes critical data movement and improves bandwidth utilization.

• Per Figure 4, the suite uses three techniques to enhance disk-to-GPU transfer efficiency in disk-offloading: blockwise transfer, multi-thread parallel transfer, and data merging.

• Blockwise Transfer: PIPO divides weight tensors into manageable blocks, transferring them in a pipelined manner across stages (disk -> CPU memory, and CPU memory -> GPU memory). For example, while one block is read into CPU memory, another block can be simultaneously transferred to GPU memory. This overlapping reduces idle time and boosts effective transfer speed (Figure 3 shows the blockwise data transfer timeline).

• Multi-thread Parallel Transfer: with each block further divided into smaller chunks, PIPO uses multiple CPU threads to load chunks from disk, while GPU threads manage their transfer to GPU memory. As soon as a CPU thread finishes its chunk, it signals the GPU thread to proceed, maintaining a continuous data flow and reducing idle time.

• Data Merging: during LLM inference, weight tensors within the same layer are loaded separately. To reduce overhead from frequent I/O requests, PIPO merges these weight tensors into a single tensor that can be loaded with a single request and further divided into multiple blocks for blockwise transfer, enabling higher parallelism and improved effective bandwidth.

• Beyond disk-to-GPU transfer, PIPO also leverages multi-thread parallel transfer and data merging for CPU-to-GPU transfers; these optimizations effectively manage data movement across each transfer stage.

3.4 Computation Optimization

• Although data transfer is the primary bottleneck in offloading inference, computation also requires careful optimization, particularly with quantized weights.
• In conventional approaches, quantized weights are dequantized into floating-point values (e.g., half-precision) before computation, incurring both time and memory overhead.
• To address this, PIPO introduces custom handwritten compute kernels that perform matrix-vector multiplication directly on 4-bit quantized weights, avoiding the dequantization operation.
• The kernels are highly optimized to leverage the full computational power of the GPU to improve performance, particularly in small-batch-size scenarios where the GPU is typically underutilized.

3.5 Automatic Configuration

• PIPO automatically configures the optimal offloading and pipeline strategies based on user-specified parameters, hardware specifications, and system load.
• Factors considered: model type, precision, batch size, prompt/generation lengths, GPU and CPU memory, PCIe bandwidth, and current resource utilization.
• Using these inputs, PIPO determines the offloading strategy by assigning weights to GPU memory, CPU memory, or disk based on memory constraints, and selects the pipeline strategy (performance-optimized vs. memory-efficient).

Configuration formalism (LLaMA3.1 family example):

• Model: l hidden layers, input dimension d, vocabulary size V; p is data type size (precision), b is batch size, s is input length (prompt + generated length).
• LLaMA3.1 uses Grouped-Query Attention (GQA), with h attention heads and h_kv KV heads.
• Hidden dimension: d_h = m * ceil( gamma * floor( (8/3) d ) / m ), where m and gamma are constant values.
• Total model weight size: W = 2*W_embed + l*(W_mha + W_mlp), where W_embed = p*d*V, W_mha = p*d*(2d + d*(h_kv/h) + 1), and W_mlp = p*d*(3*d_h + 1).
• Total KV-cache size: C = 2 * p*b*s*l * (h_kv/h).
• For peak memory, PIPO considers the prefill stage with preloading (which demands significantly more memory than decoding).
• Peak memory: M = max(M_mha, M_mlp, M_embed), where M_mha = p*b*s*(5d + hs) + W_mha + W_mlp + 2C/l, M_mlp = p*b*s*(3*d_h + 2d) + W_mha + W_mlp + C/l, M_embed = p*b*s*(V + d) + 2*W_embed (detailed in Appendix B).

System/hardware inputs and the configuration decision (Eq. 1):

• PIPO gathers: available GPU memory M_GPU, CPU memory M_CPU, GPU PCIe bandwidth B_GPU, and disk PCIe bandwidth B_SSD.

Weight on:  GPU,  if W + M < M_GPU
            CPU,  if W + C < M_CPU and B_SSD < B_GPU
            Disk, else

Pipeline:   Performance-optimized,  if M < M_GPU
            Memory-efficient,       else                          (Eq. 1)

• Once offloading and pipeline strategies are determined, PIPO adjusts parameters for optimal performance: enabling the Transfer Suite for offloading weights, determining the Block Size through experiments (Appendix A), and activating the Compute Kernel for INT4 weights to bypass dequantization overhead for batch sizes less than 16.
• PIPO delivers a fully automated, highly efficient inference, seamlessly adapting to diverse hardware and workloads with minimal manual intervention.

Algorithm 2 — PIPO Workflow:

Input: model M, batch size b, length s, precision p, CPU memory M_CPU,
       GPU memory M_GPU, GPU bandwidth B_GPU, SSD bandwidth B_SSD
S_off, S_pipe = Configure(M, b, s, p, M_CPU, M_GPU, B_GPU, B_SSD)  # Auto Config
InitModel(M, b, s, p)                       # Init Model Data
InitTransferSuitAndOperators(M, b, p, S_off)# Init PIPO components
ConstructTaskandQueue(M)                    # Build Inference Runtime
PipelineScheduling()                        # Generation (Call Algorithm 1)

4. Experiment

4.1 Experimental Setting

• Hardware: a Lenovo Thinkbook and a desktop. The Thinkbook is a typical consumer laptop equipped with NVIDIA RTX3060 (6GB), 16GB CPU memory, and 1TB M.2 SSD. (Desktop results are in Appendix C.5; Appendix D discusses combining PIPO with parallelism for multi-GPU.)
• Model: three LLM families — OPT (6.7B, 13B, 30B, 66B) to benchmark a wide range of model sizes; LLaMA3.1 (8B and 70B) for the latest models; and MoE models (results in Appendix C.4).
• Workload: text generation with varying sequence lengths and batch sizes. Prompt length set to 512 tokens; PIPO generates 32 tokens per prompt. Batch size ranges from 1 to 32. All models use FP16 (or BF16) and INT4 precision for weights, with intermediate activations in FP16 (or BF16). (More experiments on varying context lengths 512–3072 tokens are in Appendix C.3.)
• Implementation: PIPO is built upon a reconstruction and extension of FlexGen, retaining critical data structures while incorporating new modules in C++/CUDA and Python code to improve efficiency and offer extensibility.
• Baseline: FlexGen (SOTA in offloading inference). For fair comparison, the same weight storage type is used for both FlexGen and PIPO, and FlexGen is extended to support LLaMA models (which it does not natively support). All results are averaged over at least 3 independent runs.

4.2 Results

End-to-end throughput (Figure 5; X axis = weight storage type and batch size; 'G-4' = weight on GPU bs=4, 'C-8' = CPU-offloading bs=8, 'D-16' = disk-offloading bs=16):

• OPT-1.3B-FP16: PIPO achieves an average throughput improvement of 2.03x over FlexGen.
• For larger models like OPT-6.7B (~13GB weights) and LLaMA3.1-8B (>16GB weights), disk-offloading becomes necessary. PIPO achieves up to 3.10x higher throughput in disk-offloading and also outperforms FlexGen in CPU-offloading.
• Across the OPT and LLaMA3.1 families in INT4 format, PIPO outperforms FlexGen in all cases, achieving an average improvement of 1.97x and a peak improvement of 3.04x.
• Selected raw throughput numbers from Figure 5 (FlexGen / PIPO, tokens/s):
  • OPT-1.3B-FP16: G-4 (45.31 / 106.39), G-16 (66.01 / 130.13), C-48 (60.45 / 106.82), C-96 (OOM for FlexGen / 120.11).
  • LLaMa3.1-8B-FP16: D-1 (0.20 / 0.57), D-4 (0.81 / 2.51), D-8 (1.61 / 4.49), D-16 (OOM / 7.95).
  • LLaMa3.1-8B-INT4: C-4 (3.93 / 5.59), C-8 (9.61 / 9.68), D-16 (5.52 / 13.54), D-32 (18.34 / ...).
  • LLaMa3.1-70B-INT4: D-1 (0.02 / 0.06), D-4 (... / 0.14).
  • OPT-6.7B-INT4: C-4 (5.97 / 7.55), C-8 (3.99 / 12.95), D-16 (6.67 / 14.83), D-32 (10.60 / 21.65).
  • OPT-13B-INT4: D-1 (0.26 / 0.69), D-4 (1.02 / 3.12), D-8 (1.84 / 4.85), D-16 (3.14 / 7.55).
  • OPT-30B-INT4: D-1 (1.00 / 1.00), D-4 (0.44 / 1.14), D-12 (1.25 / 2.25), D-24 (2.03 / 3.70).
  • OPT-66B-INT4: D-1 (0.03 / 0.04), D-4 (0.11 / 0.14).

Transfer speed and GPU utilization (Appendix C.1):

• PIPO achieves a 26% improvement in disk-to-GPU transfer speed.
• PIPO enhances GPU utilization from 36% to 97% (Figure 8: LLaMA3.1-8B INT4 bs=16 -> 36% to 97%; OPT-30B INT4 bs=12 -> 37% to 93%).

Ablation study (Appendix C.2, Figure 9): confirms PIPO's pipeline scheduling contributes the most significant performance gain (1.97x speedup).

TTFT (Appendix C.6): PIPO achieves a 42.5% reduction in time-to-first-token.

Memory footprint (Appendix C.7, C.8): compared to a non-offload implementation, PIPO reduces VRAM usage by 66.4% with only 11.2% performance degradation.

5. Conclusion

• PIPO is a novel offloading inference framework for efficient LLM inference on consumer devices.
• By leveraging a fine-grained inference pipeline coupled with data-transfer and computation optimizations, PIPO significantly improves GPU utilization and inference throughput.
• It achieves a remarkable throughput improvement over the widely adopted FlexGen.
• PIPO's flexible architecture ensures adaptability to various LLMs, making it a promising solution for running LLMs locally on consumer desktops and laptops.

Appendix A — Experiments about Block Size

• Modern storage systems read/write in discrete units (pages/blocks); hardware is optimized for high-speed I/O on contiguous addresses. Reading a complete tensor in blocks is generally not adversely affected as long as block size is appropriately selected.
• PIPO measures transfer speeds across different data sizes on the target device (Figure 6, RTX3060 Thinkbook):
  • Disk to CPU: block size of 8MB achieves best performance (~12.2 GB/s peak; then declines toward ~9.6–10 GB/s at larger sizes).
  • CPU to GPU: block sizes above 32MB reach the bandwidth limitation (~22–23 GB/s), indicating optimal performance at this threshold.
• Hence PIPO adopts a block size of 32MB for overall data transfer on the target device.

Appendix B — Memory Constraints

• Demonstrates memory constraints of LLaMA3.1 with and without preloading for prefill and decoding stages.
• B.1 Prefill Stage base terms: M_input = pbsd, M_output = pbsd, M_qkv = 3pbsd, M_w = pbsd_h, M_attn = pbhs^2.
  • With/without preloading, full expansions are given for M_mha, M_mlp, M_embed (e.g., with preloading M_mha = pbs(5d+hs) + W_mha + W_mlp + C/l, expanding to pbs(5d+hs) + pd(2d + 2d*(h_kv/h) + 1) + pd(3d_h+1) + 2pbsd*(h_kv/h)).
• B.2 Decoding Stage (input length 1, significantly lower memory): M_input = pbd, M_output = pbd, M_qkv = 3pbd, M_w = pbd_h, M_attn = pbh. With preloading, M_mha = pb(5d+h) + W_mha + W_mlp + 2C/l, etc.

Appendix C — Supplementary Experimental Results

• C.1 Bandwidth and GPU Utilization (Figures 7, 8): PIPO's data transfer suite outperforms FlexGen (which relies on PyTorch's official implementation) when data size exceeds 8MB, maintaining high efficiency as size increases. GPU utilization increases from below 40% to over 90% (36%->97% on LLaMA3.1-8B bs=16; 37%->93% on OPT-30B bs=12), indicating faster transfer and more efficient scheduling.
• C.2 Ablation Study (Figure 9): progressively integrating PIPO's components on INT4 OPT-13B (bs=4, disk-offloading): FlexGen = 1.0x baseline; PIPO-base (pipeline scheduling alone) = 1.97x; +transfer suite = 2.41x; +compute kernel = 2.66x. Pipeline scheduling alone nearly doubles throughput.
• C.3 Extended Context Handling (Figures 10, 11): stress tests with longer prompt and generation lengths.
  • Figure 10 (LLaMA3.1-8B INT4, bs=1, SSD disk-offloading, prompt 512–3072): PIPO sustains ~1.17 down to ~0.9 tokens/s vs. FlexGen ~0.45 down to ~0.36.
  • Figure 11 (LLaMA3.1-8B INT4, bs=4, generation 128–1024): PIPO from ~3.3 down to ~2.25 tokens/s vs. FlexGen ~3.3 down to ~1.8; PIPO maintains stable inference performance.
• C.4 MoE Architecture Benchmark (Figure 12): Deploying very large dense models (LLaMA3.1-405B) on consumer devices is hard — over a 20 GB/s PCIe a 405B model takes ~40 seconds to traverse every Transformer layer. MoE models activate only a subset of experts per token. Mixtral 8x7B has 8 experts (2 selected); DeepSeek-R1 has 256 experts (8 selected -> only 30B params used per token's decoding), giving an advantage for offloading since only needed weights load. Challenge: experts used can't be predicted before the "gate" operator; opportunities for PIPO scheduling: cache loading/saving unaffected; DeepSeek-R1's shared (fixed) expert can be loaded in parallel; one expert's computation can overlap another's weight loading.
  • On an RTX3060 laptop: PIPO achieves 12.482 tokens/s on Mixtral 8x7B and finishes DeepSeek-R1 671B at 0.13 tokens/s (disk offloading for both). No existing works have deployed these models on devices with such limited VRAM (6GB) and DRAM (16GB).
  • Figure 12 (Mixtral-8x7B-INT4, cache on VRAM, 512 prompt): throughput by batch size — 4 (1.34), 8 (2.69), 16 (4.51), 32 (10.37), 64 (17.00).
• C.5 High-end Device Performance (RTX4090, 64GB): focus on larger models.
  • Table 1 — Throughput (tokens/s) on RTX4090: | Model | Weight on | FlexGen | PIPO | |---|---|---|---| | LLaMA3.1-70B-INT4 | CPU | 1.113 | 1.214 | | LLaMA3.1-70B-INT4 | Disk | 0.497 | 1.107 | | OPT-30B-FP16 | Disk | 1.113 | 1.214 |
  • Table 2 — RTX4090 with partial offloading: | Model | Weight on | FlexGen | PIPO | |---|---|---|---| | LLaMA3.1-70B-INT4 | 40% GPU, 60% CPU | 5.454 | 7.718 | | OPT-30B-FP16 | 45% GPU, 55% CPU | 6.870 | 8.665 |
• C.6 Latency Analysis — Table 3 (LLaMA3.1-8B, bs=1, disk offloading): | Context-length | TTFT FlexGen | TTFT PIPO | Decode FlexGen | Decode PIPO | |---|---|---|---|---| | 512 | 2.120 | 1.218 | 2.074 | 0.837 | | 1024 | 2.371 | 1.717 | 2.027 | 0.842 | | 1536 | 2.686 | 2.336 | 2.031 | 0.857 | | 2048 | 3.406 | 2.956 | 2.266 | 0.876 | (All seconds; PIPO cuts both TTFT and per-token decode latency substantially.)
• C.7 Offloading Overhead Analysis (Tables 4, 5): for memory-sufficient models, PIPO strategically disables offloading to maximize throughput. Applying disk/CPU offloading to memory-sufficient models reveals offloading overhead and shows PIPO's optimized disk offloading narrows the gap with CPU offloading.
  • Table 4 — LLaMA3.2-1B (FlexGen / PIPO tokens/s): GPU/GPU (132.770 / 130.716), GPU/CPU (127.392 / 128.551), CPU/GPU (32.666 / 32.636), CPU/CPU (31.227 / 31.233), Disk/GPU (3.852 / 16.982), Disk/CPU (3.622 / 16.307). Disk-offloading gives PIPO a ~4.4x advantage.
  • Table 5 — LLaMA3.1-8B (FlexGen / PIPO): CPU/GPU (5.497 / 6.058), CPU/CPU (5.449 / 6.009), Disk/GPU (1.926 / 4.925), Disk/CPU (1.765 / 4.695).
• C.8 Memory Footprint Profiling — Table 6 (OPT-6.7B INT4, prompt 256, bs=4): Under same settings PIPO consumes 200 MB more VRAM (preloading) but its transfer suite reduces DRAM usage by 2GB; with disk offloading PIPO achieves the same throughput as FlexGen while cutting DRAM usage by up to 10 GB; vs. non-offload, PIPO reduces VRAM by 66.4% at 11.2% performance degradation. | Weight/Cache | Framework | Throughput | VRAM (GB) | DRAM (GB) | |---|---|---|---|---| | GPU/GPU | FlexGen / PIPO | 7.184 / 9.174 | 5.424 / 5.666 | 8.90 / 6.30 | | GPU/CPU | FlexGen / PIPO | 6.837 / 9.172 | 5.166 / 5.342 | 7.90 / 7.30 | | CPU/GPU | FlexGen / PIPO | 7.089 / 8.151 | 1.796 / 1.904 | 16.39 / 15.18 | | CPU/CPU | FlexGen / PIPO | 6.460 / 8.028 | 1.438 / 1.486 | 16.57 / 16.49 | | Disk/GPU | FlexGen / PIPO | 2.276 / 6.729 | 1.896 / 2.012 | 9.28 / 6.35 | | Disk/CPU | FlexGen / PIPO | 2.122 / 6.428 | 1.438 / 1.522 | 9.75 / 7.28 |

Appendix D — Combination with Parallelism Techniques

• PIPO can be extended to multi-GPU by combining with Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) — simply modify the weight loading process (what is loaded and its destination). For loading one new layer during offloading: DP forces every GPU to load the layer; TP lets each GPU load its portion; PP requires only one GPU to load the layer.
• Since PCIe bandwidth is the bottleneck and inference speed hinges on data loading latency: DP performs worst, while TP achieves best performance when GPUs have isolated PCIe bandwidth. High-end consumer motherboards typically offer isolated PCIe channels; lower-end PCs may share bandwidth; servers usually provide isolated channels.
• Expert Parallelism (EP) — as in DeepSeek-R1 — can also be integrated: distribute experts within a MoE layer across GPUs and forward tokens to their experts; similar to applying TP to a whole MoE layer but only loading required experts, making it friendly to pipeline offloading.

Limitations (stated and implied)

• Single-GPU consumer device is the primary target; multi-GPU is discussed only conceptually (Appendix D), not benchmarked end-to-end.
• The MoE-offloading gate-prediction problem (experts used cannot be predicted before the gate operator) is acknowledged as an open challenge, not solved.
• Custom INT4 compute kernel benefit is scoped to small batch sizes (bs < 16); at larger batches dequantization-free kernels are not the active path.
• Block-size choice (32MB) is empirically tuned on one device (RTX3060 Thinkbook); optimal block size may differ on other storage/PCIe configurations.
• Quantization is restricted to INT4 weights and KV-cache; accuracy impact of the quantization is not quantified in the paper.
• PCIe / disk bandwidth is the fundamental ceiling; PIPO improves utilization and scheduling but cannot exceed the hardware transfer envelope.

Future Work / Open Problems

• Predicting/scheduling experts in MoE offloading before the gate operator to better overlap expert weight loading with computation.
• Deeper integration of DP/TP/PP/EP parallelism with pipelined offloading for multi-GPU consumer and server settings (Appendix D outlines but does not fully evaluate this).
• Generalizing automatic block-size and configuration selection across a wider range of consumer hardware.

NCCL / Collective Communication Relevance

• This paper does not use NCCL, AllReduce, Ring/Tree algorithms, or LL/LL128/ Simple protocols. The setting is single-GPU offloaded inference on a consumer device, where the bottleneck is the host-side PCIe/NVMe data path (disk -> CPU DRAM -> GPU VRAM), not inter-GPU collectives.
• The only contact with the collective-communication world is conceptual: Appendix D notes that DP/TP/PP/EP could extend PIPO to multi-GPU, where inter-GPU communication would re-enter the picture, but no collective is measured or tuned.