How to Split a 27B LLM Across Mac & PC via llama.cpp RPC

Watch Codacus's Step-by-Step RPC Tutorial

Follow along with Codacus as he configures and benchmarks distributed pipeline parallelism live:

Subscribe to Codacus on YouTube for more ingenious distributed computing and local AI architecture guides.

How llama.cpp RPC Works Under the Hood

Standard multi-GPU inference within a single chassis relies on high-speed PCIe switches or NVLink bridges. However, llama.cpp RPC decouples compute backends completely:

• Client (Master Node): Loads the GGUF model metadata, handles user prompts, tokenization, KV cache scheduling, and samples the final logits.
• RPC Backend (Worker Node): Runs a lightweight TCP daemon (rpc-server). It receives computational graphs, allocates layer weights directly into its own memory space (Metal or CUDA), and returns intermediate activation tensors.
• Heterogeneous Acceleration: Because the RPC protocol operates at the GGML tensor level, the Master can be an Nvidia CUDA rig while the Worker is an Apple Silicon Metal machine (or vice versa)!

Step-by-Step Setup: Step 1. Compile rpc-server on Mac (Worker)

On your Apple Silicon Mac (M1/M2/M3/M4), compile llama.cpp with Metal acceleration:

# On your Mac terminal
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j --target rpc-server

# Start the RPC worker daemon listening on all interfaces
./build/bin/rpc-server -p 50052 -H 0.0.0.0

Make sure your Mac's local IP address (e.g. 192.168.1.50) is reachable from your PC, and allow port 50052 in macOS Firewall.

Step 2. Compile llama.cpp with RPC on PC (Master)

On your Nvidia PC (Linux or Windows with CUDA toolkit), compile with both CUDA and RPC client support enabled:

# On your Linux / WSL2 Nvidia PC
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DGGML_RPC=ON
cmake --build build --config Release -j

# Execute distributed inference across both machines!
./build/bin/llama-cli   -m models/qwen2.5-27b-instruct-q4_k_m.gguf   --rpc 192.168.1.50:50052   -ngl 99   -p "Explain quantum entanglement in simple terms:"

Codacus's Empirical Performance Benchmark

Codacus tested this exact topology splitting Qwen 27B across an Nvidia desktop (RTX 4070 12GB) and an Apple Silicon Mac Mini (16GB Unified Memory):

3 Critical Optimization Rules from Codacus

1. Use Wired Ethernet or Thunderbolt Networking: Wi-Fi introduces variable ping latency. Because tensor activations pass between layers after every forward pass, consistent sub-millisecond network round trips preserve maximum generation speed.
2. Match Layer Distribution to VRAM Capacities: Use --tensor-split to divide layer weights proportionally so neither device runs out of local VRAM and spills over into swap.
3. Enable Flash Attention on Both Nodes: Pass -fa on the master node to minimize intermediate activation sizes sent across the RPC socket.

Related Engineering Deep Dives

Explore our full suite of local AI optimization tutorials:

• Is Frontier Class Local AI Finally Practical? (177B Breakdown)
• One llama.cpp Update Made Local AI 65% Faster
• How to Run Dense LLMs on Low VRAM with D-Flash Caching