How to Benchmark LLMs on a Raspberry Pi 5 (llama.cpp, Step by Step)

Most "Raspberry Pi AI benchmark" numbers on the internet are one-off runs with unknown settings. If you want numbers you can trust - and compare - you need a method. This is the one we use.

1. Install llama.cpp

sudo apt update && sudo apt install build-essential cmake git -y
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j4

Build from source. The prebuilt binaries rarely match your kernel and flags, and a mismatched build can cost you 20% throughput.

2. Pick models that actually fit

A Pi 5 has 8 GB of RAM shared with the OS. Stay under ~5 GB for the model + context or you will swap and your numbers are garbage. Reliable picks in Q4 quantization: TinyLlama 1.1B, Qwen 1.5B/3B, Phi-3-mini (tight but works), Gemma 2B. Skip anything 7B+ unless you enjoy watching swap thrash.

3. Measure honestly

Use llama-bench with a fixed prompt and generation length, and log three numbers per run: prompt processing tokens/sec, generation tokens/sec, and SoC temperature at start and end. A run without a temperature is not a benchmark - thermal throttling on an uncooled Pi 5 shows up around 85°C and can cut throughput by a third mid-run.

./build/bin/llama-bench -m models/qwen-3b-q4.gguf -p 128 -n 256 -r 3

4. Keep runs repeatable

Fix the CPU governor: sudo apt install cpufrequtils, set performance for the run, note it down.
Same prompt set every time. Different prompts = different numbers.
Let the board cool between runs, or say so in the log.
Record ambient temperature if you compare across days.

Skip the spreadsheet work

We run this protocol often enough that we packaged it. The Raspberry Pi Local AI Benchmark Workbook (€6) has the log sheets, the test matrix, and the pass/fail criteria pre-built. When you have a pile of logs, the Benchmark Report Generator (€5) turns them into a clean report. And if throttling is your problem, the Thermal Test Planner (€4) lays out the cool-down test sequence properly.