Analysis and optimization of language models on embedded platforms
Escuela Técnica Superior de Ingeniería Informática, University of Seville — Bachelor's Thesis (Computer Engineering, Computer Engineering track), within a collaboration grant, supervised by Jorge Fernández Berni and Delia Velasco Montero (Dept. of Electronics and Electromagnetism) · Project lead
A systematic study of the feasibility and optimization of inference for small language models (SLMs, one to four billion parameters) on a Raspberry Pi 5 (BCM2712 SoC, 8 GB LPDDR4X), as a local alternative to cloud dependence in resource-constrained environments.
The thesis delivers two software contributions: MonitorSystem, a C++ real-time inference and hardware metrics capture system, and monitorviz, a Python library that automates the computation of derived metrics and the generation of comparative visualizations. On top of that infrastructure, six experimental series are designed and run, evaluating respectively the baseline characterization of six SLMs, quantization level, context window size, prefill batch size, inference engine, and the Hailo-10H hardware accelerator — combining throughput, energy efficiency and parameter count into a purpose-built figure of merit.
Among the findings: active cooling is an enabling requirement (removing it cuts throughput by up to 53% and drops four of the six models below the interactive-use threshold), 4-bit quantization is the optimal trade-off on this memory-bandwidth-bound platform, and llama.cpp beats Ollama on time-to-first-token by up to a factor of six.
The thesis was awarded highest honors, with a perfect grade of 10.