Thoughts

2024.12.14

Fitting Really Big Models into Not-So-Big GPUs: Precision and Quantization

How much GPU memory a model actually needs and how to cut it down — fp32, fp16 and bf16, what quantization trades away, LLM.int8() and bitsandbytes in practice, and the KV cache that grows on top of the weights.

21 min read

2024.12.14

On-Device Large Language Models on Android

Why running LLMs on phones is worth the trouble, and two working Android demos — a RAG document-querying app built on Phi-3-mini and ONNX Runtime, and a multi-LoRA adapter-switching app on Gemma 2B.

12 min read