https://paymanshus.github.io/thoughts/ Essays on machine learning, agents, and engineering by Paymanshu Sharma. en-us Thu, 29 Oct 2026 15:38:03 GMT https://paymanshus.github.io/thoughts/llm-optimization-memory-gpu-cpu/ https://paymanshus.github.io/thoughts/llm-optimization-memory-gpu-cpu/ Sat, 14 Dec 2024 00:00:00 GMT How much GPU memory a model actually needs and how to cut it down — fp32, fp16 and bf16, what quantization trades away, LLM.int8() and bitsandbytes in practice, and the KV cache that grows on top of the weights. llm optimization quantization precision kv cache vram inference https://paymanshus.github.io/thoughts/on-device-llms-on-android/ https://paymanshus.github.io/thoughts/on-device-llms-on-android/ Sat, 14 Dec 2024 00:00:00 GMT Why running LLMs on phones is worth the trouble, and two working Android demos — a RAG document-querying app built on Phi-3-mini and ONNX Runtime, and a multi-LoRA adapter-switching app on Gemma 2B. on-device ai edge llm onnx runtime quantization android rag