Technology
WebLLM enables client-side language model execution via WebGPU
The companion engine to MLC LLM provides hardware-accelerated local inference inside browsers with OpenAI API compatibility.
The short version
- WebLLM runs open-source large language models entirely within client web browsers without server processing by utilizing WebGPU.
- Developers can integrate models like Llama 3, Mistral, Gemma, Phi, and Qwen using standard OpenAI API patterns, Web Workers, or Service Workers.
- Model caching is supported across several browser storage backends, with some features such as function calling still under active development.
Key facts
- WebLLM runs language model inference locally in web browsers using WebGPU acceleration without server dependencies.[Hacker News]
- The framework maintains API compatibility with OpenAI interfaces, including support for streaming, JSON-mode structured output, seeding, and logit-level controls.[Hacker News]
- WebLLM supports multiple open-source model families, including Llama, Phi, Gemma, Mistral, and Qwen, as well as custom models compiled in the MLC format.[Hacker News]
- The engine supports background processing through Web Workers and Service Workers, along with storage caching using the Cache API, IndexedDB, OPFS, and an experimental Chrome extension backend.[Hacker News]
What remains uncertain
- Native tool and function-calling capabilities remain a work in progress with only preliminary support currently implemented.[Hacker News]