Phi-3 Mini

Model overview

Phi-3 Mini is a 3.8-billion-parameter model from Microsoft designed from the ground up for local and edge deployment rather than server-scale serving. Despite its small size, it ships with a 128K token context window, making it useful for tasks that need to reference long documents or conversation history without needing a much larger model. It's a good fit for on-device assistants, lightweight RAG retrieval, and classification tasks embedded inside a larger pipeline, especially on hardware where a 7B+ model would be too heavy. It's released under the permissive MIT license, which makes it easier to evaluate for commercial use than many open-weight models with more restrictive terms — though you should still confirm the license text on the model card for your specific use case.

Best for

Local assistant workflows on 4–8 GB VRAM consumer GPUs, on-device inference, pipeline-embedded classification, and RAG retrieval on constrained hardware.

Who should use it

• Local assistant workflows on 4–8 GB VRAM consumer GPUs, on-device inference, pipeline-embedded classification, and RAG retrieval on constrained hardware.
• Builders who want local or self-hosted testing options.

Common workflows

• Low-resource local assistant workflows, edge AI, on-device inference
• small workflows
• edge workflows
• local workflows
• efficient workflows

Deployment and hardware notes

3.8B parameters. Q4_K_M requires approximately 4 GB VRAM — fits a 6 GB or 8 GB consumer GPU. Q8_0 requires approximately 6 GB VRAM. FP16 requires approximately 8 GB VRAM. CPU inference via llama.cpp with 8 GB system RAM minimum.

Practical hardware fit

Phi-3 Mini runs comfortably on 6-8 GB consumer GPUs at Q4_K_M quantization, and can run on CPU with roughly 8 GB of system RAM using llama.cpp — useful if you don't have a dedicated GPU at all.

License and usage notes

MIT. Open weights. Verify the exact model card and license terms for the checkpoint or hosted provider you use.

Strengths

• Open weights model option for Phi workflows.
• Local assistant workflows on 4–8 GB VRAM consumer GPUs, on-device inference, pipeline-embedded classification, and RAG retrieval on constrained hardware.
• Runs comfortably on a 6 GB or 8 GB consumer GPU at Q4_K_M. Use `ollama run phi3:mini` for the fastest local start.

Limitations

• Smaller capability headroom than 7B+ models; best for task-specific or constrained workflows rather than open-ended general assistant use.
• 3.8B parameters. Q4_K_M requires approximately 4 GB VRAM — fits a 6 GB or 8 GB consumer GPU. Q8_0 requires approximately 6 GB VRAM. FP16 requires approximately 8 GB VRAM. CPU inference via llama.cpp with 8 GB system RAM minimum.
• Context window and limits: 128K tokens.
• Verify the exact model card, provider docs, license, and serving support before production use.

Caveats

Its small parameter count means it has less capability headroom than 7B+ models for open-ended reasoning or complex multi-step tasks. It performs best on task-specific or constrained workflows rather than as a general-purpose assistant replacement.

Local workflow notes

Runs comfortably on a 6 GB or 8 GB consumer GPU at Q4_K_M. Use `ollama run phi3:mini` for the fastest local start.

Local runtimes: Ollama (phi3:mini), LM Studio, llama.cpp, Transformers

Platforms: Windows, macOS, Linux

VRAM fit by quantization level

Enter your GPU VRAM below to see which quantization of Phi-3 Mini fits and get the Ollama run command.

Sources to verify

Related resources

Continue with model source notes, local tools, and implementation guides related to this model.

Ready to run this model locally?