Choose an open model to run locally
Narrows down which open-weights model, size and quantization level fit your hardware and task, with trade-offs explained.
Help me choose an open-weights language model to run on my own machine.
Use case: {{USE_CASE}}
Hardware: {{HARDWARE}}
What matters most to me: {{PRIORITIES}}
1. Explain how model size (parameters) and quantization affect memory use, speed and quality, using my hardware as the example.
2. Recommend 2 to 3 size and quantization combinations that would fit, with a rough memory estimate for each.
3. Say which model families are known to be strong for my use case, and note that I should check current benchmarks because rankings change quickly.
4. Mention whether a smaller distilled model might be enough.
5. Suggest a simple local test: 10 prompts from my real use case and how to judge the results.
Be clear about which facts may be out of date.Example output
Shortlist: an 8B model at 4-bit (fits 16 GB, about 30 tokens/s) or a 14B model at 4-bit (tight fit, better reasoning). Avoid: 70B; needs ~40 GB even when quantised.
How to use
Give exact HARDWARE details: GPU model and VRAM, or Mac chip and unified memory, plus system RAM. Use tools like Ollama or LM Studio to try the shortlist with your own prompts.
Why it works
The prompt asks for the reasoning (memory maths, quality trade-offs) rather than just a name, so the advice still helps after new models come out. Flagging out-of-date facts matters because model rankings change every few months.