More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
Whichllm scans your hardware—GPU, CPU, RAM—and pulls live benchmark data from HuggingFace to rank the best local LLMs you can actually run. You install it with a single command (uvx whichllm@latest) or via brew and pip. Once set up, a simple whichllm or uvx whichllm@latest command shows your top picks, complete with model size, quant format, benchmark score, and estimated tokens per second. You can also simulate a hypothetical GPU (for example, RTX 4090), compare upgrade candidates side by side, and even reverse-plan the GPU you’d need to run a given model like “llama 3 70b.”
Under the hood, whichllm merges results from LiveBench, Artificial Analysis, Aider, multimodal/vision tests, Chatbot Arena ELO and the Open LLM Leaderboard. It tags each score by source—direct, variant, base, interpolated or self-reported—and applies confidence weights so stale or self-reported claims don’t outrank fresh, full-data records. Model size factors in as a log₂-scaled bonus, and runtime fit (VRAM overhead, activation memory, offload penalties) and quantization discounts adjust the final score. Speed adds a ±8 usability gate, with reported t/s, confidence ranges and notes on MoE’s active-vs-total split.
Whichllm also handles one-command chats (whichllm run “qwen 2.5 1.5b gguf”) or spits out ready-to-copy Python snippets (whichllm snippet “qwen 7b”). It auto-detects Apple Silicon, NVIDIA, AMD or CPU-only setups and caches results for offline use. For scripts, JSON output with detailed fields—speed_confidence, speed_range, estimated_tok_per_sec—lets you pipe results into jq or map HuggingFace IDs to Ollama model names. You can even filter by use case (coding, vision, math), set minimum speed thresholds, or force refreshes for the latest data.
The tool’s hardware tables show, for example, that on an RTX 4090 or 3090 you get Qwen3.6-27B in Q5_K_M at a score of 92.8 and around 27 t/s. On 8 GB VRAM (RTX 4060) the top pick is Qwen3-14B in Q3_K_M (score 71.0, ~22 t/s). CPU-only mode yields a 20B MoE model at about 6 t/s with a 45.2 score. With one command, you know not just what fits, but what delivers the best real-world performance.
Questions about this article
No questions yet.