It depends on how complex you want to make it.

The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.

Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.

You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.

Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).

If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.

Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.

I’m running it through Llama.cpp but it will likely be available via LM Studio as well.