It’s no exaggeration to say the large language models that power AI bots like ChatGPT and Gemini are changing the world. Many of us are now relying on them for coding, writing, summarizing, and searching the web. Whatever directions AI goes in over the next few years, LLMs will be part of that future.
If you use ChatGPT, Claude, Perplexity, or any of the other AI platforms, then you’re using an LLM. What’s less commonly known is that these LLMs can be run locally, on your own computer. The key benefits are offline access and greater privacy, because you’re not sending anything to the cloud for anyone else to analyze or review.
You’re also not paying any AI companies a monthly subscription or hitting any usage rates. Numerous LLMs are available to download for free, including from big names like Meta and Google. Though they tend not to be as advanced or as speedy as the LLMs inside the apps you have to pay for, they’re capable enough for everyday use, and you can pick and choose between them as needed.
There is more maintenance involved here, and you lose some of the convenience that comes with just loading up the ChatGPT app. (You’ll need to handle updates yourself, for example.) However, you get a more personalized and private AI system all to yourself, and it’s not difficult to get started.
What You Need to Get Started
You can run local LLMs on Windows, macOS, and Linux, though macOS tends to be the preferred platform for most AI enthusiasts. Everything is more unified and consistent on Macs—unlike Windows PCs, only one company makes them—and Apple Silicon chips combine the CPU, GPU, and RAM together, which AI models like.
Whichever platform you go for, having plenty of RAM helps. The bare minimum is 8 GB, though if that’s all you’ve got, you’ll be somewhat limited in terms of the size of the LLMs you can run and how fast they’ll go. Sixteen gigs is better, and 32 or more is required if you want to use the biggest and fastest models. For the best results, you need plenty of VRAM on a dedicated GPU (anything above 8 GB will make a difference), as this type of memory is optimized for the kind of tasks AI models do.
If you’re on Windows, a dedicated Nvidia GPU helps a lot: Graphics chips are better for running AI processes than standard processors, which is partly why Nvidia and the AI boom are so closely interlinked. These dedicated graphics cards will also have their own RAM, which gives AI models extra thinking space.
So while there’s no “minimum spec” for running a local LLM, as much RAM as possible and a discrete graphics card will certainly help. You then need a piece of software to run the model (to act as the outer interface), plus the model itself. There are plenty of choices for both, so you can pick and choose as required.
For those getting started, LM Studio Bionic is generally considered to be the best AI app choice on Windows and macOS, and it’s free to use. Other popular and trusted options include vLLM, Llama.cpp, Ollama, and GPT4All, which are again available for multiple operating systems. These pieces of software are a little more technical and involved, though.
With an AI app chosen, the other decision is to settle on an LLM. The software programs we’ve mentioned will guide you toward some options, and there are also online repositories of models, the most well-known being Hugging Face (which has more than 3 million models available—quite the selection).
An Example Local LLM Setup
A whole host of setups are possible when it comes to local LLMs, but we’ll take you step by step through the process of setting up LM Studio Bionic on Windows, to give you an idea of how it works. Once you’ve downloaded the installation file from the website and have LM Studio Bionic up and running, click Create Project to get started, and give your first project a name.
A blank conversation window opens: If you click Choose a model in the prompt box, then Get local models, you’ll be taken to the main model picker for LM Studio Bionic. Each of the available models is listed together with its size, its popularity, and a few details about it—keep an eye out for the staff picks if you want some recommendations to get started with. Smaller models won’t be quite as capable, but they will download faster and take up less room on your system.
Back in the chat interface, everything works similar to a standard AI chatbot app. In the prompt box at the bottom you can choose between your AI models (if you have more than one LLM installed), and on the left of the prompt box you’ve got a + (plus) icon for submitting images and files (if this is supported by the current model).
On the left of the interface is the navigation pane, where you can switch between your projects. Via the Settings link (lower left) on the navigation pane you’ve got the preferences screen for LM Studio Bionic. There are a bunch of ways to customize the software here, from how deleted chats are handled to the interface elements onscreen.
Click Library in Settings to manage your existing models or the Explore menu entry to find new ones. If you need support for images and documents, look for multimodal models. (Technically, these build on top of the capabilities of an LLM, though they often get called LLMs anyway.)
The final part of the LM Studio interface that is important to know about is the right-hand sidebar, which can be shown or hidden via the button on the very top-right corner. These give you further options for managing files across projects and giving the program access to the file system on your computer, if needed.
