How to Use Hermes Agent with Local AI Models

September 7, 2026

A complete guide to running Hermes Agent entirely on your own hardware, with zero API costs and no data leaving your machine.

Overview

Hermes Agent works with any OpenAI-compatible API endpoint. If a server implements /v1/chat/completions, you can point Hermes at it. This means you can use local models, GPU inference servers, multi-provider routers, or any third-party API.

There are two broad paths:

Managed Runtime (Desktop App): Hermes downloads and manages llama.cpp for you. You pick a model from a built-in catalog; Hermes handles memory fitting, context sizing, quantization, and GPU layer placement. Nothing leaves your computer.
Custom Endpoint (CLI / Headless): You run your own inference server (Ollama, llama.cpp, LM Studio, MLX, vLLM, etc.) and point Hermes at it via the “Custom endpoint” provider.

Installing Hermes Desktop

Download Hermes Agent from Nous Research.

Easiest way is to get Hermes Desktop.

Hermes Managed Runtime (Desktop App)

This is the simplest path. Available in the Hermes Desktop app.

If you are just installing it, you can configure your models now.

Select Run models locally

You can click set up for me and it will set up everything automatically with the chosen model.

Select the model and options

Test the agent

If you already have Hermes, follow these steps below to switch to local models.

Steps

Open Settings –> Providers –> Local Models (or choose Run models locally during onboarding).
Click Install runtime. Hermes downloads the official llama.cpp build for your hardware (a few hundred MB), verifies it, and keeps it updated.
Pick a model from the catalog and click Download.
Click Use. New chats now run on the local model.

How Hermes Chooses Models

Every model in the catalog is priced against your machine before you download:

Memory fit: Green = fits your GPU (runs entirely in GPU memory), Amber = uses system RAM (works but slower), Red = too big for this machine.
Context: The window the model starts with and the maximum it can grow to.
Download size: Of the build selected for your hardware.

Models ship in several quantizations. Hermes picks the highest-quality build that runs fully on your GPU; machines with less memory get a more compact build. Below 4-bit quantization, quality loss is too severe.

Memory Management (Automatic)

Models start at a context window that fits your GPU and grow toward their native maximum as conversations need more room.
Every recommended model gets at least a 64K context window.
When a model is larger than your GPU's memory, overflow goes to system RAM in the order that hurts least (expert weights first, never the attention cache).
Idle models unload after 15 minutes to free GPU memory; they reload automatically.

Approach 2: LM Studio

LM Studio is a desktop app for running local models with a GUI.

Setup

Open LM Studio and start the local server (the server icon).
It runs on http://localhost:1234 by default.
Load a model in the LM Studio UI.

Configure Hermes

hermes model

Select LM Studio. It auto-detects http://localhost:1234/v1. Pick a model from the discovered list.

Or in ~/.hermes/config.yaml:

model:
  default: "your-model-name"
  provider: "lmstudio"

Configuration Reference

Managed Runtime (Desktop App)

The managed runtime is controlled by the local_runtime section of config.yaml. The desktop UI writes these values for you; they're documented for CLI and headless use:

local_runtime:
  enabled: false     # true = start the managed server with Hermes
  backend: auto      # auto | cuda | metal | vulkan | hip | cpu
  tag: b10362        # pinned llama.cpp release; auto-updated

Models and runtime builds live under the Hermes home directory (models/ and runtimes/llamacpp/).

Custom Endpoint

model:
  default: "gemma4:31b"
  provider: "custom"
  base_url: "http://localhost:11434/v1"
  # api_key: optional, leave empty for local servers

Timeouts for Local Models

Hermes auto-detects local endpoints and relaxes streaming timeouts. Override if needed:

# ~/.hermes/.env
HERMES_API_TIMEOUT=1800        # 30 minutes for slow local models
HERMES_STREAM_READ_TIMEOUT=1800 # Socket-level read timeout

Running as a Gateway Bot

Expose Hermes as a Telegram bot running entirely on your hardware.

This way you can interact with it using a chat application you already use on your phone or laptop.

Create a bot via @BotFather and get the token.
Add to ~/.hermes/config.yaml:

model:
  default: "gemma4:31b"
  provider: "custom"
  base_url: "http://localhost:11434/v1"

platforms:
  telegram:
    enabled: true
    token: "YOUR_TELEGRAM_BOT_TOKEN"

Start the gateway:

hermes gateway

What You Can Do With It

You can chat with Hermes of course, but the point of an Agent is it can actually do more for you. Hermes comes with many tools for doing different things, like web search, coding, email, messaging, etc.

I asked it to build me a single page web based Towers of Hanoi game, and while it did a good job, it reversed order of the disks, putting larger on top of smaller.

Was able to quickly correct that, and after that, the game did work correctly.

Next I asked to to research NVDA stock:

Then plan an itinerary for a trip to New York City