How to Run LLMs Locally
Why “small-footprint AI” is the new competitive edge
AI is now table stakes for service desks and ops teams. Yet every cloud call costs three things: latency, money and compliance risk. Small, highly-optimised local models flip the script—you stay in control, respond faster and slash spend.
The Problem – Cloud friction vs. on-prem agility
A single round-trip to a hyperscaler often adds 600–800 ms. With thousands of daily automations that means:
- Higher cost: €0.02–€0.12 per 1 000 tokens
- Uncertain compliance: DSGVO frowns on cross-border transfers
- Dev friction: Shifting APIs, rate limits, vendor lock-in
Local LLMs remove those blockers—but which model? What GPU? How do you expose an API for n8n?
Our Solution – Ollama registry + n8n runners
S&S Technologies bundles Ollama (a lightweight model server) with hardened n8n workflow runners. Ollama hosts models on-prem; n8n orchestrates prompts, context windows and post-processing. Together they deliver governed automation at scale. Automate. Optimize. Scale.
How It Works
1. Model Classes & Use-Cases
| Model class | Typical size | Best for | Example prompt |
|---|---|---|---|
| SLMs (Small Language Models) | 3–7 B params | Real-time chat, ticket triage | “Summarise this ticket in 20 words and assign a priority.” |
| Vision-capable LLMs | 7–13 B | OCR, screenshot analysis, invoice QA | “Tell me if this invoice is missing a PO number.” |
| “Thinking” Models (reasoning-tuned) | 7–13 B with larger context | Root-cause analysis, multi-step automations | “Given these log excerpts, hypothesise the fault and propose next steps.” |
2. Choosing Models
- Phi-3-mini-4K-instruct (4 B) – Blazing-fast SLM for chat / classification
- Gemma-2-7B-it – General-purpose “thinking” model with 8 K context
- Llava-1.6-7B – Adds image understanding; ideal for GUI screenshot QA
- OpenHermes-2.5-Mistral-7B – Strong reasoning, returns well-structured JSON
Tip: Keep one SLM plus one vision model always resident; spin up a heavier “thinking” model only when needed.
3. Hardware Checks
- CPU-only pilot: Modern 8-core laptop; expect 2–4 tokens / s
- GPU minimum: NVIDIA RTX 4060 (8 GB VRAM) or better; 15–25 tokens / s for a 7 B weight
- RAM: 16 GB for a single model; 32 GB if you hot-swap several
- Disk: .gguf files are 1–2 GB each—store on NVMe for fast loads
4. Installing Ollama & n8n — quick links
# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh
# then
ollama pull gemma:2-7b-it
ollama serve # http://localhost:11434
Full guide: read the Ollama installation docs.
Running n8n next to Ollama is just as simple:
- Self-host (Docker): follow the n8n Docker quick-start
- Cloud trial: spin up a 14-day SaaS instance on n8n Cloud
Either route works—the only difference is where the workflow engine lives.
5. Wiring Into n8n — built-in AI nodes
| # | Action | Why it matters |
|---|---|---|
| 1 | In Credentials → Ollama, set Base URL to http://localhost:11434 (or the container name). Drop an Ollama Model node (completions) or Ollama Chat Model node (chat). |
Native node = no custom HTTP calls and richer error handling |
| 2 | Single-shot prompts: wrap the model in Basic LLM Chain.Multi-tool reasoning: feed the model into an AI Agent and attach tool nodes (Web search, SQL, HTTP…). | Re-creates LangChain patterns inside n8n |
| 3 | Connect the chain/agent output to your output like a ticket system - like with the S&S Technologies Matrix42 n8n node (full tutorial at MATRIX42ARTICLELINK). | One node turns AI insight into ticket enrichment |
Business Impact
| Metric | Cloud LLM | Local Ollama + n8n | Improvement |
|---|---|---|---|
| Avg. response latency | 900 ms | 120 ms | ↓ 87 % |
| Cost per 1 000 tokens | €0.06 | €0.004 (energy) | ↓ 93 % |
| Data-residency risk | External | On-prem | Eliminated |
Clients typically report 30–50 % faster ticket resolution and ≈ 80 % lower AI spend within the first month.
Next Steps
One-Week Pilot Checklist
- Verify workstation or server meets GPU/RAM targets
- Install Ollama; pull Phi-3-mini and Gemma-2-7B
- Build your own workflow or Contact our team to help you get started
- Run test tickets and invoices through the new nodes
- Monitor latency and accuracy
- Decide production rollout
Ready to reclaim speed, sovereignty and budget? Contact our team to get started!
Tags: workflow automation, n8n, AI agents, ITSM automation, Salzburg IT services
S & S Technologies GmbH – Haspingerstraße 4, 5550 Radstadt, Salzburg, Austria
UID-Nr: ATU 77676212 | FN: 571385y (LG Salzburg)