LangChainGo is a framework for developing applications powered by language models

> 🎉 **Join our new official Discord community!** Connect with other LangChain Go developers, get help and contribute: [Join Discord](https://discord.gg/t9UbBQs2rG)

# 🦜️🔗 LangChain Go

[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=flat-square)](https://pkg.go.dev/github.com/tmc/langchaingo)
[![scorecard](https://goreportcard.com/badge/github.com/tmc/langchaingo)](https://goreportcard.com/report/github.com/tmc/langchaingo)
[![](https://dcbadge.vercel.app/api/server/t9UbBQs2rG?compact=true&style=flat)](https://discord.gg/t9UbBQs2rG)
[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/tmc/langchaingo)
[<img src="https://github.com/codespaces/badge.svg" title="Open in Github Codespace" width="150" height="20">](https://codespaces.new/tmc/langchaingo)

⚡ Building applications with LLMs through composability, with Go! ⚡

## 🤔 What is this?

This is the Go language implementation of [LangChain](https://github.com/langchain-ai/langchain).

## 📖 Documentation

- [Documentation Site](https://tmc.github.io/langchaingo/docs/)
- [API Reference](https://pkg.go.dev/github.com/tmc/langchaingo)


## 🎉 Examples

See [./examples](./examples) for example usage.

```go
package main

import (
 "context"
 "fmt"
 "log"

 "github.com/tmc/langchaingo/llms"
 "github.com/tmc/langchaingo/llms/openai"
)

func main() {
 ctx := context.Background()
 llm, err := openai.New()
 if err != nil {
 log.Fatal(err)
 }
 prompt := "What would be a good company name for a company that makes colorful socks?"
 completion, err := llms.GenerateFromSinglePrompt(ctx, llm, prompt)
 if err != nil {
 log.Fatal(err)
 }
 fmt.Println(completion)
}
```

```shell
$ go run .
Socktastic
```

# Resources

Join the Discord server for support and discussions: [Join Discord](https://discord.gg/8bHGKzHBkM)

Here are some links to blog posts and articles on using Langchain Go:

- [Using Gemini models in Go with LangChainGo](https://eli.thegreenplace.net/2024/using-gemini-models-in-go-with-langchaingo/) - Jan 2024
- [Using Ollama with LangChainGo](https://eli.thegreenplace.net/2023/using-ollama-with-langchaingo/) - Nov 2023
- [Creating a simple ChatGPT clone with Go](https://sausheong.com/creating-a-simple-chatgpt-clone-with-go-c40b4bec9267?sk=53a2bcf4ce3b0cfae1a4c26897c0deb0) - Aug 2023
- [Creating a ChatGPT Clone that Runs on Your Laptop with Go](https://sausheong.com/creating-a-chatgpt-clone-that-runs-on-your-laptop-with-go-bf9d41f1cf88?sk=05dc67b60fdac6effb1aca84dd2d654e) - Aug 2023


# Contributors

There is a momentum for moving the development of langchaingo to a more community effort, if you are interested in being a maintainer or you are a contributor please join our [Discord](https://discord.gg/8bHGKzHBkM) and let us know.

<a href="https://github.com/tmc/langchaingo/graphs/contributors">
 <img src="https://contrib.rocks/image?repo=tmc/langchaingo" />
</a>

langchaingo

Open Source OpenAI alternative, self-host AI models

<h1 align="center">
 
 <img width="300" src="./core/http/static/logo.png"> 
 
</h1>


<a href="https://github.com/go-skynet/LocalAI/stargazers" target="blank">
<img src="https://img.shields.io/github/stars/go-skynet/LocalAI?style=for-the-badge" alt="LocalAI stars"/>
</a>
<a href='https://github.com/go-skynet/LocalAI/releases'>
<img src='https://img.shields.io/github/release/go-skynet/LocalAI?&label=Latest&style=for-the-badge'>
</a>
<a href="LICENSE" target="blank">
<img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge" alt="LocalAI License"/>
</a>



<a href="https://twitter.com/LocalAI_API" target="blank">
<img src="https://img.shields.io/badge/X-%23000000.svg?style=for-the-badge&logo=X&logoColor=white&label=LocalAI_API" alt="Follow LocalAI_API"/>
</a>
<a href="https://discord.gg/uJAeKSAGDy" target="blank">
<img src="https://img.shields.io/badge/dynamic/json?color=blue&label=Discord&style=for-the-badge&query=approximate_member_count&url=https%3A%2F%2Fdiscordapp.com%2Fapi%2Finvites%2FuJAeKSAGDy%3Fwith_counts%3Dtrue&logo=discord" alt="Join LocalAI Discord Community"/>
</a>



<a href="https://trendshift.io/repositories/5539" target="_blank"><img src="https://trendshift.io/api/badge/repositories/5539" alt="mudler%2FLocalAI | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>


**LocalAI** is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

- **Drop-in API compatibility** — OpenAI, Anthropic, ElevenLabs APIs
- **35+ backends** — llama.cpp, vLLM, transformers, whisper, diffusers, MLX...
- **Any hardware** — NVIDIA, AMD, Intel, Apple Silicon, Vulkan, or CPU-only
- **Multi-user ready** — API key auth, user quotas, role-based access
- **Built-in AI agents** — autonomous agents with tool use, RAG, MCP, and skills
- **Privacy-first** — your data never leaves your infrastructure

Created and maintained by [Ettore Di Giacinto](https://github.com/mudler).

> [:book: Documentation](https://localai.io/) | [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) | [💻 Quickstart](https://localai.io/basics/getting_started/) | [🖼️ Models](https://models.localai.io/) | [❓FAQ](https://localai.io/faq/)

## Guided tour

https://github.com/user-attachments/assets/08cbb692-57da-48f7-963d-2e7b43883c18

<details>

<summary>
Click to see more!
</summary>

#### User and auth

https://github.com/user-attachments/assets/228fa9ad-81a3-4d43-bfb9-31557e14a36c

#### Agents

https://github.com/user-attachments/assets/6270b331-e21d-4087-a540-6290006b381a

#### Usage metrics per user

https://github.com/user-attachments/assets/cbb03379-23b4-4e3d-bd26-d152f057007f

#### Fine-tuning and Quantization

https://github.com/user-attachments/assets/5ba4ace9-d3df-4795-b7d4-b0b404ea71ee

#### WebRTC

https://github.com/user-attachments/assets/ed88e34c-fed3-4b83-8a67-4716a9feeb7b

</details>

## Quickstart

### macOS

<a href="https://github.com/mudler/LocalAI/releases/latest/download/LocalAI.dmg">
 <img src="https://img.shields.io/badge/Download-macOS-blue?style=for-the-badge&logo=apple&logoColor=white" alt="Download LocalAI for macOS"/>
</a>

> **Note:** The DMG is not signed by Apple. After installing, run: `sudo xattr -d com.apple.quarantine /Applications/LocalAI.app`. See [#6268](https://github.com/mudler/LocalAI/issues/6268) for details.

### Containers (Docker, podman, ...)

> Already ran LocalAI before? Use `docker start -i local-ai` to restart an existing container.

#### CPU only:

```bash
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
```

#### NVIDIA GPU:

```bash
# CUDA 13
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13

# CUDA 12
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12

# NVIDIA Jetson ARM64 (CUDA 12, for AGX Orin and similar)
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64

# NVIDIA Jetson ARM64 (CUDA 13, for DGX Spark)
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13
```

#### AMD GPU (ROCm):

```bash
docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
```

#### Intel GPU (oneAPI):

```bash
docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intel
```

#### Vulkan GPU:

```bash
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan
```

### Loading models

```bash
# From the model gallery (see available models with `local-ai models list` or at https://models.localai.io)
local-ai run llama-3.2-1b-instruct:q4_k_m
# From Huggingface
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
# From the Ollama OCI registry
local-ai run ollama://gemma:2b
# From a YAML config
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
# From a standard OCI registry (e.g., Docker Hub)
local-ai run oci://localai/phi-2:latest
```

> **Automatic Backend Detection**: LocalAI automatically detects your GPU capabilities and downloads the appropriate backend. For advanced options, see [GPU Acceleration](https://localai.io/features/gpu-acceleration/).

For more details, see the [Getting Started guide](https://localai.io/basics/getting_started/).

## Latest News

- **March 2026**: [Agent management](https://github.com/mudler/LocalAI/pull/8820), [New React UI](https://github.com/mudler/LocalAI/pull/8772), [WebRTC](https://github.com/mudler/LocalAI/pull/8790), [MLX-distributed via P2P and RDMA](https://github.com/mudler/LocalAI/pull/8801), [MCP Apps, MCP Client-side](https://github.com/mudler/LocalAI/pull/8947)
- **February 2026**: [Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245), [ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396)
- **January 2026**: **LocalAI 3.10.0** — Anthropic API support, Open Responses API, video & image generation (LTX-2), unified GPU backends, tool streaming, Moonshine, Pocket-TTS. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0)
- **December 2025**: [Dynamic Memory Resource reclaimer](https://github.com/mudler/LocalAI/pull/7583), [Automatic multi-GPU model fitting (llama.cpp)](https://github.com/mudler/LocalAI/pull/7584), [Vibevoice backend](https://github.com/mudler/LocalAI/pull/7494)
- **November 2025**: [Import models via URL](https://github.com/mudler/LocalAI/pull/7245), [Multiple chats and history](https://github.com/mudler/LocalAI/pull/7325)
- **October 2025**: [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/) support for agentic capabilities
- **September 2025**: New Launcher for macOS and Linux, extended backend support for Mac and Nvidia L4T, MLX-Audio, WAN 2.2
- **August 2025**: MLX, MLX-VLM, Diffusers, llama.cpp now supported on Apple Silicon
- **July 2025**: All backends migrated outside the main binary — [lightweight, modular architecture](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)

For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [News page](https://localai.io/basics/news/).

## Features

- [Text generation](https://localai.io/features/text-generation/) (`llama.cpp`, `transformers`, `vllm` ... [and more](https://localai.io/model-compatibility/))
- [Text to Audio](https://localai.io/features/text-to-audio/)
- [Audio to Text](https://localai.io/features/audio-to-text/)
- [Image generation](https://localai.io/features/image-generation)
- [OpenAI-compatible tools API](https://localai.io/features/openai-functions/)
- [Realtime API](https://localai.io/features/openai-realtime/) (Speech-to-speech)
- [Embeddings generation](https://localai.io/features/embeddings/)
- [Constrained grammars](https://localai.io/features/constrained_grammars/)
- [Download models from Huggingface](https://localai.io/models/)
- [Vision API](https://localai.io/features/gpt-vision/)
- [Object Detection](https://localai.io/features/object-detection/)
- [Reranker API](https://localai.io/features/reranker/)
- [P2P Inferencing](https://localai.io/features/distribute/)
- [Distributed Mode](https://localai.io/features/distributed-mode/) — Horizontal scaling with PostgreSQL + NATS
- [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/)
- [Built-in Agents](https://localai.io/features/agents/) — Autonomous AI agents with tool use, RAG, skills, SSE streaming, and [Agent Hub](https://agenthub.localai.io)
- [Backend Gallery](https://localai.io/backends/) — Install/remove backends on the fly via OCI images
- Voice Activity Detection (Silero-VAD)
- Integrated WebUI

## Supported Backends & Acceleration

LocalAI supports **35+ backends** including llama.cpp, vLLM, transformers, whisper.cpp, diffusers, MLX, MLX-VLM, and many more. Hardware acceleration is available for **NVIDIA** (CUDA 12/13), **AMD** (ROCm), **Intel** (oneAPI/SYCL), **Apple Silicon** (Metal), **Vulkan**, and **NVIDIA Jetson** (L4T). All backends can be installed on-the-fly from the [Backend Gallery](https://localai.io/backends/).

See the full [Backend & Model Compatibility Table](https://localai.io/model-compatibility/) and [GPU Acceleration guide](https://localai.io/features/gpu-acceleration/).

## Resources

- [Documentation](https://localai.io/)
- [LLM fine-tuning guide](https://localai.io/docs/advanced/fine-tuning/)
- [Build from source](https://localai.io/basics/build/)
- [Kubernetes installation](https://localai.io/basics/getting_started/#run-localai-in-kubernetes)
- [Integrations & community projects](https://localai.io/docs/integrations/)
- [Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)
- [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
- [Examples](https://github.com/mudler/LocalAI-examples)

## Autonomous Development Team

LocalAI is helped being maintained by a team of autonomous AI agents led by an AI Scrum Master.

- **Live Reports**: [reports.localai.io](http://reports.localai.io)
- **Project Board**: [Agent task tracking](https://github.com/users/mudler/projects/6)
- **Blog Post**: [Learn about the experiment](https://mudler.pm/posts/2026/02/28/a-call-to-open-source-maintainers-stop-babysitting-ai-how-i-built-a-100-local-autonomous-dev-team-to-maintain-localai-and-why-you-should-too/)

## Citation

If you utilize this repository, data in a downstream project, please consider citing it with:

```
@misc{localai,
 author = {Ettore Di Giacinto},
 title = {LocalAI: The free, Open source OpenAI alternative},
 year = {2023},
 publisher = {GitHub},
 journal = {GitHub repository},
 howpublished = {\url{https://github.com/go-skynet/LocalAI}},
```

## Sponsors

> Do you find LocalAI useful?

Support the project by becoming [a backer or sponsor](https://github.com/sponsors/mudler). Your logo will show up here with a link to your website.

A huge thank you to our generous sponsors who support this project covering CI expenses, and our [Sponsor list](https://github.com/sponsors/mudler):


 <a href="https://www.spectrocloud.com/" target="blank">
 <img height="200" src="https://github.com/user-attachments/assets/72eab1dd-8b93-4fc0-9ade-84db49f24962">
 </a>
 <a href="https://www.premai.io/" target="blank">
 <img height="200" src="https://github.com/mudler/LocalAI/assets/2420543/42e4ca83-661e-4f79-8e46-ae43689683d6"> 
 </a>


### Individual sponsors

A special thanks to individual sponsors, a full list is on [GitHub](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler). Special shout out to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!

## Star history

[![LocalAI Star history Chart](https://api.star-history.com/svg?repos=go-skynet/LocalAI&type=Date)](https://star-history.com/#go-skynet/LocalAI&Date)

## License

LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/).

MIT - Author Ettore Di Giacinto <mudler@localai.io>

## Acknowledgements

LocalAI couldn't have been built without the help of great software already available from the community. Thank you!

- [llama.cpp](https://github.com/ggerganov/llama.cpp)
- https://github.com/tatsu-lab/stanford_alpaca
- https://github.com/cornelk/llama-go for the initial ideas
- https://github.com/antimatter15/alpaca.cpp
- https://github.com/EdVince/Stable-Diffusion-NCNN
- https://github.com/ggerganov/whisper.cpp
- https://github.com/rhasspy/piper
- [exo](https://github.com/exo-explore/exo) for the MLX distributed auto-parallel sharding implementation

## Contributors

This is a community project, a special thanks to our contributors!
<a href="https://github.com/go-skynet/LocalAI/graphs/contributors">
 <img src="https://contrib.rocks/image?repo=go-skynet/LocalAI" />
</a>

LocalAI

<a href="https://ollama.com">
 <img src="https://github.com/ollama/ollama/assets/3325447/0d0b44e2-8f4a-4e99-9b52-a5c1c741c8f7" alt="ollama" width="200"/>
 </a>


# Ollama

Start building with open models.

## Download

### macOS

```shell
curl -fsSL https://ollama.com/install.sh | sh
```

or [download manually](https://ollama.com/download/Ollama.dmg)

### Windows

```shell
irm https://ollama.com/install.ps1 | iex
```

or [download manually](https://ollama.com/download/OllamaSetup.exe)

### Linux

```shell
curl -fsSL https://ollama.com/install.sh | sh
```

[Manual install instructions](https://docs.ollama.com/linux#manual-install)

### Docker

The official [Ollama Docker image](https://hub.docker.com/r/ollama/ollama) `ollama/ollama` is available on Docker Hub.

### Libraries

- [ollama-python](https://github.com/ollama/ollama-python)
- [ollama-js](https://github.com/ollama/ollama-js)

### Community

- [Discord](https://discord.gg/ollama)
- [𝕏 (Twitter)](https://x.com/ollama)
- [Reddit](https://reddit.com/r/ollama)

## Get started

```
ollama
```

You'll be prompted to run a model or connect Ollama to your existing agents or applications such as `claude`, `codex`, `openclaw` and more.

### Coding

To launch a specific integration:

```
ollama launch claude
```

Supported integrations include [Claude Code](https://docs.ollama.com/integrations/claude-code), [Codex](https://docs.ollama.com/integrations/codex), [Droid](https://docs.ollama.com/integrations/droid), and [OpenCode](https://docs.ollama.com/integrations/opencode).

### AI assistant

Use [OpenClaw](https://docs.ollama.com/integrations/openclaw) to turn Ollama into a personal AI assistant across WhatsApp, Telegram, Slack, Discord, and more:

```
ollama launch openclaw
```

### Chat with a model

Run and chat with [Gemma 3](https://ollama.com/library/gemma3):

```
ollama run gemma3
```

See [ollama.com/library](https://ollama.com/library) for the full list.

See the [quickstart guide](https://docs.ollama.com/quickstart) for more details.

## REST API

Ollama has a REST API for running and managing models.

```
curl http://localhost:11434/api/chat -d '{
 "model": "gemma3",
 "messages": [{
 "role": "user",
 "content": "Why is the sky blue?"
 }],
 "stream": false
}'
```

See the [API documentation](https://docs.ollama.com/api) for all endpoints.

### Python

```
pip install ollama
```

```python
from ollama import chat

response = chat(model='gemma3', messages=[
 {
 'role': 'user',
 'content': 'Why is the sky blue?',
 },
])
print(response.message.content)
```

### JavaScript

```
npm i ollama
```

```javascript
import ollama from "ollama";

const response = await ollama.chat({
 model: "gemma3",
 messages: [{ role: "user", content: "Why is the sky blue?" }],
});
console.log(response.message.content);
```

## Supported backends

- [llama.cpp](https://github.com/ggml-org/llama.cpp) project founded by Georgi Gerganov.

## Documentation

- [CLI reference](https://docs.ollama.com/cli)
- [REST API reference](https://docs.ollama.com/api)
- [Importing models](https://docs.ollama.com/import)
- [Modelfile reference](https://docs.ollama.com/modelfile)
- [Building from source](https://github.com/ollama/ollama/blob/main/docs/development.md)

## Community Integrations

> Want to add your project? Open a pull request.

### Chat Interfaces

#### Web

- [Open WebUI](https://github.com/open-webui/open-webui) - Extensible, self-hosted AI interface
- [Onyx](https://github.com/onyx-dot-app/onyx) - Connected AI workspace
- [LibreChat](https://github.com/danny-avila/LibreChat) - Enhanced ChatGPT clone with multi-provider support
- [Lobe Chat](https://github.com/lobehub/lobe-chat) - Modern chat framework with plugin ecosystem ([docs](https://lobehub.com/docs/self-hosting/examples/ollama))
- [NextChat](https://github.com/ChatGPTNextWeb/ChatGPT-Next-Web) - Cross-platform ChatGPT UI ([docs](https://docs.nextchat.dev/models/ollama))
- [Perplexica](https://github.com/ItzCrazyKns/Perplexica) - AI-powered search engine, open-source Perplexity alternative
- [big-AGI](https://github.com/enricoros/big-AGI) - AI suite for professionals
- [Lollms WebUI](https://github.com/ParisNeo/lollms-webui) - Multi-model web interface
- [ChatOllama](https://github.com/sugarforever/chat-ollama) - Chatbot with knowledge bases
- [Bionic GPT](https://github.com/bionic-gpt/bionic-gpt) - On-premise AI platform
- [Chatbot UI](https://github.com/ivanfioravanti/chatbot-ollama) - ChatGPT-style web interface
- [Hollama](https://github.com/fmaclen/hollama) - Minimal web interface
- [Chatbox](https://github.com/Bin-Huang/Chatbox) - Desktop and web AI client
- [chat](https://github.com/swuecho/chat) - Chat web app for teams
- [Ollama RAG Chatbot](https://github.com/datvodinh/rag-chatbot.git) - Chat with multiple PDFs using RAG
- [Tkinter-based client](https://github.com/chyok/ollama-gui) - Python desktop client

#### Desktop

- [Dify.AI](https://github.com/langgenius/dify) - LLM app development platform
- [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) - All-in-one AI app for Mac, Windows, and Linux
- [Maid](https://github.com/Mobile-Artificial-Intelligence/maid) - Cross-platform mobile and desktop client
- [Witsy](https://github.com/nbonamy/witsy) - AI desktop app for Mac, Windows, and Linux
- [Cherry Studio](https://github.com/kangfenmao/cherry-studio) - Multi-provider desktop client
- [Ollama App](https://github.com/JHubi1/ollama-app) - Multi-platform client for desktop and mobile
- [PyGPT](https://github.com/szczyglis-dev/py-gpt) - AI desktop assistant for Linux, Windows, and Mac
- [Alpaca](https://github.com/Jeffser/Alpaca) - GTK4 client for Linux and macOS
- [SwiftChat](https://github.com/aws-samples/swift-chat) - Cross-platform including iOS, Android, and Apple Vision Pro
- [Enchanted](https://github.com/AugustDev/enchanted) - Native macOS and iOS client
- [RWKV-Runner](https://github.com/josStorer/RWKV-Runner) - Multi-model desktop runner
- [Ollama Grid Search](https://github.com/dezoito/ollama-grid-search) - Evaluate and compare models
- [macai](https://github.com/Renset/macai) - macOS client for Ollama and ChatGPT
- [AI Studio](https://github.com/MindWorkAI/AI-Studio) - Multi-provider desktop IDE
- [Reins](https://github.com/ibrahimcetin/reins) - Parameter tuning and reasoning model support
- [ConfiChat](https://github.com/1runeberg/confichat) - Privacy-focused with optional encryption
- [LLocal.in](https://github.com/kartikm7/llocal) - Electron desktop client
- [MindMac](https://mindmac.app) - AI chat client for Mac
- [Msty](https://msty.app) - Multi-model desktop client
- [BoltAI for Mac](https://boltai.com) - AI chat client for Mac
- [IntelliBar](https://intellibar.app/) - AI-powered assistant for macOS
- [Kerlig AI](https://www.kerlig.com/) - AI writing assistant for macOS
- [Hillnote](https://hillnote.com) - Markdown-first AI workspace
- [Perfect Memory AI](https://www.perfectmemory.ai/) - Productivity AI personalized by screen and meeting history

#### Mobile

- [Ollama Android Chat](https://github.com/sunshine0523/OllamaServer) - One-click Ollama on Android

> SwiftChat, Enchanted, Maid, Ollama App, Reins, and ConfiChat listed above also support mobile platforms.

### Code Editors & Development

- [Cline](https://github.com/cline/cline) - VS Code extension for multi-file/whole-repo coding
- [Continue](https://github.com/continuedev/continue) - Open-source AI code assistant for any IDE
- [Void](https://github.com/voideditor/void) - Open source AI code editor, Cursor alternative
- [Copilot for Obsidian](https://github.com/logancyang/obsidian-copilot) - AI assistant for Obsidian
- [twinny](https://github.com/rjmacarthy/twinny) - Copilot and Copilot chat alternative
- [gptel Emacs client](https://github.com/karthink/gptel) - LLM client for Emacs
- [Ollama Copilot](https://github.com/bernardo-bruning/ollama-copilot) - Use Ollama as GitHub Copilot
- [Obsidian Local GPT](https://github.com/pfrankov/obsidian-local-gpt) - Local AI for Obsidian
- [Ellama Emacs client](https://github.com/s-kostyaev/ellama) - LLM tool for Emacs
- [orbiton](https://github.com/xyproto/orbiton) - Config-free text editor with Ollama tab completion
- [AI ST Completion](https://github.com/yaroslavyaroslav/OpenAI-sublime-text) - Sublime Text 4 AI assistant
- [VT Code](https://github.com/vinhnx/vtcode) - Rust-based terminal coding agent with Tree-sitter
- [QodeAssist](https://github.com/Palm1r/QodeAssist) - AI coding assistant for Qt Creator
- [AI Toolkit for VS Code](https://aka.ms/ai-tooklit/ollama-docs) - Microsoft-official VS Code extension
- [Open Interpreter](https://docs.openinterpreter.com/language-model-setup/local-models/ollama) - Natural language interface for computers

### Libraries & SDKs

- [LiteLLM](https://github.com/BerriAI/litellm) - Unified API for 100+ LLM providers
- [Semantic Kernel](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/ai/ollama) - Microsoft AI orchestration SDK
- [LangChain4j](https://github.com/langchain4j/langchain4j) - Java LangChain ([example](https://github.com/langchain4j/langchain4j-examples/tree/main/ollama-examples/src/main/java))
- [LangChainGo](https://github.com/tmc/langchaingo/) - Go LangChain ([example](https://github.com/tmc/langchaingo/tree/main/examples/ollama-completion-example))
- [Spring AI](https://github.com/spring-projects/spring-ai) - Spring framework AI support ([docs](https://docs.spring.io/spring-ai/reference/api/chat/ollama-chat.html))
- [LangChain](https://python.langchain.com/docs/integrations/chat/ollama/) and [LangChain.js](https://js.langchain.com/docs/integrations/chat/ollama/) with [example](https://js.langchain.com/docs/tutorials/local_rag/)
- [Ollama for Ruby](https://github.com/crmne/ruby_llm) - Ruby LLM library
- [any-llm](https://github.com/mozilla-ai/any-llm) - Unified LLM interface by Mozilla
- [OllamaSharp for .NET](https://github.com/awaescher/OllamaSharp) - .NET SDK
- [LangChainRust](https://github.com/Abraxas-365/langchain-rust) - Rust LangChain ([example](https://github.com/Abraxas-365/langchain-rust/blob/main/examples/llm_ollama.rs))
- [Agents-Flex for Java](https://github.com/agents-flex/agents-flex) - Java agent framework ([example](https://github.com/agents-flex/agents-flex/tree/main/agents-flex-llm/agents-flex-llm-ollama/src/test/java/com/agentsflex/llm/ollama))
- [Elixir LangChain](https://github.com/brainlid/langchain) - Elixir LangChain
- [Ollama-rs for Rust](https://github.com/pepperoni21/ollama-rs) - Rust SDK
- [LangChain for .NET](https://github.com/tryAGI/LangChain) - .NET LangChain ([example](https://github.com/tryAGI/LangChain/blob/main/examples/LangChain.Samples.OpenAI/Program.cs))
- [chromem-go](https://github.com/philippgille/chromem-go) - Go vector database with Ollama embeddings ([example](https://github.com/philippgille/chromem-go/tree/v0.5.0/examples/rag-wikipedia-ollama))
- [LangChainDart](https://github.com/davidmigloz/langchain_dart) - Dart LangChain
- [LlmTornado](https://github.com/lofcz/llmtornado) - Unified C# interface for multiple inference APIs
- [Ollama4j for Java](https://github.com/ollama4j/ollama4j) - Java SDK
- [Ollama for Laravel](https://github.com/cloudstudio/ollama-laravel) - Laravel integration
- [Ollama for Swift](https://github.com/mattt/ollama-swift) - Swift SDK
- [LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/llm/ollama/) and [LlamaIndexTS](https://ts.llamaindex.ai/modules/llms/available_llms/ollama) - Data framework for LLM apps
- [Haystack](https://github.com/deepset-ai/haystack-integrations/blob/main/integrations/ollama.md) - AI pipeline framework
- [Firebase Genkit](https://firebase.google.com/docs/genkit/plugins/ollama) - Google AI framework
- [Ollama-hpp for C++](https://github.com/jmont-dev/ollama-hpp) - C++ SDK
- [PromptingTools.jl](https://github.com/svilupp/PromptingTools.jl) - Julia LLM toolkit ([example](https://svilupp.github.io/PromptingTools.jl/dev/examples/working_with_ollama))
- [Ollama for R - rollama](https://github.com/JBGruber/rollama) - R SDK
- [Portkey](https://portkey.ai/docs/welcome/integration-guides/ollama) - AI gateway
- [Testcontainers](https://testcontainers.com/modules/ollama/) - Container-based testing
- [LLPhant](https://github.com/theodo-group/LLPhant?tab=readme-ov-file#ollama) - PHP AI framework

### Frameworks & Agents

- [AutoGPT](https://github.com/Significant-Gravitas/AutoGPT/blob/master/docs/content/platform/ollama.md) - Autonomous AI agent platform
- [crewAI](https://github.com/crewAIInc/crewAI) - Multi-agent orchestration framework
- [Strands Agents](https://github.com/strands-agents/sdk-python) - Model-driven agent building by AWS
- [Cheshire Cat](https://github.com/cheshire-cat-ai/core) - AI assistant framework
- [any-agent](https://github.com/mozilla-ai/any-agent) - Unified agent framework interface by Mozilla
- [Stakpak](https://github.com/stakpak/agent) - Open source DevOps agent
- [Hexabot](https://github.com/hexastack/hexabot) - Conversational AI builder
- [Neuro SAN](https://github.com/cognizant-ai-lab/neuro-san-studio) - Multi-agent orchestration ([docs](https://github.com/cognizant-ai-lab/neuro-san-studio/blob/main/docs/user_guide.md#ollama))

### RAG & Knowledge Bases

- [RAGFlow](https://github.com/infiniflow/ragflow) - RAG engine based on deep document understanding
- [R2R](https://github.com/SciPhi-AI/R2R) - Open-source RAG engine
- [MaxKB](https://github.com/1Panel-dev/MaxKB/) - Ready-to-use RAG chatbot
- [Minima](https://github.com/dmayboroda/minima) - On-premises or fully local RAG
- [Chipper](https://github.com/TilmanGriesel/chipper) - AI interface with Haystack RAG
- [ARGO](https://github.com/xark-argo/argo) - RAG and deep research on Mac/Windows/Linux
- [Archyve](https://github.com/nickthecook/archyve) - RAG-enabling document library
- [Casibase](https://casibase.org) - AI knowledge base with RAG and SSO
- [BrainSoup](https://www.nurgo-software.com/products/brainsoup) - Native client with RAG and multi-agent automation

### Bots & Messaging

- [LangBot](https://github.com/RockChinQ/LangBot) - Multi-platform messaging bots with agents and RAG
- [AstrBot](https://github.com/Soulter/AstrBot/) - Multi-platform chatbot with RAG and plugins
- [Discord-Ollama Chat Bot](https://github.com/kevinthedang/discord-ollama) - TypeScript Discord bot
- [Ollama Telegram Bot](https://github.com/ruecat/ollama-telegram) - Telegram bot
- [LLM Telegram Bot](https://github.com/innightwolfsleep/llm_telegram_bot) - Telegram bot for roleplay

### Terminal & CLI

- [aichat](https://github.com/sigoden/aichat) - All-in-one LLM CLI with Shell Assistant, RAG, and AI tools
- [oterm](https://github.com/ggozad/oterm) - Terminal client for Ollama
- [gollama](https://github.com/sammcj/gollama) - Go-based model manager for Ollama
- [tlm](https://github.com/yusufcanb/tlm) - Local shell copilot
- [tenere](https://github.com/pythops/tenere) - TUI for LLMs
- [ParLlama](https://github.com/paulrobello/parllama) - TUI for Ollama
- [llm-ollama](https://github.com/taketwo/llm-ollama) - Plugin for [Datasette's LLM CLI](https://llm.datasette.io/en/stable/)
- [ShellOracle](https://github.com/djcopley/ShellOracle) - Shell command suggestions
- [LLM-X](https://github.com/mrdjohnson/llm-x) - Progressive web app for LLMs
- [cmdh](https://github.com/pgibler/cmdh) - Natural language to shell commands
- [VT](https://github.com/vinhnx/vt.ai) - Minimal multimodal AI chat app

### Productivity & Apps

- [AppFlowy](https://github.com/AppFlowy-IO/AppFlowy) - AI collaborative workspace, self-hostable Notion alternative
- [Screenpipe](https://github.com/mediar-ai/screenpipe) - 24/7 screen and mic recording with AI-powered search
- [Vibe](https://github.com/thewh1teagle/vibe) - Transcribe and analyze meetings
- [Page Assist](https://github.com/n4ze3m/page-assist) - Chrome extension for AI-powered browsing
- [NativeMind](https://github.com/NativeMindBrowser/NativeMindExtension) - Private, on-device browser AI assistant
- [Ollama Fortress](https://github.com/ParisNeo/ollama_proxy_server) - Security proxy for Ollama
- [1Panel](https://github.com/1Panel-dev/1Panel/) - Web-based Linux server management
- [Writeopia](https://github.com/Writeopia/Writeopia) - Text editor with Ollama integration
- [QA-Pilot](https://github.com/reid41/QA-Pilot) - GitHub code repository understanding
- [Raycast extension](https://github.com/MassimilianoPasquini97/raycast_ollama) - Ollama in Raycast
- [Painting Droid](https://github.com/mateuszmigas/painting-droid) - Painting app with AI integrations
- [Serene Pub](https://github.com/doolijb/serene-pub) - AI roleplaying app
- [Mayan EDMS](https://gitlab.com/mayan-edms/mayan-edms) - Document management with Ollama workflows
- [TagSpaces](https://www.tagspaces.org) - File management with [AI tagging](https://docs.tagspaces.org/ai/)

### Observability & Monitoring

- [Opik](https://www.comet.com/docs/opik/cookbook/ollama) - Debug, evaluate, and monitor LLM applications
- [OpenLIT](https://github.com/openlit/openlit) - OpenTelemetry-native monitoring for Ollama and GPUs
- [Lunary](https://lunary.ai/docs/integrations/ollama) - LLM observability with analytics and PII masking
- [Langfuse](https://langfuse.com/docs/integrations/ollama) - Open source LLM observability
- [HoneyHive](https://docs.honeyhive.ai/integrations/ollama) - AI observability and evaluation for agents
- [MLflow Tracing](https://mlflow.org/docs/latest/llms/tracing/index.html#automatic-tracing) - Open source LLM observability

### Database & Embeddings

- [pgai](https://github.com/timescale/pgai) - PostgreSQL as a vector database ([guide](https://github.com/timescale/pgai/blob/main/docs/vectorizer-quick-start.md))
- [MindsDB](https://github.com/mindsdb/mindsdb/blob/staging/mindsdb/integrations/handlers/ollama_handler/README.md) - Connect Ollama with 200+ data platforms
- [chromem-go](https://github.com/philippgille/chromem-go/blob/v0.5.0/embed_ollama.go) - Embeddable vector database for Go ([example](https://github.com/philippgille/chromem-go/tree/v0.5.0/examples/rag-wikipedia-ollama))
- [Kangaroo](https://github.com/dbkangaroo/kangaroo) - AI-powered SQL client

### Infrastructure & Deployment

#### Cloud

- [Google Cloud](https://cloud.google.com/run/docs/tutorials/gpu-gemma2-with-ollama)
- [Fly.io](https://fly.io/docs/python/do-more/add-ollama/)
- [Koyeb](https://www.koyeb.com/deploy/ollama)
- [Harbor](https://github.com/av/harbor) - Containerized LLM toolkit with Ollama as default backend

#### Package Managers

- [Pacman](https://archlinux.org/packages/extra/x86_64/ollama/)
- [Homebrew](https://formulae.brew.sh/formula/ollama)
- [Nix package](https://search.nixos.org/packages?show=ollama&from=0&size=50&sort=relevance&type=packages&query=ollama)
- [Helm Chart](https://artifacthub.io/packages/helm/ollama-helm/ollama)
- [Gentoo](https://github.com/gentoo/guru/tree/master/app-misc/ollama)
- [Flox](https://flox.dev/blog/ollama-part-one)
- [Guix channel](https://codeberg.org/tusharhero/ollama-guix)

Ollama

Manage, load-balance, and failover packs of Ollamas

[![Go Report Card](https://goreportcard.com/badge/github.com/presbrey/ollamafarm)](https://goreportcard.com/report/github.com/presbrey/ollamafarm)
[![codecov](https://codecov.io/gh/presbrey/ollamafarm/branch/main/graph/badge.svg)](https://codecov.io/gh/presbrey/ollamafarm)
![Go Test](https://github.com/presbrey/ollamafarm/workflows/Go%20Test/badge.svg)
[![GoDoc](https://godoc.org/github.com/presbrey/ollamafarm?status.svg)](https://godoc.org/github.com/presbrey/ollamafarm)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

# OllamaFarm

OllamaFarm is a Go package that manages multiple Ollama instances, providing a convenient way to interact with a farm of Ollama servers. It offers features like automatic offline detection and failover, model availability tracking, and server selection based on criteria such as model.

## Installation

To install OllamaFarm, use the following command:

```bash
go get github.com/presbrey/ollamafarm
```

## Usage

Here's an example of how to use OllamaFarm with multiple Ollamas in the same group and different priorities:

```go
package main

import (
    "context"
    "fmt"
    "log"

    "github.com/presbrey/ollamafarm"
    "github.com/ollama/ollama/api"
)

func main() {
    farm := ollamafarm.New()

    // Register Ollama servers in the same group with different priorities
    farm.RegisterURL("http://ollama1:11434", &ollamafarm.Properties{Group: "4090", Priority: 1})
    farm.RegisterURL("http://ollama2:11434", &ollamafarm.Properties{Group: "4090", Priority: 2})
    farm.RegisterURL("http://ollama3:11434", &ollamafarm.Properties{Group: "3090", Priority: 1})

    // Select an Ollama instance
    ollama := farm.First(&ollamafarm.Where{Model: "llama3.1:8b-instruct-fp16"})
    if ollama != nil {
        // Perform a Chat call
        req := &api.ChatRequest{
            Model: "llama3.1:8b-instruct-fp16",
            Messages: []api.Message{
                {Role: "user", Content: "How many letter R are in the word Strawberry?"},
            },
        }

        err := ollama.Client().Chat(context.Background(), req, func(resp api.ChatResponse) error {
            fmt.Print(resp.Message.Content)
            return nil
        })

        if err != nil {
            log.Fatalf("Chat error: %v", err)
        }
    }

    // Get model counts
    modelCounts := farm.ModelCounts(nil)
    fmt.Printf("Available models: %v\n", modelCounts)
}
```

Note: When an Ollama instance goes offline, OllamaFarm automatically selects the next online Ollama with the highest priority (lowest priority number) within the same group. This ensures continuous operation and optimal resource utilization without manual intervention.

## API Reference

### Types

* `Farm`: The main struct that manages multiple Ollama instances.
* `Ollama`: Represents an individual Ollama server.
* `Options`: Defines the options for a Farm. All fields are optional.
  ```go
  type Options struct {
      Client     *http.Client
      Heartbeat  time.Duration
      ModelsTTL  time.Duration
  }
  ```
* `Properties`: Defines the properties of an Ollama client. All fields are optional.
  ```go
  type Properties struct {
      Group    string
      Offline  bool
      Priority int
  }
  ```
* `Where`: Defines the selection criteria for Ollama clients.
  ```go
  type Where struct {
      Group   string
      Model   string
      Offline bool
  }
  ```

### Functions

* `New() *Farm`: Creates a new Farm instance with default options.
* `NewWithOptions(options *Options) *Farm`: Creates a new Farm instance with the given options.

### Farm Methods

* `RegisterClient(id string, client *api.Client, properties *Properties)`: Adds a new Ollama to the Farm if it doesn't already exist.
* `RegisterURL(baseURL string, properties *Properties) error`: Adds a new Ollama to the Farm using the baseURL as the ID.
* `First(where *Where) *Ollama`: Returns the first Ollama that matches the given criteria.
* `Select(where *Where) []*Ollama`: Returns a list of Ollamas that match the given criteria, sorted by ascending Priority.
* `ModelCounts(where *Where) map[string]uint`: Returns a count of all models available across all registered Ollamas.

### Ollama Methods

* `Client() *api.Client`: Returns the Ollama client.
* `Farm() *Farm`: Returns the Farm that this Ollama belongs to.
* `Group() string`: Returns the Ollama's group.
* `Online() bool`: Returns whether the Ollama is online.
* `Priority() int`: Returns the Ollama's priority.

## Contributing

Contributions to OllamaFarm are welcome! Please note the following guidelines:

1. All pull requests must maintain or improve the existing test coverage.
2. New features or changes must not break any existing APIs.
3. Write clear, concise commit messages.
4. Follow Go best practices and style guidelines.

## License

This project is licensed under the [MIT LICENSE](LICENSE.txt) file in the root directory of this repository.


OllamaFarm

Go SDK for building AI applications. One SDK, 20+ providers. Inspired by Vercel AI SDK

goai

OpenTelemetry-native LLM observability and budget guardrails for cost-constrained production environments

# Otellix

**Production-grade LLM observability for Go backends — built for cost-constrained markets.**

![LLM Observability Dashboard Mockup](assets/dashboard_mockup.png)

[![CI](https://github.com/oluwajubelo1/otellix/actions/workflows/ci.yml/badge.svg)](https://github.com/oluwajubelo1/otellix/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/oluwajubelo1/otellix/graph/badge.svg)](https://codecov.io/gh/oluwajubelo1/otellix)
[![Go Reference](https://pkg.go.dev/badge/github.com/oluwajubelo1/otellix.svg)](https://pkg.go.dev/github.com/oluwajubelo1/otellix)
[![Go Report Card](https://goreportcard.com/badge/github.com/oluwajubelo1/otellix)](https://goreportcard.com/report/github.com/oluwajubelo1/otellix)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## 📖 Comprehensive Documentation

*   [**Architecture Overview**](docs/architecture.md) — How Otellix works with OpenTelemetry.
*   [**LLM Providers**](docs/providers.md) — Setting up OpenAI, Anthropic, Gemini, and Ollama.
*   [**Budget Guardrails**](docs/budget-guardrails.md) — Implementing cost policies and mid-stream cutoffs.
*   [**Native Streaming**](docs/streaming.md) — High-performance real-time observability.

## Why Otellix

- **Go-Native first.** No Python "sidecars" or heavy frameworks needed. If you're building LLM features in Go, Otellix is your production-ready bridge between AI and Observability.
- **Standards-based.** Built entirely on **OpenTelemetry**. Traces, spans, and metrics are standard-compliant. Switch between Jaeger, Honeycomb, Datadog, or Grafana without changing your code.
- **Cost Guardrails.** The only Go SDK with **Budget Cutoffs** that block or notify mid-stream when limits are hit.
- **Ollama Ready.** Full support for local LLMs via Ollama, with real-time NDJSON streaming and zero external dependencies.

## Quick Start

```bash
go get github.com/oluwajubelo1/otellix
```

### Simple Tracing

```go
package main

import (
    "context"
    "github.com/oluwajubelo1/otellix"
    "github.com/oluwajubelo1/otellix/providers/openai"
)

func main() {
    shutdown := otellix.SetupDev()
    defer shutdown()

    provider := openai.New() // Uses OPENAI_API_KEY env

    result, err := otellix.Trace(context.Background(), provider, 
        providers.CallParams{
            Model: "gpt-4o",
            Messages: []providers.Message{{Role: "user", Content: "Hello world"}},
        },
        otellix.WithFeatureID("chat"),
        otellix.WithUserID("user_007"),
    )
    // ...
}
```

## Native Real-time Streaming

Otellix supports native Go 1.23 iterators for streaming. It tracks token accumulation in real-time and can **abort a stream mid-flight** if the user hits their budget limit.

```go
stream, err := otellix.TraceStream(ctx, provider, params,
    otellix.WithUserID("user_123"),
    otellix.WithBudgetConfig(budgetCfg),
)
defer stream.Close()

for {
    evt, err := stream.Recv()
    if err != nil {
        if strings.Contains(err.Error(), "budget") {
            fmt.Println("\n[!] CUTOFF: User ran out of money mid-response!")
        }
        break
    }
    fmt.Print(evt.Token) // Watch it stream live...
}
```

## Features

| Feature | Description | Providers |
|---|---|---|
| **Real-time Streaming** | Native Go iterators with usage tracking | OpenAI, Anthropic, Gemini, Ollama |
| **Cost Attribution** | USD cost per call/user/feature/project | All Cloud Providers |
| **Budget Guardrails** | Per-user/project daily ceilings (Block/Notify/Downgrade) | All |
| **Local LLM Native** | Direct NDJSON parsing for Ollama (no overhead) | Ollama |
| **Prompt Optimization**| Dynamic prompts based on real-time budget telemetry | All |
| **Standard OTel** | 100% compliant Spans, Attributes, and Metrics | All |

## Local LLM Observability (Ollama)

Otellix treats **Ollama** as a first-class provider. It implements a zero-dependency, high-performance NDJSON parser to ensure your local traces are as detailed as your cloud production environment.

```go
provider := ollama.New(ollama.WithBaseURL("http://localhost:11434"))
// Everything else is the same — Otellix abstracts the details.
```

## Budget Guardrails & Fallbacks

| Mode | Behaviour |
|---|---|
| `FallbackBlock` | Block the call (or abort the stream) immediately if limit is hit. |
| `FallbackNotify` | Proceed with the call but emit a `budget.warning` OTel event. |
| `FallbackDowngrade`| Automatically swap `gpt-4o` for `gpt-4o-mini` if budget is low. |

## OTel Attributes (Standardized)

Otellix populates your spans with standardized attributes for powerful querying in Grafana/Honeycomb:
*   `llm.provider`, `llm.model`, `llm.feature_id`, `llm.user_id`, `llm.project_id`
*   `llm.input_tokens`, `llm.output_tokens`, `llm.cost_usd`, `llm.latency_ms`
*   `llm.budget_blocked`, `llm.prompt_fingerprint`

## Grafana Dashboard

A pre-built dashboard is included in `examples/docker-compose`. View live token usage, cost growth, and budget utilization in seconds.

```bash
cd examples/docker-compose && docker-compose up
```

## Supported Providers

*   **Anthropic**: Claude 3.5 Sonnet, Opus, Haiku (with prompt caching metrics).
*   **OpenAI**: GPT-4o, GPT-4o-mini, GPT-3.5-Turbo.
*   **Google Gemini**: Gemini 1.5 Pro & Flash (via Native SDK).
*   **Ollama**: Any local model (Llama3, Mistral, Phi-3).

## Contributing & License

We love contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md).
Licensed under **MIT**.
LICENSE).


otellix

YAML-driven multi-agent AI runtime for Go with Erlang-style supervision, MCP tool server support, and a CLI

routex

AI gateway for routing, securing, and monitoring LLM traffic across 10+ providers. OpenAI-compatible API, WASM policy plugins, canary rollouts, real-time dashboard

AegisFlow

A Go toolkit for building AI agents and applications across multiple providers with unified LLM, embeddings, tool calling, and MCP integration

# Go AI Client Library

[![Go Reference](https://pkg.go.dev/badge/github.com/joakimcarlsson/ai.svg)](https://pkg.go.dev/github.com/joakimcarlsson/ai)
[![Go Report Card](https://goreportcard.com/badge/github.com/joakimcarlsson/ai)](https://goreportcard.com/report/github.com/joakimcarlsson/ai)
[![codecov](https://codecov.io/gh/joakimcarlsson/ai/branch/main/graph/badge.svg)](https://codecov.io/gh/joakimcarlsson/ai)

A comprehensive, multi-provider Go library for interacting with various AI models through unified interfaces. Supports LLMs, embeddings, image generation, audio generation, speech-to-text, and rerankers from 10+ providers with streaming, tool calling, structured output, and MCP integration.

**[Documentation](https://joakimcarlsson.github.io/ai)** | **[API Reference](https://pkg.go.dev/github.com/joakimcarlsson/ai)**

## Features

- **Multi-Provider Support** — Unified interface for 10+ AI providers
- **LLM Support** — Chat completions, streaming, tool calling, structured output
- **Agent Framework** — Multi-agent orchestration with sub-agents, handoffs, fan-out, session management, persistent memory, and context strategies
- **Embedding Models** — Text, multimodal, and contextualized embeddings
- **Image Generation** — Text-to-image with OpenAI, Gemini, and xAI
- **Audio** — Text-to-speech (ElevenLabs, OpenAI, Google Cloud, Azure Speech) and speech-to-text (OpenAI Whisper, ElevenLabs Scribe, Deepgram, AssemblyAI, Google Cloud)
- **Rerankers** — Document reranking with Voyage AI and Cohere
- **Batch Processing** — Async bulk requests via native batch APIs (OpenAI, Anthropic, Gemini) or bounded concurrent execution for any provider
- **MCP Integration** — Model Context Protocol support for advanced tooling
- **Cost Tracking** — Built-in token and character usage with cost calculation

## Installation

```bash
go get github.com/joakimcarlsson/ai
```

## Quick Start

```go
package main

import (
 "context"
 "fmt"
 "log"

 "github.com/joakimcarlsson/ai/message"
 "github.com/joakimcarlsson/ai/model"
 llm "github.com/joakimcarlsson/ai/providers"
)

func main() {
 client, err := llm.NewLLM(
 model.ProviderOpenAI,
 llm.WithAPIKey("api-key"),
 llm.WithModel(model.OpenAIModels[model.GPT4o]),
 )
 if err != nil {
 log.Fatal(err)
 }

 messages := []message.Message{
 message.NewUserMessage("Hello, how are you?"),
 }

 response, err := client.SendMessages(context.Background(), messages, nil)
 if err != nil {
 log.Fatal(err)
 }

 fmt.Println(response.Content)
}
```

## Supported Providers

| Provider | LLM | Embeddings | Images | TTS | STT | Rerankers |
|----------|-----|------------|--------|-----|-----|-----------|
| OpenAI | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Anthropic | ✅ | | | | | |
| Google Gemini | ✅ | ✅ | ✅ | | | |
| Google Cloud | | | | ✅ | ✅ | |
| AWS Bedrock | ✅ | ✅ | | | | |
| Azure OpenAI | ✅ | | | | | |
| Azure Speech | | | | ✅ | | |
| Vertex AI | ✅ | | | | | |
| Groq | ✅ | | | | | |
| OpenRouter | ✅ | | | | | |
| xAI | ✅ | | ✅ | | | |
| Voyage AI | | ✅ | | | | ✅ |
| Cohere | ✅ | ✅ | | | | ✅ |
| Mistral | ✅ | ✅ | | | | |
| ElevenLabs | | | | ✅ | ✅ | |
| Deepgram | | | | | ✅ | |
| AssemblyAI | | | | | ✅ | |

## Agent Framework

```go
import (
 "github.com/joakimcarlsson/ai/agent"
 "github.com/joakimcarlsson/ai/agent/session"
)

myAgent := agent.New(llmClient,
 agent.WithSystemPrompt("You are a helpful assistant."),
 agent.WithTools(&weatherTool{}),
 agent.WithSession("user-123", session.FileStore("./sessions")),
)

response, _ := myAgent.Chat(ctx, "What's the weather in Tokyo?")
```

The agent framework supports [sub-agents](https://joakimcarlsson.github.io/ai/agent/sub-agents/), [handoffs](https://joakimcarlsson.github.io/ai/agent/handoffs/), [fan-out](https://joakimcarlsson.github.io/ai/agent/fan-out/), [team coordination](https://joakimcarlsson.github.io/ai/agent/team-coordination/), [continue/resume](https://joakimcarlsson.github.io/ai/agent/continue/), [context strategies](https://joakimcarlsson.github.io/ai/agent/context-strategies/), [persistent memory](https://joakimcarlsson.github.io/ai/agent/memory/), and [instruction templates](https://joakimcarlsson.github.io/ai/agent/instruction-templates/).

## Batch Processing

Process bulk requests with native batch APIs or bounded concurrency.

```go
import (
 "github.com/joakimcarlsson/ai/batch"
 "github.com/joakimcarlsson/ai/model"
)

// Native batch API — OpenAI, Anthropic, or Gemini
proc, _ := batch.New(
 model.ProviderOpenAI,
 batch.WithAPIKey("your-api-key"),
 batch.WithModel(model.OpenAIModels[model.GPT4o]),
)

// Concurrent fallback — works with any provider
proc, _ := batch.New(
 model.ProviderGroq,
 batch.WithLLM(groqClient),
 batch.WithMaxConcurrency(10),
)

requests := []batch.Request{
 {ID: "q1", Type: batch.RequestTypeChat, Messages: msgs1},
 {ID: "q2", Type: batch.RequestTypeChat, Messages: msgs2},
}

resp, _ := proc.Process(ctx, requests)
for _, r := range resp.Results {
 fmt.Printf("[%s] %s\n", r.ID, r.ChatResponse.Content)
}
```

Per-item error handling, progress callbacks, and async channel-based tracking are all supported. See the [batch processing docs](https://joakimcarlsson.github.io/ai/advanced/batch-processing/) for details.

## Versioning

Each module is versioned independently using path-prefixed git tags. The tag
prefix **must** match the subdirectory path exactly — this is how the Go module
system resolves versions.

| Module | Tag format | Example |
|--------|-----------|---------|
| Root | `vX.Y.Z` | `v0.15.0` |
| postgres | `integrations/postgres/vX.Y.Z` | `integrations/postgres/v0.1.0` |
| sqlite | `integrations/sqlite/vX.Y.Z` | `integrations/sqlite/v1.1.0` |
| pgvector | `integrations/pgvector/vX.Y.Z` | `integrations/pgvector/v0.1.0` |

All modules follow [semantic versioning](https://semver.org). The root module
and integration modules are versioned independently.

## Release Process

Releases follow the AWS SDK v2 pattern: CI on main is the safety net, git tags
drive `go get` resolution, and dated GitHub Releases provide changelogs.

### 1. Ensure main is green

CI must pass on the latest commit before tagging.

### 2. Tag modules that changed

```bash
# Tag a single module (dry-run — creates local tag only)
scripts/release.sh tag -m postgres -v v0.1.0

# Tag and push
make release-tag MODULE=postgres VERSION=v0.1.0
```

For integration modules, the `require` version for the root module in
`integrations/<name>/go.mod` must match the latest published root tag.
The script warns if this is stale.

### 3. Warm the Go module proxy

```bash
scripts/release.sh warm -t integrations/postgres/v0.1.0
```

This ensures the tagged version is immediately available via `go get`.

### 4. Create a dated GitHub Release

```bash
# Dry-run (shows what would be published)
scripts/release.sh release

# Publish
make release-publish
```

This creates a `release-YYYY-MM-DD` tag and a GitHub Release listing all
module versions tagged since the previous release.

## License

See [LICENSE](LICENSE) file.

Embeddable vector database for Go with Chroma-like interface and zero third-party dependencies. In-memory with optional persistence

# chromem-go

[![Go Reference](https://pkg.go.dev/badge/github.com/philippgille/chromem-go.svg)](https://pkg.go.dev/github.com/philippgille/chromem-go)
[![Build status](https://github.com/philippgille/chromem-go/actions/workflows/go.yml/badge.svg)](https://github.com/philippgille/chromem-go/actions/workflows/go.yml)
[![Go Report Card](https://goreportcard.com/badge/github.com/philippgille/chromem-go)](https://goreportcard.com/report/github.com/philippgille/chromem-go)
[![GitHub Releases](https://img.shields.io/github/release/philippgille/chromem-go.svg)](https://github.com/philippgille/chromem-go/releases)

Embeddable vector database for Go with Chroma-like interface and zero third-party dependencies. In-memory with optional persistence.

Because `chromem-go` is embeddable it enables you to add retrieval augmented generation (RAG) and similar embeddings-based features into your Go app *without having to run a separate database*. Like when using SQLite instead of PostgreSQL/MySQL/etc.

It's *not* a library to connect to Chroma and also not a reimplementation of it in Go. It's a database on its own.

The focus is not scale (millions of documents) or number of features, but simplicity and performance for the most common use cases. On a mid-range 2020 Intel laptop CPU you can query 1,000 documents in 0.3 ms and 100,000 documents in 40 ms, with very few and small memory allocations. See [Benchmarks](#benchmarks) for details.

> ⚠️ The project is in beta, under heavy construction, and may introduce breaking changes in releases before `v1.0.0`. All changes are documented in the [`CHANGELOG`](./CHANGELOG.md).

## Contents

1. [Use cases](#use-cases)
2. [Interface](#interface)
3. [Features + Roadmap](#features)
4. [Installation](#installation)
5. [Usage](#usage)
6. [Benchmarks](#benchmarks)
7. [Development](#development)
8. [Motivation](#motivation)
9. [Related projects](#related-projects)

## Use cases

With a vector database you can do various things:

- Retrieval augmented generation (RAG), question answering (Q&A)
- Text and code search
- Recommendation systems
- Classification
- Clustering

Let's look at the RAG use case in more detail:

### RAG

The knowledge of large language models (LLMs) - even the ones with 30 billion, 70 billion parameters and more - is limited. They don't know anything about what happened after their training ended, they don't know anything about data they were not trained with (like your company's intranet, Jira / bug tracker, wiki or other kinds of knowledge bases), and even the data they *do* know they often can't reproduce it *exactly*, but start to *hallucinate* instead.

Fine-tuning an LLM can help a bit, but it's more meant to improve the LLMs reasoning about specific topics, or reproduce the style of written text or code. Fine-tuning does *not* add knowledge *1:1* into the model. Details are lost or mixed up. And knowledge cutoff (about anything that happened after the fine-tuning) isn't solved either.

=> A vector database can act as the up-to-date, precise knowledge for LLMs:

1. You store relevant documents that you want the LLM to know in the database.
2. The database stores the *embeddings* alongside the documents, which you can either provide or can be created by specific "embedding models" like OpenAI's `text-embedding-3-small`.
 - `chromem-go` can do this for you and supports multiple embedding providers and models out-of-the-box.
3. Later, when you want to talk to the LLM, you first send the question to the vector DB to find *similar*/*related* content. This is called "nearest neighbor search".
4. In the question to the LLM, you provide this content alongside your question.
5. The LLM can take this up-to-date precise content into account when answering.

Check out the [example code](examples) to see it in action!

## Interface

Our original inspiration was the [Chroma](https://www.trychroma.com/) interface, whose core API is the following (taken from their [README](https://github.com/chroma-core/chroma/blob/0.4.21/README.md)):

<details><summary>Chroma core interface</summary>

```python
import chromadb
# setup Chroma in-memory, for easy prototyping. Can add persistence easily!
client = chromadb.Client()

# Create collection. get_collection, get_or_create_collection, delete_collection also available!
collection = client.create_collection("all-my-documents")

# Add docs to the collection. Can also update and delete. Row-based API coming soon!
collection.add(
 documents=["This is document1", "This is document2"], # we handle tokenization, embedding, and indexing automatically. You can skip that and add your own embeddings as well
 metadatas=[{"source": "notion"}, {"source": "google-docs"}], # filter on these!
 ids=["doc1", "doc2"], # unique for each doc
)

# Query/search 2 most similar results. You can also .get by id
results = collection.query(
 query_texts=["This is a query document"],
 n_results=2,
 # where={"metadata_field": "is_equal_to_this"}, # optional filter
 # where_document={"$contains":"search_string"} # optional filter
)
```

</details>

Our Go library exposes the same interface:

<details><summary>chromem-go equivalent</summary>

```go
package main

import "github.com/philippgille/chromem-go"

func main() {
 // Set up chromem-go in-memory, for easy prototyping. Can add persistence easily!
 // We call it DB instead of client because there's no client-server separation. The DB is embedded.
 db := chromem.NewDB()

 // Create collection. GetCollection, GetOrCreateCollection, DeleteCollection also available!
 collection, _ := db.CreateCollection("all-my-documents", nil, nil)

 // Add docs to the collection. Update and delete will be added in the future.
 // Can be multi-threaded with AddConcurrently()!
 // We're showing the Chroma-like method here, but more Go-idiomatic methods are also available!
 _ = collection.Add(ctx,
 []string{"doc1", "doc2"}, // unique ID for each doc
 nil, // We handle embedding automatically. You can skip that and add your own embeddings as well.
 []map[string]string{{"source": "notion"}, {"source": "google-docs"}}, // Filter on these!
 []string{"This is document1", "This is document2"},
 )

 // Query/search 2 most similar results. You can also get by ID.
 results, _ := collection.Query(ctx,
 "This is a query document",
 2,
 map[string]string{"metadata_field": "is_equal_to_this"}, // optional filter
 map[string]string{"$contains": "search_string"}, // optional filter
 )
}
```

</details>

Initially `chromem-go` started with just the four core methods, but we added more over time. We intentionally don't want to cover 100% of Chroma's API surface though. 
We're providing some alternative methods that are more Go-idiomatic instead.

For the full interface see the Godoc: <https://pkg.go.dev/github.com/philippgille/chromem-go>

## Features

- [X] Zero dependencies on third party libraries
- [X] Embeddable (like SQLite, i.e. no client-server model, no separate DB to maintain)
- [X] Multithreaded processing (when adding and querying documents), making use of Go's native concurrency features
- [X] Experimental WebAssembly binding
- Embedding creators:
 - Hosted:
 - [X] [OpenAI](https://platform.openai.com/docs/guides/embeddings/embedding-models) (default)
 - [X] [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/embeddings)
 - [X] [GCP Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings)
 - [X] [Cohere](https://cohere.com/models/embed)
 - [X] [Mistral](https://docs.mistral.ai/platform/endpoints/#embedding-models)
 - [X] [Jina](https://jina.ai/embeddings)
 - [X] [mixedbread.ai](https://www.mixedbread.ai/)
 - Local:
 - [X] [Ollama](https://github.com/ollama/ollama)
 - [X] [LocalAI](https://github.com/mudler/LocalAI)
 - Bring your own (implement [`chromem.EmbeddingFunc`](https://pkg.go.dev/github.com/philippgille/chromem-go#EmbeddingFunc))
 - You can also pass existing embeddings when adding documents to a collection, instead of letting `chromem-go` create them
- Similarity search:
 - [X] Exhaustive nearest neighbor search using cosine similarity (sometimes also called exact search or brute-force search or FLAT index)
- Filters:
 - [X] Document filters: `$contains`, `$not_contains`
 - [X] Metadata filters: Exact matches
- Storage:
 - [X] In-memory
 - [X] Optional immediate persistence (writes one file for each added collection and document, encoded as [gob](https://go.dev/blog/gob), optionally gzip-compressed)
 - [X] Backups: Export and import of the entire DB to/from a single file (encoded as [gob](https://go.dev/blog/gob), optionally gzip-compressed and AES-GCM encrypted)
 - Includes methods for generic `io.Writer`/`io.Reader` so you can plug S3 buckets and other blob storage, see [examples/s3-export-import](examples/s3-export-import) for example code
- Data types:
 - [X] Documents (text)

### Roadmap

- Performance:
 - Use SIMD for dot product calculation on supported CPUs (draft PR: [#48](https://github.com/philippgille/chromem-go/pull/48))
 - Add [roaring bitmaps](https://github.com/RoaringBitmap/roaring) to speed up full text filtering
- Embedding creators:
 - Add an `EmbeddingFunc` that downloads and shells out to [llamafile](https://github.com/Mozilla-Ocho/llamafile)
- Similarity search:
 - Approximate nearest neighbor search with index (ANN)
 - Hierarchical Navigable Small World (HNSW)
 - Inverted file flat (IVFFlat)
- Filters:
 - Operators (`$and`, `$or` etc.)
- Storage:
 - JSON as second encoding format
 - Write-ahead log (WAL) as second file format
 - Optional remote storage (S3, PostgreSQL, ...)
- Data types:
 - Images
 - Videos

## Installation

`go get github.com/philippgille/chromem-go@latest`

## Usage

See the Godoc for a reference: <https://pkg.go.dev/github.com/philippgille/chromem-go>

For full, working examples, using the vector database for retrieval augmented generation (RAG) and semantic search and using either OpenAI or locally running the embeddings model and LLM (in Ollama), see the [example code](examples).

### Quickstart

This is taken from the ["minimal" example](examples/minimal):

```go
package main

import (
 "context"
 "fmt"
 "runtime"

 "github.com/philippgille/chromem-go"
)

func main() {
 ctx := context.Background()

 db := chromem.NewDB()

 // Passing nil as embedding function leads to OpenAI being used and requires
 // "OPENAI_API_KEY" env var to be set. Other providers are supported as well.
 // For example pass `chromem.NewEmbeddingFuncOllama(...)` to use Ollama.
 c, err := db.CreateCollection("knowledge-base", nil, nil)
 if err != nil {
 panic(err)
 }

 err = c.AddDocuments(ctx, []chromem.Document{
 {
 ID: "1",
 Content: "The sky is blue because of Rayleigh scattering.",
 },
 {
 ID: "2",
 Content: "Leaves are green because chlorophyll absorbs red and blue light.",
 },
 }, runtime.NumCPU())
 if err != nil {
 panic(err)
 }

 res, err := c.Query(ctx, "Why is the sky blue?", 1, nil, nil)
 if err != nil {
 panic(err)
 }

 fmt.Printf("ID: %v\nSimilarity: %v\nContent: %v\n", res[0].ID, res[0].Similarity, res[0].Content)
}
```

Output:

```text
ID: 1
Similarity: 0.6833369
Content: The sky is blue because of Rayleigh scattering.
```

## Benchmarks

Benchmarked on 2024-03-17 with:

- Computer: Framework Laptop 13 (first generation, 2021)
- CPU: 11th Gen Intel Core i5-1135G7 (2020)
- Memory: 32 GB
- OS: Fedora Linux 39
 - Kernel: 6.7

```console
$ go test -benchmem -run=^$ -bench .
goos: linux
goarch: amd64
pkg: github.com/philippgille/chromem-go
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkCollection_Query_NoContent_100-8 13164 90276 ns/op 5176 B/op 95 allocs/op
BenchmarkCollection_Query_NoContent_1000-8 2142 520261 ns/op 13558 B/op 141 allocs/op
BenchmarkCollection_Query_NoContent_5000-8 561 2150354 ns/op 47096 B/op 173 allocs/op
BenchmarkCollection_Query_NoContent_25000-8 120 9890177 ns/op 211783 B/op 208 allocs/op
BenchmarkCollection_Query_NoContent_100000-8 30 39574238 ns/op 810370 B/op 232 allocs/op
BenchmarkCollection_Query_100-8 13225 91058 ns/op 5177 B/op 95 allocs/op
BenchmarkCollection_Query_1000-8 2226 519693 ns/op 13552 B/op 140 allocs/op
BenchmarkCollection_Query_5000-8 550 2128121 ns/op 47108 B/op 173 allocs/op
BenchmarkCollection_Query_25000-8 100 10063260 ns/op 211705 B/op 205 allocs/op
BenchmarkCollection_Query_100000-8 30 39404005 ns/op 810295 B/op 229 allocs/op
PASS
ok github.com/philippgille/chromem-go 28.402s
```

## Development

- Build: `go build ./...`
- Test: `go test -v -race -count 1 ./...`
- Benchmark:
 - `go test -benchmem -run=^$ -bench .` (add `> bench.out` or similar to write to a file)
 - With profiling: `go test -benchmem -run ^$ -cpuprofile cpu.out -bench .`
 - (profiles: `-cpuprofile`, `-memprofile`, `-blockprofile`, `-mutexprofile`)
- Compare benchmarks:
 1. Install `benchstat`: `go install golang.org/x/perf/cmd/benchstat@latest`
 2. Compare two benchmark results: `benchstat before.out after.out`

## Motivation

In December 2023, when I wanted to play around with retrieval augmented generation (RAG) in a Go program, I looked for a vector database that could be embedded in the Go program, just like you would embed SQLite in order to not require any separate DB setup and maintenance. I was surprised when I didn't find any, given the abundance of embedded key-value stores in the Go ecosystem.

At the time most of the popular vector databases like Pinecone, Qdrant, Milvus, Chroma, Weaviate and others were not embeddable at all or only in Python or JavaScript/TypeScript.

Then I found [@eliben](https://github.com/eliben)'s [blog post](https://eli.thegreenplace.net/2023/retrieval-augmented-generation-in-go/) and [example code](https://github.com/eliben/code-for-blog/tree/eda87b87dad9ed8bd45d1c8d6395efba3741ed39/2023/go-rag-openai) which showed that with very little Go code you could create a very basic PoC of a vector database.

That's when I decided to build my own vector database, embeddable in Go, inspired by the ChromaDB interface. ChromaDB stood out for being embeddable (in Python), and by showing its core API in 4 commands on their README and on the landing page of their website.

## Related projects

- Shoutout to [@eliben](https://github.com/eliben) whose [blog post](https://eli.thegreenplace.net/2023/retrieval-augmented-generation-in-go/) and [example code](https://github.com/eliben/code-for-blog/tree/eda87b87dad9ed8bd45d1c8d6395efba3741ed39/2023/go-rag-openai) inspired me to start this project!
- [Chroma](https://github.com/chroma-core/chroma): Looking at Pinecone, Qdrant, Milvus, Weaviate and others, Chroma stood out by showing its core API in 4 commands on their README and on the landing page of their website. It was also putting the most emphasis on its embeddability (in Python).
- The big, full-fledged client-server-based vector databases for maximum scale and performance:
 - [Pinecone](https://www.pinecone.io/): Closed source
 - [Qdrant](https://github.com/qdrant/qdrant): Written in Rust, not embeddable in Go
 - [Milvus](https://github.com/milvus-io/milvus): Written in Go and C++, but not embeddable as of December 2023
 - [Weaviate](https://github.com/weaviate/weaviate): Written in Go, but not embeddable in Go as of March 2024 (only in Python and JavaScript/TypeScript and that's experimental)
- Some non-specialized SQL, NoSQL and Key-Value databases added support for storing vectors and (some of them) querying based on similarity:
 - [pgvector](https://github.com/pgvector/pgvector) extension for [PostgreSQL](https://www.postgresql.org/): Client-server model
 - [Redis](https://github.com/redis/redis) ([1](https://redis.io/docs/interact/search-and-query/query/vector-search/), [2](https://redis.io/docs/interact/search-and-query/advanced-concepts/vectors/)): Client-server model
 - [sqlite-vss](https://github.com/asg017/sqlite-vss) extension for [SQLite](https://www.sqlite.org/): Embedded, but the [Go bindings](https://github.com/asg017/sqlite-vss/tree/8fc44301843029a13a474d1f292378485e1fdd62/bindings/go) require CGO. There's a [CGO-free Go library](https://gitlab.com/cznic/sqlite) for SQLite, but then it's without the vector search extension.
 - [DuckDB](https://github.com/duckdb/duckdb) has a function to calculate cosine similarity ([1](https://duckdb.org/docs/sql/functions/nested)): Embedded, but the Go bindings use CGO
 - [MongoDB](https://github.com/mongodb/mongo)'s cloud platform offers a vector search product ([1](https://www.mongodb.com/products/platform/atlas-vector-search)): Client-server model
- Some libraries for vector similarity search:
 - [Faiss](https://github.com/facebookresearch/faiss): Written in C++; 3rd party Go bindings use CGO
 - [Annoy](https://github.com/spotify/annoy): Written in C++; Go bindings use CGO ([1](https://github.com/spotify/annoy/blob/2be37c9e015544be2cf60c431f0cccc076151a2d/README_GO.rst))
 - [USearch](https://github.com/unum-cloud/usearch): Written in C++; Go bindings use CGO
- Some orchestration libraries, inspired by the Python library [LangChain](https://github.com/langchain-ai/langchain), but with no or only rudimentary embedded vector DB:
 - [LangChain Go](https://github.com/tmc/langchaingo)
 - [LinGoose](https://github.com/henomis/lingoose)
 - [GoLC](https://github.com/hupe1980/golc)

chromem-go

AI Agent runtime engine with long-lived sessions for Claude Code, OpenCode, pi-mono and other CLI AI tools. Provides full-duplex streaming, multi-platform integrations, and secure sandbox

hotplex

A Go library for building stateful, multi-actor applications with LLMs, built on the concept of LangGraph，with a lot of builtin Agent architectures

# LangGraphGo
![](https://github.com/smallnest/lango-website/blob/master/images/logo/lango5.png)

> Abbreviated as `lango`, 中文: `懒狗`

[![License](https://img.shields.io/:license-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=flat-square)](https://pkg.go.dev/github.com/smallnest/langgraphgo) [![github actions](https://github.com/smallnest/langgraphgo/actions/workflows/go.yaml/badge.svg)](https://github.com/smallnest/langgraphgo/actions) [![Go Report Card](https://goreportcard.com/badge/github.com/smallnest/langgraphgo)](https://goreportcard.com/report/github.com/smallnest/langgraphgo) [![Coverage Status](https://coveralls.io/repos/github/smallnest/langgraphgo/badge.svg?branch=master)](https://coveralls.io/github/smallnest/langgraphgo?branch=master)

[English](./README.md) | [简体中文](./README_CN.md)

Website: [http://lango.rpcx.io](http://lango.rpcx.io)


> 🔀 **Forked from [paulnegz/langgraphgo](https://github.com/paulnegz/langgraphgo)** - Enhanced with streaming, visualization, observability, and production-ready features.
>
> This fork aims for **feature parity with the Python LangGraph library**, adding support for parallel execution, persistence, advanced state management, pre-built agents, and human-in-the-loop workflows.

## Test coverage

![](coverage.svg)

## 🌐 Websites Built with LangGraphGo

Real-world applications built with LangGraphGo:

| [Insight](https://insight.rpcx.io) | [NoteX](https://notex.rpcx.io) |
| :--------------------------------: | :----------------------------: |
| ![](docs/insight.png) | ![](docs/notex.png) |

**Insight** - An AI-powered knowledge management and insight generation platform that uses LangGraphGo to build intelligent analysis workflows, helping users extract key insights from massive amounts of information.

**NoteX** - An intelligent note-taking and knowledge organization tool that leverages AI for automatic categorization, tag extraction, and content association, making knowledge management more efficient.

## 📦 Installation

```bash
go get github.com/smallnest/langgraphgo
```

**Note**: This repository uses Git submodules for the `showcases` directory. When cloning, use one of the following methods:

```bash
# Method 1: Clone with submodules
git clone --recurse-submodules https://github.com/smallnest/langgraphgo

# Method 2: Clone first, then initialize submodules
git clone https://github.com/smallnest/langgraphgo
cd langgraphgo
git submodule update --init --recursive
```

## 🚀 Features

- **Core Runtime**:
 - **Parallel Execution**: Concurrent node execution (fan-out) with thread-safe state merging.
 - **Runtime Configuration**: Propagate callbacks, tags, and metadata via `RunnableConfig`.
 - **Generic Types**: Type-safe state management with generic StateGraph implementations.
 - **LangChain Compatible**: Works seamlessly with `langchaingo`.

- **Persistence & Reliability**:
 - **Checkpointers**: Redis, Postgres, SQLite, and File implementations for durable state.
 - **File Checkpointing**: Lightweight file-based checkpointing without external dependencies.
 - **State Recovery**: Pause and resume execution from checkpoints.

- **Advanced Capabilities**:
 - **State Schema**: Granular state updates with custom reducers (e.g., `AppendReducer`).
 - **Smart Messages**: Intelligent message merging with ID-based upserts (`AddMessages`).
 - **Command API**: Dynamic control flow and state updates directly from nodes.
 - **Ephemeral Channels**: Temporary state values that clear automatically after each step.
 - **Subgraphs**: Compose complex agents by nesting graphs within graphs.
 - **Enhanced Streaming**: Real-time event streaming with multiple modes (`updates`, `values`, `messages`).
 - **Pre-built Agents**: Ready-to-use `ReAct`, `CreateAgent`, and `Supervisor` agent factories.
 - **Programmatic Tool Calling (PTC)**: LLM generates code that calls tools programmatically, reducing latency and token usage by 10x.

- **Developer Experience**:
 - **Visualization**: Export graphs to Mermaid, DOT, and ASCII with conditional edge support.
 - **Human-in-the-loop (HITL)**: Interrupt execution, inspect state, edit history (`UpdateState`), and resume.
 - **Observability**: Built-in tracing and metrics support.
 - **Tools**: Integrated `Tavily` and `Exa` search tools.

## 🎯 Quick Start

```go
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/smallnest/langgraphgo/graph"
	"github.com/tmc/langchaingo/llms"
	"github.com/tmc/langchaingo/llms/openai"
)

func main() {
	ctx := context.Background()
	model, _ := openai.New()

	// 1. Create Graph
	g := graph.NewStateGraph[map[string]any]()

	// 2. Add Nodes
	g.AddNode("generate", "generate", func(ctx context.Context, state map[string]any) (map[string]any, error) {
		input, ok := state["input"].(string)
		if !ok {
			return nil, fmt.Errorf("invalid input")
		}

		response, err := model.Call(ctx, input)
		if err != nil {
			return nil, err
		}

		state["output"] = response
		return state, nil
	})

	// 3. Define Edges
	g.AddEdge("generate", graph.END)
	g.SetEntryPoint("generate")

	// 4. Compile
	runnable, _ := g.Compile()

	// 5. Invoke
	initialState := map[string]any{
		"input": "Hello, LangGraphGo!",
	}
	result, _ := runnable.Invoke(ctx, initialState)

	fmt.Println(result)
}
```

## 📚 Examples

This project includes **90+ comprehensive examples** organized into categories:

### Featured Examples

- **[ReAct Agent](./examples/react_agent/)** - Reason and Action agent using tools
- **[RAG Pipeline](./examples/rag_pipeline/)** - Complete retrieval-augmented generation
- **[Chat Agent](./examples/chat_agent/)** - Multi-turn conversation with session management
- **[Supervisor](./examples/supervisor/)** - Multi-agent orchestration
- **[Tree of Thoughts](./examples/tree_of_thoughts/)** - Search-based reasoning with multiple solution paths
- **[Planning Agent](./examples/planning_agent/)** - Dynamic workflow plan creation
- **[PEV Agent](./examples/pev_agent/)** - Plan-Execute-Verify with self-correction
- **[Reflection Agent](./examples/reflection_agent/)** - Iterative improvement through self-reflection
- **[Mental Loop](./examples/mental_loop/)** - Simulator-in-the-loop for safe action testing
- **[Reflexive Metacognitive Agent](./examples/reflexive_metacognitive/)** - Self-aware agent with explicit capabilities model

### Example Categories

- **[Basic Concepts](./examples/README.md#basic-concepts)** - Simple LLM integration, LangChain compatibility
- **[State Management](./examples/README.md#state-management)** - State schema, custom reducers, smart messages
- **[Graph Structure](./examples/README.md#graph-structure--routing)** - Conditional routing, subgraphs, generics
- **[Parallel Execution](./examples/README.md#parallel-execution)** - Fan-out/fan-in with state merging
- **[Streaming & Events](./examples/README.md#streaming--events)** - Real-time updates, listeners, logging
- **[Persistence](./examples/README.md#persistence-checkpointing)** - Checkpointing with file, memory, databases
- **[Human-in-the-Loop](./examples/README.md#human-in-the-loop)** - Interrupts, approval, time travel
- **[Pre-built Agents](./examples/README.md#pre-built-agents)** - ReAct, Supervisor, Chat, Planning agents
- **[Programmatic Tool Calling](./examples/README.md#programmatic-tool-calling-ptc)** - PTC for 10x latency reduction
- **[Memory](./examples/README.md#memory)** - Buffer, sliding window, summarization strategies
- **[RAG](./examples/README.md#rag-retrieval-augmented-generation)** - Vector stores, GraphRAG with FalkorDB
- **[Tools & Integrations](./examples/README.md#tools--integrations)** - Search tools, GoSkills, MCP

**[View All 90+ Examples →](./examples/README.md)**

## 🔧 Key Concepts

### Parallel Execution
LangGraphGo automatically executes nodes in parallel when they share the same starting node. Results are merged using the graph's state merger or schema.

```go
g.AddEdge("start", "branch_a")
g.AddEdge("start", "branch_b")
// branch_a and branch_b run concurrently
```

### Human-in-the-loop (HITL)
Pause execution to allow for human approval or input.

```go
config := &graph.Config{
 InterruptBefore: []string{"human_review"},
}

// Execution stops before "human_review" node
state, err := runnable.InvokeWithConfig(ctx, input, config)

// Resume execution
resumeConfig := &graph.Config{
 ResumeFrom: []string{"human_review"},
}
runnable.InvokeWithConfig(ctx, state, resumeConfig)
```

### Pre-built Agents
Quickly create complex agents using factory functions.

```go
// Create a ReAct agent
agent, err := prebuilt.CreateReactAgent(model, tools)

// Create an agent with options
agent, err := prebuilt.CreateAgent(model, tools, prebuilt.WithSystemMessage("System prompt"))

// Create a Supervisor agent
supervisor, err := prebuilt.CreateSupervisor(model, agents)
```

### Programmatic Tool Calling (PTC)
Generate code that calls tools directly, reducing API round-trips and token usage.

```go
// Create a PTC agent
agent, err := ptc.CreatePTCAgent(ptc.PTCAgentConfig{
 Model: model,
 Tools: toolList,
 Language: ptc.LanguagePython, // or ptc.LanguageGo
 ExecutionMode: ptc.ModeDirect, // Subprocess (default) or ModeServer
 MaxIterations: 10,
})

// LLM generates code that calls tools programmatically
result, err := agent.Invoke(ctx, initialState)
```

See the [PTC README](./ptc/README.md) for detailed documentation.

## 🎨 Graph Visualization

```go
exporter := runnable.GetGraph()
fmt.Println(exporter.DrawMermaid()) // Generates Mermaid flowchart
```

## 📈 Performance

- **Graph Operations**: ~14-94μs depending on format
- **Tracing Overhead**: ~4μs per execution
- **Event Processing**: 1000+ events/second
- **Streaming Latency**: <100ms

## 🧪 Testing

```bash
go test ./... -v
```

# Contributors

This project is open for contributions! if you are interested in being a contributor please create feature issues first, then submit PRs..	


## 📄 License

MIT License - see original repository for details.

langgraphgo

Local compatibility proxy for the Gemini and OpenAI APIs. Run one container locally and test both SDK protocol shapes on the same port without API keys or network access

# localaik

[![CI](https://github.com/harshaneel/localaik/actions/workflows/release.yml/badge.svg)](https://github.com/harshaneel/localaik/actions/workflows/release.yml)
[![Docker Hub](https://img.shields.io/docker/v/gokhalh/localaik?sort=semver&label=Docker%20Hub)](https://hub.docker.com/r/gokhalh/localaik)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Go Report Card](https://goreportcard.com/badge/github.com/harshaneel/localaik)](https://goreportcard.com/report/github.com/harshaneel/localaik)
[![Go Version](https://img.shields.io/github/go-mod/go-version/harshaneel/localaik)](https://github.com/harshaneel/localaik/blob/main/go.mod)
[![Go Reference](https://pkg.go.dev/badge/github.com/harshaneel/localaik.svg)](https://pkg.go.dev/github.com/harshaneel/localaik)

A local compatibility server for the Gemini and OpenAI APIs. Run one container, point your SDK at `http://localhost:8090`, and get both protocol shapes on the same port for tests and development.

## Motivation

Testing code that calls Gemini or OpenAI is painful: real API calls are slow, cost money, and need network access. localaik gives you a single Docker container that speaks both protocols backed by a local model — no API key, no internet, deterministic enough for CI.

## Architecture

```
┌────────────────────────────────────────────────────────┐
│  localaik container                                    │
│                                                        │
│  ┌──────────────────────────┐    ┌──────────────────┐  │
│  │  localaik proxy (:8090)  │    │ llama.cpp (:8080)│  │
│  │                          │    │                  │  │
│  │  /v1beta/* (Gemini)  ────┼──▶ │  Gemma 3 model   │  │
│  │  /v1/*     (OpenAI)  ────┼──▶ │                  │  │
│  │                          │    └──────────────────┘  │
│  │                          │                          │
│  │                          │    ┌──────────────────┐  │
│  │  PDF uploads ────────────┼──▶ │    pdftoppm      │  │
│  │                          │    │  PDF ─▶ images   │  │
│  └──────────────────────────┘    └──────────────────┘  │
└────────────────────────────────────────────────────────┘
```

SDK requests hit the localaik proxy, which translates Gemini or OpenAI wire format and forwards to the local llama.cpp server running a Gemma 3 model.

## Quick start

```bash
docker run -d -p 8090:8090 gokhalh/localaik
```

Or with Docker Compose:

```yaml
services:
  localaik:
    image: gokhalh/localaik
    ports:
      - "8090:8090"
```

localaik is a plain HTTP server, so any language or SDK that can set a base URL will work.

More runnable samples (curl, Go, Python, JavaScript, Java) live under **[examples/](examples/README.md)**.

### Gemini SDK

**Go:**

```go
client, err := genai.NewClient(ctx, &genai.ClientConfig{
    APIKey:      "test",
    HTTPOptions: genai.HTTPOptions{BaseURL: "http://localhost:8090"},
})
```

**Python:**

```python
from google import genai

client = genai.Client(
    api_key="test",
    http_options=genai.types.HttpOptions(api_version="v1beta", base_url="http://localhost:8090"),
)
```

Or set the environment variable for any language:

```bash
export GOOGLE_GEMINI_BASE_URL=http://localhost:8090
```

### OpenAI SDK

**Python:**

```python
from openai import OpenAI

client = OpenAI(api_key="test", base_url="http://localhost:8090/v1")
```

**Go:**

```go
client := openai.NewClient(
    option.WithAPIKey("test"),
    option.WithBaseURL("http://localhost:8090/v1"),
)
```

## Docker tags


| Tag                   | Model              | Image size |
| --------------------- | ------------------ | ---------- |
| `latest`, `gemma3-4b` | Gemma 3 4B Q4_K_M  | ~3 GB      |
| `gemma3-12b`          | Gemma 3 12B Q4_K_M | ~7 GB      |


Version-pinned tags follow the pattern `v0.1.1-gemma3-4b`, `v0.1.1-gemma3-12b`.

## Tuning (v0.1.3 onwards)

Pass environment variables to tune the underlying model server:

```bash
docker run -d -p 8090:8090 \
  -e LK_THREADS=8 \
  -e LK_CTX_SIZE=4096 \
  -e LK_FLASH_ATTN=1 \
  -e LK_CONT_BATCHING=1 \
  -e LK_PARALLEL=2 \
  gokhalh/localaik
```

Or with Docker Compose:

```yaml
services:
  localaik:
    image: gokhalh/localaik
    ports:
      - "8090:8090"
    environment:
      LK_THREADS: 8
      LK_CTX_SIZE: 4096
      LK_FLASH_ATTN: 1
      LK_CONT_BATCHING: 1
      LK_PARALLEL: 2
```


| Variable           | Default         | Description                         |
| ------------------ | --------------- | ----------------------------------- |
| `LK_CTX_SIZE`      | 8192            | Context window in tokens            |
| `LK_THREADS`       | auto            | CPU threads for inference           |
| `LK_THREADS_BATCH` | same as threads | CPU threads for prompt processing   |
| `LK_BATCH_SIZE`    | 2048            | Prompt processing batch size        |
| `LK_UBATCH_SIZE`   | 512             | Micro-batch size                    |
| `LK_GPU_LAYERS`    | 0               | Layers offloaded to GPU (99 = all)  |
| `LK_PARALLEL`      | 1               | Max concurrent request slots        |
| `LK_FLASH_ATTN`    | 0 (off)         | Flash attention (`1` to enable)     |
| `LK_CONT_BATCHING` | 0 (off)         | Continuous batching (`1` to enable) |
| `LK_MLOCK`         | 0 (off)         | Lock model in RAM (`1` to enable)   |


## Implemented routes


| Route                                               | Used by                        | Notes                                   |
| --------------------------------------------------- | ------------------------------ | --------------------------------------- |
| `POST /v1beta/models/{model}:generateContent`       | Gemini `GenerateContent`       | Translated to upstream chat completions |
| `POST /v1beta/models/{model}:streamGenerateContent` | Gemini `GenerateContentStream` | Gemini-style SSE (typically `?alt=sse`) |
| `POST /v1/chat/completions`                         | OpenAI chat completions        | Forwarded to upstream                   |
| `GET /health`                                       | Health checks                  | Custom route                            |


All other API routes return `404`.

## Tested SDKs

Automated contract tests validate against:

- `google.golang.org/genai` v1.51.0
- `github.com/openai/openai-go/v3` v3.30.0

Other SDK versions and languages may work if they emit the same HTTP shapes.

## Use in CI

Run localaik as a GitHub Actions service container so your tests hit a real local model instead of mocks:

```yaml
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      localaik:
        image: gokhalh/localaik
        ports:
          - 8090:8090
        options: >-
          --health-cmd "curl -f http://localhost:8090/health"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 30
    steps:
      - uses: actions/checkout@v4
      - run: go test ./...
        env:
          GOOGLE_GEMINI_BASE_URL: http://localhost:8090
          OPENAI_BASE_URL: http://localhost:8090/v1
```

## Gemini compatibility

**Supported features:**

- Text, image (`inlineData`), and PDF input (auto-converted to page images)
- `fileData` for image URLs and local/`data:`-URI PDF/text files
- `systemInstruction`
- `generationConfig`: temperature, topP, topK, candidateCount, maxOutputTokens, stopSequences, responseLogprobs, logprobs, presencePenalty, frequencyPenalty, seed
- Structured output via `responseMimeType`, `responseSchema`, `responseJsonSchema`
- Function declarations via `tools`, function calling config via `toolConfig`
- `functionCall` and `functionResponse` parts
- Streaming SSE responses
- Usage metadata and finish reasons

**Partial support:**

- `top_k`, `n`, logprobs, and tool choice behavior depends on the upstream runtime
- `executableCode`, `codeExecutionResult`, `toolCall`, `toolResponse` parts preserved as text context

**Not supported:**

- SDK methods outside `GenerateContent` / `GenerateContentStream`
- Non-function tools (Google Search, Maps, URL context, code execution)
- Embeddings, token counting, cached content, live/bidi sessions, uploads

## OpenAI compatibility

**Supported:** text chat completions, structured output, vision inputs, tool-related fields (all passed through to upstream).

**Not supported:** Responses API, Assistants, Embeddings, Images, Audio, Files, Vector stores.

## Development

> **Tip:** Run `make docker-up` to build and start the localaik container, which includes a local llama.cpp server with a bundled model. This is the easiest way to get a working upstream for development.

```bash
# Run the proxy locally (requires a running llama.cpp server)
go run ./cmd/localaik --port 8090 --upstream http://127.0.0.1:8080/v1

# Common commands
make help              # Show all targets
make lint              # Format check + go vet
make test-unit         # Unit tests
make test-integration  # Integration tests (requires docker-up)
make test              # All of the above
make docker-up         # Build and start container
make docker-down       # Stop container
```

### Building the image

```bash
# Default (Gemma 3 4B)
docker build -t gokhalh/localaik .

# Custom model
docker build \
  --build-arg MODEL_URL=... \
  --build-arg MODEL_SHA256=... \
  --build-arg MMPROJ_URL=... \
  --build-arg MMPROJ_SHA256=... \
  -t gokhalh/localaik:custom .
```

## Limitations

- Intended for tests and development, not production
- Image size is dominated by model weights
- Cold starts can take tens of seconds while the model loads
- PDF rendering adds latency per page



The Best Go Libraries For Artificial Intelligence (13)