Awesome Go

ai-gateway

CategoryArtificial Intelligence
SubcategoryArtificial Intelligence
Stars266
GitHubhttps://github.com/ferro-labs/ai-gateway

OpenAI-compatible LLM gateway that routes requests across 30 providers with fallback, rate limiting, budgets, guardrails, and observability

About ai-gateway

This large README is shown as a plain-text preview. Read the full README at the source

<div align="center">

<p align="right">
  <a href="README.md">English</a> | <a href="README.zh-CN.md">δΈ­ζ–‡</a>
</p>

<table border="0" cellspacing="0" cellpadding="0"><tr>
  <td rowspan="2"><img src="docs/logo.png" alt="Ferro Labs AI Gateway" width="64" /></td>
  <td align="center"><h1>Ferro Labs AI Gateway</h1></td>
</tr><tr>
  <td align="center"><strong>Open-Source, OpenAI-Compatible LLM Gateway</strong></td>
</tr></table>

**High-performance AI gateway in Go. Route LLM requests across 30 providers via a single OpenAI-compatible API.**

[![Deploy on Railway: SQLite](https://railway.com/button.svg)](https://railway.com/deploy/ferro-labs-ai-sqlite-storage?referralCode=KblxKX&utm_medium=integration&utm_source=template&utm_campaign=generic)
[![Deploy on Railway: PostgreSQL](https://railway.com/button.svg)](https://railway.com/deploy/ferro-labs-ai-postgresql-storage?referralCode=KblxKX&utm_medium=integration&utm_source=template&utm_campaign=generic)
[![Deploy to Render: PostgreSQL](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy?repo=https://github.com/ferro-labs/ai-gateway)

[![Go](https://img.shields.io/badge/go-1.25+-00ADD8.svg)](https://go.dev)
[![Go Reference](https://pkg.go.dev/badge/github.com/ferro-labs/ai-gateway.svg)](https://pkg.go.dev/github.com/ferro-labs/ai-gateway)
[![codecov](https://codecov.io/gh/ferro-labs/ai-gateway/branch/main/graph/badge.svg)](https://codecov.io/gh/ferro-labs/ai-gateway)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![GitHub Stars](https://img.shields.io/github/stars/ferro-labs/ai-gateway?style=flat&color=yellow)](https://github.com/ferro-labs/ai-gateway/stargazers)
[![CI](https://github.com/ferro-labs/ai-gateway/actions/workflows/ci.yml/badge.svg)](https://github.com/ferro-labs/ai-gateway/actions/workflows/ci.yml)
[![Code Scanning](https://github.com/ferro-labs/ai-gateway/actions/workflows/code-scanning.yml/badge.svg)](https://github.com/ferro-labs/ai-gateway/actions/workflows/code-scanning.yml)
[![Ask DeepWiki](https://deepwiki.com/badge.svg?url=https%3A%2F%2Fdeepwiki.com%2Fferro-labs%2Fai-gateway)](https://deepwiki.com/ferro-labs/ai-gateway)
[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/ferro-labs)](https://artifacthub.io/packages/search?org=ferro-labs)
[![Docs](https://img.shields.io/badge/docs-ferrolabs.ai-2ea44f)](https://docs.ferrolabs.ai)
[![Discord](https://img.shields.io/badge/Discord-Join%20Us-5865F2?logo=discord&logoColor=white)](https://discord.gg/yCAeYvJeDV)

πŸ“– **Documentation:** [docs.ferrolabs.ai](https://docs.ferrolabs.ai)

πŸ”€ **30 providers, 2,500+ models β€” one API**<br/>
⚑ **13,925 RPS at 1,000 concurrent users** ([v1.0.0 benchmark](#performance))<br/>
πŸ“¦ **Single static binary, no external services required, 32 MB base memory**

<img src="docs/architecture.svg" alt="Ferro Labs AI Gateway Architecture" width="100%" />

</div>

---

## Quick Start

Under two minutes from install to first response.

| Platform / tool | Install |
|:---|:---|
| macOS, Linux | `curl -fsSL https://get.ferrolabs.ai \| sh` |
| Windows | `irm https://get.ferrolabs.ai/install.ps1 \| iex` |
| Homebrew | `brew install ferro-labs/tap/ferrogw` |
| Scoop | `scoop bucket add ferrolabs https://github.com/ferro-labs/homebrew-tap` then `scoop install ferrogw` |
| npm | `npm install -g ferrogw` |
| Python | `uv tool install ferrogw` |
| Docker | `docker run -p 8080:8080 ghcr.io/ferro-labs/ai-gateway:latest` |
| Go | `go install github.com/ferro-labs/ai-gateway/cmd/ferrogw@latest` β€” builds from source, without the dashboard |
| Debian, RPM, Alpine | `.deb`, `.rpm` and `.apk` packages on the [releases page](https://github.com/ferro-labs/ai-gateway/releases/latest) |

Then go from nothing to a served request:

```bash
export OPENAI_API_KEY=sk-your-key     # ferrogw init detects this and writes the matching target
ferrogw init                          # writes config.yaml, prints your master key
export GATEWAY_CONFIG=./config.yaml   # the server reads a config file only when this is set
export MASTER_KEY=fgw_your-master-key # the key ferrogw init printed
ferrogw serve                         # starts the server on :8080
```

`ferrogw init` prints the master key **once** and never writes it to disk β€” save
it yourself, in your `.env` file or a secret manager.

Docker runs the server itself, so there is no `ferrogw init` to print a master
key β€” choose your own. Pass both variables by name so their values stay off the
command line, where `ps` would show them:

```bash
export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw-any-strong-secret-you-choose
docker run -p 8080:8080 -e OPENAI_API_KEY -e MASTER_KEY ghcr.io/ferro-labs/ai-gateway:latest
```

Passing them by name keeps the values out of process arguments only β€” they are
still readable in the container's environment and through `docker inspect`, so
use Docker secrets or your platform's secret storage in production.

<div align="center">
  <img src="docs/demo.gif" alt="Installing Ferro Labs AI Gateway with one command, running ferrogw init, starting the server, and getting a completed chat response" width="100%" />
</div>

### First request

```bash
export MASTER_KEY=fgw_your-master-key   # the key ferrogw init printed, in whichever shell you curl from

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello from Ferro Labs AI Gateway"}]
  }' | jq
```

Every release is signed with keyless cosign and ships an SPDX SBOM β€”
verification steps are in [SECURITY.md](SECURITY.md#verifying-releases).

---

## Why Ferro Labs AI Gateway

AI gateways differ most under load, in throughput, latency and memory. Ferro Labs AI Gateway is written in Go from the ground up for real-world throughput β€” a single binary that routes LLM requests with predictable latency and minimal resource usage.

| Feature          | Ferro Labs  | LiteLLM | Bifrost    | Kong AI     |
|:-----------------|:------------|:--------|:-----------|:------------|
| Language         | Go          | Python  | Go         | Go/Lua      |
| Single binary    | βœ…          | ❌      | βœ…         | ❌          |
| Providers        | 30          | 100+    | 20+        | 10+         |
| MCP support      | βœ…          | ❌      | βœ…         | ❌          |
| Response cache   | βœ…          | βœ…      | βœ…         | ❌ (paid)   |
| Guardrails       | βœ…          | βœ…      | ❌         | ❌ (paid)   |
| OSS license      | Apache 2.0  | MIT     | Apache 2.0 | Apache 2.0  |
| Managed cloud    | Coming Soon | βœ…      | βœ…         | βœ…          |

---

## Features

| Capability | What it does | Reference |
|:---|:---|:---|
| πŸ”€ **Routing** | 8 strategies β€” single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional β€” with per-target retry, failover, and model aliases | [Docs β†’](https://docs.ferrolabs.ai/routing/) |
| πŸ”Œ **30 providers** | Chat and streaming everywhere; embeddings, images, rerank, moderations, speech-to-text, text-to-speech, and batch where the vendor offers them | [Docs β†’](https://docs.ferrolabs.ai/providers/) |
| πŸ›‘οΈ **Guardrails & plugins** | Eleven built in β€” word filter, regex guard, PII redaction, secret scanning, prompt-injection shield, response schema validation, token/message limits, response cache, rate limiting, per-key budgets, request logging β€” and the plugin framework is public for writing your own | [Docs β†’](https://docs.ferrolabs.ai/plugins/) |
| 🎯 **Capability matrix** | One declarative record of which OpenAI parameters each provider forwards, translates, or cannot express, served by `GET /v1/capabilities` | [Docs β†’](https://docs.ferrolabs.ai/guides/provider-capabilities/) |
| πŸ€– **MCP** | Connects to stdio and Streamable HTTP tool servers, injects their tools into chat completions, and drives the agentic `tool_calls` loop itself | [Docs β†’](https://docs.ferrolabs.ai/guides/mcp/) |
| πŸ“Š **Observability** | OpenTelemetry tracing and Prometheus metrics, one trace ID across logs and spans β€” a zero-allocation no-op until enabled | [Docs β†’](https://docs.ferrolabs.ai/guides/observability/) |
| πŸ–₯️ **Dashboard** | Operations console compiled into the binary and served at `/` β€” traffic, spend, provider health, request logs, audit trail | [Docs β†’](https://docs.ferrolabs.ai/guides/dashboard/) |

---

## Dashboard

Every release binary serves a built-in operations console at `/` β€” same port as
the API, compiled in with `go:embed`, no second image and no second origin. Sign
in with your `MASTER_KEY` or any admin / read-only key and it reads the live
gateway: traffic, spend, provider health, routing, plugins, request logs, and the
audit trail.

The bundle is produced by the release pipeline, so a binary built with
`go install` is the one exception: it answers `/` with a placeholder instead.
Any other install method above ships the console.

<div align="center">
  <img src="docs/dashboard.gif" alt="Ferro Labs AI Gateway operations console: Overview, Analytics, Providers, Routing Strategies, Plugins, Playground, Tracing, Request Logs, Audit Trail, Configuration, and API Keys" width="100%" />
</div>

Overview, Analytics (latency/TTFT/cost percentiles), Providers, Routing,
Plugins, a Playground over the real routing path, Tracing, Request Logs, the
Audit Trail, Configuration with history/rollback, and scoped API key management.

Run the gateway and open <http://localhost:8080>. To see it filled like the
recording above, bring up the self-contained demo stack:

```bash
make up-fullstack   # gateway + Postgres + Jaeger + Prometheus + Grafana + mock upstream + load generator
# then open http://localhost:8080
```

Build and development details are in [web/README.md](web/README.md).

---

## Documentation

The root README is the overview; each subsystem keeps its own reference beside
its code:

| Reference | Covers |
|:---|:---|
| [providers/README.md](providers/README.md) | The 30 providers, the per-provider endpoint matrix, and every `/v1/*` surface |
| [config/README.md](config/README.md) | Config loading, validation, `${VAR}` secrets, declared models, trusted proxies |
| [internal/strategies/README.md](internal/strategies/README.md) | All 8 routing strategies and their failure semantics |
| [plugin/README.md](plugin/README.md) | The plugin framework and the eleven built-in plugins |
| [mcp/README.md](mcp/README.md) | MCP tool servers, transports, the subprocess trust boundary, readiness |
| [observability/README.md](observability/README.md) | Tracing setup, managed backends, emitted attributes, privacy levels, exporters |
| [deploy/README.md](deploy/README.md) | Dockerfiles, Compose files, the fullstack demo stack |
| [web/README.md](web/README.md) | Dashboard development and the embed contract |
| [AGENTS.md](AGENTS.md) | The complete operator/developer reference: architecture, every env var, request flow |
| [SECURITY.md](SECURITY.md) | Reporting, security posture, release verification |
| [CONTRIBUTING.md](CONTRIBUTING.md) | Branch strategy, commit conventions, provider/plugin checklists |

---

## Examples

Integration examples for common use cases are in [ferro-labs/ai-gateway-examples](https://github.com/ferro-labs/ai-gateway-examples):

| Example | Description |
|:--------|:------------|
| [basic](https://github.com/ferro-labs/ai-gateway-examples/tree/main/basic) | Single chat completion to the first configured provider |
| [fallback](https://github.com/ferro-labs/ai-gateway-examples/tree/main/fallback) | Fallback strategy β€” try providers in order with retries |
| [loadbalance](https://github.com/ferro-labs/ai-gateway-examples/tree/main/loadbalance) | Weighted load balancing across targets (70/30 split) |
| [with-guardrails](https://github.com/ferro-labs/ai-gateway-examples/tree/main/with-guardrails) | Built-in word-filter and max-token guardrail plugins |
| [with-mcp](https://github.com/ferro-labs/ai-gateway-examples/tree/main/with-mcp) | Local MCP server with tool-calling integration |
| [embedded](https://github.com/ferro-labs/ai-gateway-examples/tree/main/embedded) | Embed the gateway as an HTTP handler inside an existing server |

---

## Configuration

One YAML/JSON file, named by `GATEWAY_CONFIG`, drives routing, guardrails, MCP
tools, and observability:

```yaml
strategy:
  mode: fallback              # 8 modes β€” see internal/strategies/README.md

targets:
  - virtual_key: openai       # an allowlist: only listed providers are routable
    retry: { attempts: 3 }    # per target, honoured under every routing mode
    concurrency: { max_concurrency: 32, queue_size: 1000 }
  - virtual_key: anthropic

aliases:
  fast: gpt-4o-mini

plugins:                      # guardrails first, then cache β€” see plugin/README.md
  - name: word-filter
    type: guardrail
    stage: before_request
    enabled: true
    config: { blocked_words: ["password", "secret"] }

mcp_servers:                  # tool servers β€” see mcp/README.md
  - name: search
    url: https://mcp.example.com/mcp
    headers: { Authorization: "Bearer ${SEARCH_TOKEN}" }
```

`${VAR}` references (braced form only) resolve when a component is constructed,
never at file load β€” so secrets are never stored, served by `GET /admin/config`,
or restored by a rollback. A bare `$` is data; an undefined variable is an
error.

The full annotated reference with every option is
[config.example.yaml](config.example.yaml) /
[config.example.json](config.example.json), and the schema guide is
[config/README.md](config/README.md). `ferrogw validate` checks a file without
starting the server.

### Key environment variables

| Variable | Purpose |
|----------|---------|
| `MASTER_KEY` | Bootstrap and break-glass admin credential (generated by `ferrogw init`); give each operator their own key from `POST /admin/keys` for day-to-day use |
| `GATEWAY_CONFIG` | Path to config YAML/JSON |
| `GATEWAY_ENV` | Set to `production` to enable production-mode safety guards: it refuses to start on `ALLOW_UNAUTHENTICATED_PROXY=true` or a `*` entry in `CORS_ORIGINS`, and warns when per-IP rate limiting is off, pprof is mounted, or the API key store is in-memory |
| `PORT` | Server port (default: `8080`) |
| `ALLOW_UNAUTHENTICATED_PROXY` | Set to `true` to disable proxy-route auth (dev only; blocked when `GATEWAY_ENV=production`) |
| `CORS_ORIGINS` | Comma-separated allowed CORS origins; cross-origin is denied when unset. Matched **literally** β€” there is no wildcard, so list each origin explicitly |
| `TRUSTED_PROXIES` | CIDRs of trusted reverse proxies; forwarded headers are honored only from these (default: loopback). See [config/README.md](config/README.md#trusted-proxies-trusted_proxies) |
| `<PROVIDER>_BASE_URL` | Points a provider at a proxy, self-hosted server, or regional endpoint. It is the **API root**, used verbatim β€” write it exactly as the vendor documents it, version segment included (`https://api.groq.com/openai/v1`); a bare host resolves to the provider's own version segment |

See [AGENTS.md](AGENTS.md) for the full environment variable reference including provider API keys, store backends, and OTel settings.

---

## Observability

**See everything your gateway does** β€” every request, what it cost, how long it took, which provider served it, and which guardrails ran. Ferro Labs AI Gateway ships first-class **OpenTelemetry tracing** and **Prometheus metrics** out of the box, and stays at a **zero-allocation no-op until you turn it on**. Point it at **Jaeger, Grafana, New Relic, LangSmith, Datadog, or Honeycomb** β€” anything that speaks OTLP β€” and every request emits a `gateway.request` span carrying GenAI semantic conventions (`gen_ai.*`) plus `ferro.*` extensions for cost, routing, MCP tool calls, and stream timings. The same trace ID threads your logs, spans, and the `X-Request-ID` response header. Requests can name the end user and session they belong to (`user`, `X-User-ID`, `X-Session-ID`, or W3C baggage), and that identity rides the request span, every event, and request-log rows.

> πŸ“ˆ **Full observability guide β†’ [observability/README.md](observability/README.md)** β€” managed-backend setup, endpoint & transport rules, every emitted attribute, privacy levels, and exporter plugins.

Bring up the gateway wired to a full monitoring stack β€” **Prometheus, Grafana, and Jaeger** β€” driven by generated traffic, in one command:

```bash
make up-fullstack   # then open Grafana at http://localhost:3000
```

<p align="center">
  <img src="docs/observability/grafana-dashboard.gif" alt="Grafana dashboard: per-provider request rate, latency percentiles, token cost, and circuit-breaker state" width="100%" />
  <br/>
  <em>Grafana β€” request rate, latency percentiles, per-provider breakdown, token cost, and circuit-breaker state, all from the gateway's Prometheus metrics.</em>
</p>

<p align="center">
  <img src="docs/observability/jaeger-trace.gif" alt="Jaeger trace: one gateway.request span expanding to show its gen_ai.* and ferro.* attributes" width="100%" />
  <br/>
  <em>Jaeger β€” one request's <code>gateway.request</code> span, opened to reveal its <code>gen_ai.*</code> and <code>ferro.*</code> attributes.</em>
</p>

Enable tracing with one variable (or the `observability:` config block β€”
endpoint, protocol, sampling, privacy, headers are all documented in the guide):

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
ferrogw serve
```

---

## CLI

`ferrogw` is a single binary β€” no separate CLI tool required.

| Command | Description |
|:--------|:------------|
| `ferrogw` | Start the gateway server (default) |
| `ferrogw serve` | Start the gateway server (explicit) |
| `ferrogw init` | First-run setup β€” generate master key and config |
| `ferrogw validate` | Validate a config file without starting |
| `ferrogw doctor` | Check environment (API keys, config, connectivity) |
| `ferrogw status` | Show gateway health and provider status |
| `ferrogw version` | Print version, commit, and build info |
| `ferrogw admin keys list` | List API keys |
| `ferrogw admin keys create --name <name>` | Create an API key (`--scope`, `--expires-in`) |
| `ferrogw admin logs stats` | Show request log statistics |
| `ferrogw plugins` | List registered plugins |

Global flags available on all subcommands: `--gateway-url`, `--api-key`, `--format` (table/json/yaml).

---

## Deployment

### Local development

```bash
export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw_your-master-key
export GATEWAY_CONFIG=./config.yaml
make build && ./bin/ferrogw
```

### Railway & Render

The deploy buttons at the top of this README provision either platform: Railway
with SQLite on a volume (point `API_KEY_STORE_DSN`, `CONFIG_STORE_DSN` and
`REQUEST_LOG_STORE_DSN` at paths under `/data`) or PostgreSQL, and Render from the repo's `render.yaml` Blueprint,
which generates `MASTER_KEY` and wires the store DSNs to a managed Postgres
automatically.

### Docker Compose

Three Compose files in `deploy/` follow the standard override pattern β€” a shared
base, a dev override that builds from source, and a prod override with a pinned
tag, health check, and resource limits. Run everything from the repository root:

```bash
make up             # dev: builds from source
IMAGE_TAG=v1.4.5 CORS_ORIGINS=https://your-domain.com make up-prod
make down           # tears down either
```

One container serves both the API and the dashboard β€” no second image, no second
origin. Provider keys go in a repository-root `.env` or the environment.
[deploy/README.md](deploy/README.md) has the full reference, including a
self-contained PostgreSQL pairing and the fullstack observability stack.

### Kubernetes via Helm

```bash
helm repo add ferro-labs https://ferro-labs.github.io/helm-charts
helm repo update
helm install ferro-gw ferro-labs/ai-gateway \
  --set env.OPENAI_API_KEY=sk-your-key
```

Frequently Asked Questions

What is ai-gateway?

ai-gateway is a Artificial Intelligence library for the Go programming language. OpenAI-compatible LLM gateway that routes requests across 30 providers with fallback, rate limiting, budgets, guardrails, and observability

How do I install ai-gateway?

Install ai-gateway with the Go module system using `go get ferro-labs/ai-gateway`. Check the repository for the current installation instructions.

What category does ai-gateway belong to?

ai-gateway is listed under Artificial Intelligence, specifically Artificial Intelligence.

← Back to Artificial Intelligence