dbt
A framework for running self-updating signed binaries from a central, trusted repository
About dbt
This large README is shown as a plain-text preview. Read the full README at the source
# Dynamic Binary Toolkit (DBT)
**A secure, self-updating binary distribution system for organizations.**
DBT automatically downloads, verifies, and runs signed executables from a trusted repository, ensuring your team always uses the latest, secure versions of your tools without manual intervention.
[](https://img.shields.io/github/release/nikogura/dbt.svg)
[](https://github.com/nikogura/dynamic-binary-toolkit/actions/workflows/ci.yml)
[](https://goreportcard.com/report/github.com/nikogura/dynamic-binary-toolkit)
[](http://godoc.org/github.com/nikogura/dynamic-binary-toolkit/pkg/dbt)
[](https://codecov.io/gh/nikogura/dbt)
[](https://github.com/avelino/awesome-go)
## How It Works
DBT is a secure downloader and verifier for executable files. When a user runs a tool through DBT:
1. **Self-update**: DBT checks if it needs to update itself
2. **Download**: Fetches the requested tool from your repository
3. **Verify**: Checks the SHA256 checksum and GPG signature against your truststore
4. **Execute**: Runs the verified tool, replacing itself in the process table
If any verification step fails, DBT stops immediately and reports the error. No unverified code ever runs.
```mermaid
sequenceDiagram
participant DBT
participant Tool
participant Repository
DBT-->>Repository: Get truststore from Repository (public keys of trusted tool authors)
DBT-->>Repository: What's latest version of dbt, and what's its sha256 checksum?
loop DBT Integrity Check
DBT->>DBT: Calculate my own checksum
DBT->>DBT: Compare calculated checksum against downloaded checksum
DBT->>DBT: Verify signature of DBT itself
end
Note over DBT,Repository: If validation fails, download the latest version.<br>Validate that, and if it passes, execute it with the original arguments.<br> The original process exits. The child takes over parent's pid.<br>Lather, rinse, and repeat.
DBT-->>Repository: Is there a tool called <tool name>?
DBT-->>Repository: What's the latest version of <tool name>, and what's its sha256 checksum?
loop Tool Integrity Check
DBT->>Tool: Is <tool name> already on disk?
Note over DBT,Repository: If not, download it, its checksum, and its signature
DBT->>Tool: Calculate sha256 checksum of Tool
DBT->>Tool: Compare calculated checksum against downloaded checksum
DBT->>Tool: Verify signature of Tool
end
DBT-->>Tool: Run <tool name> with provided arguments
Note over DBT,Repository: DBT exits. Tool takes DBT's pid in the process table
```
DBT can distribute any single-file executable: Go binaries, Python tools built with PyInstaller, shell scripts with detached signatures, or anything else that compiles to a single file.
## Why DBT?
Software distribution in organizations often involves manual updates users forget, security verification that's too complex for regular use, and "push model" updates that break workflows.
DBT implements a "pull model" where users get the latest version automatically, administrators get cryptographic verification and controlled distribution, and organizations avoid complex MDM infrastructure. Users can pin to specific versions when needed (`dbt -v 1.2.3 -- tool`).
## Setting Up DBT
Setting up DBT for your organization requires four steps:
1. **Deploy a repository** — somewhere to store your binaries
2. **Create your signing keys** — GPG keys for artifact verification
3. **Publish DBT itself** — get dbt and catalog onto your repository
4. **Build your organization's installer** — a single binary that bootstraps users
### Step 1: Deploy a Repository
DBT needs a file server that supports HTTP GET (for downloads) and ideally HTTP PUT (for publishing). There are several options.
#### Option A: Kubernetes Reposerver (Recommended)
The built-in reposerver is a purpose-built Go HTTP server with authentication, and ships as a distroless container image. Kustomize manifests are included in the `kubernetes/` directory.
```bash
# Deploy the base configuration
kubectl apply -k kubernetes/base
# Or create a customized overlay (recommended)
cp -r kubernetes/overlays/example kubernetes/overlays/myenv
# Edit kubernetes/overlays/myenv/kustomization.yaml and reposerver.json
kubectl apply -k kubernetes/overlays/myenv
```
Set the reposerver version in your overlay's `kustomization.yaml`:
```yaml
images:
- name: ghcr.io/nikogura/dbt-reposerver
newTag: "3.7.16"
```
The base deployment creates a StatefulSet with a 10Gi PersistentVolumeClaim. Adjust the size in your overlay to fit your artifact volume.
See `kubernetes/README.md` for the full Kubernetes deployment guide, including ingress configuration and WAF considerations.
#### Option B: Docker
Run the reposerver directly with Docker for development or small deployments:
```bash
# Run with default settings (serves /var/dbt on port 9999)
docker run -d -p 9999:9999 \
-v /path/to/repo:/var/dbt \
ghcr.io/nikogura/dbt-reposerver:latest
# Run with a config file for authentication
docker run -d -p 9999:9999 \
-v /path/to/repo:/var/dbt \
-v /path/to/config.json:/etc/dbt/reposerver.json \
ghcr.io/nikogura/dbt-reposerver:latest \
-a 0.0.0.0 -p 9999 -r /var/dbt -f /etc/dbt/reposerver.json
```
**Command-line flags** (passed after the image name):
- `-a`: Listen address (default: `0.0.0.0`)
- `-p`: Listen port (default: `9999`)
- `-r`: Server root directory (default: `/var/dbt`)
- `-f`: Config file path (optional, required for authentication)
This is a distroless image with no shell. Configuration is done via command-line flags, environment variables, or a config file.
#### Environment Variable Configuration
The reposerver can be configured entirely via `REPOSERVER_` environment variables, removing the need for a JSON config file. This is the idiomatic approach for container deployments.
Priority order: **config file (`-f`)** > **CLI flags (`-a`, `-p`, `-r`)** > **environment variables**.
| Variable | Default | Description |
|----------|---------|-------------|
| `REPOSERVER_ADDRESS` | `0.0.0.0` | Listen address |
| `REPOSERVER_PORT` | `9999` | Listen port |
| `REPOSERVER_ROOT` | _(empty)_ | Server root directory |
| `REPOSERVER_AUTH_GETS` | `false` | Require auth for GET requests |
| `REPOSERVER_AUTH_TYPE_GET` | _(empty)_ | Auth method for GET (e.g. `oidc`, `static-token`) |
| `REPOSERVER_AUTH_TYPE_PUT` | _(empty)_ | Auth method for PUT (comma-separated for multi-auth) |
| `REPOSERVER_IDP_FILE_GET` | _(empty)_ | Identity provider file for GET auth |
| `REPOSERVER_IDP_FILE_PUT` | _(empty)_ | Identity provider file for PUT auth |
| `REPOSERVER_IDP_FUNC_GET` | _(empty)_ | Identity provider function for GET auth |
| `REPOSERVER_IDP_FUNC_PUT` | _(empty)_ | Identity provider function for PUT auth |
| `REPOSERVER_STATIC_TOKEN_GET` | _(empty)_ | Static bearer token for GET auth |
| `REPOSERVER_STATIC_TOKEN_PUT` | _(empty)_ | Static bearer token for PUT auth |
| `REPOSERVER_OIDC_ISSUER_URL_GET` | _(empty)_ | OIDC issuer URL for GET |
| `REPOSERVER_OIDC_AUDIENCES_GET` | _(empty)_ | OIDC audiences for GET (comma-separated) |
| `REPOSERVER_OIDC_USERNAME_CLAIM_GET` | _(empty)_ | OIDC username claim for GET |
| `REPOSERVER_OIDC_ALLOWED_GROUPS_GET` | _(empty)_ | Allowed OIDC groups for GET (comma-separated) |
| `REPOSERVER_OIDC_SKIP_ISSUER_VERIFY_GET` | `false` | Skip OIDC issuer verification for GET |
| `REPOSERVER_OIDC_JWKS_CACHE_GET` | `0` | JWKS cache seconds for GET |
| `REPOSERVER_OIDC_ISSUER_URL_PUT` | _(empty)_ | OIDC issuer URL for PUT |
| `REPOSERVER_OIDC_AUDIENCES_PUT` | _(empty)_ | OIDC audiences for PUT (comma-separated) |
| `REPOSERVER_OIDC_USERNAME_CLAIM_PUT` | _(empty)_ | OIDC username claim for PUT |
| `REPOSERVER_OIDC_ALLOWED_GROUPS_PUT` | _(empty)_ | Allowed OIDC groups for PUT (comma-separated) |
| `REPOSERVER_OIDC_SKIP_ISSUER_VERIFY_PUT` | `false` | Skip OIDC issuer verification for PUT |
| `REPOSERVER_OIDC_JWKS_CACHE_PUT` | `0` | JWKS cache seconds for PUT |
**Kubernetes example — no ConfigMap needed:**
```yaml
containers:
- name: reposerver
image: ghcr.io/nikogura/dbt-reposerver:latest
env:
- name: REPOSERVER_ADDRESS
value: "0.0.0.0"
- name: REPOSERVER_PORT
value: "9999"
- name: REPOSERVER_ROOT
value: "/var/dbt"
- name: REPOSERVER_AUTH_TYPE_PUT
value: "static-token"
- name: REPOSERVER_STATIC_TOKEN_PUT
valueFrom:
secretKeyRef:
name: dbt-reposerver
key: publish-token
```
**Docker example:**
```bash
docker run -d -p 9999:9999 \
-v /path/to/repo:/var/dbt \
-e REPOSERVER_ROOT=/var/dbt \
-e REPOSERVER_AUTH_TYPE_PUT=static-token \
-e REPOSERVER_STATIC_TOKEN_PUT=my-secret-token \
ghcr.io/nikogura/dbt-reposerver:latest
```
#### Option C: Amazon S3
Use an S3 bucket as a repository. No server to run — publishing is done with the AWS CLI, and DBT reads directly from S3 using standard AWS credentials.
```bash
# Create the bucket
aws s3 mb s3://my-dbt-repo
# S3 URLs must use virtual host format in DBT config:
# https://my-dbt-repo.s3.us-east-1.amazonaws.com
```
S3 authentication uses your existing AWS credential configuration (`~/.aws/`, environment variables, IAM roles, or `credential_process`).
#### Option D: Artifactory or Any WebDAV Server
Any server that supports HTTP GET and PUT works as a DBT repository. [Artifactory Open Source](https://www.jfrog.com/open-source), Apache with WebDAV, nginx with dav module — all are viable options.
### Step 2: Secure Your Repository
> "DBT is as secure as the repository you trust and how well you protect your signing keys."
If your repository is the reposerver (options A or B above), you should configure authentication. The reposerver supports separate auth methods for read (GET) and write (PUT/DELETE) operations.
#### Choosing an Auth Method
| Method | Best For | Config Value |
|--------|----------|--------------|
| **SSH-OIDC** (Recommended) | Passwordless CLI auth via SSH keys + Dex — no browser, no passwords | `oidc` |
| **Static token** | CI/CD pipelines, automated publishing | `static-token` |
| **SSH-agent JWT** | Direct SSH key verification without an OIDC provider | `ssh-agent-file` |
| **Basic auth (htpasswd)** | Small teams, simple setups | `basic-htpasswd` |
| **Browser-based OIDC** | Keycloak, Okta, Auth0 (requires browser redirect — poor CLI UX) | `oidc` |
You can combine multiple methods. For example, OIDC for user downloads and a static token for CI/CD publishing. Separate multiple auth types with commas: `"authTypePut": "static-token,oidc"`.
#### Example: Open Reads, Token-Protected Writes
The simplest production setup — anyone can download tools, but publishing requires a token:
```json
{
"address": "0.0.0.0",
"port": 9999,
"serverRoot": "/var/dbt",
"authGets": false,
"authTypePut": "static-token",
"authOptsPut": {
"staticTokenEnv": "DBT_PUBLISH_TOKEN"
}
}
```
Set the `DBT_PUBLISH_TOKEN` environment variable on the reposerver container. Clients publish using `Authorization: Bearer <token>`.
The `staticTokenEnv` field reads the token from an environment variable (recommended). You can also use `staticToken` to set the value directly in config, but environment variables are safer for production.
#### Example: OIDC for Everything
Full OIDC authentication for both reads and writes, with group-based authorization for publishing:
```json
{
"address": "0.0.0.0",
"port": 9999,
"serverRoot": "/var/dbt",
"authGets": true,
"authTypeGet": "oidc",
"authOptsGet": {
"oidc": {
"issuerUrl": "https://dex.example.com",
"audiences": ["dbt-server"],
"usernameClaimKey": "email"
}
},
"authTypePut": "oidc",
"authOptsPut": {
"oidc": {
"issuerUrl": "https://dex.example.com",
"audiences": ["dbt-server"],
"usernameClaimKey": "email",
"allowedGroups": ["dbt-publishers"]
}
}
}
```
**OIDC options:**
- **issuerUrl**: OIDC provider URL (fetches `.well-known/openid-configuration` automatically)
- **audiences**: Expected `aud` claims in tokens
- **usernameClaimKey**: Claim to extract the username from (`sub`, `email`, `preferred_username`, or `name`)
- **allowedGroups**: Groups authorized to access (empty = all authenticated users)
- **requiredClaims**: Additional claims that must match exactly
This works with any OIDC provider: Dex, Keycloak, Okta, Auth0, etc.
#### Example: OIDC Reads + Token Writes (Recommended for CI/CD)
Users authenticate via OIDC to download tools. CI/CD publishes via static token:
```json
{
"address": "0.0.0.0",
"port": 9999,
"serverRoot": "/var/dbt",
"authGets": true,
"authTypeGet": "oidc",
"authOptsGet": {
"oidc": {
"issuerUrl": "https://dex.example.com",
"audiences": ["dbt-server"]
}
},
"authTypePut": "static-token",
"authOptsPut": {
"staticTokenEnv": "DBT_PUBLISH_TOKEN"
}
}
```
#### Example: Basic Auth (htpasswd)
For small teams or environments without an identity provider:
```json
{
"address": "0.0.0.0",
"port": 9999,
"serverRoot": "/var/dbt",
"authGets": false,
"authTypePut": "basic-htpasswd",
"authOptsPut": {
"idpFile": "/etc/dbt/htpasswd"
}
}
```
Generate the htpasswd file with: `htpasswd -c /etc/dbt/htpasswd username`
#### S3 Security
For S3 repositories, security is handled by AWS IAM. Use IAM policies to control who can read from and write to the bucket.
### Step 3: Create Your Signing Keys
DBT verifies every binary using GPG signatures. You need a GPG key to sign artifacts and a **truststore** containing the public key for clients to verify against.
#### Generate a Signing Key
```bash
# Generate a new key (use your organization's email)
gpg --full-generate-key
# List your keys to find the key ID
gpg --list-secret-keys --keyid-format long
```
#### Create the Truststore
The truststore is a file containing one or more PGP public keys in ASCII-armored format. DBT clients download this file and use it to verify signatures.
```bash
# Export your public key
gpg --armor --export [email protected] > truststore
# Multiple signers? Concatenate their public keys:
gpg --armor --export [email protected] >> truststore
gpg --armor --export [email protected] >> truststore
```
#### Upload the Truststore to Your Repository
The truststore must be accessible at the `truststore` path relative to your dbt repository URL.
**For the reposerver (HTTP PUT):**
```bash
curl -X PUT \
-H "Authorization: Bearer $DBT_PUBLISH_TOKEN" \
--data-binary @truststore \
https://dbt.example.com/dbt/truststore
```
**For S3:**
```bash
aws s3 cp truststore s3://my-dbt-repo/dbt/truststore
```
**For a plain filesystem (Docker volume, NFS, etc.):**
```bash
cp truststore /path/to/repo/dbt/truststore
```
#### Protect Your Signing Key
Your signing key is the root of trust for your entire DBT deployment. If it is compromised, an attacker can sign malicious binaries that all your users will trust.
- Store the private key in a secrets manager or hardware security module
- Never commit it to Git unencrypted
- For CI/CD, store it as a base64-encoded secret and import it at runtime:
```bash
echo -n "$GPG_SIGNING_KEY" | base64 -d | gpg --batch --import
```
### Step 4: Publish DBT to Your Repository
Your repository needs dbt and catalog binaries before users can install anything. The easiest way is to republish the upstream releases from GitHub.
#### Repository Directory Layout
The reposerver expects this directory structure under its server root (`/var/dbt` by default):
```
/var/dbt/
├── dbt/
│ ├── truststore # GPG public keys
│ ├── <version>/
│ │ └── <os>/<arch>/
│ │ ├── dbt # Binary
│ │ ├── dbt.asc # GPG signature
│ │ └── dbt.sha256 # SHA256 checksum
│ └── installer/
│ └── <os>/<arch>/
│ └── dbt-installer # Pre-built installer
└── dbt-tools/
└── <toolname>/
└── <version>/
├── description.txt # Tool description (shown in catalog)
├── description.txt.asc
└── <os>/<arch>/
├── <toolname> # Binary
├── <toolname>.asc # GPG signature
├── <toolname>.sha256 # SHA256 checksum
└── <toolname>.md5 # MD5 checksum (optional)
```
#### Path Separation: `/dbt/` vs `/dbt-tools/`
The two top-level directories serve different trust levels:
- **`/dbt/`** contains dbt itself and the truststore. These are the root of trust — if either is compromised, all downstream tool verification is meaningless. Only administrators should have write access to this path.
- **`/dbt-tools/`** contains tools published by developers. Because every tool is verified against the truststore (which lives under `/dbt/`), publishing a tool does not grant trust — the tool's GPG signature must still match a key in the truststore.
This separation enables **delegated publishing**: administrators gate dbt and the truststore, while developers publish their own tools independently. The two paths can be served from different backends entirely — different servers, different S3 buckets, or different auth policies. The client `toolsRepository` config field controls where tools are fetched from, independent of the dbt repository URL.
**Note:** For rebranded distributions (see [Rebranding](#rebranding)), these paths change to `/{BrandBinary}/` and `/{BrandToolsPath}/` respectively. For example, a distribution branded as `myorg` would use `/myorg/` and `/myorg-tools/`.
#### Option A: Downstream Publishing Script (Recommended)
The `scripts/publish-downstream.sh` script downloads the latest release from GitHub, signs each artifact with your GPG key, and uploads everything to your repository:
```bash
# Bearer token auth (CI/CD)
./scripts/publish-downstream.sh \
--http https://dbt.example.com \
--auth bearer --token "$DBT_PUBLISH_TOKEN" \
-y
# Specific version
./scripts/publish-downstream.sh \
--http https://dbt.example.com \
--auth bearer --token "$DBT_PUBLISH_TOKEN" \
-v 3.7.16 -y
# Publish to S3 instead
export DBT_S3_BUCKET="my-dbt-repo"
export TOOLS_S3_BUCKET="my-dbt-tools-repo"
export S3_REGION="us-east-1"
./scripts/publish-downstream.sh --s3 -v 3.7.16 -y
# Dry run (see what would be published)
./scripts/publish-downstream.sh \
--http https://dbt.example.com \
--auth bearer --token "$DBT_PUBLISH_TOKEN" -d
```
**Authentication options for the script:**
| Method | Flag | Description |
|--------|------|-------------|
| Bearer token | `--auth bearer --token TOKEN` | Static token (CI/CD pipelines) |
| OIDC | `--auth oidc --oidc-issuer URL` | Interactive authentication |
| Basic auth | `--auth basic` | Username/password (via env vars) |
| None | `--auth none` | No authentication (unsecured repos) |
**Requirements:** `curl`, `jq`, `gpg`. For S3: `aws` CLI.
The script publishes dbt binaries, catalog binaries, installer scripts, and (if built) pre-built installer binaries for all supported platforms.
#### Option B: Manual Upload (curl)
Upload individual artifacts directly:
```bash
VERSION="3.7.16"
PLATFORM="linux/amd64"
# Sign the binary
gpg --detach-sign --armor dbt
# Generate checksum
sha256sum dbt | cut -d' ' -f1 > dbt.sha256
# Upload binary, signature, and checksumFrequently Asked Questions
What is dbt?
dbt is a Utilities library for the Go programming language. A framework for running self-updating signed binaries from a central, trusted repository
How do I install dbt?
Install dbt with the Go module system using `go get nikogura/dbt`. Check the repository for the current installation instructions.
What category does dbt belong to?
dbt is listed under Utilities, specifically Utilities.