Dagu

About Dagu

This large README is shown as a plain-text preview. Read the full README at the source

<div align="center">
  <a href="https://dagu.sh">
    <img src="./assets/images/hero-logo.png" width="720" alt="Dagu: built for teams whose main work is not orchestration">
  </a>
  <p>
    <a href="https://docs.dagu.sh">Docs</a> ·
    <a href="https://docs.dagu.sh/getting-started/cli">CLI</a> ·
    <a href="https://petstore.swagger.io/?url=https://raw.githubusercontent.com/dagucloud/dagu/main/api/v1/api.yaml">API</a> ·
    <a href="https://docs.dagu.sh/writing-workflows/examples">Examples</a> ·
    <a href="https://dagu-demo-f5e33d0e.dagu.sh">Live demo</a>
    <code>(username/password: demouser)</code> ·
    <a href="https://discord.gg/gpahPUjGRk">Discord</a>
  </p>
</div>

<h1>Dagu</h1>

Dagu is a local-first workflow engine for operations and internal automation. It is open source and self-hostable: a single binary with a built-in Web UI, no external database or message broker, running on Linux, macOS, and Windows. Define [DAGs](https://en.wikipedia.org/wiki/Directed_acyclic_graph) in a declarative YAML format. It natively supports shell commands, Docker containers, Kubernetes Jobs, remote commands via SSH, and more through Dagu Actions.

Dagu turns existing scripts and runbooks into production workflows with scheduling, retries, human tasks, and run history. It runs where your data and credentials live: on-prem, air-gapped, edge, or cloud, and scales from a single node to a fleet of workers.

**Highlights:**

- Single binary installation.
- Self-contained: no external DBMS or message broker required.
- Runs on Linux, macOS, and Windows.
- Declarative YAML format for defining DAGs.
- Run existing shell commands, Docker containers, Kubernetes Jobs, and remote commands over SSH without modifications.
- Compose reusable Sub-DAGs and run work in parallel with concurrency controls.
- Schedule workflows with cron syntax, timezones, overlap policies, and catch-up windows.
- Keep logs, run history, retries, notifications, and webhook triggers in one place.
- Built-in MCP server for inspecting workflows and runs, maintaining Wiki pages, applying changes, and controlling runs.

## Quick Look

For a quick look at how workflows are defined, see the examples.

<div align="center">
  <a href="./assets/images/dagu-demo.mp4?raw=1">
    <img src="./assets/images/cockpit-demo-poster.jpg" width="720" alt="Dagu Cockpit showing queued, running, completed, and failed workflow runs">
  </a>
</div>

| Run Details | Step Logs | Wiki |
|---|---|---|
| ![Run details in dark mode](./assets/images/readme-run-details-dark.png) | ![Workflow logs in dark mode](./assets/images/readme-logs-dark.png) | ![Workflow Wiki in dark mode](./assets/images/readme-wiki-dark.png) |

**Try it live:** [Live Demo](https://dagu-demo-f5e33d0e.dagu.sh) (credentials: `demouser` / `demouser`)

## Why Dagu?

Orchestration is not your main work. You have scripts and containers that already work. You want a schedule, retries, dependencies, and a place to see logs. The usual options each have a cost:

- **cron** runs commands, but gives you no dependencies, no retries, no history.
- **Airflow** orchestrates, but you operate a platform for it (scheduler, metadata database, workers, a Python environment), and your jobs get rewritten as `@dag`/`@task` framework code.
- **Temporal** gives durable execution, but your business logic moves into its SDK and programming model.

You wanted to schedule some jobs. Now you operate a second system, and the orchestrator lives inside the code it was supposed to serve.

Dagu treats workflow structure as configuration, not code. Order, dependencies, retries, schedules, and human tasks go in one YAML file next to your scripts; the engine that runs them is a single process:

```sh
  Traditional Orchestrator          Dagu
  ┌────────────────────────┐        ┌──────────────────┐
  │  Web Server            │        │                  │
  │  Scheduler             │        │  dagu start-all  │
  │  Worker(s)             │        │                  │
  │  PostgreSQL            │        └──────────────────┘
  │  Redis / RabbitMQ      │         Single binary.
  │  Python Runtime        │         Self-hosted.
  └────────────────────────┘         Adds scheduling, retries, and human tasks around existing automation.
    6+ services to manage
```

Your scripts never import the orchestrator. Delete the YAML and they run exactly as before. Keep it, and every run gets a dependency graph, retries, per-step logs, history, and a Web UI.

## Performance

Dagu stores state in local files and reaches production throughput without external services.

- **Throughput:** A single machine can run thousands of workflow runs per day. Actual capacity depends on CPU, memory, disk, and workflow shape.
- **Load control:** Queues, concurrency limits, and resource limits control how many runs execute at once and where they run.
- **Scale out:** Workers spread execution across machines when one node is not enough.

## Real-World Use Cases

| Use Case | How Dagu Helps |
| --- | --- |
| ETL and data operations | Turn data extraction scripts, SQL queries, dbt commands, and data-processing runbooks into observable pipelines with durable execution. |
| Legacy scripts and scheduled jobs | Turn interdependent scripts into maintainable DAGs with a UI, automatic logging, retries, and notifications instead of opaque cron jobs. |
| Media conversion | Run `ffmpeg` for video transcoding and format conversion. File-backed state allows workers to run heavy conversions in parallel without single-machine bottlenecks or external databases. |
| Infrastructure and server automation | Run any command or script over SSH on remote servers, keeping logs, results, and notifications in one place. |
| GitHub-driven workflows | Trigger workflows from GitHub events to run automation on private infrastructure without exposing servers to the public internet. |
| Container and Kubernetes workflows | Run Docker containers and Kubernetes Jobs as steps in your workflows without building a custom control plane around containers. |
| Customer support automation | Provide self-service workflows that non-engineering teams can run for diagnostics, database queries, and routine operations without escalating to engineering. |
| IoT and edge workflows | Run sensor polling, local ML inference, data preprocessing, backups, offline sync, and health checks close to the data source with Web UI visibility. |

## Quick Start

### Install

**macOS/Linux:**

```sh
curl -fsSL https://raw.githubusercontent.com/dagucloud/dagu/main/scripts/installer.sh | bash
```

**Homebrew:**

```sh
brew install dagu
```

**npm:**

```sh
npm install -g --ignore-scripts=false @dagucloud/dagu
```

**Windows (PowerShell):**

```powershell
irm https://raw.githubusercontent.com/dagucloud/dagu/main/scripts/installer.ps1 | iex
```

**Docker:**

```sh
docker run --rm -v ~/.dagu:/var/lib/dagu -p 8080:8080 ghcr.io/dagucloud/dagu:latest dagu start-all
```

> This command does not expose the host Docker daemon to Dagu. Workflows that
> use `container:` or `action: docker.run` need the
> [container-step Docker setup](https://docs.dagu.sh/getting-started/installation/docker#run-container-steps-when-dagu-runs-in-docker).
> Mounting the Docker socket grants workflows control of the host daemon.

**Kubernetes (Helm):**

```sh
helm repo add dagu https://dagucloud.github.io/dagu
helm repo update
helm install dagu dagu/dagu --set persistence.storageClass=<your-rwx-storage-class>
```

> Replace `<your-rwx-storage-class>` with a StorageClass that supports `ReadWriteMany`. See [charts/dagu/README.md](./charts/dagu/README.md) for chart configuration.

The script installers run a guided wizard that can add Dagu to your PATH, set it up as a background service, and create the initial admin account. Homebrew, npm, Docker, and Helm install without the wizard. See the [Installation documentation](https://docs.dagu.sh/getting-started/installation/) for all options.

### Create and run a workflow

Create `hello.yaml`:

```yaml
steps:
  - id: hello
    run: echo "hello from Dagu"
```

Run the workflow with:

```sh
dagu start hello.yaml
```

### Start the server

```sh
dagu start-all --dags .
```

Visit http://localhost:8080

## How You Run Dagu

Dagu runs on one machine, on temporary workers your platform creates for each run, or on workers you keep running. All three are self-hosted, and the same workflow YAML runs on any of them. See the [Deployment Models guide](https://docs.dagu.sh/overview/deployment-models).

<table>
  <tr>
    <td width="50%" align="center" valign="top">
      <strong>Single server</strong><br>
      <img src="./assets/images/deployment-model-local.gif" width="100%" alt="Single-server deployment model with one Dagu server handling scheduling and execution.">
    </td>
    <td width="50%" align="center" valign="top">
      <strong>Temporary workers</strong><br>
      <img src="./assets/images/deployment-model-shared-volume.gif" width="100%" alt="Deployment model where a launcher provisions a temporary worker per run, the worker writes state to a shared volume and is destroyed, and an always-on Dagu server reads that state.">
    </td>
  </tr>
  <tr>
    <td width="50%" align="center" valign="top">
      <strong>Distributed workers</strong><br>
      <img src="./assets/images/deployment-model-self-hosted.gif" width="100%" alt="Distributed-worker deployment where the Dagu server dispatches tasks into a coordinator and workers on separate hosts poll it over gRPC, reporting status and logs back, with the server and persistent volume sharing the same data.">
    </td>
    <td width="50%"></td>
  </tr>
</table>

| Topology | Execution | Best for |
|----------|-----------|----------|
| **Single server** | `dagu start-all` runs the server, scheduler, and steps in one process on one machine. | Development, single-machine scheduled workloads, edge jobs, and internal automation. |
| **Temporary workers** | Cloud Run Jobs, Kubernetes Jobs, or CI provision a worker per run that invokes the binary and is destroyed when the run ends. The server reads run state from a shared volume. | Ephemeral compute, capacity that falls to zero between jobs, and launchers you already operate. |
| **Distributed workers** | Workers you keep running poll a coordinator over gRPC and are routed work by label. | Docker and private-network steps, warm toolchains, and multiple execution hosts. |

### Licensing

- **Community self-host:** No license key required. You operate the server, storage, upgrades, networking, and workers. Start with the [installation guide](https://docs.dagu.sh/getting-started/installation/).
- **Self-host license:** Adds SSO, RBAC, audit logging, and incident SaaS integration to Dagu. See [self-host licensing](https://dagu.sh/pricing#self-host).

## Key Features

- **Observability:** Shared workflows and scheduling with clear visualizations, status tracking, and logs in the Web UI.
- **Language-agnostic:** No framework required. Define workflow steps using shell commands, Docker containers, Kubernetes Jobs, SQL queries, HTTP requests, and any other tool via official and third-party Dagu Actions.
- **Build workflows:** Reuse a step's result when its command and files have not changed. Dagu can also infer dependencies from matching file paths.
- **Reproducibility:** Reproducible runs with pinned tools, plus automatic installation and caching on workers, eliminating the need to manually install dependencies on the server or workers.
- **Human Tasks:** Pause a workflow for acknowledgement or typed operator input, then expose the response to downstream steps.
- **Secret management:** Built-in secret management with secure log masking, preventing credentials from leaking into logs or the Web UI.
- **Self-hosted:** A single binary that runs on Linux, macOS, and Windows. Execution scales out to a fleet of workers.
- **Permission Control:** RBAC and SSO support for team environments, controlling who can view, run, and edit workflows through granular permissions and audit logging.
- **MCP Server:** Authenticated MCP clients can inspect workflows and runs, maintain Wiki pages, apply changes, and control runs.

## Architecture

One binary carries every role. Which roles you start, and where, is what the [deployment models](#how-you-run-dagu) differ on.

- **Server** serves the Web UI and REST API.
- **Scheduler** owns `schedule:` and drains the queue.
- **Coordinator** is the gRPC endpoint workers poll. It also persists what they report: run status, streamed logs, and artifacts.
- **Worker** polls a coordinator, executes dispatched runs locally, and reports back. Routed by labels.
- `dagu start-all` runs the server, scheduler, and coordinator in one process.

Set `DAGU_HEADLESS=true` to run without the Web UI, which applies to any of the topologies and suits CI or CLI-only environments.

```sh
Single server:

  ┌─────────────────────────────────────────┐
  │  dagu start-all                         │
  │  ┌───────────┐ ┌───────────┐ ┌────────┐ │
  │  │ HTTP / UI │ │ Scheduler │ │Executor│ │
  │  └───────────┘ └───────────┘ └────────┘ │
  │  File-based storage (logs, state, queue)│
  └─────────────────────────────────────────┘

Distributed workers:

  ┌────────────┐                   ┌────────────┐
  │ Scheduler  │                   │ HTTP / UI  │
  │            │                   │            │
  │ ┌────────┐ │                   └─────┬──────┘
  │ │ Queue  │ │  Dispatch (gRPC)        │ Dispatch / GetWorkers
  │ │(file)  │ │─────────┐               │ (gRPC)
  │ └────────┘ │         │               │
  └────────────┘         ▼               ▼
                    ┌─────────────────────────┐
                    │      Coordinator        │
                    │  ┌───────────────────┐  │
                    │  │ Dispatch Task     │  │
                    │  │ Store (pending/   │  │
                    │  │ claimed)          │  │
                    │  └───────────────────┘  │
                    └────────▲────────────────┘
                             │
                   Worker poll / task response
                   Heartbeat / ReportStatus /
                   StreamLogs (gRPC)
                             │
               ┌─────────────┴─────────────┐
               │             │             │
          ┌────┴───┐    ┌────┴───┐    ┌────┴───┐
          │Worker 1│    │Worker 2│    │Worker N│ Sandbox execution of DAGs
          │        │    │        │    │        │
          └────────┘    └────────┘    └────────┘

Temporary workers:

  ┌────────────┐   provisions   ┌──────────────┐
  │  Launcher  │───────────────▶│  dagu start  │
  │ Cloud Run  │                │  exits when  │
  │ K8s Job/CI │                │ the run ends │
  └────────────┘                └──────┬───────┘
                                       │ writes
                                       ▼
  ┌────────────┐     reads      ┌──────────────────┐
  │ Dagu server│◀───────────────│  Shared volume   │
  │  UI / API  │                │ dags/state/logs  │
  └────────────┘                └──────────────────┘

  No coordinator, and no network path between the two.
```

## Parameter Definition

Workflows can define parameters that render as typed input forms in the Web UI and can be referenced by steps.

```yaml
params:
  - name: customer_id
    type: string
    description: Customer or account identifier
  - name: change_scope
    type: string
    description: What the repair is allowed to change
    enum:
      - metadata_only
      - permissions
      - full_account
    default: metadata_only
  - name: dry_run
    type: boolean
    default: true

steps:
  - id: extract
    run: >-
      ./scripts/extract.sh
      --customer "${params.customer_id}"
      --scope "${params.change_scope}"
      --dry-run="${params.dry_run}"
    retry_policy:
      limit: 3
      interval_sec: 30
```

<div align="center">
  <img src="./assets/images/ui-params.webp" width="720" alt="Generated parameter input form in the Dagu Web UI">
</div>

## Workflow Examples

### Docker step

When Dagu itself runs in Docker, enable
[Docker daemon access](https://docs.dagu.sh/getting-started/installation/docker#run-container-steps-when-dagu-runs-in-docker)
before using container steps.

Pass standard `docker run` options directly in YAML, including the image, pull policy, platform, volume mounts, working directory, and resource limits:

```yaml
resources:
  limits:
    cpu: 500m
    memory: 512Mi

steps:
  - id: report
    action: docker.run
    with:
      image: ghcr.io/acme/reporting:1.4.2
      pull: always
      platform: linux/amd64
      working_dir: /work
      volumes:
        - ~/orders:/work/data
      auto_remove: true
      command: python generate_report.py --input /work/data/orders.csv
```

See the [Docker](https://docs.dagu.sh/step-types/docker) and [DAG Run Resource Limits](https://docs.dagu.sh/writing-workflows/dag-run-resource-limits) documentation for all configuration options.

### Parallel Sub-DAG execution

The parent invokes the same child DAG for multiple targets and limits concurrent child runs:

```yaml
steps:
  - id: patch
    action: dag.run
    with:
      dag: patch-host
      params:
        host: ${ITEM}
    parallel:
      items:
        - web-1.internal
        - web-2.internal
        - db-1.internal
      max_concurrent: 2

---

name: patch-host
params:
  - name: host
    type: string
ssh:
  user: deploy
  host: ${params.host}
steps:
  - id: apply
    run: apt-get update -q && apt-get upgrade -y
```

See the [Sub-DAGs](https://docs.dagu.sh/writing-workflows/sub-dags) documentation for parameter passing and fan-out options.

### SSH remote execution

```yaml
ssh:
  user: deploy
  host: web-1.internal
  key: ~/.ssh/deploy_key

steps:
  - id: health
    run: curl -f http://localhost:8080/health
    retry_policy:
      limit: 3
      interval_sec: 10

  - id: restart
    run: systemctl restart myapp
    depends: health
```

See the [SSH](https://docs.dagu.sh/step-types/ssh) documentation for authentication and connection options.

### Scheduling with overlap control and catch-up

```yaml
schedule:
  - "0 */6 * * *"          # Every 6 hours
overlap_policy: skip       # Skip if previous run is still active
catchup_window: "5h"       # Catch up missed runs when scheduler is down for up to 5 hours

timeout_sec: 3600
handler_on:
  failure:
    run: notify-team.sh
  exit:
    run: cleanup.sh
```

See the [Scheduling](https://docs.dagu.sh/writing-workflows/scheduling) and [Lifecycle Handlers](https://docs.dagu.sh/writing-workflows/lifecycle-handlers) documentation for all options.

### Retry and error handling

```yaml
steps:
  - name: flaky-api-call
    run: curl -f https://api.example.com/data
    retry_policy:
      limit: 3
      interval_sec: 10
    continue_on:
      failure: true
```

See the [Durable Execution](https://docs.dagu.sh/writing-workflows/durable-execution) and [Continue On](https://docs.dagu.sh/writing-workflows/continue-on) documentation for retry policies and failure handling.

## More Workflow Examples

### Parallel executions

```yaml
steps:
  - id: extract
    run: ./extract.sh

  - id: transform_a
    run: ./transform_a.sh
    depends: extract

  - id: transform_b
    run: ./transform_b.sh
    depends: extract

  - id: load
    run: ./load.sh
    depends: [transform_a, transform_b]
```

```mermaid
%%{init: {'theme': 'base', 'themeVariables': {'background': '#18181B', 'primaryTextColor': '#fff', 'lineColor': '#888'}}}%%
graph LR
    A[extract] --> B[transform_a]
    A --> C[transform_b]
    B --> D[load]
    C --> D
    style A fill:#18181B,stroke:#22C55E,stroke-width:1.6px,color:#fff
    style B fill:#18181B,stroke:#22C55E,stroke-width:1.6px,color:#fff
    style C fill:#18181B,stroke:#22C55E,stroke-width:1.6px,color:#fff
    style D fill:#18181B,stroke:#3B82F6,stroke-width:1.6px,color:#fff
```

Frequently Asked Questions