Fast and Distributed Actor framework using protocol buffers as message for Golang

<h2 align="center">
  <img src="docs/assets/goakt-messaging-distributed-go.png" alt="GoAkt - Distributed Actor framework for Go" width="800"/><br />
  Distributed Actor framework for Go
</h2>

<p align="center">
  <a href="https://github.com/Tochemey/goakt/actions/workflows/build.yml"><img src="https://img.shields.io/github/actions/workflow/status/Tochemey/goakt/build.yml?branch=main" alt="build" /></a>
  <a href="https://pkg.go.dev/github.com/tochemey/goakt"><img src="https://pkg.go.dev/badge/github.com/tochemey/goakt.svg" alt="Go Reference" /></a>
  <a href="https://goreportcard.com/report/github.com/tochemey/goakt"><img src="https://goreportcard.com/badge/github.com/tochemey/goakt" alt="Go Report Card" /></a>
  <a href="https://codecov.io/gh/Tochemey/goakt"><img src="https://codecov.io/gh/Tochemey/goakt/graph/badge.svg?token=J0p9MzwSRH" alt="codecov" /></a>
  <a href="https://go.dev/doc/install"><img src="https://img.shields.io/github/go-mod/go-version/Tochemey/goakt" alt="GitHub go.mod Go version" /></a>
  <a href="https://github.com/avelino/awesome-go"><img src="https://awesome.re/badge.svg" alt="Awesome" /></a>
  <a href="https://www.bestpractices.dev/projects/9248"><img src="https://www.bestpractices.dev/projects/9248/badge" alt="OpenSSF Best Practices" /></a>
</p>

Distributed [Go](https://go.dev/) actor framework to build a reactive and distributed system in Go with typed actor messages.
GoAkt is highly scalable and available when running in cluster mode. It comes with the necessary features require to
build a distributed actor-based system without sacrificing performance and reliability. With GoAkt, you can instantly create a fast, scalable, distributed system
across a cluster of computers.

If you are not familiar with the actor model, the blog post from Brian Storti [here](https://www.brianstorti.com/the-actor-model/) is an excellent and short introduction to the actor model.
Also, check the reference section at the end of the post for more material regarding the actor model.

> **Version & branches**
>
> - **v3** — Protocol buffers for actor messages. Branch: [`release/3.14`](https://github.com/Tochemey/goakt/tree/release/3.14).
> - **v4** — Typed messages with `any`, unified APIs, pluggable serializers (Proto/CBOR), and config-only remoting. See [CHANGELOG.md](./CHANGELOG.md) for migration; [Docs](https://docs.goakt.dev/) for API reference.

## Features

| Feature                   | Description                                                                                                                                                                                                                                                                           |
|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Actor Model**           | Build concurrent and distributed systems using the actor model with typed messages.                                                                                                                                                                                                   |
| **Messaging**             | Tell/Ask APIs for fire-and-forget or request/response flows.                                                                                                                                                                                                                          |
| **Reentrancy**            | Reentrancy-enabled async request messaging with configurable modes and per-call overrides.                                                                                                                                                                                            |
| **Supervision**           | One-for-one/one-for-all strategies, directives, and retry windows for fault tolerance.                                                                                                                                                                                                |
| **Remoting**              | Seamless communication between actors across nodes over TCP.                                                                                                                                                                                                                          |
| **Context Propagation**   | Pluggable context propagation for request-scoped metadata.                                                                                                                                                                                                                            |
| **Clustering**            | Multiple discovery backends (Consul, etcd, Kubernetes, NATS, mDNS, static).                                                                                                                                                                                                           |
| **Location Transparency** | Interact with actors without needing to know their physical location.                                                                                                                                                                                                                 |
| **Relocation**            | Automatic actor relocation on node failure with configurable policies.                                                                                                                                                                                                                |
| **Cluster Singletons**    | Run a single instance across the cluster with guardian-based lifecycle.                                                                                                                                                                                                               |
| **Grains**                | Virtual actors capabilities.                                                                                                                                                                                                                                                          |
| **Passivation**           | Automatically stop idle actors and reclaim resources.                                                                                                                                                                                                                                 |
| **Routers**               | Routing strategies such as round robin, random, and fan-out.                                                                                                                                                                                                                          |
| **Scheduling**            | Timers and delayed messaging built into the runtime.                                                                                                                                                                                                                                  |
| **Stashing & Mailboxes**  | Stash buffers and customizable mailboxes (bounded/unbounded, priority).                                                                                                                                                                                                               |
| **Dependency Injection**  | Attach runtime dependencies to actors at spawn time.                                                                                                                                                                                                                                  |
| **Observability**         | OpenTelemetry metrics, event stream, and dead letters.                                                                                                                                                                                                                                |
| **Extensions**            | Pluggable APIs for cross-cutting capabilities.                                                                                                                                                                                                                                        |
| **Data Center**           | Multi-datacenter support with DC-transparent messaging, pluggable control plane (NATS JetStream, Etcd), DC-aware placement, and cross-DC actor/grain communication.                                                                                                                   |
| **Distributed Data**      | Built-in CRDTs (GCounter, PNCounter, LWWRegister, MVRegister, ORSet, ORMap, Flag) with delta-based replication and anti-entropy synchronization. Configurable coordination levels, key subscriptions, tombstone-based deletion, BoltDB snapshots, and role-based replication scoping. |
| **Reactive Streams**      | Actor-backed, backpressure-aware stream processing with a composable DSL. Rich sources, transformations (map, filter, flatMap, batch, throttle, scan, parallel map), sinks, fan-out/fan-in, graph topologies, error/overflow strategies, stage fusion, and built-in metrics/tracing.  |

## Installation

```bash
# v3.x.x — stable, used in production
go get github.com/tochemey/goakt/v3

# v4.x.x — production ready
go get github.com/tochemey/goakt/v4
```

## Documentations

- **v4**: [v4](https://docs.goakt.dev)
- **v3**: [v3](https://tochemey.gitbook.io/goakt)

## Examples

Kindly check out the [examples](https://github.com/Tochemey/goakt-examples)' repository.

## Support

GoAkt is free and open source. If you need priority support on complex topics or request new features, please consider [sponsorship](https://github.com/sponsors/Tochemey).

## Security

Applications using this library should routinely upgrade their **Go version** and refresh **dependencies** as needed to mitigate security vulnerabilities. **GoAkt** will make a best effort to keep dependencies current and perform vulnerability checks whenever necessary.

## Community

You can join these groups and chat to discuss and ask GoAkt related questions on:

[![GitHub Discussions](https://img.shields.io/github/discussions/Tochemey/goakt)](https://github.com/Tochemey/goakt/discussions)
[![GitHub Issues or Pull Requests](https://img.shields.io/github/issues/Tochemey/goakt)](https://github.com/Tochemey/goakt/issues)

## Contribution

We welcome contributions—bug fixes, new features, and documentation improvements. Before diving in, read the [Architecture Document](./arch/ARCHITECTURE.md) to understand the codebase. We use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) and [Earthly](https://earthly.dev/get-earthly) for builds.

See [contributing.md](./CONTRIBUTING.md) for prerequisites, setup, and the full contribution workflow.

## In Production

This framework is used in production by the following projects/companies:

- [Baki Money](https://www.baki.money/): AI-powered Expense Tracking platform that turns receipts into stories...
- [Event Processor](https://www.v-app.io/iot-builder-3/): Clustered Complex Event Processor (CEP) for IoT data streams.

## Feedback

Kindly use this [issue](https://github.com/Tochemey/goakt/issues/948) to give us your feedback that can help us enhance the framework.

## Benchmark

One can find the benchmark tests here: [Benchmark](./benchmark/)

## Architecture

One can find the architecture documents here: [Architecture](./architecture/)

## Sponsors

<a href="https://github.com/andrew-werdna"><img src="https:&#x2F;&#x2F;github.com&#x2F;andrew-werdna.png" width="60px" alt="User avatar: Andrew Brown" /></a>

Goakt

Distributed actors for Go, C#, and Java/Kotlin

[![Go Report Card](https://goreportcard.com/badge/github.com/asynkron/protoactor-go)](https://goreportcard.com/report/github.com/asynkron/protoactor-go)
[![GoDoc](https://godoc.org/github.com/asynkron/protoactor-go?status.svg)](https://godoc.org/github.com/asynkron/protoactor-go)
[![checks](https://github.com/asynkron/protoactor-go/actions/workflows/checks.yml/badge.svg)](https://github.com/asynkron/protoactor-go/actions/workflows/checks.yml)
[![Sourcegraph](https://sourcegraph.com/github.com/asynkron/protoactor-go/-/badge.svg)](https://sourcegraph.com/github.com/asynkron/protoactor-go?badge)

### [Join our Slack channel](https://join.slack.com/t/asynkron/shared_invite/zt-ko824601-yGN1d3GHF9jzZX2VtONodQ)

# Cross platform actors

Introducing cross platform actor support between Go and C#.

Can I use this?
The Go implementation is still in beta, there are users using Proto Actor for Go in production already.
But be aware that the API might change over time until 1.0.

## Sourcecode - Go

This is the Go repository for Proto Actor.

The C# implementation can be found
here [https://github.com/asynkron/protoactor-dotnet](https://github.com/asynkron/protoactor-dotnet)

## Design principles:

**Minimalistic API** -
The API should be small and easy to use.
Avoid enterprisey JVM like containers and configurations.

**Build on existing technologies** - There are already a lot of great tech for e.g. networking and clustering, build on
those.
e.g. gRPC streams for networking, Consul.IO for clustering.

**Pass data, not objects** - Serialization is an explicit concern, don't try to hide it.
Protobuf all the way.

**Be fast** - Do not trade performance for magic API trickery.

Ultra fast remoting, Proto Actor currently manages to pass over **two million messages per second** between nodes using
only two actors, while still preserving message order!
This is six times more the new super advanced UDP based Artery transport for Scala Akka, and 30 times faster than
Akka.NET.

```text
:> node1.exe
Started EndpointManager
Started Activator
Starting Proto.Actor server address="127.0.0.1:8081"
Started EndpointWatcher address="127.0.0.1:8080"
Started EndpointWriter address="127.0.0.1:8080"
EndpointWriter connecting address="127.0.0.1:8080"
EndpointWriter connected address="127.0.0.1:8080"
2020/06/22 10:45:20 Starting to send
2020/06/22 10:45:20 50000
2020/06/22 10:45:20 100000
2020/06/22 10:45:20 150000
... snip ...
2020/06/22 10:45:21 900000
2020/06/22 10:45:21 950000
2020/06/22 10:45:21 1000000
2020/06/22 10:45:21 Elapsed 732.9921ms
2020/06/22 10:45:21 Msg per sec 2728542 <--

```

## History

As the creator of the Akka.NET project, I have come to some distinct conclusions while being involved in that project.
In Akka.NET we created our own thread pool, our own networking layer, our own serialization support, our own
configuration support etc. etc.
This was all fun and challenging, it is however now my firm opinion that this is the wrong way to go about things.

**If possible, software should be composed, not built**, only add code to glue existing pieces together.
This yields a much better time to market, and allows us to focus on solving the actual problem at hand, in this case
concurrency and distributed programming.

Proto Actor builds on existing technologies, Protobuf for serialization, gRPC streams for network transport.
This ensures cross platform compatibility, network protocol version tolerance and battle proven stability.

Another extremely important factor here is business agility and having an exit strategy.
By being cross platform, your organization is no longer tied into a specific platform, if you are migrating from .NET to
Go,
This can be done while still allowing actor based services to communicate between platforms.

Reinvent by not reinventing.

---

## Why Actors

![batman](/resources/batman.jpg)

- Decoupled Concurrency
- Distributed by default
- Fault tolerance

For a more indepth description of the differences, see this
thread [Actors vs. CSP](https://www.quora.com/Go-programming-language-How-are-Akka-actors-are-different-than-Goroutines-and-Channels)

## Building

You need to ensure that your `$GOPATH` variable is properly set.

Next, install the [standard protocol buffer implementation](https://github.com/google/protobuf) and run the following
commands to get all the necessary tooling:

```
go get github.com/asynkron/protoactor-go/...
cd $GOPATH/src/github.com/asynkron/protoactor-go
go get ./...
make
```

After invoking last command you will have generated protobuf definitions and built the project.

Windows users can use Cygwin to run make: [www.cygwin.com](https://www.cygwin.com/)

## Testing

This command exectutes all tests in the repository except for consul integration tests (you need consul for running
those tests). We also skip directories that don't contain any tests.

```
go test `go list ./... | grep -v "/examples/" | grep -v "/persistence" | grep -v "/scheduler"`
```
## Hello world

```go
type Hello struct{ Who string }
type HelloActor struct{}

func (state *HelloActor) Receive(context actor.Context) {
    switch msg := context.Message().(type) {
    case Hello:
        fmt.Printf("Hello %v\n", msg.Who)
    }
}

func main() {
    context := actor.EmptyRootContext
    props := actor.PropsFromProducer(func() actor.Actor { return &HelloActor{} })
    pid, err := context.Spawn(props)
    if err != nil {
        panic(err)
    }
    context.Send(pid, Hello{Who: "Roger"})
    console.ReadLine()
}
```

## State machines / SetBehavior, PushBehavior and PopBehavior

```go
type Hello struct{ Who string }
type SetBehaviorActor struct{}

func (state *SetBehaviorActor) Receive(context actor.Context) {
    switch msg := context.Message().(type) {
    case Hello:
        fmt.Printf("Hello %v\n", msg.Who)
        context.SetBehavior(state.Other)
    }
}

func (state *SetBehaviorActor) Other(context actor.Context) {
    switch msg := context.Message().(type) {
    case Hello:
        fmt.Printf("%v, ey we are now handling messages in another behavior", msg.Who)
    }
}

func NewSetBehaviorActor() actor.Actor {
    return &SetBehaviorActor{}
}

func main() {
    context := actor.EmptyRootContext
    props := actor.PropsFromProducer(NewSetBehaviorActor)
    pid, err := context.Spawn(props)
    if err != nil {
        panic(err)
    }
    context.Send(pid, Hello{Who: "Roger"})
    context.Send(pid, Hello{Who: "Roger"})
    console.ReadLine()
}
```

## Lifecycle events

Unlike Akka, Proto Actor uses messages for lifecycle events instead of OOP method overrides

```go
type Hello struct{ Who string }
type HelloActor struct{}

func (state *HelloActor) Receive(context actor.Context) {
    switch msg := context.Message().(type) {
    case *actor.Started:
        fmt.Println("Started, initialize actor here")
    case *actor.Stopping:
        fmt.Println("Stopping, actor is about shut down")
    case *actor.Stopped:
        fmt.Println("Stopped, actor and its children are stopped")
    case *actor.Restarting:
        fmt.Println("Restarting, actor is about restart")
    case Hello:
        fmt.Printf("Hello %v\n", msg.Who)
    }
}

func main() {
    context := actor.EmptyRootContext
    props := actor.PropsFromProducer(func() actor.Actor { return &HelloActor{} })
    pid, err := context.Spawn(props)
    if err != nil {
        panic(err)
    }
    context.Send(pid, Hello{Who: "Roger"})

    // why wait?
    // Stop is a system message and is not processed through the user message mailbox
    // thus, it will be handled _before_ any user message
    // we only do this to show the correct order of events in the console
    time.Sleep(1 * time.Second)
    context.Stop(pid)

    console.ReadLine()
}
```

## Supervision

Root actors are supervised by the `actor.DefaultSupervisionStrategy()`, which always issues a `actor.RestartDirective`
for failing actors
Child actors are supervised by their parents.
Parents can customize their child supervisor strategy using `Proto Actor.Props`

### Example

```go
type Hello struct{ Who string }
type ParentActor struct{}

func (state *ParentActor) Receive(context actor.Context) {
    switch msg := context.Message().(type) {
    case Hello:
        props := actor.PropsFromProducer(NewChildActor)
        child := context.Spawn(props)
        context.Send(child, msg)
    }
}

func NewParentActor() actor.Actor {
    return &ParentActor{}
}

type ChildActor struct{}

func (state *ChildActor) Receive(context actor.Context) {
    switch msg := context.Message().(type) {
    case *actor.Started:
        fmt.Println("Starting, initialize actor here")
    case *actor.Stopping:
        fmt.Println("Stopping, actor is about shut down")
    case *actor.Stopped:
        fmt.Println("Stopped, actor and its children are stopped")
    case *actor.Restarting:
        fmt.Println("Restarting, actor is about restart")
    case Hello:
        fmt.Printf("Hello %v\n", msg.Who)
        panic("Ouch")
    }
}

func NewChildActor() actor.Actor {
    return &ChildActor{}
}

func main() {
	decider := func(reason interface{}) actor.Directive {
		log.Printf("handling failure for child. reason:%v", reason)

		// return actor.StopDirective
		return actor.RestartDirective
	}
	supervisor := actor.NewOneForOneStrategy(10, 1000, decider)

	ctx := actor.NewActorSystem().Root
	props := actor.PropsFromProducer(NewParentActor).WithSupervisor(supervisor)

	pid := ctx.Spawn(props)
	ctx.Send(pid, Hello{Who: "Roger"})

	console.ReadLine()
}
```

## Networking / Remoting

Proto Actor's networking layer is built as a thin wrapper ontop of gRPC and message serialization is built on Protocol
Buffers<br/>

### Example

#### Node 1

```go
type MyActor struct {
    count int
}

func (state *MyActor) Receive(context actor.Context) {
    switch context.Message().(type) {
    case *messages.Response:
        state.count++
        fmt.Println(state.count)
    }
}

func main() {
    remote.Start("localhost:8090")

    context := actor.EmptyRootContext
    props := actor.PropsFromProducer(func() actor.Actor { return &MyActor{} })
    pid, _ := context.Spawn(props)
    message := &messages.Echo{Message: "hej", Sender: pid}

    // this is to spawn remote actor we want to communicate with
    spawnResponse, _ := remote.SpawnNamed("localhost:8091", "myactor", "hello", time.Second)

    // get spawned PID
    spawnedPID := spawnResponse.Pid
    for i := 0; i < 10; i++ {
        context.Send(spawnedPID, message)
    }

    console.ReadLine()
}
```

#### Node 2

```go
type MyActor struct{}

func (*MyActor) Receive(context actor.Context) {
    switch msg := context.Message().(type) {
    case *messages.Echo:
        context.Send(msg.Sender, &messages.Response{
            SomeValue: "result",
        })
    }
}

func main() {
    remote.Start("localhost:8091")

    // register a name for our local actor so that it can be spawned remotely
    remote.Register("hello", actor.PropsFromProducer(func() actor.Actor { return &MyActor{} }))
    console.ReadLine()
}
```

### Message Contracts

```proto
syntax = "proto3";
package messages;
import "actor.proto"; // we need to import actor.proto, so our messages can include PID's

// this is the message the actor on node 1 will send to the remote actor on node 2
message Echo {
  actor.PID Sender = 1; // this is the PID the remote actor should reply to
  string Message = 2;
}

// this is the message the remote actor should reply with
message Response {
  string SomeValue = 1;
}
```

Notice: always use "gogoslick_out" instead of "go_out" when generating proto code. "gogoslick_out" will create type
names which will be used during serialization.

For more examples, see the example folder in this repository.

## Contributors

<a href="https://github.com/asynkron/protoactor-go/graphs/contributors">
  <img src="https://contributors-img.web.app/image?repo=asynkron/protoactor-go" />
</a>

Made with [contributors-img](https://contributors-img.web.app). 


ProtoActor

Blazingly fast and light-weight Actor engine written in Golang

[![Go Report Card](https://goreportcard.com/badge/github.com/anthdm/hollywood)](https://goreportcard.com/report/github.com/anthdm/hollywood)
![example workflow](https://github.com/anthdm/hollywood/actions/workflows/build.yml/badge.svg?branch=master)
<a href="https://discord.gg/gdwXmXYNTh">
	<img src="https://discordapp.com/api/guilds/1025692014903316490/widget.png?style=shield" alt="Discord Shield"/>
</a>

# Blazingly fast, low latency actors for Golang

Hollywood is an ULTRA fast actor engine build for speed and low-latency applications. Think about game servers,
advertising brokers, trading engines, etc... It can handle **10 million messages in under 1 second**.

## What is the actor model?

The Actor Model is a computational model used to build highly concurrent and distributed systems. It was introduced by
Carl Hewitt in 1973 as a way to handle complex systems in a more scalable and fault-tolerant manner.

In the Actor Model, the basic building block is an actor, sometimes referred to as a receiver in Hollywood, 
which is an independent unit of computation that communicates with other actors by exchanging messages. 
Each actor has its own state and behavior, and
can only communicate with other actors by sending messages. This message-passing paradigm allows for a highly
decentralized and fault-tolerant system, as actors can continue to operate independently even if other actors fail or
become unavailable.

Actors can be organized into hierarchies, with higher-level actors supervising and coordinating lower-level actors. This
allows for the creation of complex systems that can handle failures and errors in a graceful and predictable way.

By using the Actor Model in your application, you can build highly scalable and fault-tolerant systems that can handle a
large number of concurrent users and complex interactions.

## Features

Compiles to WASM! Both GOOS=js and GOOS=wasm32

- Guaranteed message delivery on actor failure (buffer mechanism)
- Fire & forget or request & response messaging, or both
- High performance dRPC as the transport layer
- Optimized proto buffers without reflection
- Lightweight and highly customizable
- Cluster support for writing distributed self discovering actors 

# Benchmarks

```
make bench
```

```
spawned 10 engines
spawned 2000 actors per engine
Send storm starting, will send for 10s using 20 workers
Messages sent per second 3244217
..
Messages sent per second 3387478
Concurrent senders: 20 messages sent 35116641, messages received 35116641 - duration: 10s
messages per second: 3511664
deadletters: 0
```

# Installation

```
go get github.com/anthdm/hollywood/...
```

> Hollywood requires Golang version `1.21`

# Quickstart

We recommend you start out by writing a few examples that run locally. Running locally is a bit simpler as the compiler
is able to figure out the types used. When running remotely, you'll need to provide protobuffer definitions for the
compiler.

## Hello world.

Let's go through a Hello world message. The complete example is available in the 
[hello world](examples/helloworld) folder. Let's start in main:
```go
engine, err := actor.NewEngine(actor.NewEngineConfig())
```
This creates a new engine. The engine is the core of Hollywood. It's responsible for spawning actors, sending messages
and handling the lifecycle of actors. If Hollywood fails to create the engine it'll return an error. For development 
you shouldn't use to pass any options to the engine so you can pass nil. We'll look at the options later.

Next we'll need to create an actor. These are some times referred to as `Receivers` after the interface they must 
implement. Let's create a new actor that will print a message when it receives a message. 

```go
pid := engine.Spawn(newHelloer, "hello")
```
This will cause the engine to spawn an actor with the ID "hello". The actor will be created by the provided 
function `newHelloer`. Ids must be unique. It will return a pointer to a PID. A PID is a process identifier. It's a unique identifier for the actor. Most of
the time you'll use the PID to send messages to the actor. Against remote systems you'll use the ID to send messages, 
but on local systems you'll mostly use the PID.

Let's look at the `newHelloer` function and the actor it returns. 

```go
type helloer struct{}

func newHelloer() actor.Receiver {
	return &helloer{}
}
```

Simple enough. The `newHelloer` function returns a new actor. The actor is a struct that implements the actor.Receiver.
Lets look at the `Receive` method.

```go
type message struct {
	data string
}

func (h *helloer) Receive(ctx *actor.Context) {
	switch msg := ctx.Message().(type) {
	case actor.Initialized:
		fmt.Println("helloer has initialized")
	case actor.Started:
		fmt.Println("helloer has started")
	case actor.Stopped:
		fmt.Println("helloer has stopped")
	case *message:
		fmt.Println("hello world", msg.data)
	}
}
```

You can see we define a message struct. This is the message we'll send to the actor later. The Receive method
also handles a few other messages. These lifecycle messages are sent by the engine to the actor, you'll use these to 
initialize your actor

The engine passes an actor.Context to the `Receive` method. This context contains the message, the PID of the sender and
some other dependencies that you can use.

Now, lets send a message to the actor. We'll send a `message`, but you can send any type of message you want. The only
requirement is that the actor must be able to handle the message. For messages to be able to cross the wire 
they must be serializable. For protobuf to be able to serialize the message it must be a pointer. 
Local messages can be of any type.

Finally, lets send a message to the actor.

```go
engine.Send(pid, &message{data: "hello, world!"})
```

This will send a message to the actor. Hollywood will route the message to the correct actor. The actor will then print
a message to the console.

The **[examples](examples/)** folder is the best place to learn and
explore Hollywood further.


## Spawning actors

When you spawn an actor you'll need to provide a function that returns a new actor. As the actor is spawn there are a few
tunable options you can provide.

### With default configuration
```go
e.Spawn(newFoo, "myactorname")
```

### Passing arguments to the constructor

Sometimes you'll want to pass arguments to the actor constructor. This can be done by using a closure. There is
an example of this in the [request example](examples/request). Let's look at the code.

The default constructor will look something like this:
```go
func newNameResponder() actor.Receiver {
	return &nameResponder{name: "noname"}
}
```

To build a new actor with a name you can do the following:
```go
func newCustomNameResponder(name string) actor.Producer {
	return func() actor.Receiver {
		return &nameResponder{name}
	}
}

```
You can then spawn the actor with the following code:
```go
pid := engine.Spawn(newCustomNameResponder("anthony"), "name-responder")
```

### With custom configuration
```go
e.Spawn(newFoo, "myactorname",
	actor.WithMaxRestarts(4),
		actor.WithInboxSize(1024 * 2),
		actor.WithId("bar"),
	)
)
```
The options should be pretty self explanatory. You can set the maximum number of restarts, which tells the engine
how many times the given actor should be restarted in case of panic, the size of the inbox, which sets a limit on how
and unprocessed messages the inbox can hold before it will start to block.

### As a stateless function 
Actors without state can be spawned as a function, because its quick and simple.
```go
e.SpawnFunc(func(c *actor.Context) {
	switch msg := c.Message().(type) {
	case actor.Started:
		fmt.Println("started")
		_ = msg
	}
}, "foo")
```

## Remote actors
Actors can communicate with each other over the network with the Remote package. 
This works the same as local actors but "over the wire". Hollywood supports serialization with protobuf.

### Configuration

remote.New() takes a listen address and a remote.Config struct.

You'll instantiate a new remote with the following code:
```go
tlsConfig := TlsConfig: &tls.Config{
	Certificates: []tls.Certificate{cert},
}

config := remote.NewConfig().WithTLS(tlsConfig)
remote := remote.New("0.0.0.0:2222", config)

engine, err := actor.NewEngine(actor.NewEngineConfig().WithRemote(remote))
```

Look at the [Remote actor examples](examples/remote) and the [Chat client & Server](examples/chat) for more information.

## Eventstream

In a production system thing will eventually go wrong. Actors will crash, machines will fail, messages will end up in
the deadletter queue. You can build software that can handle these events in a graceful and predictable way by using
the event stream.

The Eventstream is a powerful abstraction that allows you to build flexible and pluggable systems without dependencies. 

1. Subscribe any actor to a various list of system events
2. Broadcast your custom events to all subscribers 

Note that events that are not handled by any actor will be dropped. You should have an actor subscribed to the event 
stream in order to receive events. As a bare minimum, you'll want  to handle `DeadLetterEvent`. If Hollywood fails to 
deliver a message to an actor it will send a `DeadLetterEvent` to the event stream. 

Any event that fulfills the `actor.LogEvent` interface will be logged to the default logger, with the severity level, 
message and the attributes of the event set by the `actor.LogEvent` `log()` method.

### List of internal system events 
* `actor.ActorInitializedEvent`, an actor has been initialized but did not processed its `actor.Started message`
* `actor.ActorStartedEvent`, an actor has started
* `actor.ActorStoppedEvent`, an actor has stopped
* `actor.DeadLetterEvent`, a message was not delivered to an actor
* `actor.ActorRestartedEvent`, an actor has restarted after a crash/panic.
* `actor.RemoteUnreachableEvent`, sending a message over the wire to a remote that is not reachable.
* `cluster.MemberJoinEvent`, a new member joins the cluster 
* `cluster.MemberLeaveEvent`, a new member left the cluster 
* `cluster.ActivationEvent`, a new actor is activated on the cluster 
* `cluster.DeactivationEvent`, an actor is deactivated on the cluster 

### Eventstream example

There is a [eventstream monitoring example](examples/eventstream-monitor) which shows you how to use the event stream.
It features two actors, one is unstable and will crash every second. The other actor is subscribed to the event stream
and maintains a few counters for different events such as crashes, etc. 

The application will run for a few seconds and the poison the unstable actor. It'll then query the monitor with a 
request. As actors are floating around inside the engine, this is the way you interact with them. main will then print
the result of the query and the application will exit.

## Customizing the Engine

We're using the function option pattern. All function options are in the actor package and start their name with
"EngineOpt". Currently, the only option is to provide a remote. This is done by

```go
r := remote.New(remote.Config{ListenAddr: addr})
engine, err := actor.NewEngine(actor.EngineOptRemote(r))
```
addr is a string with the format "host:port".

## Middleware

You can add custom middleware to your Receivers. This can be useful for storing metrics, saving and loading data for
your Receivers on `actor.Started` and `actor.Stopped`.

For examples on how to implement custom middleware, check out the middleware folder in the ***[examples](examples/middleware)***

## Logging

Hollywood has some built in logging. It will use the default logger from the `log/slog` package. You can configure the
logger to your liking by setting the default logger using `slog.SetDefaultLogger()`. This will allow you to customize 
the log level, format and output. Please see the `slog` package for more information.

Note that some events might be logged to the default logger, such as `DeadLetterEvent` and `ActorStartedEvent` as these
events fulfill the `actor.LogEvent` interface. See the Eventstream section above for more information.

# Test

```
make test
```

# Community and discussions
Join our Discord community with over 2000 members for questions and a nice chat.
<br>
<a href="https://discord.gg/gdwXmXYNTh">
	<img src="https://discordapp.com/api/guilds/1025692014903316490/widget.png?style=banner2" alt="Discord Banner"/>
</a>

# Used in Production By

This project is currently used in production by the following organizations/projects:

- [Sensora IoT](https://sensora.io)
- [Market Monkey Terminal](https://marketmonkeyterminal.com)

# License

Hollywood is licensed under the MIT licence.


Hollywood

Graph control flow library (AOP, actor, state-machine)

[![](https://goreportcard.com/badge/github.com/pancsta/asyncmachine-go)](https://goreportcard.com/report/github.com/pancsta/asyncmachine-go)
[![](https://pkg.go.dev/badge/github.com/pancsta/asyncmachine-go.svg)](https://pkg.go.dev/github.com/pancsta/asyncmachine-go)
[![website](https://img.shields.io/badge/asyncmachine-.dev-blue)](https://asyncmachine.dev)
![](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/pancsta/c6032233dc1d632732ecdc1a4c119850/raw/loc-pkg.json)
![](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/pancsta/c6032233dc1d632732ecdc1a4c119850/raw/loc-tools.json)
![](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/pancsta/c6032233dc1d632732ecdc1a4c119850/raw/tests.json)
![](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/pancsta/c6032233dc1d632732ecdc1a4c119850/raw/tests-pkg.json)
![](https://img.shields.io/github/v/release/pancsta/asyncmachine-go)
[![](https://img.shields.io/github/last-commit/pancsta/asyncmachine-go/main)](https://github.com/pancsta/asyncmachine-go/commits/main/)
[![](https://matrix.to/img/matrix-badge.svg)](https://matrix.to/#/#room:asyncmachine)

<div align="center">
    <a href="#samples">Samples</a> .
    <a href="#getting-started">Getting Started</a> .
    <a href="#packages">Packages</a> .
    <a href="#devtools">Devtools</a> .
    <a href="#apps">Apps</a> .
    <a href="#documentation">Docs</a> .
    <a href="#faq">FAQ</a> .
    <a href="#changes">Changes</a>
    <br />
</div>

# <img src="https://pancsta.github.io/assets/asyncmachine-go/logo.png" height="25" width="25" /> asyncmachine-go

<div align="center"><picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://pancsta.github.io/assets/asyncmachine-go/lifecycle.dark.png">
  <source media="(prefers-color-scheme: light)" srcset="https://pancsta.github.io/assets/asyncmachine-go/lifecycle.light.png">
  <img alt="OpenTelemetry traces in Jaeger" src="https://pancsta.github.io/assets/asyncmachine-go/lifecycle.light.png">
</picture></div>

> [!NOTE]
> State machines communicate through states.

**asyncmachine-go** is a distributed workflow engine (technically a pathless control-flow graph with a consensus),
which implements [AOP](https://en.wikipedia.org/wiki/Aspect-oriented_programming) and [actor model](https://en.wikipedia.org/wiki/Actor_model)
through a **[clock-based state-machine](/pkg/machine/README.md)**. It features [atomic transitions](/docs/manual.md#transition-lifecycle),
[relations](/docs/manual.md#relations), [transparent RPC](/pkg/rpc/README.md), [TUI debugger](/tools/cmd/am-dbg/README.md),
[telemetry](/pkg/telemetry/README.md), [REPL](/tools/cmd/arpc/README.md), [selective distribution](/pkg/rpc/README.md#selective-distribution),
[diagrams](/tools/cmd/am-vis/README.md), and [WASM](/docs/wasm.md) support.

As a control flow library, it decides about running of predefined bits of code (transition handlers) - their order and
which ones to run, according to currently active states (flags). Thanks to a [novel state machine](/pkg/machine/README.md),
the number of handlers can be minimized while maximizing scenario coverage. It's lightweight, fault-tolerant by design,
has rule-based mutations, and can target virtually *any* step-in-time, in *any* workflow. It's a low-level
tool with acceptable performance.

**asyncmachine-go** takes care of `context`, `select`, and `panic`, while allowing for graph-structured concurrency
with [goroutine cancelation](https://github.com/pancsta/asyncmachine-go/pull/261). The history log and relations have
vector formats. It aims to create **autonomous** workflows with **organic** control flow and **stateful** APIs.

<div align="center" class="video">
    <a href="https://github.com/pancsta/asyncmachine-go/blob/main/tools/cmd/am-dbg/README.md">
        <img style="min-height: 820px"
            src="https://github.com/user-attachments/assets/84613447-87dc-48da-9e76-7b4c76705fd3"
            alt="TUI Debugger" />
    </a>
</div>

> [!NOTE]
> `git clone https://github.com/pancsta/asyncmachine-go.git`

#### *Each state represents*

- binary flag
- node with relations
- AOP aspect
- ***logical clock***
- subscription topic
- multiple methods
- metric
- trace
- lock
- breakpoint

Besides the main use-case of **workflows**, it can be used for **stateful applications of any size** - daemons, UIs,
configs, bots, firewalls, synchronization consensus, games, smart graphs, microservice orchestration, robots, contracts,
streams, DI containers, message broking, test scenarios, simulators, as well as **"real-time" systems** which rely on
instant cancelation.

<div align="center">
    <a href="https://github.com/pancsta/assets/blob/main/asyncmachine-go/am-vis.svg?raw=true">
        <img style="min-height: 263px"
            src="https://pancsta.github.io/assets/asyncmachine-go/am-vis.svg?raw=true" alt="am-vis" />
    </a>
</div>

> [!NOTE]
> Flow is state, and state is flow, in a graph.

## Samples

**Minimal** - an untyped definition of 2 states and 1 relation, then 1 mutation and a check.

```go
import am "github.com/pancsta/asyncmachine-go/pkg/machine"
// ...
mach := am.New(nil, am.Schema{
    "Foo": {Require: am.S{"Bar"}},
    "Bar": {},
}, nil)
mach.Add1("Foo", nil)
mach.Is1("Foo") // false
```

**Complicated** - wait on a multi state (event) and the Ready state with a 1s
timeout, then mutate with typed args, on top of a state context.

```go
// state ctx is an expiration ctx
ctx := client.Mach.NewStateCtx(ssC.WorkerReady)
// clock-based subscription
whenPayload := client.Mach.WhenTicks(ssC.WorkerPayload, 1, ctx)
// state mutation
client.RpcWorker.NetMach.Add1(ssW.WorkRequested, Pass(&A{
    Input: 2}))
// WaitFor* wraps select statements
err := amhelp.WaitForAll(ctx, time.Second,
    // post-mutation subscription
    mach.When1(ss.BasicStatesDef.Ready, nil),
    // pre-mutation subscription
    whenPayload)
// check cancelation
if ctx.Err() != nil {
    return // state ctx expired
}
// check error
if err != nil {
    // error state mutation
    client.Mach.AddErr(err, nil)
    return // no err required
}
// client/WorkerPayload and mach/Ready activated
```

<div align="center">
    <a href="https://github.com/pancsta/asyncmachine-go/blob/main/tools/cmd/arpc/README.md">
        <img src="https://pancsta.github.io/assets/asyncmachine-go/videos/repl-demo1.gif"
            alt="REPL" />
    </a>
</div>

> [!NOTE]
> Clock-based navigation in time.

**Handlers** - [Aspect Oriented](https://en.wikipedia.org/wiki/Aspect-oriented_programming) transition handlers.

```go
// can Foo activate?
func (h *Handlers) FooEnter(e *am.Event) bool {
    return true
}
// with Foo active, can Bar activate?
func (h *Handlers) FooBar(e *am.Event) bool {
    return true
}
// Foo activates
func (h *Handlers) FooState(e *am.Event) {
    h.foo = NewConn()
}
// Foo de-activates
func (h *Handlers) FooEnd(e *am.Event) {
    h.foo.Close()
}
```

**Schemas** - relational schemas ([aRPC server](/pkg/rpc/states/ss_rpc.go)).

```go
var ServerSchema = am.Schema{
    ssF.ClientConnected: {Require: S{ssS.RpcReady}},
    ssF.ErrDelivery:     {Require: S{ssS.Exception}},
    ssF.ErrHandlerTimeout: {
        Add:     S{ssS.Exception},
        Multi:   true,
        Require: S{ssS.Exception},
    },
    ssF.ErrNetwork: {
        Remove:  S{ssS.ClientConnected},
        Require: S{ssS.Exception},
    },
    ssF.ErrNetworkTimeout: {Require: S{ssS.Exception}},
    ssF.ErrOnClient:       {Require: S{ssS.Exception}},
    ssF.ErrProviding:      {Require: S{ssS.Exception}},
    ssF.ErrRpc:            {Require: S{ssS.Exception}},
    ssF.ErrSendPayload:    {Require: S{ssS.Exception}},
    ssF.Exception:         {Multi: true},
    ssF.HandshakeDone: {
        Remove:  S{ssS.Handshaking, ssS.HandshakeDone, ssS.Exception},
        Require: S{ssS.Start, ssS.ClientConnected},
    },
    ssF.Handshaking: {
        Remove:  S{ssS.Handshaking, ssS.HandshakeDone},
        Require: S{ssS.Start},
    },
    ssF.Healthcheck: {Multi: true},
    ssF.Heartbeat:   {},
    ssF.MetricSync:  {Multi: true},
    ssF.Ready: {
        Auto:    true,
        Require: S{ssS.HandshakeDone, ssS.RpcReady},
    },
    ssF.RpcAccepting: {
        Remove:  S{ssS.RpcStarting, ssS.RpcAccepting, ssS.RpcReady},
        Require: S{ssS.Start},
    },
    ssF.RpcReady: {
        Remove:  S{ssS.RpcStarting, ssS.RpcAccepting, ssS.RpcReady},
        Require: S{ssS.Start},
    },
    ssF.RpcStarting: {
        Remove:  S{ssS.RpcStarting, ssS.RpcAccepting, ssS.RpcReady},
        Require: S{ssS.Start},
    },
    ssF.SendPayload:     {Multi: true},
    ssF.Start:           {Add: S{ssS.RpcStarting}},
    ssF.WebSocketTunnel: {},
}
```

All examples and benchmarks can be found in [`/examples`](/examples/README.md).

<div align="center">
    <a href="https://github.com/pancsta/asyncmachine-go/blob/main/pkg/telemetry/README.md#open-telemetry-traces">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://pancsta.github.io/assets/asyncmachine-go/otel-jaeger.dark.png">
  <source media="(prefers-color-scheme: light)" srcset="https://pancsta.github.io/assets/asyncmachine-go/otel-jaeger.light.png">
  <img alt="OpenTelemetry traces in Jaeger" src="https://pancsta.github.io/assets/asyncmachine-go/otel-jaeger.light.png">
</picture>
</a></div>

## Getting Started

- 🦾 **[`/pkg/machine`](pkg/machine/README.md)** is the main package
- [`/docs/manual.md`](/docs/manual.md) is the go-to
- [`/docs/diagrams.md`](/docs/diagrams.md) try to explain things visually
- [`/examples`](/examples/README.md) show use cases and integrations
    - with [`/examples/mach_template`](/examples/mach_template) being ready for copy-paste
    - also [Basic](/examples/basic), [CLI Daemon](/examples/cli_daemon), [aRPC](/examples/arpc), [WASM](/examples/wasm),
      [WASM Workflow](/examples/wasm_workflow), [TUI](/examples/tui)
- [`/tools/cmd/am-gen`](/tools/cmd/am-gen) will bootstrap
- [`/tools/cmd/am-dbg`](/tools/cmd/am-dbg/README.md) will record every detail
- code in [`/pkg/node`](pkg/node) shows a high-level usage
- and [reading tests](https://github.com/search?q=repo%3Apancsta%2Fasyncmachine-go+path%3A%2F.*_test.go%2F&type=code)
  is always a good idea

## [Packages](/pkg)

This monorepo offers the following importable packages, especially:

- 🦾 **[`/pkg/machine`](/pkg/machine) State machine, dependency free, semver compatible.**
- [`/pkg/states`](/pkg/states) Reusable state schemas, handlers, and piping.
- [`/pkg/helpers`](/pkg/helpers) Useful functions when working with async state machines.
- [`/pkg/telemetry`](/pkg/telemetry) Telemetry exporters for dbg, metrics, traces, and logs.

Other packages:

- [`/pkg/rpc`](/pkg/rpc) Remote state machines, with the same API as local ones.
- [`/pkg/history`](/pkg/history) History tracking and traversal, including Key-Value and SQL.
- [`/pkg/integrations`](/pkg/integrations) Integrations with NATS and JSON.
- [`/pkg/graph`](/pkg/graph) Directional multigraph of connected state machines.
- [`/pkg/node`](/pkg/node) Distributed worker pools with supervisors.
- [`/pkg/pubsub`](/pkg/pubsub) Decentralized PubSub based on libp2p gossipsub.

## [Devtools](/tools)

- [`/tools/cmd/am-dbg`](/tools/cmd/am-dbg/README.md) Multi-client TUI debugger.
- [`/tools/cmd/am-gen`](/tools/cmd/am-gen) Generates schema files and Grafana dashboards.
- [`/tools/cmd/arpc`](/tools/cmd/arpc) Network-native REPL and CLI.
- [`/tools/cmd/am-vis`](/tools/cmd/am-vis) Generates D2 diagrams.
- [`/tools/cmd/am-relay`](/tools/cmd/am-relay) Rotates logs and relays WASM.

<div align="center">
    <a href="/tools/cmd/am-dbg/README.md">
        <img src="https://pancsta.github.io/assets/asyncmachine-go/am-dbg-dashboard.png"
            alt="am-dbg">
    </a>
</div>

> [!NOTE]
> Inspecting cause-and-effect in distributed systems.

## Apps

**asyncmachine-go** synchronizes state for the following projects:

- [secai](https://github.com/pancsta/secai) - AI Workflows framework
- [secai Web UI](https://github.com/pancsta/secai/tree/main/web) - WebAssembly [go-app](https://go-app.dev/) PWA
- Self-hosting of [pkg/rpc](pkg/rpc/states), [pkg/node](pkg/node/states), [pkg/pubsub](pkg/pubsub/states)
- [arpc REPL](/tools/repl/states) - Cobra-based REPL
- [am-dbg TUI Debugger](/tools/debugger/states) - Single state-machine TUI app
- [libp2p PubSub Simulator](https://github.com/pancsta/go-libp2p-pubsub-benchmark/#libp2p-pubsub-simulator) - Sandbox
  simulator for libp2p-pubsub
- [libp2p PubSub Benchmark](https://github.com/pancsta/go-libp2p-pubsub-benchmark/#libp2p-pubsub-benchmark) -
  Benchmark of libp2p-pubsub ported to asyncmachine-go

## Documentation

- API: [pkg.go.dev](https://pkg.go.dev/github.com/pancsta/asyncmachine-go/pkg/machine) / [code.asyncmachine.dev](https://code.asyncmachine.dev)
- [diagrams](/docs/diagrams.md) / [cookbook](/docs/cookbook.md)
- [manual MD](/docs/manual.md) / [manual PDF](https://pancsta.github.io/assets/asyncmachine-go/manual.pdf)
    - [Machine and States](/docs/manual.md#machine-and-states)
    - [Changing State](/docs/manual.md#changing-state)
    - [Advanced Topics](/docs/manual.md#advanced-topics)
    - [Cheatsheet](/docs/manual.md#cheatsheet)

## Goals

- scale up, not down
- defaults work by default
- everything can be traced and debugged
- automation is evolution
- state != data

## Community

- [GitHub discussions](https://github.com/pancsta/asyncmachine-go/discussions)
- [Matrix chat](https://matrix.to/#/#room:asyncmachine)
- [Author's RSS](https://blogic.tech/feed)

<div align="center">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://asyncmachine.dev/chart_dark.png?nocache">
  <source media="(prefers-color-scheme: light)" srcset="https://asyncmachine.dev/chart_light.png?nocache">
  <img alt="OpenTelemetry traces in Jaeger" src="https://asyncmachine.dev/chart_light.png?nocache">
</picture></div>

> [!NOTE]
> Hundreds of clones.

## Status

Under development, status depends on each package. The bottom layers seem prod grade, the top ones are alpha or testing.

<div align="center">
    <a href="https://pancsta.github.io/assets/asyncmachine-go/grafana.dark.png">
        <img style="min-height: 397px"
            src="https://pancsta.github.io/assets/asyncmachine-go/grafana.dark.png"
            alt="grafana dashboard" />
    </a>
</div>

> [!NOTE]
> Managing distributed concurrency.

## Development

- [good first issues](https://github.com/pancsta/asyncmachine-go/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22)
- before
    - `./scripts/dep-taskfile.sh`
    - `task install-deps`
- after
    - `task test`
    - `task format`
    - `task lint`
    - `task precommit`

### Roadmap

- more tooling and diagrams
- bug fixes, optimizations
- network security, ACLs
- [ROADMAP.md](/ROADMAP.md)

<div align="center">
    <a href="https://github.com/pancsta/asyncmachine-go/blob/main/tools/cmd/am-vis/README.md#transition-sequence">
        <img style="min-height: 397px"
            src="https://pancsta.github.io/assets/asyncmachine-go/am-vis/tx.svg"
            alt="tx sequence diagram" />
    </a>
</div>

> [!NOTE]
> Step by step.

## [FAQ](/FAQ.md)

### How does asyncmachine work?

It calls struct methods according to conventions, a schema, and currently active states (eg `BarEnter`, `FooFoo`,
`FooBar`, `BarState`). It tackles nondeterminism by embracing it - like an UDP event stream with structure.

### What is a "state" in asyncmachine?

State is a binary ID as in status / switch / flag, eg "process RUNNING" or "car BROKEN".

### What does "clock-based" mean?

Each state has a counter of activations & deactivations, and all state counters create "machine time". These are logical
clocks, and the queue is also (partially) counted.

### What's the difference between states and events?

The same event happening many times will cause only 1 state activation, until the state becomes inactive.

The complete FAQ is available at [FAQ.md](/FAQ.md).

## Changes

- [Changelog](/CHANGELOG.md)
- [Breaking changes](/BREAKING.md)
- [Repo traffic](https://github.com/pancsta/asyncmachine-go/pulse)
- [Release feed](https://github.com/pancsta/asyncmachine-go/releases.atom)

<div align="center">
    <img src="https://pancsta.github.io/assets/asyncmachine-go/video.gif?raw=true"
        alt="TUI Debugger" />
</div>

> [!NOTE]
> Don't lose your sync.


asyncmachine-go/pkg/machine

An actor-based Framework with network transparency for creating event-driven architecture in Golang. Inspired by Erlang

<h1><a href="https://ergo.services"><img src=".github/images/logo.svg" alt="Ergo Framework" width="159" height="49"></a></h1>

[![Gitbook Documentation](https://img.shields.io/badge/GitBook-Documentation-f37f40?style=plastic&logo=gitbook&logoColor=white&style=flat)](https://docs.ergo.services)
[![GoDoc](https://pkg.go.dev/badge/ergo-services/ergo)](https://pkg.go.dev/ergo.services/ergo)
[![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT)
[![Telegram Community](https://img.shields.io/badge/Telegram-ergo__services-229ed9?style=flat&logo=telegram&logoColor=white)](https://t.me/ergo_services)
[![Reddit](https://img.shields.io/badge/Reddit-r/ergo__services-ff4500?style=plastic&logo=reddit&logoColor=white&style=flat)](https://reddit.com/r/ergo_services)

The Ergo Framework is an implementation of ideas, technologies, and design patterns from the Erlang world in the Go programming language. It is based on the actor model, network transparency, and a set of ready-to-use components for development. This significantly simplifies the creation of complex and distributed solutions while maintaining a high level of reliability and performance.

### Features ###

1. **Actor Model**: isolated processes communicate through message passing, handling messages sequentially in their own mailbox with four priority queues. Processes support both asynchronous messaging and synchronous request-response patterns, enabling flexible communication while maintaining the actor model guarantees.

2. **Network Transparency**: actors interact the same way whether local or remote. The framework uses custom serialization and protocol for [efficient](https://github.com/ergo-services/benchmarks) distributed communication with connection pooling, compression, and type caching, making network location transparent to application code.

3. **Supervision Trees**: hierarchical fault recovery where supervisors monitor child processes and apply restart strategies when failures occur. Supports multiple supervision types (One For One, All For One, Rest For One, Simple One For One) and restart strategies (Transient, Temporary, Permanent) for building self-healing systems.

4. **Meta Processes**: bridge blocking I/O with the actor model through dedicated meta processes that handle TCP, UDP, Port, and Web protocols. Meta processes run blocking operations without affecting regular actor message processing.

5. **Distributed Systems**: service discovery through embedded or external registrars (etcd, Saturn), distributed publish/subscribe events with token-based authorization and buffering, remote process spawning with factory-based permissions, and remote application orchestration across nodes.

6. **Ready-to-use Components**: core framework includes Actor, Supervisor, Pool, and WebWorker actors plus TCP, UDP, Port, and Web meta processes. Extra library provides Leader, Metrics actors, WebSocket, SSE meta processes, Observer application, Colored and Rotate loggers, Erlang protocol support.

7. **Flexibility**: customize network stack components, certificate management, compression and message priorities, logging, distributed events, and meta processes. Framework supports mTLS, NAT traversal, important delivery for guaranteed messaging, and Cron-based scheduling.

Examples demonstrating the framework's capabilities are available in the [examples repository](https://github.com/ergo-services/examples).

### Benchmarks ###

On a 64-core processor, Ergo Framework demonstrates a performance of **over 21 million messages per second locally** and **nearly 5 million messages per second over the network**.

![image](https://github.com/user-attachments/assets/813900ad-ff79-4cc7-b1e6-396c5781acb1)

Available benchmarks can be found in the [benchmarks repository](https://github.com/ergo-services/benchmarks).

* Messaging performance (local, network)

* Memory consumption per process (demonstrates framework memory footprint)

* Serialization performance comparison: EDF vs Protobuf vs Gob

* Distributed Pub/Sub (event delivery to 1,000,000 subscribers across 10 nodes)

### Observer ###
To inspect the node, network stack, running applications, and processes, you can use the [`observer`](https://github.com/ergo-services/tools/) tool

<img src="https://github.com/user-attachments/assets/1cb83305-6c56-4eb7-b567-76f3c551c176" width="500">

To install the Observer tool, you need to have the Go compiler version 1.20 or higher. Run the following command:

```
$ go install ergo.tools/observer@latest
```

You can also embed the [Observer application](https://docs.ergo.services/extra-library/applications/observer) into your node. To see it in action, see the [demo example](https://github.com/ergo-services/examples/tree/master/demo). For more information, visit the [Observer documentation](https://docs.ergo.services/tools/observer). 



### Quick start ###

For a quick start, use the [`ergo`](https://docs.ergo.services/tools/ergo) tool — a command-line utility designed to simplify the process of generating boilerplate code for your project based on the Ergo Framework. With this tool, you can rapidly create a complete project structure, including applications, actors, supervisors, network components, and more. It offers a set of arguments that allow you to customize the project according to specific requirements, ensuring it is ready for immediate development.

To install use the following command:

```
$ go install ergo.tools/ergo@latest
```

Now, you can create your project with just one command. Here is example:

Supervision tree

```
  mynode
  ├─ myapp
  │  │
  │  └─ mysup
  │     │
  │     └─ myactor
  ├─ myweb
  └─ myactor2
```

To generate project for this design use the following command:

```
$ ergo -init MyNode \
      -with-app MyApp \
      -with-sup MyApp:MySup \
      -with-actor MySup:MyActor \
      -with-web MyWeb \
      -with-actor MyActor2 \
      -with-observer 
```

as a result you will get generated project:

```
  mynode
  ├── apps
  │  └── myapp
  │     ├── myactor.go
  │     ├── myapp.go
  │     └── mysup.go
  ├── cmd
  │  ├── myactor2.go
  │  ├── mynode.go
  │  ├── myweb.go
  │  └── myweb_worker.go
  ├── go.mod
  ├── go.sum
  └── README.md
```

to try it:

```
$ cd mynode
$ go run ./cmd
```

Since we included Observer application, open http://localhost:9911 to inspect your node and running processes.

### Erlang support ###

Starting from version 3.0.0, support for the Erlang network stack has been moved to a [separate module](https://github.com/ergo-services/proto). Version 3.0 was distributed under the BSL 1.1 license, but starting from version 3.1 it is available under the MIT license. Detailed information is available in the [Erlang protocol documentation](https://docs.ergo.services/extra-library/network-protocols/erlang).

### Requirements ###

* Go 1.20.x and above

### Changelog ###

Fully detailed changelog see in the [ChangeLog](CHANGELOG.md) file.

#### [v3.2.0](https://github.com/ergo-services/ergo/releases/tag/v1.999.320) 2026-02-04 [tag version v1.999.320] ####

* Introduced **mTLS support** - new `gen.CertAuthManager` interface for mutual TLS with CA pool management (`ClientCAs`, `RootCAs`, `ClientAuth`, `ServerName`). See [Mutual TLS](https://docs.ergo.services/networking/mutual-tls) documentation
* Introduced **NAT support** - new `RouteHost` and `RoutePort` options in `gen.AcceptorOptions` for nodes behind NAT or load balancers. See [Behind the NAT](https://docs.ergo.services/networking/behind-the-nat) documentation
* Introduced **spawn time control** - `InitTimeout` option in `gen.ProcessOptions` limits `ProcessInit` duration for both local and remote spawn. Remote spawn and application processes limited to max 15 seconds. See [Process](https://docs.ergo.services/basics/process) documentation
* Introduced **zip-bomb protection** - decompression size limits to prevent memory exhaustion attacks
* Added `gen.Ref` methods for request timeout tracking. See [Generic Types](https://docs.ergo.services/basics/generic-types#gen.ref):
  - `Deadline` - returns deadline timestamp stored in reference
  - `IsAlive` - checks if reference is still valid (deadline not exceeded)
* Added `gen.Node` methods. See [Node](https://docs.ergo.services/basics/node) documentation:
  - `ProcessPID` / `ProcessName` - resolve process PID by name and vice versa
  - `Call`, `CallWithTimeout`, `CallWithPriority`, `CallImportant`, `CallPID`, `CallProcessID`, `CallAlias` - synchronous requests from Node interface
  - `Inspect` / `InspectMeta` - inspect processes and meta processes
  - `MakeRefWithDeadline` - create reference with embedded deadline
* Added `gen.RemoteNode.ApplicationInfo` - query application information from remote nodes. See [Remote Start Application](https://docs.ergo.services/networking/remote-start-application) documentation
* Added `gen.Process` methods. See [Process](https://docs.ergo.services/basics/process) documentation:
  - `SendWithPriorityAfter` - delayed send with priority
  - `SendExitAfter` / `SendExitMetaAfter` - delayed exit signals
  - `SendResponseImportant` / `SendResponseErrorImportant` - important delivery for responses
* Added `gen.Meta` methods. See [Meta Process](https://docs.ergo.services/basics/meta-process) documentation:
  - `SendResponse` / `SendResponseError` - respond to requests from meta process
  - `SendPriority` / `SetSendPriority` - message priority control
  - `Compression` / `SetCompression` - compression settings
  - `EnvDefault` - get environment variable with default value
* Added `gen.ApplicationSpec` / `gen.ApplicationInfo` fields:
  - `Tags` - labels for instance selection (blue/green, canary, maintenance). See [Tags for Instance Selection](https://docs.ergo.services/basics/application#tags-for-instance-selection)
  - `Map` - logical role to process name mapping. See [Process Role Mapping](https://docs.ergo.services/basics/application#process-role-mapping)
* Added **HandleInspect** implementations for all supervisor types (OFO, ARFO, SOFO)
* Fixed **LinkChild** in `RemoteNode.Spawn` / `RemoteNode.SpawnRegister`
* Fixed **args persistence** for Simple One For One supervisor - child processes now restart with their original spawn arguments
* Fixed **critical bug**: terminate signals (Link/Monitor exits) were incorrectly rejected due to wrong incarnation validation in network layer. Thanks to [@qjpcpu](https://github.com/qjpcpu) for reporting [#248](https://github.com/ergo-services/ergo/issues/248)
* Completely reworked internal **Target Manager** (`node/tm/`) - improved architecture for process, event, and node target management with comprehensive test coverage
* Completely reworked internal **Pub/Sub** mechanism - improved reliability and performance
* Improved **ProcessInit state** - more `gen.Process` methods now available during initialization:
  - `Link*`, `Unlink*`, `Monitor*`, `Demonitor*`
  - `Call*`, `Inspect`, `InspectMeta`
  - `RegisterName`, `UnregisterName`, `RegisterEvent`, `UnregisterEvent`
  - `SendResponse*`, `SendResponseError*`
  - `CreateAlias`, `DeleteAlias`
* Introduced **shutdown timeout** - `ShutdownTimeout` option in `gen.NodeOptions` (default 3 minutes). During graceful shutdown, pending processes are logged every 5 seconds with state and queue info. After timeout, node force exits with error code 1. See [Node](https://docs.ergo.services/basics/node) documentation
* Added **pprof labels** for actor and meta process goroutines (with `--tags pprof`) - each process goroutine is labeled with its PID, each meta process with its Alias, making it easy to identify stuck processes in pprof output
* Improved API documentation - comprehensive godoc comments for all public interfaces
* **Documentation rewritten** - complete documentation now included in the repository (`docs/`) and available at [docs.ergo.services](https://docs.ergo.services)
* New documentation articles:
  - [Project Structure](https://docs.ergo.services/basics/project-structure) - organizing projects with message isolation levels, deployment patterns, and evolution strategies
  - [Building a Cluster](https://docs.ergo.services/advanced/building-a-cluster) - step-by-step guide to distributed systems with service discovery, load balancing, and failover
  - [Message Versioning](https://docs.ergo.services/advanced/message-versioning) - evolving message contracts in distributed clusters with explicit versioning strategies
  - [Handle Sync](https://docs.ergo.services/advanced/handle-sync) - synchronous message handling patterns
  - [Important Delivery](https://docs.ergo.services/advanced/important-delivery) - guaranteed delivery mechanism
  - [Pub/Sub Internals](https://docs.ergo.services/advanced/pub-sub-internals) - event system architecture
  - [Debugging](https://docs.ergo.services/advanced/debugging) - build tags, pprof integration, troubleshooting stuck processes

* **Extra Library - Actors** (https://github.com/ergo-services/actor):
  - Introduced **Leader** actor - distributed leader election with Raft-inspired consensus algorithm. Features: term-based disambiguation, automatic failover, split-brain prevention through majority quorum, dynamic peer discovery. See [documentation](https://docs.ergo.services/extra-library/actors/leader)
  - Introduced **Metrics** actor - Prometheus metrics exporter that collects node/network telemetry via HTTP endpoint. Features: automatic collection of node metrics (uptime, processes, memory), network metrics per remote node, extensible for custom metrics. See [documentation](https://docs.ergo.services/extra-library/actors/metrics)

* **Extra Library - Meta Processes** (https://github.com/ergo-services/meta):
  - Introduced **SSE** (Server-Sent Events) meta-process - unidirectional server-to-client streaming over HTTP. Features: server handler for accepting connections, client connection for external SSE endpoints, full SSE spec support (event types, IDs, retry hints, multi-line data), process pool with round-robin load balancing, Last-Event-ID for reconnection. See [documentation](https://docs.ergo.services/extra-library/meta-processes/sse)

* **Benchmarks** (https://github.com/ergo-services/benchmarks):
  - Introduced **Distributed Pub/Sub** benchmark - demonstrates event delivery to 1,000,000 subscribers across 10 nodes. Achieves 2.9M msg/sec delivery rate with only 10 network messages (one per consumer node) instead of 1M


### Development and debugging ###

To enable Golang profiler just add `--tags pprof` in your `go run` or `go build` (profiler runs at
`http://localhost:9009/debug/pprof`). Use `PPROF_HOST` and `PPROF_PORT` environment variables to customize the address.

With `--tags pprof`, each actor goroutine is labeled with its PID and each meta process with its Alias for easy identification in pprof output:

```
curl -s "http://localhost:9009/debug/pprof/goroutine?debug=1" | grep -B5 'labels:.*pid'
curl -s "http://localhost:9009/debug/pprof/goroutine?debug=1" | grep -B5 'labels:.*meta'
```

Output:
```
1 @ 0x100c17fa0 ...
# labels: {"pid":"<ABC123.0.1005>"}
#   main.(*Worker).HandleMessage+0x27  /path/worker.go:45
```

This helps identify stuck processes during shutdown by matching PIDs/Aliases from the shutdown log with goroutine stack traces.

To disable panic recovery use `--tags norecover`.

To enable trace logging level for the internals (node, network,...) use `--tags trace` and set the log level `gen.LogLevelTrace` for your node.

For detailed debugging techniques, troubleshooting scenarios, and best practices, see the [Debugging](https://docs.ergo.services/advanced/debugging) documentation.

To run tests with cleaned test cache:

```
go vet
go clean -testcache
go test -v ./testing/tests/...
```

### Commercial support

please, contact support@ergo.services for more information


The Best Go Libraries For Actor Model (5)

Goakt

ProtoActor

Hollywood

asyncmachine-go/pkg/machine

Ergo