gnata

About gnata

This large README is shown as a plain-text preview. Read the full README at the source

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/gnata-dark.png" />
    <source media="(prefers-color-scheme: light)" srcset="assets/gnata-light.png" />
    <img src="assets/gnata-light.png" alt="gnata" width="720" />
  </picture>
</p>

<p align="center">
  A full JSONata 2.x implementation in Go, built for production streaming workloads.
</p>

<p align="center">
  <a href="https://github.com/RecoLabs/gnata/actions/workflows/ci.yml"><img src="https://github.com/RecoLabs/gnata/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
  <a href="https://github.com/RecoLabs/gnata/actions/workflows/ci.yml"><img src="https://img.shields.io/badge/coverage-93.8%25-brightgreen" alt="Coverage" /></a>
  <a href="https://pkg.go.dev/github.com/recolabs/gnata"><img src="https://pkg.go.dev/badge/github.com/recolabs/gnata.svg" alt="Go Reference" /></a>
  <a href="https://github.com/RecoLabs/gnata/blob/main/go.mod"><img src="https://img.shields.io/github/go-mod/go-version/RecoLabs/gnata" alt="Go version" /></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License" /></a>
  <a href="https://www.npmjs.com/package/gnata-js"><img src="https://img.shields.io/npm/v/gnata-js" alt="npm version" /></a>
</p>

<p align="center">
  <a href="#quick-start">Quick Start</a> &middot;
  <a href="#streaming-api">Streaming API</a> &middot;
  <a href="#metrics--observability">Metrics</a> &middot;
  <a href="#performance">Performance</a> &middot;
  <a href="#jsonata-compatibility">Compatibility</a> &middot;
  <a href="#wasm">WASM Playground</a> &middot;
  <a href="CONTRIBUTING.md">Contributing</a> &middot;
  <a href="SECURITY.md">Security</a>
</p>

---

## What is gnata?

[JSONata](https://jsonata.org) is a lightweight query and transformation language for JSON data — think "jq meets XPath with lambda functions." gnata brings the full JSONata 2.x specification to Go, with a production-grade streaming tier designed for evaluating thousands of expressions against millions of events per day with zero contention.

## Features

- **Full JSONata 2.x** — path navigation, wildcards, descendants, predicates, sorting, grouping, lambdas, closures, higher-order functions, transforms, regex, and the complete 50+ function standard library.
- **Two-tier evaluation** — simple expressions use a zero-copy fast path (GJSON); complex expressions fall back to a full AST evaluator.
- **Lock-free streaming** — `StreamEvaluator` batches multiple expressions per event with schema-keyed plan caching. After warm-up, the hot path uses only atomic loads — no mutexes, no RWLocks, no channels.
- **Zero allocations** — simple field comparisons like `user.email = "[email protected]"` evaluate with **0 heap allocations** via GJSON zero-copy string views.
- **Bounded memory** — schema plan cache uses a FIFO ring-buffer with configurable capacity (`WithMaxCachedSchemas`), evicting the oldest entry on overflow.
- **Context-aware** — all evaluation methods accept `context.Context` for cancellation and timeouts. Long-running expressions check context at loop boundaries.
- **Linear-time regex** — uses Go's standard `regexp` (RE2 engine) for guaranteed linear-time matching with no timeouts or backtracking.
- **1,778 test cases** — ported from the official jsonata-js test suite (0 failures, 0 skips).
- **One dependency** — [`tidwall/gjson`](https://github.com/tidwall/gjson) for fast-path byte-level field extraction.
- **~13K lines of Go** — complete implementation with no code generation.
- **WASM support** — compile to WebAssembly for an in-browser playground, or with [TinyGo](https://tinygo.org) for WASI hosts (several times smaller binaries).

## Quick Start

```sh
go get github.com/recolabs/gnata
```

```go
package main

import (
    "context"
    "fmt"
    "github.com/recolabs/gnata"
)

func main() {
    expr, _ := gnata.Compile(`Account.Order.Product.Price`)

    data := map[string]any{
        "Account": map[string]any{
            "Order": []any{
                map[string]any{"Product": map[string]any{"Price": 34.45}},
                map[string]any{"Product": map[string]any{"Price": 21.67}},
            },
        },
    }

    result, _ := expr.Eval(context.Background(), data)
    fmt.Println(result) // [34.45 21.67]
}
```

## API Overview

gnata provides three evaluation tiers, each building on the previous:

### Tier 1 — `Eval(context.Context, any)`

Evaluate against pre-parsed Go values. Pass a context for cancellation and timeouts.

```go
expr, err := gnata.Compile(`$sum(orders.amount)`)
result, err := expr.Eval(ctx, data) // data is map[string]any
```

### Tier 2 — `EvalBytes(context.Context, json.RawMessage)`

Evaluate directly against raw JSON bytes. For fast-path-eligible expressions, fields are extracted via GJSON with zero-copy — the entire document is never materialized.

```go
expr, _ := gnata.Compile(`user.email = "[email protected]"`)
result, _ := expr.EvalBytes(ctx, rawJSON) // rawJSON is json.RawMessage
fmt.Println(expr.IsComparisonFastPath())  // true — zero-copy evaluation
```

### Tier 3 — `StreamEvaluator`

Batch-evaluate multiple compiled expressions against each event in a streaming pipeline. Schema-keyed plan caching deduplicates field extraction across expressions. Lock-free after warm-up.

```go
se := gnata.NewStreamEvaluator(nil,
    gnata.WithPoolSize(500),
    gnata.WithMaxCachedSchemas(50000),
)

// Compile expressions (goroutine-safe)
indices := make([]int, len(rules))
for i, rule := range rules {
    indices[i], _ = se.Compile(rule.Expr)
}

// Hot path — millions of times, hundreds of goroutines
results, _ := se.EvalMany(ctx, eventBytes, schemaKey, indices)
for i, result := range results {
    if result != nil {
        handleMatch(indices[i], result)
    }
}

// Alternative: pre-decoded map input (avoids re-serialization)
results, _ = se.EvalMap(ctx, fieldMap, schemaKey, indices)

// Alternative: already-unmarshaled Go value (skips JSON decoding)
results, _ = se.EvalPreparsed(ctx, eventMap, schemaKey, indices)
```

### Custom Function Registration

Register domain-specific functions via `WithCustomFunctions` on `StreamEvaluator` or `NewCustomEnv` for standalone expressions. See [Custom Functions](#custom-functions) below.

## Streaming API

The `StreamEvaluator` is designed for high-throughput event processing where the same expressions are evaluated against millions of structurally similar events.

### How It Works

```
Startup (once)
  Compile N expressions ──> Analyze AST: classify fast/full
  Configure stable routing: schemaKey -> exprIndices

Hot Path (millions/day, lock-free)
  Raw json.RawMessage event + schemaKey
    ├── BoundedCache lookup (atomic pointer read)
    │   ├── HIT  ──> Immutable GroupPlan
    │   └── MISS ──> Build plan (merge GJSON paths, atomic CAS store)
    ├── gjson.GetBytes per fast-path expression
    ├── Fast-path expressions: distribute extracted results (0 allocs)
    ├── Full-path expressions: selective unmarshal + AST eval
    └── results[]
```

### Key Properties

- **Efficient JSON field extraction** — fast-path expressions use `gjson.GetBytes` for zero-copy path lookups directly on raw JSON bytes.
- **Schema-keyed caching** — the `GroupPlan` (merged paths, expression groupings, selective unmarshal targets) is computed once per schema key and reused immutably.
- **Lock-free reads** — `BoundedCache` publishes an `atomic.Pointer` snapshot on every write; reads scan the snapshot without acquiring a lock. Writes are serialised by a mutex.
- **Selective unmarshal** — full-path expressions unmarshal only the subtrees they need (e.g., just the `items` array from a 10KB event), not the entire document.
- **Pre-decoded map input** — `EvalMap` accepts `map[string]json.RawMessage` directly, skipping full-document serialization when the caller already has individually-encoded fields. Fast paths resolve top-level keys via O(1) map lookup.
- **Pre-parsed Go values** — `EvalPreparsed` accepts `map[string]any`, `[]any`, or any other decoded value the AST evaluator already handles, skipping JSON decoding. GJSON fast paths are not used (they need raw bytes).
- **Dynamic mutation** — `Replace`, `Remove`, and `Reset` methods allow modifying registered expressions at runtime with automatic cache invalidation.
- **Observability** — implement `MetricsHook` to receive per-evaluation callbacks for cache hits/misses, eval latency, fast-path usage, and errors.

## Custom Functions

gnata supports registering user-defined functions that extend the standard JSONata library.

### Defining Custom Functions

Custom functions implement the `CustomFunc` signature:

```go
type CustomFunc func(args []any, focus any) (any, error)
```

Where `args` are the evaluated arguments passed by the JSONata expression and `focus` is the current context value. `focus` follows jsonata-js:

- it is `nil` when the function is passed as an argument, to `$map` or to a lambda, or applied by a bare `x ~> $f`;
- a call in tail position of a lambda body gets the context of the call that entered the lambda.

A custom function named `clone` replaces `$clone`, including the copy the transform operator (`~> |…|…|`) makes of its input.

### Regex Arguments (breaking change in v0.6.0)

A regex is a function inside an expression, as in jsonata-js. A custom function receives a regex argument as a `map[string]any{"pattern": …, "flags": …}`, and returning **that same map** returns the regex. Any other map of that shape is a plain object: one the function builds, a copy of the map it received, and one from input data, even when the function passes it through.

Before v0.6.0 a regex was such a map everywhere, so any `{pattern, flags}` map a custom function returned acted as a regex:

```go
// v0.5.x: $match("abc", $regex("b")) matched. Since v0.6.0 it raises T0410.
"regex": func(args []any, _ any) (any, error) {
    return map[string]any{"pattern": args[0], "flags": "g"}, nil
},
```

To migrate, take the regex as an argument and return the value you received; a custom function that passes an argument through, such as a `$coalesce(a, /b/)`, keeps working:

```go
// $match("abc", $pick(cond, /b/, /c/)) matches with either regex.
"pick": func(args []any, _ any) (any, error) {
    if args[0] == true {
        return args[1], nil
    }
    return args[2], nil
},
```

### Lambda Context (breaking change in v0.6.0)

A lambda's body evaluates against the context the lambda was defined in, as in jsonata-js. Before v0.6.0, a call that passed arguments evaluated the body against the context of the call, so over `{"name": "root", "items": [{"name": "a"}, {"name": "b"}]}`:

```
($f := function($x){ name & $x }; items.$f("!"))
// v0.5.x: ["a!", "b!"]. Since v0.6.0: ["root!", "root!"].
```

To read the item a call is made on, pass it as an argument, as in `($f := function($v, $x){ $v.name & $x }; items.$f($, "!"))`, which gives `["a!", "b!"]`. A lambda defined in the path step, as in `items.(function($x){ name & $x })("!")`, also reads each item.

### `functions.EvalFn` (breaking change in v0.6.0)

`functions.EvalFn`, the callback `functions.RegisterAll` takes to apply a function value, lost its `focus` parameter: it is now `func(fn any, args []any, env *evaluator.Environment) (any, error)`. As in jsonata-js, the standard library applies a function it is given, such as a `$map` callback, with a null context. `EvalFn` refers to the internal `evaluator.Environment` type, so code outside this module cannot implement it in practice.

### Registration

Register custom functions via `WithCustomFunctions` when creating a `StreamEvaluator`:

```go
customFuncs := map[string]gnata.CustomFunc{
    "md5": func(args []any, focus any) (any, error) {
        if len(args) == 0 || args[0] == nil {
            return nil, nil
        }
        h := md5.Sum([]byte(fmt.Sprint(args[0])))
        return fmt.Sprintf("%x", h), nil
    },
    "parseEpochSeconds": func(args []any, focus any) (any, error) {
        // Convert epoch seconds to ISO 8601
        f, ok := args[0].(float64)
        if !ok {
            return nil, nil
        }
        return time.Unix(int64(f), 0).UTC().Format(time.RFC3339), nil
    },
}

se := gnata.NewStreamEvaluator(nil,
    gnata.WithCustomFunctions(customFuncs),
    gnata.WithMaxCachedSchemas(10000),
)

// Expressions can now use $md5() and $parseEpochSeconds()
idx, _ := se.Compile(`$md5(user.email)`)
result, _ := se.EvalOne(ctx, eventJSON, "schema1", idx)
```

### Standalone Expression Evaluation

For one-off evaluations with custom functions, create a custom environment and pass it to `EvalWithCustomFuncs`:

```go
env := gnata.NewCustomEnv(customFuncs)
expr, _ := gnata.Compile(`$md5(payload.email)`)
result, _ := expr.EvalWithCustomFuncs(ctx, data, env)
```

The environment should be created once and reused across evaluations for best performance.

### Argument Copying

Object arguments reach custom functions as `map[string]any`. By default every call receives freshly copied maps, so a function may modify the maps it is given. Arrays are passed through without copying when they hold only scalars, so a function must not modify an array argument (or one nested inside a map) in place.

When many expressions pass the same payload objects to custom functions, the copies add up. `WithReadOnlyCustomFuncArgs` caches the normalized view on each object decoded from the input and shares it between calls:

```go
se := gnata.NewStreamEvaluator(nil,
    gnata.WithCustomFunctions(customFuncs),
    gnata.WithReadOnlyCustomFuncArgs(), // custom functions must not modify their arguments
)
```

Only objects gnata decodes itself from the input of `EvalMany`, `EvalManyWithVars`, `EvalOne` and `EvalMap` are cached; values you decode with `DecodeJSON` and pass to `EvalPreparsed` or as variables never are. Objects returned in results may carry the cache, so don't mutate them and feed them back in while the option is set.

## Metrics & Observability

The `StreamEvaluator` accepts an optional `MetricsHook` (via `WithMetricsHook`) for production telemetry. Implement the interface and wire it in — a nil hook (the default) adds zero overhead.

| Callback | Arguments | What to monitor |
|---|---|---|
| `OnEval` | `exprIndex`, `fastPath`, `duration`, `err` | Fast-path ratio; per-expression latency |
| `OnCacheHit` | `schemaKey` | Cache hit rate — should approach 100% after warm-up |
| `OnCacheMiss` | `schemaKey` | Cache misses trigger plan rebuilds |
| `OnEviction` | — | Cache at capacity, evicting plans; increase `WithMaxCachedSchemas` |

For point-in-time cache stats without a hook, use `se.Stats()` which returns hit/miss/entry/eviction counts.

## Performance

All benchmarks on Apple M4 Pro. gnata is compared against the reference [jsonata-js](https://github.com/jsonata-js/jsonata) implementation running in Node.js, evaluation time only (no RPC/transport overhead). Simple field lookups and comparisons hit a zero-copy GJSON fast path — the JSON document is never fully parsed — and functions on a pure path (e.g. `$exists(a.b)`, `$lowercase(name)`) get a similar fast path via a single `gjson.GetBytes` call.

| Category | gnata latency | Speedup vs JSONata |
|---|---|---|
| Fast Path (GJSON zero-copy) | 41 ns – 142 ns | 570x – 1,500x* |
| Boolean Logic | 127 ns – 573 ns | 43x – 270x |
| Numeric & Arithmetic | 177 ns – 530 ns | 32x – 200x* |
| String Functions | 73 ns – 580 ns | 7x – 360x* |
| Array & Filtering | 195 ns – 573 ns | 24x – 140x |
| Higher-Order Functions & Lambdas | 425 ns – 1.5 µs | 8x – 88x* |
| Joins (`@` operator) | 545 ns – 1.6 µs | up to 19x* |
| Conditionals & Blocks | 170 ns – 7.1 µs | 3x – 68x* |
| Regex | 497 ns – 947 ns | up to 6x* |
| Complex Boolean Expressions | 202 ns – 928 ns | up to 31x* |

\* Some expressions in this category evaluate faster than JSONata's measurement floor (< 1 µs) — actual speedup is understated.

Fast-path expressions typically achieve **0-2 allocations** and **0-40 bytes** per evaluation. Supported fast-path functions (21): `$exists`, `$contains`, `$string`, `$boolean`, `$number`, `$keys`, `$distinct`, `$not`, `$lowercase`, `$uppercase`, `$trim`, `$length`, `$type`, `$abs`, `$floor`, `$ceil`, `$sqrt`, `$count`, `$reverse`, `$sum`, `$max`, `$min`, `$average`.

### StreamEvaluator (batch of 4 expressions)

| Metric | Value |
|---|---|
| ns/op | 20,500 |
| throughput | 29 MB/s |
| allocs/op | 517 |
| cache hit rate | 100% (after warm-up) |

## JSONata Compatibility

gnata targets full compatibility with [JSONata 2.x](https://docs.jsonata.org), validated against **1,704 test cases** from the official [jsonata-js test suite](https://github.com/jsonata-js/jsonata/tree/master/test/test-suite) at **v2.2.2** — **0 failures, 0 skips**.

### Supported Features

- Path navigation, wildcards (`*`), descendants (`**`), parent (`%`)
- Array predicates, numeric indexing, filter expressions
- Sorting (`^`), grouping (`{}`), transforms (`|...|...|`)
- Lambda functions, closures, tail-call optimization
- Partial application, function composition (`~>`)
- Focus (`@`) and index (`#`) variable binding
- Conditional expressions (`? :`)
- Range operator (`..`), string concatenation (`&`)
- Join operator (`@`) with lateral-join semantics

### Standard Library (50+ functions)

| Category | Functions |
|---|---|
| **String** | `$string` `$length` `$substring` `$uppercase` `$lowercase` `$trim` `$pad` `$contains` `$split` `$join` `$match` `$replace` `$eval` `$base64encode` `$base64decode` `$encodeUrl` `$decodeUrl` `$encodeUrlComponent` `$decodeUrlComponent` `$formatNumber` `$formatBase` `$formatInteger` `$parseInteger` |
| **Numeric** | `$number` `$abs` `$floor` `$ceil` `$round` `$power` `$sqrt` `$random` `$sum` `$max` `$min` `$average` |
| **Array** | `$count` `$append` `$sort` `$reverse` `$shuffle` `$distinct` `$flatten` `$zip` |
| **Object** | `$keys` `$values` `$lookup` `$spread` `$merge` `$each` `$sift` `$type` `$error` `$assert` |
| **Boolean** | `$boolean` `$not` `$exists` |
| **Higher-Order** | `$map` `$filter` `$reduce` `$single` |
| **Date/Time** | `$now` `$millis` `$fromMillis` `$toMillis` |

### Guardrails

`Compile` accepts optional resource guardrails, matching jsonata-js 2.2's `stack` / `timeout` / `sequence` options:

```go
expr, err := gnata.Compile(userExpr,
    gnata.WithStack(100),                // error D1011 beyond this recursion depth
    gnata.WithTimeout(500*time.Millisecond), // error D1012 if evaluation runs longer
    gnata.WithSequence(1_000_000),       // error D2015 if a built sequence grows past this
)
```

`WithTimeout` is enforced by checking a deadline on every function call and periodically between expression nodes, rather than with a timer, so it also applies inside a synchronous call on single-threaded WebAssembly hosts. A single builtin or custom function call is not interrupted, so evaluation can exceed the timeout by at most the duration of the call in progress.

Frequently Asked Questions