metricsd

About metricsd

This large README is shown as a plain-text preview. Read the full README at the source

# Metrics Collector Service (metricsd)

[![Go Version](https://img.shields.io/badge/go-1.27+-blue.svg)](https://golang.org/dl/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Go Report Card](https://goreportcard.com/badge/github.com/0x524A/metricsd)](https://goreportcard.com/report/github.com/0x524A/metricsd)
[![GitHub Release](https://img.shields.io/github/v/release/0x524A/metricsd)](https://github.com/0x524A/metricsd/releases)
[![Docker Image Size](https://img.shields.io/docker/image-size/0x524a/metricsd/latest)](https://hub.docker.com/r/0x524a/metricsd)
[![GitHub Issues](https://img.shields.io/github/issues/0x524A/metricsd)](https://github.com/0x524A/metricsd/issues)
[![GitHub Stars](https://img.shields.io/github/stars/0x524A/metricsd?style=social)](https://github.com/0x524A/metricsd/stargazers)

A production-ready, high-performance metrics collector service written in Go that collects system and application metrics and ships them to remote endpoints with enterprise-grade security.

> **🚀 Features**: System metrics (CPU, Memory, Disk, Network) • GPU monitoring (NVIDIA) • Application endpoint scraping • TLS/mTLS support • Prometheus & HTTP JSON shipping • Docker & Kubernetes ready

## Table of Contents

- [Features](#features)
- [Quick Start](#quick-start)
- [Architecture](#architecture)
- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
- [Shipper Types](#shipper-types)
- [TLS Configuration](#tls-configuration)
- [Collected Metrics](#collected-metrics)
- [Security Considerations](#security-considerations)
- [Deployment](#deployment)
- [Performance Tuning](#performance-tuning)
- [Development](#development)
- [Troubleshooting](#troubleshooting)
- [FAQ](#faq)
- [Contributing](#contributing)
- [License](#license)

## Quick Start

Get metricsd up and running in 5 minutes:

```bash
# Clone and build
git clone https://github.com/0x524A/metricsd.git
cd metricsd
go build -o bin/metricsd cmd/metricsd/main.go

# Create configuration
cp config.example.json config.json

# Edit config.json to set your endpoint
# For example, change endpoint to your Prometheus or metrics collector URL

# Run the service
./bin/metricsd -config config.json

# Check health
curl http://localhost:8080/health
```

**With TLS:**
```bash
# Generate self-signed certificates (for testing)
mkdir -p certs && cd certs
openssl req -x509 -newkey rsa:4096 -keyout client.key -out client.crt -days 365 -nodes \
  -subj "/CN=metricsd-client"
cd ..

# Update config.json to enable TLS
# Set shipper.tls.enabled to true
# Set certificate paths in shipper.tls section

# Run with TLS
./bin/metricsd -config config.json
```

**With Docker:**
```bash
docker build -t metricsd:latest .
docker run -d -p 8080:8080 -v $(pwd)/config.json:/etc/metricsd/config.json:ro metricsd:latest
```

## Features

- **Comprehensive Metrics Collection**
  - CPU usage (per-core and total utilization)
  - Memory usage (RAM and swap statistics)
  - Disk I/O and usage statistics
  - Network I/O statistics
  - GPU metrics via NVIDIA NVML (optional)
  - Custom application endpoint scraping

- **Application Metrics Collection**
  - HTTP endpoint scraping for application metrics
  - Support for multiple application endpoints
  - JSON-based metrics format
  - Configurable timeout and retry logic

- **Flexible Shipping Options**
  - Prometheus Remote Write protocol with Snappy compression
  - HTTP JSON POST
  - Advanced TLS/SSL support for secure transmission
  - Configurable request timeouts

- **Enterprise-Grade Security**
  - Full TLS 1.2/1.3 support with custom configuration
  - Client certificate authentication (mTLS)
  - Custom CA certificate support
  - Configurable cipher suites
  - SNI (Server Name Indication) support
  - TLS version pinning (min/max)
  - Session ticket management
  - Optional certificate verification bypass for testing

- **Configurable & Extensible**
  - JSON configuration with environment variable overrides
  - Adjustable collection intervals
  - Enable/disable specific metric collectors
  - Health endpoint for monitoring
  - Flexible shipper interface for custom backends

- **Plugin System**
  - Shell script plugins with JSON output
  - Automatic plugin discovery from directory
  - Per-plugin timeout and interval scheduling
  - Circuit breaker for failing plugins
  - Compile-time Go plugin extension point
  - Security: path validation, sandboxed execution environment

- **Splunk Integration**
  - Splunk HEC (HTTP Event Collector) shipper
  - JSON file shipper for Splunk Universal Forwarder
  - Single-metric and multi-metric JSON formats

- **Debian Packaging**
  - `.deb` packages for amd64 and arm64
  - systemd service with security hardening
  - Automatic user/group creation

- **Production-Ready**
  - Structured logging with zerolog
  - Graceful shutdown with cleanup
  - Error handling and resilience
  - SOLID design principles
  - Resource cleanup and leak prevention

## Architecture

The service follows SOLID principles with a clean architecture:

```
metricsd/
├── cmd/metricsd/           # Application entry point
├── internal/
│   ├── collector/          # Collector interface, registry, system/GPU/HTTP collectors
│   ├── plugin/             # Plugin manager, exec plugin, discovery, security, Go registry
│   ├── config/             # Configuration management
│   ├── shipper/            # Prometheus, HTTP JSON, Splunk HEC, file shippers
│   ├── orchestrator/       # Collection orchestration (parallel, retry)
│   └── server/             # HTTP health endpoint
├── plugins/                # Shell script plugins + sidecar configs
├── packaging/debian/       # Debian package scripts + systemd service
└── docs/                   # Plugin authoring guide, design specs
```

## Installation

### Prerequisites

- Go 1.27 or later
- NVIDIA drivers and CUDA (optional, for GPU metrics)

### Build from Source

```bash
# Clone the repository
git clone https://github.com/0x524A/metricsd.git
cd metricsd

# Download dependencies
go mod download

# Build the binary
go build -o bin/metricsd cmd/metricsd/main.go
```

## Configuration

Create a `config.json` file based on the example:

```bash
cp config.example.json config.json
```

### Configuration Options

```json
{
  "server": {
    "host": "0.0.0.0",
    "port": 8080
  },
  "collector": {
    "interval_seconds": 60,
    "enable_cpu": true,
    "enable_memory": true,
    "enable_disk": true,
    "enable_network": true,
    "enable_gpu": false,
    "plugins": {
      "enabled": false,
      "plugins_dir": "./plugins",
      "default_timeout_seconds": 30,
      "validate_on_startup": true
    }
  },
  "shipper": {
    "type": "http_json",
    "endpoint": "https://collector.example.com:9090/api/v1/metrics",
    "timeout": 30000000000,
    "tls": {
      "enabled": true,
      "cert_file": "/path/to/client-cert.pem",
      "key_file": "/path/to/client-key.pem",
      "ca_file": "/path/to/ca.pem",
      "insecure_skip_verify": false,
      "server_name": "collector.example.com",
      "min_version": "TLS1.2",
      "max_version": "TLS1.3",
      "cipher_suites": [
        "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
        "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
      ],
      "session_tickets": true
    }
  },
  "endpoints": [
    {
      "name": "app1",
      "url": "http://localhost:3000/metrics"
    }
  ]
}
```

### Configuration Fields

| Field | Description | Default |
|-------|-------------|---------|
| `server.host` | HTTP server bind address | `0.0.0.0` |
| `server.port` | HTTP server port | `8080` |
| `collector.interval_seconds` | Collection interval in seconds | `60` |
| `collector.enable_cpu` | Enable CPU metrics collection | `true` |
| `collector.enable_memory` | Enable memory metrics collection | `true` |
| `collector.enable_disk` | Enable disk metrics collection | `true` |
| `collector.enable_network` | Enable network metrics collection | `true` |
| `collector.enable_gpu` | Enable GPU metrics collection (requires NVIDIA GPU) | `false` |
| `shipper.type` | Shipper type: `prometheus_remote_write`, `http_json`, or `json_file` | - |
| `shipper.endpoint` | Remote endpoint URL | - |
| `shipper.timeout` | Request timeout in nanoseconds | `30000000000` (30s) |
| `shipper.tls.enabled` | Enable TLS/SSL | `false` |
| `shipper.tls.cert_file` | Path to client certificate file (PEM) | - |
| `shipper.tls.key_file` | Path to client private key file (PEM) | - |
| `shipper.tls.ca_file` | Path to CA certificate file for server verification | - |
| `shipper.tls.insecure_skip_verify` | Skip server certificate verification (not recommended) | `false` |
| `shipper.tls.server_name` | Server name for SNI (overrides hostname from endpoint) | - |
| `shipper.tls.min_version` | Minimum TLS version: `TLS1.0`, `TLS1.1`, `TLS1.2`, `TLS1.3` | `TLS1.2` |
| `shipper.tls.max_version` | Maximum TLS version: `TLS1.0`, `TLS1.1`, `TLS1.2`, `TLS1.3` | `TLS1.3` |
| `shipper.tls.cipher_suites` | Array of allowed cipher suites (see Cipher Suites section) | System defaults |
| `shipper.tls.session_tickets` | Enable TLS session ticket resumption | `true` |
| `endpoints` | Array of application HTTP endpoints to scrape | `[]` |

### Environment Variable Overrides

You can override configuration values using environment variables:

| Environment Variable | Description | Example |
|---------------------|-------------|---------|
| `MC_SERVER_HOST` | Server bind address | `0.0.0.0` |
| `MC_SERVER_PORT` | Server port number | `8080` |
| `MC_COLLECTOR_INTERVAL` | Collection interval in seconds | `60` |
| `MC_SHIPPER_TYPE` | Shipper type | `prometheus_remote_write` |
| `MC_SHIPPER_ENDPOINT` | Shipper endpoint URL | `https://metrics.example.com/write` |
| `MC_TLS_ENABLED` | Enable TLS | `true` |
| `MC_TLS_CERT_FILE` | Client certificate file path | `/etc/metricsd/certs/client.crt` |
| `MC_TLS_KEY_FILE` | Client private key file path | `/etc/metricsd/certs/client.key` |
| `MC_TLS_CA_FILE` | CA certificate file path | `/etc/metricsd/certs/ca.crt` |
| `MC_TLS_SERVER_NAME` | SNI server name | `collector.example.com` |
| `MC_TLS_MIN_VERSION` | Minimum TLS version | `TLS1.2` |
| `MC_TLS_INSECURE_SKIP_VERIFY` | Skip certificate verification | `false` |
| `MC_FILE_PATH` | File shipper output path | `/var/log/metricsd/metrics.json` |
| `MC_FILE_MAX_SIZE_MB` | Maximum file size before rotation (MB) | `100` |
| `MC_FILE_MAX_FILES` | Number of rotated files to keep | `5` |

## Plugin System

metricsd supports shell script plugins that output JSON metrics. Plugins are automatically discovered from the configured plugins directory.

### Writing Plugins

See [Plugin Authoring Guide](docs/plugin-authoring.md) for full documentation.

Plugins are executable scripts that output a JSON array:

```bash
#!/bin/bash
echo '[{"name": "my_metric", "value": 42.5, "type": "gauge", "labels": {"env": "prod"}}]'
```

### Configuration

Each plugin can have a sidecar `.json` config file:

```json
{
  "name": "my_plugin",
  "timeout": 30,
  "enabled": true,
  "interval_seconds": 60
}
```

### Go Plugin Extension

For compile-time Go plugins, implement the `collector.Collector` interface and register via `plugin.RegisterGoPlugin()`. See the design spec for details.

## Usage

### Basic Usage

```bash
# Run with default config.json
./bin/metricsd

# Run with custom config file
./bin/metricsd -config /path/to/config.json

# Set log level
./bin/metricsd -log-level debug
```

### Log Levels

- `debug` - Detailed debugging information
- `info` - General informational messages (default)
- `warn` - Warning messages
- `error` - Error messages only

### Health Check

The service exposes a health endpoint:

```bash
curl http://localhost:8080/health
```

Response:
```json
{
  "status": "healthy",
  "timestamp": "2025-11-05T12:34:56Z",
  "uptime": "1h23m45s"
}
```

## Shipper Types

### Prometheus Remote Write

Ships metrics using the Prometheus remote write protocol with Snappy compression.

```json
{
  "shipper": {
    "type": "prometheus_remote_write",
    "endpoint": "http://prometheus:9090/api/v1/write"
  }
}
```

### HTTP JSON

Ships metrics as JSON via HTTP POST.

```json
{
  "shipper": {
    "type": "http_json",
    "endpoint": "http://collector:8080/api/v1/metrics"
  }
}
```

Payload format:
```json
{
  "timestamp": 1699185296,
  "metrics": [
    {
      "name": "system_cpu_usage_percent",
      "value": 45.2,
      "type": "gauge",
      "labels": {
        "core": "0"
      }
    }
  ]
}
```

### JSON File (File Shipper)

Ships metrics as JSON to a local file with automatic rotation. Ideal for Splunk Universal Forwarder integration or local storage.

```json
{
  "shipper": {
    "type": "json_file",
    "file": {
      "path": "/var/log/metricsd/metrics.json",
      "max_size_mb": 100,
      "max_files": 5
    }
  }
}
```

**Configuration Options:**
- `path`: Output file path (required)
- `max_size_mb`: Maximum file size before rotation in MB (default: 100)
- `max_files`: Number of rotated files to keep (default: 5)

**Use Cases:**
- Integration with Splunk Universal Forwarder
- Local metric storage and backup
- Offline metric collection
- Log aggregation pipelines

## TLS Configuration

The service supports advanced TLS configuration for secure communication with remote endpoints. This includes mutual TLS (mTLS), custom cipher suites, and version pinning.

### Basic TLS Setup

For simple TLS with server certificate verification:

```json
{
  "shipper": {
    "type": "prometheus_remote_write",
    "endpoint": "https://metrics.example.com/api/v1/write",
    "tls": {
      "enabled": true,
      "ca_file": "/etc/metricsd/certs/ca.pem"
    }
  }
}
```

### Mutual TLS (mTLS)

For client certificate authentication:

```json
{
  "shipper": {
    "type": "http_json",
    "endpoint": "https://secure-collector.example.com/metrics",
    "tls": {
      "enabled": true,
      "cert_file": "/etc/metricsd/certs/client.crt",
      "key_file": "/etc/metricsd/certs/client.key",
      "ca_file": "/etc/metricsd/certs/ca.crt",
      "server_name": "secure-collector.example.com"
    }
  }
}
```

### Advanced TLS Configuration

Full control over TLS parameters:

```json
{
  "shipper": {
    "tls": {
      "enabled": true,
      "cert_file": "/etc/metricsd/certs/client.crt",
      "key_file": "/etc/metricsd/certs/client.key",
      "ca_file": "/etc/metricsd/certs/ca.crt",
      "server_name": "metrics.internal.example.com",
      "min_version": "TLS1.2",
      "max_version": "TLS1.3",
      "cipher_suites": [
        "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
        "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
        "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
        "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
      ],
      "session_tickets": true,
      "insecure_skip_verify": false
    }
  }
}
```

### TLS Configuration Options

| Option | Description | Values |
|--------|-------------|--------|
| `enabled` | Enable/disable TLS | `true`, `false` |
| `cert_file` | Client certificate for mTLS | Path to PEM file |
| `key_file` | Client private key for mTLS | Path to PEM file |
| `ca_file` | CA certificate for server verification | Path to PEM file |
| `server_name` | SNI hostname override | Domain name |
| `min_version` | Minimum TLS version | `TLS1.0`, `TLS1.1`, `TLS1.2`, `TLS1.3` |
| `max_version` | Maximum TLS version | `TLS1.0`, `TLS1.1`, `TLS1.2`, `TLS1.3` |
| `cipher_suites` | Allowed cipher suites | Array of suite names |
| `session_tickets` | Enable session resumption | `true`, `false` |
| `insecure_skip_verify` | Skip certificate verification | `true`, `false` (not recommended for production) |

### Supported Cipher Suites

**TLS 1.3 Cipher Suites:**
- `TLS_AES_128_GCM_SHA256`
- `TLS_AES_256_GCM_SHA384`
- `TLS_CHACHA20_POLY1305_SHA256`

**TLS 1.2 Cipher Suites (Recommended):**
- `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`
- `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`
- `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`
- `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`
- `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256`
- `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256`

**Additional TLS 1.2 Cipher Suites:**
- `TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256`
- `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256`
- `TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA`
- `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA`
- `TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA`
- `TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA`
- `TLS_RSA_WITH_AES_128_GCM_SHA256`
- `TLS_RSA_WITH_AES_256_GCM_SHA384`
- `TLS_RSA_WITH_AES_128_CBC_SHA256`
- `TLS_RSA_WITH_AES_128_CBC_SHA`
- `TLS_RSA_WITH_AES_256_CBC_SHA`

> **Note:** If cipher suites are not specified, Go's default secure cipher suite list will be used. TLS 1.3 cipher suites cannot be configured in Go and use the protocol's default settings.

### TLS Best Practices

1. **Use TLS 1.2 or higher** - Set `min_version` to `TLS1.2` minimum
2. **Enable mTLS** - Use client certificates for mutual authentication
3. **Verify certificates** - Keep `insecure_skip_verify` as `false` in production
4. **Use strong cipher suites** - Prefer ECDHE and AEAD ciphers
5. **Configure SNI** - Set `server_name` when using name-based virtual hosting
6. **Rotate certificates** - Implement a certificate rotation strategy
7. **Secure key storage** - Protect private keys with appropriate file permissions

### Certificate Generation Examples

**Generate self-signed CA:**
```bash
openssl req -x509 -new -nodes -keyout ca.key -sha256 -days 1825 -out ca.crt \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=CA"
```

**Generate client certificate:**
```bash
# Generate private key
openssl genrsa -out client.key 2048

# Generate certificate signing request
openssl req -new -key client.key -out client.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=metricsd-client"

# Sign with CA
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt -days 825 -sha256
```

**Set secure file permissions:**
```bash
chmod 600 /etc/metricsd/certs/*.key
chmod 644 /etc/metricsd/certs/*.crt
chown metricsd:metricsd /etc/metricsd/certs/*
```

### Troubleshooting TLS

**Certificate verification failed:**
- Ensure CA certificate includes the full chain
- Verify `server_name` matches the certificate CN or SAN
- Check certificate expiration dates

**Handshake failure:**
- Verify cipher suites are compatible with server
- Check TLS version compatibility (min/max versions)
- Ensure client certificate is valid and trusted by server

**Enable debug logging:**
```bash
./bin/metricsd -log-level debug
```

## Collected Metrics

### System Metrics

**CPU:**
- `system_cpu_usage_percent` - Per-core CPU usage
- `system_cpu_usage_total_percent` - Overall CPU usage
- `system_cpu_count` - Number of CPU cores

**Memory:**
- `system_memory_total_bytes` - Total memory
- `system_memory_used_bytes` - Used memory
- `system_memory_available_bytes` - Available memory
- `system_memory_usage_percent` - Memory usage percentage
- `system_swap_total_bytes` - Total swap space
- `system_swap_used_bytes` - Used swap space
- `system_swap_usage_percent` - Swap usage percentage

**Disk:**
- `system_disk_total_bytes` - Total disk space
- `system_disk_used_bytes` - Used disk space
- `system_disk_free_bytes` - Free disk space
- `system_disk_usage_percent` - Disk usage percentage
- `system_disk_read_bytes_total` - Total bytes read
- `system_disk_write_bytes_total` - Total bytes written
- `system_disk_read_count_total` - Total read operations
- `system_disk_write_count_total` - Total write operations

**Network:**
- `system_network_bytes_sent_total` - Total bytes sent
- `system_network_bytes_recv_total` - Total bytes received
- `system_network_packets_sent_total` - Total packets sent
- `system_network_packets_recv_total` - Total packets received
- `system_network_errors_in_total` - Total input errors
- `system_network_errors_out_total` - Total output errors
- `system_network_drop_in_total` - Total input drops
- `system_network_drop_out_total` - Total output drops

**GPU (NVIDIA):**

Frequently Asked Questions