dnsmonster
Passive DNS Capture/Monitoring Framework
About dnsmonster
This large README is shown as a plain-text preview. Read the full README at the source
> **We're exploring a managed SaaS solution for dnsmonster!** Help shape the future of passive DNS monitoring by sharing your feedback and requirements: [Take our quick survey](https://tally.so/r/2EAxBe)
<img src="static/dnsmonster-logo.png" width=60%>
Table of Contents
- [Dnsmonster](#dnsmonster)
- [Main features](#main-features)
- [Installation](#installation)
- [Linux](#linux)
- [Container](#container)
- [Build manually](#build-manually)
- [Build statically](#build-statically)
- [Windows](#windows)
- [FreeBSD and MacOS](#freebsd-and-macos)
- [Architecture](#architecture)
- [All-in-one Installation using Docker](#all-in-one-installation-using-docker)
- [All-in-one Demo](#all-in-one-demo)
- [Enterprise Deployment](#enterprise-deployment)
- [Configuration](#configuration)
- [Command line options](#command-line-options)
- [Environment variables](#environment-variables)
- [Configuration file](#configuration-file)
- [What's the retention policy](#whats-the-retention-policy)
- [Sampling and Skipping](#sampling-and-skipping)
- [pre-process sampling](#pre-process-sampling)
- [skip domains](#skip-domains)
- [allow domains](#allow-domains)
- [SAMPLE in clickhouse SELECT queries](#sample-in-clickhouse-select-queries)
- [Supported Inputs](#supported-inputs)
- [Supported Outputs](#supported-outputs)
- [Roadmap](#roadmap)
- [Related projects](#related-projects)
# Dnsmonster
Passive DNS monitoring framework built on Golang.
`dnsmonster` implements a packet sniffer for DNS traffic. It can accept traffic from a `pcap` file, a live interface or a `dnstap` socket,
and can be used to index and store hundreds of thousands of DNS queries per second as it has shown to be capable of indexing 200k+ DNS queries per second on a commodity computer. It aims to be scalable, simple, and easy to use, and help
security teams to understand the details about an enterprise's DNS traffic. `dnsmonster` doesn't look to follow DNS conversations, rather it aims to index DNS packets as soon as they come in. It also doesn't aim to breach
the privacy of the end-users, with the ability to mask Layer 3 IPs (IPv4 and IPv6), enabling teams to perform trend analysis on aggregated data without being able to trace back the queries to an individual. [Blogpost](https://blog.n0p.me/dnsmonster/)
The code before version 1.x is considered beta quality and is subject to breaking changes. Please visit the release notes for each tag to see the list of breaking scenarios between each release, and how to mitigate potential data loss.
```mermaid
graph TD
subgraph Input
B1["network input"]
B2["pcap file"]
B3["dnstap socket"]
end
subgraph "Process"
C1["Sampling based of ratio"]
C2["Packet Process"]
C3["Dispatcher"]
O11["Output1"]
O12["Domain Skip (optional)"]
O13["Domain Allow (optional)"]
O21["Output2"]
O22["Domain Skip (optional)"]
O23["Domain Allow (optional)"]
O31["Output3"]
O32["Domain Skip (optional)"]
O33["Domain Allow (optional)"]
end
B1 --> Process
B2 --> Process
B3 --> Process
C1 --> C2
C2 --> C3
C3 --> O11
C3 --> O21
C3 --> O31
O11 --> O12 --> O13
O21 --> O22 --> O23
O31 --> O32 --> O33
subgraph Output
Splunk
Syslog
H["ClickHouse"]
Postgres
Kafka
I["JSON File"]
Influx
Elastic
J["stdout"]
Parquet
Sentinel
end
O13 --> H
O23 --> I
O33 --> J
```
# Main features
- Ability to use Linux's `afpacket` and zero-copy packet capture.
- Supports BPF
- Ability to mask IP address to enhance privacy
- Ability to have a pre-processing sampling ratio
- Ability to have a list of "skip" `fqdn`s to avoid writing some domains/suffix/prefix to storage
- Ability to have a list of "allow" domains, used to log access to certain domains
- Hot-reload of skip and allow domain files/urls
- Modular output with configurable logic per output stream.
- Automatic data retention policy using ClickHouse's TTL attribute
- Built-in Grafana dashboard for ClickHouse output.
- Ability to be shipped as a single, statically linked binary
- Ability to be configured using environment variables, command line options or configuration file
- Ability to sample outputs using ClickHouse's SAMPLE capability
- Ability to send metrics using `prometheus` and `statstd`
- High compression ratio thanks to ClickHouse's built-in LZ4 storage
- Supports DNS Over TCP, Fragmented DNS (udp/tcp) and IPv6
- Supports [dnstap](https://github.com/dnstap/golang-dnstap) over Unix socket or TCP
- built-in SIEM integration with Splunk and Microsoft Sentinel
# Installation
## Linux
Best way to get started with `dnsmonster` is to download the binary from the release section. The binary is statically built against `musl`, hence it should work out of the box for many distros. For `afpacket` support, you must use kernel 3.x+. Any modern Linux distribution (CentOS/RHEL 7+, Ubuntu 14.0.4.2+, Debian 7+) is shipped with a 3.x+ version so it should work out of the box. If your distro isn't working with the pre-compiled version, please submit an issue with the details, and build `dnsmonster` manually using this section [Build Manually](#build-manually).
### Container
Since `dnsmonster` uses raw packet capture funcationality, Docker/Podman daemon must grant the capability to the container
```
sudo docker run --rm -it --net=host --cap-add NET_RAW --cap-add NET_ADMIN --name dnsmonster ghcr.io/mosajjal/dnsmonster:latest --devName lo --stdoutOutputType=1
```
### Build manually
- with `libpcap`:
Make sure you have `go`, `libpcap-devel` and `linux-headers` packages installed. The name of the packages might differ based on your distribution. After this, simply clone the repository and run `go build ./cmd/dnsmonster`
```sh
git clone https://github.com/mosajjal/dnsmonster --depth 1 /tmp/dnsmonster
cd /tmp/dnsmonster
go get
go build -o dnsmonster ./cmd/dnsmonster
```
- without `libpcap`:
`dnsmonster` only uses one function from `libpcap`, and that's converting the `tcpdump`-style filters into BPF bytecode. If you can live with no BPF support, you can build `dnsmonster` without `libpcap`. Note that for any other platform, the packet capture falls back to `libpcap` so it becomes a hard dependency (*BSD, Windows, Darwin)
```sh
git clone https://github.com/mosajjal/dnsmonster --depth 1 /tmp/dnsmonster
cd /tmp/dnsmonster
go get
go build -o dnsmonster -tags nolibpcap ./cmd/dnsmonster
```
The above build also works on ARMv7 (RPi4) and AArch64.
### Build statically
If you have a copy of `libpcap.a`, you can build the statically link it to `dnsmonster` and build it fully statically. In the code below, please change `/root/libpcap-1.9.1/libpcap.a` to the location of your copy.
```
git clone https://github.com/mosajjal/dnsmonster --depth 1 /tmp/dnsmonster
cd /tmp/dnsmonster/
go get
go build --ldflags "-L /root/libpcap-1.9.1/libpcap.a -linkmode external -extldflags \"-I/usr/include/libnl3 -lnl-genl-3 -lnl-3 -static\"" -a -o dnsmonster ./cmd/dnsmonster
```
For more information on how the statically linked binary is created, take a look at [this](Dockerfile) Dockerfile.
## Windows
Bulding on Windows is much the same as Linux. Just make sure that you have `npcap`. Clone the repository (`--history 1` works), and run `go get` and `go build ./cmd/dnsmonster`
As mentioned, Windows release of the binary depends on [npcap](https://nmap.org/npcap/#download) to be installed. After installation, the binary should work out of the box. It's been tested in a Windows 10 environment and it executed without an issue. To find interface names to give `--devName` parameter and start sniffing, you'll need to do the following:
- open cmd.exe as Administrator and run the following: `getmac.exe`, you'll see a table with your interfaces' MAC address and a Transport Name column with something like this: `\Device\Tcpip_{16000000-0000-0000-0000-145C4638064C}`
- run `dnsmonster.exe` in `cmd.exe` like this:
```sh
dnsmonster.exe --devName \Device\NPF_{16000000-0000-0000-0000-145C4638064C}
```
Note that you must change `\Tcpip` from `getmac.exe` to `\NPF` and then pass it to `dnsmonster.exe`.
## FreeBSD and MacOS
Much the same as Linux and Windows, make sure you have `git`, `libpcap` and `go` installed, then follow the same instructions:
```sh
git clone https://github.com/mosajjal/dnsmonster --depth 1 /tmp/dnsmonster
cd /tmp/dnsmonster
go get
go build -o dnsmonster ./cmd/dnsmonster
```
# Architecture
## All-in-one Installation using Docker
In the example diagram, the egress/ingress of the DNS server traffic is captured, after that, an optional layer of packet aggregation is added before hitting the DNSMonster Server. The outbound data going out of DNS Servers is quite useful to perform cache and performance analysis on the DNS fleet. If an aggregator isn't available for you, you can have both TAPs connected directly to DNSMonster and have two DNSMonster Agents looking at the traffic.
running `./autobuild.sh` creates multiple containers:
* multiple instances of `dnsmonster` to look at the traffic on any interface. Interface list will be prompted as part of `autobuild.sh`
* an instance of `clickhouse` to collect `dnsmonster`'s output and saves all the logs/data to a data and logs directory. Both will be prompted as part of `autobuild.sh`
* an instance of `grafana` looking at the `clickhouse` data with pre-built dashboard.
### All-in-one Demo
[](static/aio_demo.svg)
## Enterprise Deployment
# Configuration
DNSMonster can be configured using 3 different methods. Command line options, Environment variables and configuration file. Order of precedence:
- Command line options (Case-insensitive)
- Environment variables (Always upper-case)
- Configuration file (Case-sensitive, lowercase)
- Default values (No configuration)
## Command line options
Note that command line arguments are case-insensitive as of v0.9.5
[//]: <> (start of command line options)
```sh
# [capture]
# Device used to capture
--devname=
# Pcap filename to run
--pcapfile=
# dnstap socket path. Example: unix:///tmp/dnstap.sock, tcp://127.0.0.1:8080
--dnstapsocket=
# Port selected to filter packets
--port=53
# Capture Sampling by a:b. eg sampleRatio of 1:100 will process 1 percent of the incoming packets
--sampleratio=1:1
# Cleans up packet hash table used for deduplication
--dedupcleanupinterval=1m0s
# Set the dnstap socket permission, only applicable when unix:// is used
--dnstappermission=755
# Number of routines used to handle received packets
--packethandlercount=2
# Size of the tcp assembler
--tcpassemblychannelsize=10000
# Size of the tcp result channel
--tcpresultchannelsize=10000
# Number of routines used to handle tcp packets
--tcphandlercount=1
# Size of the channel to send packets to be defragged
--defraggerchannelsize=10000
# Size of the channel where the defragged packets are returned
--defraggerchannelreturnsize=10000
# Size of the packet handler channel
--packetchannelsize=1000
# Afpacket Buffersize in MB
--afpacketbuffersizemb=64
# BPF filter applied to the packet stream. If port is selected, the packets will not be defragged.
--filter=((ip and (ip[9] == 6 or ip[9] == 17)) or (ip6 and (ip6[6] == 17 or ip6[6] == 6 or ip6[6] == 44)))
# Use AFPacket for live captures. Supported on Linux 3.0+ only
--useafpacket
# The PCAP capture does not contain ethernet frames
--noetherframe
# Deduplicate incoming packets, Only supported with --devName and --pcapFile. Experimental
--dedup
# Do not put the interface in promiscuous mode
--nopromiscuous
# [clickhouse_output]
# Address of the clickhouse database to save the results. multiple values can be provided.
--clickhouseaddress=localhost:9000
# Username to connect to the clickhouse database
--clickhouseusername=
# Password to connect to the clickhouse database
--clickhousepassword=
# Database to connect to the clickhouse database
--clickhousedatabase=default
# Table which data will be stored on clickhouse database
--clickhousetable=DNS_LOG
# Interval between sending results to ClickHouse. If non-0, Batch size is ignored and batch delay is used
--clickhousedelay=0s
# Clickhouse connection LZ4 compression level, 0 means no compression
--clickhousecompress=0
# Debug Clickhouse connection
--clickhousedebug
# Use TLS for Clickhouse connection
--clickhousesecure
# Save full packet query and response in JSON format.
--clickhousesavefullquery
# Use DNSTap identity field instead of ServerName for the identity field in ClickHouse
--clickhouseusednstapidentity
# What should be written to clickhouse. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--clickhouseoutputtype=0
# Minimum capacity of the cache array used to send data to clickhouse. Set close to the queries per second received to prevent allocations
--clickhousebatchsize=100000
# Number of Clickhouse output Workers
--clickhouseworkers=1
# Channel Size for each Clickhouse Worker
--clickhouseworkerchannelsize=100000
# [elastic_output]
# What should be written to elastic. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--elasticoutputtype=0
# elastic endpoint address, example: http://127.0.0.1:9200. Used if elasticOutputType is not none
--elasticoutputendpoint=
# elastic index
--elasticoutputindex=default
# Send data to Elastic in batch sizes
--elasticbatchsize=1000
# Interval between sending results to Elastic if Batch size is not filled
--elasticbatchdelay=1s
# [file_output]
# What should be written to file. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--fileoutputtype=0
# Path to output folder. Used if fileoutputType is not none
--fileoutputpath=
# Interval to rotate the file in cron format
--fileoutputrotatecron=0 0 * * *
# Number of files to keep. 0 to disable rotation
--fileoutputrotatecount=4
# Output format for file. options:json, csv, csv_no_header, gotemplate. note that the csv splits the datetime format into multiple fields
--fileoutputformat=json
# Go Template to format the output as needed
--fileoutputgotemplate={{.}}
# [influx_output]
# What should be written to influx. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--influxoutputtype=0
# influx Server address, example: http://localhost:8086. Used if influxOutputType is not none
--influxoutputserver=
# Influx Server Auth Token
--influxoutputtoken=dnsmonster
# Influx Server Bucket
--influxoutputbucket=dnsmonster
# Influx Server Org
--influxoutputorg=dnsmonster
# Minimum capacity of the cache array used to send data to Influx
--influxoutputworkers=8
# Minimum capacity of the cache array used to send data to Influx
--influxbatchsize=1000
# [kafka_output]
# What should be written to kafka. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--kafkaoutputtype=0
# kafka broker address(es), example: 127.0.0.1:9092. Used if kafkaOutputType is not none
--kafkaoutputbroker=
# Kafka topic for logging
--kafkaoutputtopic=dnsmonster
# Minimum capacity of the cache array used to send data to Kafka
--kafkabatchsize=1000
# Output format. options:json, gob.
--kafkaoutputformat=json
# Kafka connection timeout in seconds
--kafkatimeout=3
# Interval between sending results to Kafka if Batch size is not filled
--kafkabatchdelay=1s
# Compress Kafka connection
--kafkacompress
# Compression Type Kafka connection [snappy gzip lz4 zstd]; default(snappy).
--kafkacompressiontype=snappy
# Use TLS for kafka connection
--kafkasecure
# Path of CA certificate that signs Kafka broker certificate
--kafkacacertificatepath=
# Path of TLS certificate to present to broker
--kafkatlscertificatepath=
# Path of TLS certificate key
--kafkatlskeypath=
# [parquet_output]
# What should be written to parquet file. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--parquetoutputtype=0
# Path to output folder. Used if parquetoutputtype is not none
--parquetoutputpath=
# Number of records to write to parquet file before flushing
--parquetflushbatchsize=10000
# Number of workers to write to parquet file
--parquetworkercount=4
# Size of the write buffer in bytes
--parquetwritebuffersize=256000
# [psql_output]
# What should be written to Microsoft Psql. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--psqloutputtype=0
# Psql endpoint used. must be in uri format. example: postgres://username:password@hostname:port/database?sslmode=disable
--psqlendpoint=
# Psql table which data will be stored on database
--psqltable=DNS_LOG
# Number of PSQL workers
--psqlworkers=1
# Psql Batch Size
--psqlbatchsize=1
# Interval between sending results to Psql if Batch size is not filled. Any value larger than zero takes precedence over Batch Size
--psqlbatchdelay=0s
# Timeout for any INSERT operation before we consider them failed
--psqlbatchtimeout=5s
# Save full packet query and response in JSON format.
--psqlsavefullquery
# [sentinel_output]
# What should be written to Microsoft Sentinel. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logic
# 3: Enable Output and apply allowdomains logic
# 4: Enable Output and apply both skip and allow domains logic
--sentineloutputtype=0
# Sentinel Shared Key, either the primary or secondary, can be found in Agents Management page under Log Analytics workspace
--sentineloutputsharedkey=
# Sentinel Customer Id. can be found in Agents Management page under Log Analytics workspace
--sentineloutputcustomerid=
# Sentinel Output LogType
--sentineloutputlogtype=dnsmonster
# Sentinel Output Proxy in URI format
--sentineloutputproxy=
# Sentinel Batch Size
--sentinelbatchsize=100
# Interval between sending results to Sentinel if Batch size is not filled. Any value larger than zero takes precedence over Batch Size
--sentinelbatchdelay=0s
# [splunk_output]
# What should be written to HEC. options:
# 0: Disable Output
# 1: Enable Output without any filters
# 2: Enable Output and apply skipdomains logicFrequently Asked Questions
What is dnsmonster?
dnsmonster is a Networking library for the Go programming language. Passive DNS Capture/Monitoring Framework
How do I install dnsmonster?
Install dnsmonster with the Go module system using `go get mosajjal/dnsmonster`. Check the repository for the current installation instructions.
What category does dnsmonster belong to?
dnsmonster is listed under Networking, specifically Networking.