Using InfluxDB, Telegraf, Grafana & Tor .Onion’s To Build a Surveillance Resistant Distributed Metrics Infrastructure
At Ablative Hosting we prefer to use .onion networking everywhere we can. .onion’s ensure that our web properties can never be blocked, our employees can trust the end-to-end nature of the connectivity, our endpoints aren’t exposed in Certificate Transparency Logs. Most importantly; Tor prevents adversaries from tapping the upstream of our metrics or log servers and trivially discovering the IP addresses of all our other servers (or employees) when they connect.
This blog post will take you through the creation of a distributed metrics system that routes all traffic via Tor .onions.
We will be working with 2 nodes, A and B. Server A is the hub where all metrics are sent and Server B is an example of one of what could be many servers.
Whilst we normally host all our servers on OpenBSD this guide will focus on Fedora Server (32) because the external linked documentation maps well to all other Linux distros.
At a high level the design we’re aiming for is something akin to this;
+------------+ +-----------+ +--------------+
| Tor SOCKS5 | ------- Tor ------- | .onion | -----Tor ---- | +--------------+
| ^ | | # | +-| +--------------+
| Privoxy | | InfluxDB | +-| Microservice |
| ^ | | # | +--------------+
| Telegraf | | Grafana |
+------------+ | # | +-------------+
| .onion | ------ Tor ------ | Tor Browser |
+-----------+ +-------------+
[ Server B ] [ Server A ] [Ablative Staff]
Tor
On both servers A and B install Tor with sudo dnf install tor or download the latest tarball from the TorProject.org website.
We will configure the torrc on each server at a later stage.
Server A | Grafana
Grafana is described as ‘The open observability platform’ and allows you to create beautiful dashboards from a variety of data sources. We gather hundreds of metrics from each of our servers, routers, switches and microservices so we can monitor the health of the service we provide to you.
Grafana is hosted on Server A. To get started we’ll add the Grafana YUM Repository to
the system by copying the following text to /etc/yum.repos.d/grafana.repo;
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
Issue sudo dnf install grafana grafana-influxdb to install Grafana itself and a plugin to talk to InfluxDB.
Once installed you need to edit the file at /etc/grafana/grafana.ini to make at least the following changes;
[server]
protocol = socket
enforce_domain = false
# This can be any directory but ensure you configure
# the permissions correctly
socket = /var/lib/grafana/socket/grafana.socket
Please ensure you read the Configuration Guide and make other changes as appropriate to your requirements (e.g. disabling self signup if neccessary etc), the changes above are the bare minimum to get through this tutorial.
OPSEC Note: By setting Grafana to listen on a UNIX socket we prevent the accidental exposure of the web interface to the clearnet, such a mistake
has caught many a .onion operator out.
Start Grafana.
Server A | InfluxDB
InfluxDB is an open source time series database, perfect for storing metrics.
The Download Portal provides a variety of download options, we’re going to grab the RPM;
# Grab the RPM
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.8.0.x86_64.rpm
# Check the SHA256 hash
# !!! DON'T TRUST US - PLEASE GOTO influxdata.com AND CHECK FOR YOURSELF !!!
echo "bab2228b37d6034b3a325240c7b5b51cbfd86eb8e4cf4622e2fc33e1f348cbf4" \
&& sha256sum influxdb-1.8.0.x86_64.rpm
# Install the RPM
sudo yum localinstall influxdb-1.8.0.x86_64.rpm
Once installed edit /etc/influxdb/influxdb.conf to make at least the following changes;
[http]
unix-socket-enabled = true
bind-socket = "/var/run/influxdb.sock"
Start InfluxDB.
Server A | Telegraf
Telegraf describes itself as the open source server agent to help you collect metrics from your stacks, sensors and systems.
We want Telegraf both for its native metric collection features and its ability to act as an aggregator that can relay data back to InfluxDB over Tor (although we’ll only use that on Servers B through n).
As with InfluxDB Telegraf can be acquired from the Influx Download Portal;
# Grab the Telegraf RPM
wget https://dl.influxdata.com/telegraf/releases/telegraf-1.14.2-1.x86_64.rpm
# Check the SHA256 hash
# !!! DON'T TRUST US - PLEASE GOTO influxdata.com AND CHECK FOR YOURSELF !!!
echo "7ccf7f558b4d50650908ed8866765208747d7fa8b56cb3c87d6b97b7e8da9512" \
&& sha256sum telegraf-1.14.2-1.x86_64.rpm
# Install telegraf
sudo yum localinstall telegraf-1.14.2-1.x86_64.rpm
Once installed edit the configuration file at /etc/telegraf/telegraf.conf with at least the following changes;
[outputs.influxdb]]
urls = ["unix:///var/run/influxdb.sock"]
database = "telegraf"
[[inputs.http_listener_v2]]
service_address = ":8080"
path = "/telegraf"
methods = ["POST", "PUT"]
data_format = "influx"
This configuration allows us to write data to InfluxDB, this will be from Telegraf’s native collectors (e.g. CPU, Memeory on the server it is intalled on) as well as enabling our Microservices to send metrics using the HTTP Listener v2 format.
OPSEC NOTE: Notice that the inputs.http_listener_v2 does not have the option to listen on a unix socket. Depending on your threat model you may want to avoid using the HTTP v2 listener and talk directly to InfluxDB using the line protocol.
Start Telegraf.
Server A | Tor
Edit /etc/tor/torrc to add a HiddenService block that will publish Grafana, the Telegraf HTTP v2 Listener and InfluxDB;
HiddenServiceDir /var/lib/tor/grafana/
# Grafana
HiddenServicePort 80 unix:/var/lib/grafana/socket/grafana.socket
# Telegraf http v2
HiddenServicePort 8080 127.0.0.1:8080
# Influx forwarder
HiddenServicePort 8186 unix:/var/run/influxdb.sock
(Re)start Tor and grab the hostname from /var/lib/tor/grafana/hostname.
OPSEC NOTE: Hopefully you have looked at the various authentication options that Influx, Telegraf and Grafana offer and considered one of those on top of the ‘Security through Obscurity’ that a v3 .onion would provide. Configuring the .onion to use Client Authorization is probably a worthwhile defense in depth approach.
With all your services restarted you can choose to add your InfluxDB datasource to Grafana now or later, regardless you are ready to start sending metrics from other services via Tor .onions.
Server B (and n other nodes)
Server B | Telegraf (with privoxy)
As described in the Server A Telegraf section We want Telegraf for its native features but its ability to act as an aggregator that can relay data back to InfluxDB over Tor is what we really need here.
As with with Server A InfluxDB Telegraf can be acquired from the Influx Download Portal, but this time we’ll also be installing Privoxy;
# Grab the Telegraf RPM
wget https://dl.influxdata.com/telegraf/releases/telegraf-1.14.2-1.x86_64.rpm
# Check the SHA256 hash
# !!! DON'T TRUST US - PLEASE GOTO influxdata.com AND CHECK FOR YOURSELF !!!
echo "7ccf7f558b4d50650908ed8866765208747d7fa8b56cb3c87d6b97b7e8da9512" \
&& sha256sum telegraf-1.14.2-1.x86_64.rpm
# Install telegraf
sudo yum localinstall telegraf-1.14.2-1.x86_64.rpm
# Install Privoxy
sudo yum install privoxy
Once installed, edit the configuration file at /etc/telegraf/telegraf.conf with at least the following changes;
[outputs.influxdb]]
urls = ["http://YOURv3ONIONADDRESS.onion:8186"]
database = "telegraf"
http_proxy = "http://localhost:8118"
The sharp eyed will have noticed that tcp/8118 is not the Tor daemon SOCKS5 proxy, it is the port that Privoxy will be presenting a HTTP proxy on.
To configure Privoxy to forward HTTP proxy requests to the Tor daemon’s SOCKS5 proxy edit the config file at /etc/privoxy/config, find the line with forward-socks5t / 127.0.0.1:9050 . and uncomment it.
(Re)start Tor (Tor will listen on socks5://localhost/9050 by default), Privoxy and Telegraf.
Telegraf’s default config includes inputs for gathering metrics about CPU, memory disks and other related aspects of your server, within a few seconds those metrics will be sent to InfluxDB via the .onion address.
You should be able to navigate to your .onion and start drawing beautiful dashboards.