Ablative Hosting needs to monitor our servers and microservices for any strange or malicious behaviour, in certain cases we also need to be able to send logs across the Internet without exposing a link between the servers in question and our attributed infrastructure.

To prevent either side of a log transmission event from knowing too much about each other and provide end-to-end transport encryption as well we can use Tor.

Splunk is our SIEM of choice and is available under a Free or a paid “Enterprise” license.

The free license is limited to 500Mb/day of data ingestion and, more importantly, disables any form of authentication.

As a matter of course Ablative only logs a tiny amount of data;

  • Microservice activity
    • service restarts
    • errors, warnings
    • state changes
  • Syslog events from our servers (the servers that run the website etc)
    • auth.*
    • secure.*
  • Syslog events from shared single hop servers
    • auth.*
    • secure.*
  • Ansible callbacks

We do not log;

  • Anything in/from Single Hop or MultiHop servers
    • Syslog is disabled
    • httpd logging is disabled
    • outbound firewalling prevents data egress

As such the 500Mb/day limit is not a problem for us.

The lack of authentication however would be unacceptable. Thankfully Tor helps us with this too!

Splunk HTTP Collection

Configuring Splunk for HTTP Event Collection is a trivial task. Once the splunkd daemon is listening on 127.0.0.1:8088 we can create any Event Collector Tokens we need and then point a .onion at 127.0.0.1:8088.

This still leaves the problem that the webui listening on tcp/80 has no authentication. We could firewall the server, use HTTP basic auth or a great many other options but they’d all require HTTPS and with certificate transparency being a very useful form of OSINT / target reconnaissance it’ll be far safer for us to use an authenticated .onion.

Authenticated v3 Onion

Kushal Das has an excellent blog post on this topic.

Once the SOC team have committed the public portion of their auth keys to our ‘Infrastructure as Code’ git respository they need to update their sys-whonix qube to pass the cookie when they try to reach Splunk.

Configuring sys-whonix

For whonix 15 on Qubes the sys-whonix menu option should have a menu path as follows

  • Q
    • Service: sys-whonix
      • sys-whonix: Tor User Config

Clicking this will launch a text editor with the following content;

# Tor user specific configuration file
#
# Add user modifications below this line:
############################################

They then add ClientOnionAuthDir /var/lib/tor/authdir/ under the line and save the file.

The auth_private file would be stored in /var/lib/tor/authdir/ as discussed in Kushal Das' blog post and sys-whonix would need restarting.

Up and Running

Now Ablative Hosting staff (and only Ablative Staff) can access our SIEM over Tor to monitor what is happening with the infrastructure and respond appropriately.

Coupled with our internal version of OnionWatch we can see any abnormal event that might affect customers and all without having to ship any data in clear text and all via Tor .onions!