Advantages
- Guarantee that an audit record is created for all actions performed on your servers.
- Configure servers to block and report actions that weren’t already logged.
- Any malicious actions are permanently logged, increasing the likelihood of discovery.
- You can integrate additional code to monitor the log for potentially malicious actions.
- It simplifies incident response and forensic analysis as you’ve got a complete log you can rely on.
- You can prove to auditors exactly what actions were performed on each server.
How it works
The user wants to run a command that modifies the server:
- The user’s client submits the command to the verifiable log
- The user’s computer sends the command to the agent running on the server
- The agent server verifies that the command has been added to the log, and simultaneously verifies the integrity of the log
- The agent runs the command on the server. In order for a command to run on a server, it must first appear in the log. This ensures a user cannot run a malicious command without that command being logged.
Each agent keeps a record of previously seen tree heads. If an agent finds an inconsistency with a previous tree head - implying the log has been tampered with - it stops working and raises alarms. Therefore all running agents act as a distributed network, continuously verifying the append-only nature of the log.
The verifiability of the log makes it extremely difficult to tamper with the log to cover up the malicious command.
Limitations
Users can still run malicious commands - the log doesn’t know whether a command is malicious or not, it simply stores everything. However:
- It’s now possible to write verifier software to continuously monitor the log for commands which look suspicious.
- Users may be dissuaded from running malicious commands since they’ll appear in the log and be very difficult to delete.
The verifiable log can be tampered with. However:
- The agents (and any other verifier code) can detect when that has happened, and decline to run the command.
- To undetectably tamper with the log’s history, an attacker would have to simultaneously tamper with the log and all verifiers.
How to implement this
You need to create four software components:
- Trillian personality - to provide the external interface (API) for logging commands and querying the log with cryptographic proof. Together, Trillian and the personality make up the verifiable log in the diagram above.
- Submitting component - for submitting commands to the log before they are sent to the agent.
- Agent component - for the agent to verify that received commands are in the log.
- Verifiers - to regularly check the integrity of the verifiable log, and potentially inspect log entries for malicious commands.
Consider integrating the submitting and agent components as Git hooks in an existing workflow.
Other possible kinds of actions
This use case described running commands on servers, but the same principle can be applied to guard any kind of change to a system. Other possible actions include:
- API calls to Kubernetes, for example. A service sits in front of the API and clients communicate with that instead of calling the API directly. The service verifies that requests from clients exist in the log before calling the API. Alternatively, the service enters requests into the log itself.
- Configuration diffs for Puppet or other infrastructure-as-code solutions. An agent verifies the git commit exists in the log before applying the changes.
How to further strengthen logging
- The verifiable log has a signed tree head which can be used to verify the entire data structure. The signed tree head could be regularly published online, encouraging others to store a copy of all the signed tree heads. These can be used to subsequently carry out consistency proofs: confirming that the log was append-only since the tree head was stored.
- You could share the verifiable log’s entire contents privately with a third party verifier who could monitor for suspicious commands and continuously verify the integrity of the log.
- You could add multi-party authentication for high risk commands. The agent component would ensure that that log received separate additional approvals before a high-risk command would be executed.
Examples
A payment service provider uses Trillian to log all configuration changes to hosts in their production environment. When a configuration change is received, an agent running on the host ensures that the request has been logged in their verifiable log before applying the configuration. The agents store a local copy of signed tree heads so they can cryptographically verify the integrity of the log each time they query it.
Further resources
- Puppet, Chef, Ansible and SaltStack are well-known applications for automating provisioning of servers (infrastructure as code)
- Git hooks can be used to enforce rules before code changes are committed to version control.
- Change management auditing (Wikipedia) - describes the broader practice of monitoring and auditing changes to systems. Spiceworks Help Desk is an example of traditional change-control software.
