Ctrl.blog

Legal basis for collecting and storing logs without consent

You can’t collect and store any personal data without having obtained, and being able to document that you obtained, consent from the persons you’re collecting data from. You can, however, still collect and store personal data in your server logs for the limited and legitimate purpose of detecting and preventing fraud and unauthorized system access, and ensuring the security of your systems.

Here are the relevant excerpts from the GDPR that allows data collection for this type of purposes:

Notably, this doesn’t exempt such collection from the strict requirements of the GDPR. Gandalf the Grey offers some great advice for how you should treat personal data to achieve GDPR compliance in your organization:

Encryption, access restriction, and timely erasure

The specific requirements under the GDPR that apply to your organization depend on the scope and type of data you collect set against the needs to store the data. The regulation with all its recitals is 54 800 words long, but I’ll try to summarize some practical implementation requirements from the regulation:

There are no specific technical details offered regarding how these requirements should be implemented besides the suggestion to use “encryption” in Article 6, Paragraph 4, Point E and Recital 83. The take-away is still clear: data should be secured, access should be limited even within your organization, and data should be deleted (including from backups) when there’s no longer a need to retain it.

Utility of the day: logrotate (+ gnupg)

logrotate is a very useful tool that can be used to encrypt logs in storage even on edge servers, and can help automate the deletion of old log files. It can also be used to encrypt log files in storage which when combined with organizational measures can limit access to decrypting the log files.

Unless you encrypt your logs, an unauthorized third-party who gained access to your servers could extract a lot of data about your users from your logs. Depending on how much private information is stored in your log files and the potential sensitive nature of your business, you shouldn’t store log files for more than a few hours or days unless you take measures to protect them.

Managing PGP keys in GnuPG is beyond the scope of this article. In short, you would create a key-pair on a secure machine, and then import the public-key into the GnuPG key chain on your servers while storing the private-key on a secure medium with limited access for authorized employees only, e.g. printed on paper or kept on a removable storage media.

The server can then use the public key to encrypt its log files without with public key cryptography; resulting in the server being able to encrypt the data without being able to decrypt it without the private. The log files could even be transferred to a centralized log-storage server for cold storage.

I believe such a setup could be used to achieve GDPR compliance while still maintaining auditable logs in the event of a breach of server security or other incidents that would require a log trail.

The following logrotate configuration example demonstrates secure encrypted storage (using GnuPG) erasure after time intervals (rotate in days) appropriate to how important it’s to store the various log files following a security incident.

daily
shred
compress
compresscmd /usr/bin/gpg2
compressoptions --encrypt --default-key your-key-id
compressext .gpg

/var/log/httpd/access* {
  rotate 100
}

/var/log/httpd/error* {
  rotate 200
}

/var/log/httpd/mod_security* {
  rotate 400
}

In the above example, access logs are deleted after 100 days, error logs after 200 days, and ModSecurity logs (which would only contain suspicious activity), is retained for 400 days. After this time, the logs would be securely erased using the shred utility.

The logs are still kept unencrypted for up to 24-hours when they were first recorded. This is a small time window when the data isn’t stored encrypted, but it’s required to allow human technicians and automated log analyzing tools (like SSHGuard or Fail2Ban) to process the data and act upon it to help detect and prevent unauthorized or unlawful system access.

You can reduce the time window when data is kept unencrypted by rotating logs hourly instead of daily, or by piping logs directly from your web server into an encrypted storage. This may have a serious performance impact on your server and complicate the configuration of automated security monitoring tools.

Summary

Any identifier, including network or equipment identifiers like an IP address, are considered personal data. Don’t store server logs if you don’t have to. Encrypt logs in storage and limit access to decryption credentials. Delete logs as early as possible, including from any backups. Document what steps you’ve taken to secure data and limit the impact in the case of a server breach.

This article has focused on server logs as they’re something every organization with a website or an online service will have to deal with.

However, the same principles and even stricter requirements apply to other types of data that your organization keeps on people. The deadline for GDPR compliance is May 25th 2018, and that’s barely enough time to read through the 54 800 word regulation.

With fines up in the 20 Million Euro range; you better get started auditing personal information collection and storage in your organization right away. This is the perfect time to rethink old decisions regarding what data your organization needs to keep and for how long.