The case against .env files for configuration

Overview

Preface

The twelve factor app is a very well-known manifest that proposes specific guidelines when designing SaaS applications, written by engineers with extensive experience in this field. While it is formulated as a basis for discussion (its actually there in the frontpage), it is often taken as dogma, and many inexperienced developers tend to follow it blindly.

The manifesto itself is quite insightful and covers some of the common "best-practice" rules to build, package and deploy SaaS applications in Docker environments - which by itself is a bit of a letdown, as this small detail is not mentioned explicitly. Maybe software-as-a-service is synonym with Docker or Heroku nowadays, maybe I'm just old.

Regardless, there is a topic that proposes a solution that has much more problems than the ones it tries to solve - Configuration. So, we are going to dissect why their environment variable proposition is actually bad, and what we can do about it (besides always having critical thinking on proposed best practices).

Why not, pt. I: Security

Storing sensitive information (such as credentials or certificates) in environment variables is something any seasoned Unix professional will frown upon. The problem is, environment variables are passed down to spawned processes - so if your application forks to execute some other binary, the other process (and their eventual children) will have access to your environment variables.

So, are you invoking a cli such as wkhtml2pdf to generate your database reports? Well, wkhtml2pdf receives a copy of your secrets. Neat, huh?

If your code doesn't spawn foreign child processes, it is still a bad idea. Every dependency you use has unfettered access to environment variables due to the fact they're often accessed via base system libraries. You're just a malicious commit away (in every single external dependency you may have) of leaking those secret credentials. Any other configuration method will require the malicious code to identify specific application context to retrieve this, instead of just invoking some os-related library.

This may also affect CI/CD systems, as it has become somewhat common to use this mechanism to inject credentials of third party services, such as object storage systems, automation APIs or mailing services, that can be abused if leaked.

Why not, pt. II: Portability

Contrary to what is stated in the manifesto, this is not os-agnostic. At all. In fact, even within Unix-compatible systems such as Linux, you have different ways of defining environment variables, depending on the shell. And in Windows, where you actually have a system-wide configuration database to store settings (called Registry), the process of defining environment variables is also different:

Defining an environment variable in bash:

1export VARIABLE=value 

Defining an environment variable in csh:

1setenv VARIABLE value 

Defining an environment variable in Windows (CMD.EXE):

1SET VARIABLE=value

Defining an environment variable in Windows (PowerShell):

1$env:VARIABLE = "VALUE"

Why not, pt. III: Security (again)

One practical issue the environment variable approach tries to solve is to avoid committing credentials and other sensitive information to a repository, by using code files or configuration files. While this is an actual valid issue, using environment variables is possibly the worst way to try to prevent this - in fact, instead of risking comitting configuration files by mistake, you will be commiting shell files and Windows batch files by mistake. In the end, it doesn't prevent anything, just makes it a bit more difficult to understand what is shell script boilerplate and what is sensitive configuration.

Why not, pt. IV: Structure

Environment variables are a nice way to pass non-structured information to processes. However, its key-value nature is not flexible, specially if settings are nested and cascaded configurations are desirable. Scoping may be also a problem - too generic naming, and you may be colliding with other piece of software you will be executing as a child process; too specific naming, and suddenly your configuration keys look a bit like full sentences yelled by a speaking-challenged robot.

There are also plenty of scenarios where an application may require dozens or even hundreds of configuration parameters. While simple frontend applications can easily work with a couple of configuration parameters, this is not always the case; as an example, plenty of backend systems such as Kafka or PostgreSQL require extensive configuration, and don't even use environment variables. The "one true size fits all" speech is not only misleading, it is wrong - even considering the Docker world.

Simple environment-based configuration:

 1APP_DB_HOST="localhost"
 2APP_DB_USER="someUser"
 3APP_DB_PWD="somePassword"
 4APP_DB_SCHEMA="mydb"
 5
 6APP_CACHE_ADAPTER="redis"
 7APP_CACHE_REDIS_HOST="localhost"
 8APP_CACHE_REDIS_USER="someUser"
 9APP_CACHE_REDIS_PWD="somePassword"
10APP_CACHE_REDIS_DB="2"

Simple INI-based configuration:

 1[app.db]
 2host="localhost"
 3user="someUser"
 4pwd="somePassword"
 5schema="mydb"
 6
 7[app.cache]
 8adapter="redis"
 9redis.host="localhost"
10redis.user="someUser"
11redis.pwd="somePassword"
12redis.db=2

Which one is easier to read, parse, validate and transpose to internal runtime structures? Yeah.

Why not, pt. V: Did I mention security?

It is not uncommon for environment variables to be dumped on log files for error detection, troubleshooting, etc. You may be using a fancy cloud-based application monitoring solution that will happily store all those dumped values. You may just be collecting it in a somewhat insecure fashion on a local disk, or even an S3 bucket. Unless your SaaS product is airtight, there is a good chance for environment variables to end up on log files. This is one more thing to check, to test, to possibly fix and to deply.

Why not, pt. VI: Manual verification is harder

The proposed method uses a granular approach that alleviates a non-existing problem - the proliferation of the concept of environment. By ditching environments altogether, it is up to the poor fellow trying to diagnose a problem to understand if those are production credentials, test credentials or some local developer credentials. And by - in practice - removing cascading configuration, people often forget to bundle a template file with all the available variables, leaving these pesky details such as tunables for poor or non-existing documentation, or better yet - code dumpster-diving that transform many debug sessions into an excruciating experience.

Why not, pt. VII: Automation is harder

While this approach is very common in the Docker world, not everyone is running their payloads in containers, and often with good reason; Many elastic workloads rely on complex proxy and virtual machine interactions, often orchestrated by automation mechanisms such as ansible or Chef. By having application-specific prefixing in keys, it may require further customization on configuration templates to ensure correct generation of .env files, while at the same time making everything ambiguous: is it APP_DB_HOST or DB_APP_HOST?

What is the RIGHT way?

There is no one true right way of doing things, just wrong ways of doing things. If a certain approach is appropriate to a given project, and its not blatantly wrong, then it is probably valid - even if, in that specific case - the suitable approach relies on environment variables.

Regardless, there are a couple of tips that may help avoid common pitfalls:

Prevent developers from comitting configuration files

There are several ways of preventing developers from comitting their own configuration files. When using Git, the most obvious one is to use a .gitignore strategy such as shown below.

Using a global .gitignore file to ignore everything inside the config/ folder, except for the template file:

1$ mkdir config
2$ touch config/config.template
3$ git add config/config.template
4$ echo "config/" >> .gitignore
5$ git add .gitignore

Other approaches within the GIT universe include using pre-commit hooks and tools such as git-secrets or detect-secrets to scan commits.

Static Code Analysis tools used in CI/CD context often provide plugins to detect credentials in code, such as SonarQube.

Use hard boundaries for environment configuration

Environments can be created within a config/ folder, either as separate files or separate child folders, bearing the environment name. The environment choice can - by itself - be provided either via command line argument or via environment variable. The application will then load configuration from the appropriate configuration file or directory.

Use Docker secrets if possible

Docker provides a mechanism to feed sensitive information to applications in containers, when using Swarm mode. Check Docker secrets documentation for more details.

If credentials are required during the building of images, Docker BuildKit has several features that allow the secure passing of credentials to build stages.

Use a secret store

There is a plethora of secret store products available, such as HashiCorp Vault, AWS Secrets Manager, Kubernetes Secrets as well as many others.

Use a text-based configuration format suitable for the product

There are too many popular configuration formats, ranging from xml to using source code files. Just like environment variables sometimes make sense, using source code files can also be a valid choice for some projects, specially when the configuration is parsed at every request and non-native formats cause a non-trivial delay, such as in typical PHP applications. When using source code configuration files, use a separate file, with names dependant on the environment, and make sure they only contain the configuration dictionary. There is no "one size fits all" solution.