Terraform

• Terraform is a tool to help you manage infrastructure. It is based on a declarative model in which Terraform users define a config file declaring what they want there infrastructure to look like. Terraform then computes a diff between this “goal state” and the “actual state” to generate a plan to move from the actual state to the goal state. The Terraform user reviews the plan and then applies it. During the application of a plan Terraform will preform a long running workflow to move the infrastructure to the goal state.
• Terraform is based on plugins. These plugins are provided by providers and are called resources. A provider is a company like Amazon that offers several resources. A resource is a product offering provided by a provider. So for example the provider AWS has a plugin that enables Terraform to operate on EC2 resources. Terraform manages a repository of these plugins and as long as the resource you want to mange can satisfy the simple API required of all resources you can implement a resource.
• When using Terraform you declare resources in configuration blocks. Everything in Terraform is based on config blocks. The resource block is the most important one of these blocks because its the block that actually creates / updates infrastructure. A resource block is composed of a resource name which identifies the provider/resource type, a human friendly name which can be used to reference the resource block like a variable name and a set of key value pairs that are used to configure the resource block. The following is an example of what a resource block looks like

resource "aws_instance" "web" {
  ami           = "ami-a1b2c3d4"
  instance_type = "t2.micro"
}

• A surprising number of things can be modeled using this provider resource pattern. For example suppose you want a random number or a random password. This is modeled in Terraform as a provider called random and the random provider has resources to create random numbers and passwords.
• Terraform also offers the ability to specify variables. You can declare variables in configuration files, at the command line or through environment variables. You can then consume these variables in resources and other configuration blocks. This is useful because variables can include default values, type constraints and validations.
• Resources can reference each other. So for example suppose you want to create a database but that database needs a password that you also want to manage in Terraform. The database has a dependency on the password resource. So you can reference the password resource from the database resource. Since resources can depend on each other, you can model all resources as a DAG. When Terraform runs plan it needs to do a topological traversal of this DAG in order to create all the resources.
• When one resource explicitly uses another resource Terraform knows that represents an edge in the graph, but you could also have implicit edges in the graph. This is useful when one resource does not explicitly depend on another resource, but there is an implicit dependency that needs to be respected in the Terraform plan.
• Terraform keeps track of what the ground reality looks like in a file called the state file. It would seem that when you run plan or apply Terraform would just need to look at the local configuration, compare it to the actual resources and generate/apply the diff. But there are a few reasons that state in Terraform is actually required. The first is that Terraform needs to keep track of a mapping from configured resources to actual resource ids. For example suppose I update the configuration of an AWS server. That configuration does not contain any resource ID which is required to call the AWS API. This means Terraform must map my logical resource in config to the real world resource. That mapping is stored in state. It is possible to use tags on some cloud providers to approximate this behavior but that does not scale well to all cloud provider types. The next reason that state is actually required is because when resources are deleted, Terraform needs to still know the DAG ordering so it can know in what order to delete resources. When resources are declared in config figuring out the DAG ordering is simple, but when resources are being deleted its not clear which order Terraform should traverse the DAG to do this removal safely. This information get stored in the state file. It is possible that Terraform could hard code the implied ordering for every possible resource type combination but that quickly explodes. The last reason that this state file is needed is to enable a performance gain by treating this state file as the ground reality rather than reconstructing the state of the world on every plan/apply. This is not strictly required but is helpful for large infrastructure. For large infrastructure you can run plan or apply with refresh state flag off in which case the plan/apply will be generated against the state file rather than reconstructing the state of the world.
• Terraform state can be stored either locally or in the cloud. But it should be stored in the cloud because you do not want to enable the state file (representation of ground reality) to drift between different users on the same team. By putting it in the cloud you can leverage a lock on the file to ensure you serialize updates to the state file.
• Terraform also supports the concept of data sources. Data sources enable pulling information from the outside world into Terraform. This is useful when there is a resource you don’t want to manage with Terraform but you need to access some of its properties in order to configure other infrastructure. A data source block looks a lot like a resource block but it only contains the information required to read the resource’s configuration — not to write the configuration.
• Importantly. Terraform supports loops, lists, counts and maps. These basically all enable creating similarly shaped resources without having to declare them N times in configuration.
• Terraform supports the concept of modules. Whenever you run plan/apply you are running these from a root Terraform module. A module is basically a collection of related resources. A module is composed of a single top level root module as well as all child modules that root references. So think of a module as a tree and when you invoke Terraform you operate at the root of the tree. Terraform enables information to be passed up or down the tree — so the parent can pass input arguments to the children and the children can respond back to the parent with results that can be later used in the parent.

Reflection

• Terraform seems like a powerful tool and I like that it operates in a declarative model. At its core I think Terraform is basically two things (1) A query planner which generates the plan (2) A workflow orchestration engine which takes the plan as input and realizes it.
• It is very interesting to me that a state file is needed. It seems like you could just get away with diffing the ground reality from the goal state, but that does not work for a handful of reasons — I am curious if other declarative tooling also have something like a state file?
• The provider / resource / plugin model seems surprisingly powerful. Its interesting how many different types of things can be modeled this way. These plugins also really make Terraform a platform — it does not couple to any specific infrastructure, as long as a provider satisfies the expected resource contract, Terraform can is actually indifferent to what the resource is — that is very cool.
• This plan/apply pattern seems powerful and I am interested to see if this is used in other declarative tools.