Terraform vs CloudFormation: AWS resources added over the last year

Some people are all like:

it takes CloudFormation FOREVER to support new AWS features!

But other people are all like:

it takes Terraform FOREVER to support new AWS features!

So I decided to collect some data to gain better visibility into how the two infrastructure-as-code tools compare along this dimension.

Methodology

I looked at every AWS resource for which either Terraform (changelog) or CloudFormation (changelog) added support in the last ~year: the first day of April 2018 through the last day of April 2019.

For each of those AWS resources, I looked up whether each tool added support, and if so, logged the date(s).

So I'm not really answering the original question "how long does it take $IAC_TOOL to support new AWS features," but instead getting a broad comparative view of the two tools' AWS resources supported over the last year. (This data could be used to answer the original question for a given set of features, if they were released over the last year).

Limitations

  • Only looking at newly-added resources in Terraform or CloudFormation, not updates to existing ones
  • Terraform or CloudFormation may "add support" but it is only partial, and that nuance is lost here
  • Because of the limited time frame, some of the most painful longstanding non-supported AWS resources fall through the cracks of this analysis
  • There isn't always a perfect 1:1:1 mapping between an underlying AWS resource and the CFN/TF resources that represent it, but there usually is and when there isn't I used my best judgment to keep the comparisons accurate.
  • I'm not measuring time between a new feature being released by AWS and CFN/TF adding support, because that seemed like a more complicated question to answer meaningfully (they release a lot of stuff).

Results

You can see the raw data, and the script used to generate the visualizations below, in this gist.

In summary:

  • CloudFormation added support for 131 AWS resources.
  • Terraform added support for 151 AWS resources.
  • CloudFormation added support for 70 AWS resources that Terraform does not support. High level view of the services under which those resources live:
  • Terraform added support for 84 AWS resources that CloudFormation does not support. High level view of the services under which those resources live:

  • CloudFormation and Terraform both added support for 2 AWS resources on the same day (notably, EKS cluster).
  • For the set of AWS resources that both tools support, there was a subset of 48 resources for which CloudFormation added support first. The maximum number of days by which CloudFormation support preceded Terraform support was 2686 (lol CFN has been around for a while!) and then median number of days was 103.
  • For the set of AWS resources that both tools support, there was a subset of 28 resources for which Terraform added support first. The maximum number of days by which Terraform support preceded CloudFormation support was 279 and then median number of days was 105.

Here's the number of days by which each AWS resource in question was implemented by one tool first (yellow for CFN first, purple for TF first):

Conclusions

The overall picture painted by this slice of data, to me, is that there is no simple narrative around which instrastructure-as-code tool is "better" at supporting AWS services and resources in a timely manner.

During the timeframe, both projects cranked out support for a similar number of AWS resources, and when you compare the time deltas and consider that Terraform is much newer, I don't see a clear "winner" – it depends on which services you care about.

And ultimately, both tools provide reasonable mechanisms to roll your own support for unsupported AWS resources. For this reason, my perspective is that there are several dimensions to the Terraform vs CloudFormation comparison that are more interesting, such as:

  • the workflow for {defining, peer-reviewing, deploying, destroying} infrastructure
  • validation
  • handling failures
  • detecting and fixing configuration drift
  • describing infrastructure with a DSL vs YAML/JSON config files
  • interaction with / ability to import and modify pre-existing AWS resources
  • integration with other tooling
  • documentation
  • open- vs closed- source

If there's anything interesting to you that I've overlooked in this data collection and analysis, let me know!

Show Comments