Project templating for cloud

So you’ve started using AWS CDK, you’ve shipped a few projects, you’ve shared your findings with your team and now they’ve started using CDK…then you start to notice the subtle differences in how developers have created their projects, dot files, style, folder structure and the usage of custom constructs.

How do you start a new project? “Easy as, I just copy and paste it from a previous project and change out the bits I need”

You start to see the problem, If we use a typical CDK project as an example, you might have AWS CDK constructs, your constructs and your team dot files plus any other custom configs needed, CI/CD etc.

How do you promote consistency for apps outside of just constructs that isn’t too hard to maintain and easily shared?

What can we do?

I reached out to the devtools folk in AWS Community Builder, surely there are others who have faced the same problem? What kind of tools were others usings? Not too surprising, projen got a mention as did cdk init and older popular choices like yeoman

I next combined the options from above with others found online:

Key considerations

When I talk about templating or scaffolding there are a few things that come to mind that would be considered, key considerations. It can be easy to suggest that templating is a simple file copy and the need for more developed tools might be unnecessary. Also worth noting that often templating, scaffolding, project generators are synonymous with each other. I’ll try to highlight some of the key considerations but as you read you might start to think of more:

Composition

Something that I identified in the earlier example is code reuse, once your team have a working project that is agreed within the team, we just want to use that for all similar projects moving forward. We want to layer or compose many templates or files to produce a single project output.

Something I touched on earlier is code reuse, once you have a working project you want to identify the different layers of your template. The project generator that you end up using should help support that workflow. We want to compose many layers or templates that produce a single project output. An example for CDK might be an empty CDK project and then more complex patterns on top e.g. web app, web API etc. The final template is one you’d likely change the most and the base CDK template could be updated, those changes would be consistent across all projects that use the base layer.

Placeholders

When I refer to placeholders I refer to string replacement at a minimum but often this includes transformation e.g. to uppercase. Because our templates are generic you will often want to replace generic placeholders like a project name with the actual name. You may have seen examples of this such as handlebars e.g. {{ swapThis }}

Global vs Local

A global generator would live outside of the project, its likely installed as an NPM package and you can call it whenever you want to produce a whole project (can do partials, but I’ll focus on complete). Local generators often reside within the project, they are maintained in the same repo, they will often be able to render components.

Package

Once a new template has been created we want to be able to distribute, maintain and version the templates. This will likely be through the use of existing package managers such as NPM.

Let’s do some stuff

I’m going to look at Projen, Hygen and Yeoman. The built-in CDK scaffolder can be used but isn’t strictly a project templating or generator tool. When you run cdk init for a typescript CDK app project you’ll get something like this:

Cdk init will be OK for plenty of people, use what works for you and your team.

There isn’t any other customization here for CDK init, job done, this is why we won’t be looking at it any more detail and moving onto the others.

Projen

I recommend visiting the github page for projen and flicking through the documentation as I won’t do it justice. Projen aims to:

“Define and maintain complex project configuration through code”, sounds great!

If you’ve been using CDK for a while you’ve probably heard of or used projen, it’s getting lots of traction and is popular in the community.

Let’s take a look at the basics:

Install

npx projen

New project

Creating a project, as in working project output to be used by users is simple enough:

npx projen new awscdk-app-ts

There are multiple project types not limited to CDK, check out the full list of project types.

How projen works

In the example above we’ve create a CDK app, not a custom construct. The way projen handles configuration of your project is through the .projenrc.js file located in the root of the project.

The projen way is to handle all configuration of your project through code, you will notice that all of the dot files are now read-only. Instead if you want to modify the package.json for example you will need to use the .projenrc.js to define your properties:

const { awscdk } = require("projen");
const project = new awscdk.AwsCdkTypeScriptApp({
  cdkVersion: "2.1.0",
  defaultReleaseBranch: "main",
  name: "projen-example",

  deps: ["prettier"], // simple example of adding dependency
});
project.synth();

Once you’ve updated the projen file you can re-run projen to apply those changes:

npx projen

The API reference for projen is extensive and there is a fair bit of chatter on the internet available to get you going. But, if you like idea of boilerplate projects and the projen ways I recommend checking it out. Even if you don’t it’s worth keeping any eye on the project to see what’s coming or if you can help out.

Hygen

I stumbled across hygen while looking for solutions to project generators and was pleasantly surprised. Like the previous example, checkout the project page to get the full write up, here is what hygen is about:

“Hygen is the simple, fast, and scalable code generator that lives in your project.”

What I like about this one is that everything in the quote about is so true once you start to unpack everything hygen has to offer.

Install

The quickstart guide on github has a bunch of different ways to hit the ground running, I’ll show you using npm:

npm install -g hygen

New Project

I’m going to step through this at a high-level showing project local mode, this is what hygen is all about but know that it isn’t limited to local and docs show you how to create a global generator. When in project local mode your template lives a long side your project.

hygen init self

You’ll notice when you do this you get a bunch of files and folders you’ve never seen before, this is the hygen templating at work. Your core project would live at the top level and hygen would read the _templates folder (if present) for available options.

How Hygen works

As mentioned above, the default is to run in project local mode, the files added to your local project instruct hygen how to run. Out of the box you get generator documentation and prompting which provides a UI for users to step through. Hygen handles file copying and manipulation using frontmatter:

---

to: src/stack-<%=name%>.ts

---

import { App, Stack, StackProps } from 'aws-cdk-lib';

import { Construct } from 'constructs';

export class <%=name%> extends Stack
{ constructor(scope: Construct, id: string, props: StackProps = {})
{ super(scope, id, props);
// define resources here...
}}

You’ll notice the — delimiter in the header, this is frontmatter. The basic example above is showing that this file will be copied into the root source directory and a variable for name has been used. The variable placeholders are using ejs, which is a popular templating framework and quite powerful.

Now that you’ve added a source template in hygen you can commit your local generator with your project and teams can initialize the project at any time using:

hygen generator new --name talkncloud

This is a basic example, but you can start to see how you might create CDK templates and share them with your team. I recommend checking out the hygen website, it’s an active project and the documentation is quite good.

Yeoman

We’ve reached the end and I saved yeoman for last for many reasons, but the main one being yeoman is like the og of generators, it’s been around for donkeys, well before CDK came about at least. So, what does yeoman try to achieve:

Yeoman helps you to kickstart new projects, prescribing best practices and tools to help you stay productive.

The yeoman website does a better job at detailing this than I’ll do, check it out when you get time.

Install

npm install -g yo

New project

Once you have install yeoman you can run the cli using the yo command. You’ll be presented with a few options:

yeoman-example git:(main) ✗ yo

? 'Allo mike! What would you like to do? (Use arrow keys)

Run a generator

──────────────

❯ Install a generator

Find some help

Get me out of here!

──────────────

Yeoman has a public repository for community provided generators (templates), from the cli you can search those. We’ll be making our own basic generator and to do that we are going to use the yeoman [generator-generator](npm install -g yo generator-generator “yeoman generator-generator”) package…

Source: https://github.com/yeoman/generator-generator (this always cracks me up, hahaha)

npm install -g yo generator-generator

The generator above has provided a boilerplate generator we can use to create our first generator. The dummy file included is part of the app generator, I’ve added the CDK example from the previous examples, hopefully you can see how the generators/app/templates folder is where you’d place your template files for the appgenerator. App might be a CDK API GW Lambda Route53 pattern that you’d like to share with your team.

Note: If you’d like to use typescript there is an example project here to get you started.

How it works

There is quite a bit to yeoman under the hood, check out the getting started guide for authoring generators. You may have noticed in the generator above there are several methods:

prompting();

writing();

install();

As you would have guessed it, yeoman also provides UI for prompting users for options that you can use in your template. These can then be used to conditionally do whatever you want or perform string replacement / transformation using ejs templating. Then you can write out your files and perform installation of package dependencies, what this does is once someone uses your generators it will run npm install (or whatever package manager).

You can start to see that there has been a fair amount of thought put into yeoman in how generators need to work which is why this is often a popular choice, you may also find more out of the box integration options with other services.

Supporting these projects

The projects mentioned today are all open-source projects, you can help out these projects by:

Summary

I’m sure if you’re reading this you’ve probably used various scaffolders in the past and maybe thought it would be good to extend that for your own use. The generators discussed are a few of the popular ones and there are others, not all project generators are the same and some will work better for your team than others. I do know that using project generators will help your team and organization to produce consistent projects with all of the sensible defaults you want to provide which is a great place to be.

The code samples mentioned in this article can be found in the talkncloud github repository, while these aren’t complete CDK projects the purpose is to show the outputs from the various generators:

talkncloud/aws

all things cloud from IaC to full apps. Contribute to talkncloud/aws development by creating an account on GitHub.

talkncloud GitHub

Credits: Article banner photo by Nik on Unsplash