Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automated build process generation exploration #859

Draft
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

danielrbradley
Copy link
Member

@danielrbradley danielrbradley commented May 26, 2022

This PR contains a comparible implementation of the file-target based makefile in taskfile. It also contains a prototype of how we could generate makefiles by building simple funtions (like we do for our yaml GH actions).

Considering the elements of cross-platform, up-to-date target tracking, the ability to centrally generate and parallelism; there's currently no selling point strong enough for us to shift away from make.

Make

The makefile in awsx is already in a very good place - it uses file targets which means it's trivial for make to optimize which targets need to run based on any file in the project changing. This gives us a good baseline to compare against taskfile.dev.

Taskfile

Overall, taskfile offers very similar features to make with a yaml based language which would be simpler to generate.

Factfile (true as of 3.12.1)

  • For checking up to date files, Taskfile uses file hashes by default but also supports timestamp.
  • If using timestamps, you have to specify "generates" as well as "sources" for each target.
  • If "generates" is specified, it also triggers the task if the files listed are missing.
  • Source globbing doesn't support exclusions.
  • No alternative to make's --touch to track back through dependencies and mark as built.
  • The file yaml format uses more lines of code.
  • Dependencies are moddled separately to sources (which are the same thing in make)
  • Concurrency attributes allow a clean to be run correctly in parallel at the same time as a build (unlike make).
  • The concurrency attributes must be specified on every target to behave correctly.
  • Though the CLI tool runs cross-platform, the shell commands within the taskfile are still platform dependant.

The lack of --touch alternative is a blocker for efficient (& correct) CI. When running in CI we want to:

  1. Run the first half of the build in the first job
  2. Upload the binary assets of the build result
  3. Run parallel jobs which download the binary artefacts
  4. We use --touch after restoring the artefacts so make will create stub files to make the dependencies appear up-to-date
  5. When the second half of the build runs, the existing binaries will be used.

All targets always self-describe all prerequisites so In local situations where we don't use --touch dependencies will be followed as usual ensureing no internal knowledge of the build process for end-users.

Makefile generation

If we can't use taskfile due to the CI limitations, we can instead just generate makefiles dynamically using a quick DSL (< 80 LOC).

Demo program:

import { writeFileSync } from "fs";
import { Makefile, render, Target } from "./makefile";

// Write functions to parametrize targets
function myTarget(arg: string): Target {
  return {
    name: `bin/${arg}`, // Use file targets internally
    commands: [`build ${arg} bin/${arg} $(VERSION)`], // Mix string interpolation and make variables
  };
}

const buildMyProgram = myTarget("my-program"); // Create a target

const makefile: Makefile = {
  variables: {
    // Define make variables
    VERSION: "$(shell pulumictl get version)",
  },
  targets: [
    buildMyProgram, // Add targets to the makefile model
    {
      name: "build",
      dependencies: [buildMyProgram], // Depend on other target objects
      phony: true, // Inline phony attribute
    },
  ],
};

const source = render(makefile);
writeFileSync("Makefile", source);

Output:

VERSION := $(shell pulumictl get version)

bin/my-program:: 
	build my-program bin/my-program $(VERSION)

build:: bin/my-program

.PHONY:: build

- Don't need "generates" with checksums ... and the "generates" and "sources" is relative to the "dir".
- Just use relative paths rather than messy `{{ .CWD }}`
We should still add "generates" because it will then run if the file is missing. It doesn't however fingerprint the generated files to detect manual changes - it's only to check for existence.
Use task namespaces to group provider tasks
Source excludes are not currently supported so we can't have everything except `bin` using something like `"!(bin)/*".
- Rename TS build to transpile.
Use `run: once` to avoid rebuild of shared dependencies.
Pull build target and output into variables with defaults.
Use multiple 'deps' to run provider-specific builds in parallel.
- Add clean task too to be able to quick perf testing.
- Use variable overrides to pack each binary.
- Use stub "dist-dir" target to ensure dist directory exists.
- Use `--no-progress` for all yarn installs as this doesn't work well in parallel.
- Set make default flags to mirror parallelism and provide additional warnings
- Create lightweight makefile model which is rendered at end.
- Don't try to do everything make can, just model how we use it i.e. all VARS at top.
- Write common target patterns as functions.
- Explore using higher-level functions like `cwd(dir, ...cmds)` for expressiveness.
- Allow dependencies to be a string or an instance of another target for briefness.
Dynamically building out full repeated steps rather than odd makefile hacks.
Use target property to generate .PHONY automatically
- makefile for the core model types and render function.
- makefile-builders for the functions to build specific makefiles.
This now creates a valid makefile which completes the whole provider part of the build.
@danielrbradley danielrbradley self-assigned this May 26, 2022
@danielrbradley danielrbradley changed the title Taskfile exploration Automated build process generation exploration May 26, 2022
@danielrbradley danielrbradley added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label May 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
impact/no-changelog-required This issue doesn't require a CHANGELOG update
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant