Skip to content

Commit

Permalink
fix: article redo
Browse files Browse the repository at this point in the history
  • Loading branch information
@thimaisogilis
thimaisogilis committed Oct 15, 2019
1 parent f899dfe commit c9ddd81
Showing 1 changed file with 124 additions and 121 deletions.
245 changes: 124 additions & 121 deletions site/content/posts/2014-03-06-building-files-redo-simple-branch.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
---
title: 'Building files: redo, the simple branch'
author: Tiphaine
title: "Building files: redo, the simple branch"
author: Shanti
date: 2014-03-06T09:04:00+00:00
featured_image: /wp-content/uploads/2016/04/2.La-vie-a-Sogilis.jpg
tumblr_sogilisblog_permalink:
- http://sogilisblog.tumblr.com/post/78735922317/building-files-redo-the-simple-branch
tumblr_sogilisblog_id:
- 78735922317
featured_image: /img/2016/04/2.La-vie-a-Sogilis.jpg
pyre_show_first_featured_image:
- no
pyre_portfolio_width_100:
Expand Down Expand Up @@ -93,44 +89,42 @@ tags:
- code
- développement
- redo

---
## **Make and Redo**

## Make and Redo

For a long time, developers have been using `make` as their build system. However, this very simple program have many issues. To help with these, many front-ends have been written and used, such as automake and cmake. Some tried to replace it completely like scons.

<!-- more -->

Let’s build a list of things we are dissatisfied with make:

* Recursive make is broken
* Make language is implemented differently on different platforms
* Make uses shell that is also incompatible between platforms
* Targets with multiple outputs are broken
* Make is a template language for shell: it doesn’t know shell syntax and things like filenames with spaces or special characters are plain impossible to get right.
* Dependencies must be specified manually for every target
* Make is not as powerful as it should be
* Make is not as simple as it should be
* Make is yet another language
* Make do not guarantee atomic operations when building files
- Recursive make is broken
- Make language is implemented differently on different platforms
- Make uses shell that is also incompatible between platforms
- Targets with multiple outputs are broken
- Make is a template language for shell: it doesn’t know shell syntax and things like filenames with spaces or special characters are plain impossible to get right.
- Dependencies must be specified manually for every target
- Make is not as powerful as it should be
- Make is not as simple as it should be
- Make is yet another language
- Make do not guarantee atomic operations when building files

There are alternatives, right? They are good, but not completely satisfying. Many are just layers on top of make and do not solve all its problems. They feature support for many languages, compilers, but are not as simple as a `Makefile` when it come to a simple build job. Some are built on top of a programming language while others create another language for the occasion.

To introduce redo, I’d like to quote both the <span style="text-decoration: underline;"><a href="https://github.com/mildred/redo/blob/master/README.md" target="_blank">README of the project</a></span>:
To introduce redo, I’d like to quote both the [README of the project](https://github.com/mildred/redo/blob/master/README.md):

> Although I wrote redo and I would love to take credit for it, the magical simplicity and flexibility comes because I copied verbatim a design by Daniel J. Bernstein (creator of qmail and djbdns, among many other useful things). He posted some very terse notes on his web site at one point (there is no date) with the unassuming title, “<span style="text-decoration: underline;"><a href="http://cr.yp.to/redo.html" target="_blank">Rebuilding target files when source files have changed.</a></span>” Those notes are enough information to understand how the system is supposed to work; unfortunately there’s no code to go with it. I get the impression that the hypothetical “djb redo” is incomplete and Bernstein doesn’t yet consider it ready for the real world.
>
> Although I wrote redo and I would love to take credit for it, the magical simplicity and flexibility comes because I copied verbatim a design by Daniel J. Bernstein (creator of qmail and djbdns, among many other useful things). He posted some very terse notes on his web site at one point (there is no date) with the unassuming title, “[Rebuilding target files when source files have changed.](http://cr.yp.to/redo.html)” Those notes are enough information to understand how the system is supposed to work; unfortunately there’s no code to go with it. I get the impression that the hypothetical “djb redo” is incomplete and Bernstein doesn’t yet consider it ready for the real world.
>
> After I found out about djb redo, I searched the Internet for any sign that other people had discovered what I had: a hidden, unimplemented gem of brilliant code design. I found only one interesting link: Alan Grosskurth, whose Master’s thesis at the University of Waterloo was about top-down software rebuilding, that is, djb redo. He wrote his own (admittedly slow) implementation in about 250 lines of shell script.
>
>
> If you’ve ever thought about rewriting GNU make from scratch, the idea of doing it in 250 lines of shell script probably didn’t occur to you. redo is so simple that it’s actually possible. For testing, I actually wrote an even more minimal version, which always rebuilds everything instead of checking dependencies, in 150 lines of shell (about 3 kbytes).
>
>
> The design is simply that good.
Have you found yet the <span style="text-decoration: underline;"><a href="https://github.com/apenwarr/redo" target="_blank">home of this redo implementation</a></span>?
Have you found yet the [home of this redo implementation](https://github.com/apenwarr/redo)?

&nbsp;

## **Redo in five minutes**
## Redo in five minutes

Redo is a top down build system. You must know what you want to build and how to build it. Then, building the files you asked, and then the dependencies if they are not up to date, it builds everything you might want to have.

Expand All @@ -140,27 +134,26 @@ Declaring dependencies is done by executing `redo-ifchange` with the files neces

State of files must be kept persistently on the filesystem, in a `.redo` directory.

If that triggers your interest, you can look at the <span style="text-decoration: underline;"><a href="https://github.com/mildred/redo/blob/master/README.md" target="_blank">project’s README</a></span>.

&nbsp;
If that triggers your interest, you can look at the [project’s README](https://github.com/mildred/redo/blob/master/README.md).

## **Simple examples**
## Simple examples

**<span style="text-decoration: underline;">Build documentation</span>**
### Build documentation

Let’s say you want to build `README.md` in `README.html`. Have a file called `default.html.do`:

<pre class="wp-code-highlight prettyprint">redo-ifchange "$2.md"
Markdown.pl &lt;"$2.md" &gt;"$3"
</pre>
{{< highlight >}}
redo-ifchange "$2.md"
Markdown.pl <"$2.md" >"\$3"
{{< /highlight >}}

Then, you simply run `redo README.html`. It will execute the script with three parameters: `README.html`, `README` and a temporary file.

* `redo-ifchange` will declare a dependency to `README.html` (and build it if not up to date)
* and `Markdown.pl` will build the markdown in the temporary file.
* Redo will then rename atomically the temporary file to `README.html`.
- `redo-ifchange` will declare a dependency to `README.html` (and build it if not up to date)
- and `Markdown.pl` will build the markdown in the temporary file.
- Redo will then rename atomically the temporary file to `README.html`.

**<span style="text-decoration: underline;">Find dependencies from C / C++</span>**
### Find dependencies from C / C++

Redo doesn’t feature any automatic `#include` detection for C source files. Then, how to record that information, manually?

Expand All @@ -170,112 +163,122 @@ This solution is very elegant because the source is only parsed once. The file i

Let’s see how a `default.o.do` would look like:

<pre class="wp-code-highlight prettyprint">redo-ifchange "$2.c"
{{< highlight >}}
redo-ifchange "$2.c"
gcc -MD -MF "$2.d" -c -o "$3" "$2.c"
read DEPS &lt;"$2.d"
redo-ifchange ${DEPS#*:}
</pre>
read DEPS <"$2.d"
redo-ifchange ${DEPS#\*:}
{{< /highlight >}}

* First, the source file itself is declared as a dependency.
* Then, the file is built
* When the file is built, all dependencies parsed by gcc are recorded as well
- First, the source file itself is declared as a dependency.
- Then, the file is built
- When the file is built, all dependencies parsed by gcc are recorded as well

If none of the dependencies first recorded have been modified, none of this is executed. Is is made possible because redo keeps a persistent store of dependencies.

**<span style="text-decoration: underline;">Using hash instead of timestamp</span>**
### Using hash instead of timestamp

By default, redo uses file timestamps to deterine if a file is up to date. Sometimes, a file may be rebuilt identically (let’s say `config.h`) and you don’t want to rebuild everything just because a file was rewritten with the exact same content.

When building a target, you can use redo-stamp to record a hash instead of a timestamp. It could be a hash of everything, although it is generally a hash of the output file. But if the output file contains a comment containing the current date, you can use any other data for input as redo-stamp.

Let’s look then at `config.h.do`:

<pre class="wp-code-highlight prettyprint">detect-config-h &gt;"$3"
redo-stamp &lt;"$3"
</pre>
{{< highlight >}}
detect-config-h >"$3"
redo-stamp <"$3"
{{< /highlight >}}

&nbsp;

## **What’s the simple branch?**
## What’s the simple branch?

This version of redo was built with python and SQLite. This was seen as quite complex and a big refactoring was started in the simple branch, but was never really completed. I completed it, mostly to add new features.

The simple branch features a drastic change of implementation. It’s still python (I hop there will be a C version one day) but there is no more SQLite. It only feature simple well organized files for the datastore.

**<span style="text-decoration: underline;">File format</span>**
### File format

For each file recorded in redo, there is a corresponding file in the `.redo` directory with the same name, but ending with `.deps`. For example, you’d have for `src/main.c` a file `src/.redo/main.c.deps`. It would look like:

<pre class="wp-code-highlight prettyprint">redo.0
{{< highlight >}}
redo.0
41 2848026
1386591363.1-1386591363.1-818-41-1796582 default.c.do
1386591363.1-1386591363.1-37-41-1796583 lib.h
7bc09e7900651e7b8682eadc926105a868cf74cc .
1387535039.05-1387535039.03-96-41-2848029+1387535038
0
</pre>
{{< /highlight >}}

* `redo.0` is a constant tag containing the version of the format
* `41 2848026` is the device id and inode of the `.deps` file to detect if the file has been copied or shared using a virtual device. In that case, the file is considered missing.
* The list of dependencies: <pre class="wp-code-highlight prettyprint">1386591363.1-1386591363.1-818-41-1796582 default.c.do
1386591363.1-1386591363.1-37-41-1796583 lib.h
</pre>

The beginning of the line is the stamp for the dependency. On the right, there is the file name.</li>

* The stamp hash generated by `redo-stamp` (optional, line may be missing): <pre class="wp-code-highlight prettyprint">7bc09e7900651e7b8682eadc926105a868cf74cc .
</pre>

* The stamp based on file metadata of the last generated file: <pre class="wp-code-highlight prettyprint">1387535039.05-1387535039.03-96-41-2848029+1387535038
</pre>

* The exit code of the last build of the file: `` for success.</ul>

File stamps are generally some fields returned by the `stat()` system call separated by dashes:

* the created unix time
* the modified unix time
* the file size
* the device id
* the file inode

It can also be a special value:

* `` if the file is non existent
* `dir` if the file is a directory
* `old` if the .deps file is outdated (from old redo version, or with wrong inode / device id)

Also, it is possible the dependencies will refer to the `//ALWAYS` file, which is considered invalid. This is generated by `redo-always` which always triggers a rebuild.

&nbsp;

## **How does redo manages multiple outputs**

This is only a recently implemented feature for the simple branch. It works by having a master recipe building more than one file, and declaring the extra files it built. For completeness, slave recipes can delegate the build to a master recipe that is known to build the file.

Let’s say you have a program called `run-template` that generates a `testing.c` file and a `testing.h` file from `testing.tmpl`. You declare the master recipe (`testing.c.do`) this way:

<pre class="wp-code-highlight prettyprint">redo-ifchange "$2.tmpl"
run-template --c-file="$3" --h-file="${3%.c}.h" &lt;"$2.tmpl"
</pre>

What it does is depending on the `testing.tmpl` file and then telling `run-template` to put the master file (the `testing.c` file) in `$3` (the temporary file in the .redo directory). It also tells `run-template` to place the `testing.h` file in the same directory as the `testing.c` file. Placing a file here is enough as redo will scan this directory for extra targets.

The slave recipe (`testing.h.do`) is just delegating build to `testing.c`:

<pre class="wp-code-highlight prettyprint">redo-delegate "testing.c"
</pre>

An outline of the files involved:

* `testing.tmpl`: the template
* `testing.c.do`: the recipe for `testing.c` (and `testing.h` as a secondary target)
* `testing.h.do`: the slave recipe for `testing.h`
* `.redo/testing.c.out/testing.c`: the temporary file (`$3`) during build
* `.redo/testing.c.out/testing.h`: the secondary target (`${3%.c}.h`) during build
* `testing.c`: the generated `.c` file after the build
* `testing.h`: the generated `.h` file after the build

The key is that all files placed in the same directory as `$3` (the temporary file) will be treated as secondary targets and will override files already in the project.

**Shanti**
- `redo.0` is a constant tag containing the version of the format

- `41 2848026` is the device id and inode of the `.deps` file to detect if the file has been copied or shared using a virtual device. In that case, the file is considered missing.

- The list of dependencies:

{{< highlight >}}
1386591363.1-1386591363.1-818-41-1796582 default.c.do
1386591363.1-1386591363.1-37-41-1796583 lib.h
{{< /highlight >}}

The beginning of the line is the stamp for the dependency. On the right, there is the file name.

- The stamp hash generated by `redo-stamp` (optional, line may be missing):

{{< highlight >}}
7bc09e7900651e7b8682eadc926105a868cf74cc .
{{< /highlight >}}

- The stamp based on file metadata of the last generated file:

{{< highlight >}}
1387535039.05-1387535039.03-96-41-2848029+1387535038
{{< /highlight >}}

- The exit code of the last build of the file: `0` for success.

File stamps are generally some fields returned by the `stat()` system call separated by dashes:

- the created unix time
- the modified unix time
- the file size
- the device id
- the file inode

It can also be a special value:

- `` if the file is non existent
- `dir` if the file is a directory
- `old` if the .deps file is outdated (from old redo version, or with wrong inode / device id)

Also, it is possible the dependencies will refer to the `//ALWAYS` file, which is considered invalid. This is generated by `redo-always` which always triggers a rebuild.

## How does redo manages multiple outputs

This is only a recently implemented feature for the simple branch. It works by having a master recipe building more than one file, and declaring the extra files it built. For completeness, slave recipes can delegate the build to a master recipe that is known to build the file.

Let’s say you have a program called `run-template` that generates a `testing.c` file and a `testing.h` file from `testing.tmpl`. You declare the master recipe (`testing.c.do`) this way:

{{< highlight >}}
redo-ifchange "$2.tmpl"
run-template --c-file="$3" --h-file="${3%.c}.h" <"$2.tmpl"
{{< /highlight >}}

What it does is depending on the `testing.tmpl` file and then telling `run-template` to put the master file (the `testing.c` file) in `$3` (the temporary file in the .redo directory). It also tells `run-template` to place the `testing.h` file in the same directory as the `testing.c` file. Placing a file here is enough as redo will scan this directory for extra targets.

The slave recipe (`testing.h.do`) is just delegating build to `testing.c`:

{{< highlight >}}
redo-delegate "testing.c"
{{< /highlight >}}

An outline of the files involved:

- `testing.tmpl`: the template
- `testing.c.do`: the recipe for `testing.c` (and `testing.h` as a secondary target)
- `testing.h.do`: the slave recipe for `testing.h`
- `.redo/testing.c.out/testing.c`: the temporary file (`$3`) during build
- `.redo/testing.c.out/testing.h`: the secondary target (`${3%.c}.h`) during build
- `testing.c`: the generated `.c` file after the build
- `testing.h`: the generated `.h` file after the build

The key is that all files placed in the same directory as `$3` (the temporary file) will be treated as secondary targets and will override files already in the project.

0 comments on commit c9ddd81

Please sign in to comment.