Parallel building

[junaruga]

The ticket is from sclorg/rhscl-rebuild-recipes#14 .

As a possibility, it may be possible as below ways.

1. Analyze the recipe YAML file
2. Generate build dependency tree object data for the building like a decision tree [1]
3. Build packages by following the build dependency tree data. It can be a parallel building.

[1] https://en.wikipedia.org/wiki/Decision_tree

[ncoghlan]

A potentially simpler approach might be to push the decision back on to the build recipe author, and add a %concurrent-build directive that indicated the packages nested under that node should be built concurrently:

packages:
  - python
  - %concurrent-build:
    - python-docutils
    - python-markupsafe
    - python-coverage
 - etc...

The prefix used to indicate a directive (rather than a package name) would be relatively arbitrary, so I'd suggest % for consistency with RPM specfile macros and section headers.

[junaruga]

Technically RPM List Builder can detect which package can be built in parallel automatically.
Why do you like to specify it manually by %concurrent-build?

[hroncok]

Because build order is part of the data, the recipe. Parallel build information is part of the build order.

[hroncok]

The prefix used to indicate a directive (rather than a package name) would be relatively arbitrary, so I'd suggest % for consistency with RPM specfile macros and section headers.

AFAIK that would need escaping:

ScannerError: while scanning for the next token
found character '%' that cannot start any token
  in "<unicode string>", line 3, column 5:
      - %concurrent-build:

[khardix]

AFAIK that would need escaping:

The YAML tags might be of use:

packages:
  - python
  - !concurrent
    - python-docutils
    - python-markupsafe
  - etc...

[junaruga]

Because build order is part of the data, the recipe. Parallel build information is part of the build order.

I guess that we want to reduce the data recorded in recipe file and make it simple as much as possible?

My question is the RPM List Builder can detect which packages can be the parallel build by analyzing the package dependency tree from the list of RPM spec files of the packages (BuildRequires, Requires)?

packages:
  - python (A)
  - python-docutils (B)
  - python-markupsafe (C)
  - python-coverage (D)
  - package (E) 

For example RPM List Builder can detect that E depends on B, C and D checking E's RPM spec file.

[hroncok]

If you want to detect the build order, you need no recipe files (in theory). Also macros and other bootstrapping can alter the build order.

Fact 1: The build order is in the recipe. Fact 2: rpm-list-builder follows the recipe. Conclusion: Parallel build information should be stored in the recipe. Making a guess-the-build-order magic tool might be useful, but let's keep this KISS.

[junaruga]

Okay, if we will follow my suggesting way, we do not need the recipe file. That's truth.
That's different concept from current RPM List Builder.
Ok, I agree with your idea, and the new tag for parallel building.

Thanks for the explanation.

[ncoghlan]

Right, one of the key benefits of RPM List Builder is that the recipe file makes it possible to resolve circular dependencies by defining a bootstrapping order.

Where an automatic build ordering might eventually make sense is inside a !concurrent directive, as that would allow clean builds even when packages listed that way depend on each other, and an early error if any circular dependencies are found.