Support incorporation of git hash and status into build

[claridge]

It would be helpful to support incorporation of the git hash and some parsed from (perhaps client-specified) of the git status into the build.

See https://groups.google.com/forum/#!topic/bazel-discuss/_VVkuYz0BnU for discussion.

[hanwen]

I'm happy to provide a workspace status command to emit useful Git data, but until we emit buildstamp artifacts, it's moot.

[mzhaom]

I don't know what buildstamp artifact is about, but as far as I can tell based on my test, the code is in place to translate stable-status.txt and volatile-status.txt to language specific files(header for C, properties for Java), so what's missing now is just the right option to allow users to provide a wrapper script to generate scm status and then let BazelWorkspaceStatusAction write it to the status file.

[hanwen]

right. What is needed is adding a

@OverRide
public void execute(ActionExecutionContext actionExecutionContext)
throws ActionExecutionException, InterruptedException {

to BazelWorkspaceStatusAction. The function should create a command line ("new BaseSpawn.Local" to run in SpawnActionContext), perhaps to a script under tools/buildstamp/ . The workspace directory can be get gotten from

    runtime.getDirectories().getWorkspace(),

inside BazelWorkspaceModule, and passed to the WorkspaceStatusAction.

The data that is written directly now, should be passed as data to the shell script.

The script should probably run "git status" and "git describe" to describe the state of the tree.

[mzhaom]

Thanks for the pointers.
I think I have a pending cl which is almost there. I'll send it out so that we can figure out what to do next.

Any suggestion regarding how to expose these information to skylark? They can probably access bazel-out/{stable,volatile}-status.txt directly. But I'm not sure whether that's the preferred approach.

[guoshimin]

The commit message says workspace_status_command defaults to tools/buildstamp/get_workspace_status. However,

$ bazel help build | grep workspace_status_command
  --workspace_status_command (a path; default: "")

What am I missing?

[ulfjack]

The documentation is incorrect - we currently use code to set the default
values for flags that are different between internal and external, and that
code isn't visible to the documentation printer, so it incorrectly shows up
as "". We think we have a way to fix that now; we're working on it.

You can check with:
bazel build -s

That should print the workspace status command that is actually run.

On Sat, Feb 20, 2016 at 5:39 AM, guoshimin [email protected] wrote:

The commit message says workspace_status_command defaults to
tools/buildstamp/get_workspace_status. However,

$ bazel help build | grep workspace_status_command
--workspace_status_command (a path; default: "")

What am I missing?

—
Reply to this email directly or view it on GitHub
#216 (comment).

[guoshimin]

The supposed default doesn't seem to work. I have this genrule target:

genrule(
  name = "test",
  outs = ["foo"],
  cmd = """touch $@; for file in bazel-out/*-status.txt; do echo "$$file: "; cat $$file; echo; done""",
  stamp = 1,
)

Basically it prints the content of bazel-out/*-status.txt during the build.

When I don't specify --workspace_status_command, I get

$ bazel clean && bazel build -s :test
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
INFO: Found 1 target...
>>>>> # //tools:test [action 'Executing genrule //tools:test']
(cd /private/var/tmp/_bazel_shimin/3a4c5c5e79ba6f8a3fd88b3eacc43127/universe2 && \
  exec env - \
    PATH=/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/Users/shimin/gopath/bin:/Users/shimin/gopath/bin \
    TMPDIR=/var/folders/gz/71h5bskj4x94m63ddqbc3jpw0000gq/T/ \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; touch bazel-out/local_darwin-fastbuild/genfiles/tools/foo; for file in bazel-out/*-status.txt; do echo "$file: "; cat $file; echo; done')
INFO: From Executing genrule //tools:test:
bazel-out/stable-status.txt:
BUILD_EMBED_LABEL
BUILD_HOST canard.local
BUILD_USER shimin
bazel-out/volatile-status.txt:
BUILD_TIMESTAMP 1456166941841

Target //tools:test up-to-date:
  bazel-genfiles/tools/foo
INFO: Elapsed time: 1.815s, Critical Path: 0.02s

When I specify --workspace_status_command manually, it works as expected:

$ bazel clean && bazel build -s :test --workspace_status_command tools/buildstamp/get_workspace_status
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
INFO: Found 1 target...
>>>>> # //tools:test [action 'Executing genrule //tools:test']
(cd /private/var/tmp/_bazel_shimin/3a4c5c5e79ba6f8a3fd88b3eacc43127/universe2 && \
  exec env - \
    PATH=/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/Users/shimin/gopath/bin:/Users/shimin/gopath/bin \
    TMPDIR=/var/folders/gz/71h5bskj4x94m63ddqbc3jpw0000gq/T/ \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; touch bazel-out/local_darwin-fastbuild/genfiles/tools/foo; for file in bazel-out/*-status.txt; do echo "$file: "; cat $file; echo; done')
INFO: From Executing genrule //tools:test:
bazel-out/stable-status.txt:
BUILD_EMBED_LABEL
BUILD_HOST canard.local
BUILD_USER shimin
bazel-out/volatile-status.txt:
BUILD_TIMESTAMP 1456167095265
BUILD_SCM_REVISION 74bbe13197c18d8defb74885bf3d6ec3e71ae8c1
BUILD_SCM_STATUS Modified

Target //tools:test up-to-date:
  bazel-genfiles/tools/foo
INFO: Elapsed time: 1.750s, Critical Path: 0.06s

This is my bazel version:

Build label: 0.1.5
Build target: bazel-out/local_darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Feb 9 19:11:16 2016 (1455045076)
Build timestamp: 1455045076
Build timestamp as int: 1455045076

Am I doing something wrong?

[ulfjack]

It looks like a bug to me. I'm also surprised it doesn't print the status
command execution.

On Mon, Feb 22, 2016 at 7:53 PM, guoshimin [email protected] wrote:

The supposed default doesn't seem to work. I have this genrule target:

genrule(
name = "test",
outs = ["foo"],
cmd = """touch $@; for file in bazel-out/*-status.txt; do echo "$$file: "; cat $$file; echo; done""",
stamp = 1,
)

Basically it prints the content of bazel-out/*-status.txt during the
build.

When I don't specify --workspace_status_command, I get

$ bazel clean && bazel build -s :test
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
INFO: Found 1 target...

//tools:test [action 'Executing genrule //tools:test']

(cd /private/var/tmp/_bazel_shimin/3a4c5c5e79ba6f8a3fd88b3eacc43127/universe2 &&
exec env -
PATH=/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/Users/shimin/gopath/bin:/Users/shimin/gopath/bin
TMPDIR=/var/folders/gz/71h5bskj4x94m63ddqbc3jpw0000gq/T/
/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; touch bazel-out/local_darwin-fastbuild/genfiles/tools/foo; for file in bazel-out/*-status.txt; do echo "$file: "; cat $file; echo; done')
INFO: From Executing genrule //tools:test:
bazel-out/stable-status.txt:
BUILD_EMBED_LABEL
BUILD_HOST canard.local
BUILD_USER shimin
bazel-out/volatile-status.txt:
BUILD_TIMESTAMP 1456166941841

Target //tools:test up-to-date:
bazel-genfiles/tools/foo
INFO: Elapsed time: 1.815s, Critical Path: 0.02s

When I specify --workspace_status_command manually, it works as expected:

$ bazel clean && bazel build -s :test --workspace_status_command tools/buildstamp/get_workspace_status
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
INFO: Found 1 target...

//tools:test [action 'Executing genrule //tools:test']

(cd /private/var/tmp/_bazel_shimin/3a4c5c5e79ba6f8a3fd88b3eacc43127/universe2 &&
exec env -
PATH=/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/usr/local/opt/openssl/bin:/opt/local/bin:/opt/local/sbin:/Users/shimin/local/bin:/Users/shimin/Library/Haskell/bin:/Users/shimin/bin:/usr/local/aws/bin:/Users/shimin/gopath/bin:/Users/shimin/gopath/bin
TMPDIR=/var/folders/gz/71h5bskj4x94m63ddqbc3jpw0000gq/T/
/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; touch bazel-out/local_darwin-fastbuild/genfiles/tools/foo; for file in bazel-out/*-status.txt; do echo "$file: "; cat $file; echo; done')
INFO: From Executing genrule //tools:test:
bazel-out/stable-status.txt:
BUILD_EMBED_LABEL
BUILD_HOST canard.local
BUILD_USER shimin
bazel-out/volatile-status.txt:
BUILD_TIMESTAMP 1456167095265
BUILD_SCM_REVISION 74bbe13197c18d8defb74885bf3d6ec3e71ae8c1
BUILD_SCM_STATUS Modified

Target //tools:test up-to-date:
bazel-genfiles/tools/foo
INFO: Elapsed time: 1.750s, Critical Path: 0.06s

This is my bazel version:

Build label: 0.1.5
Build target: bazel-out/local_darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Feb 9 19:11:16 2016 (1455045076)
Build timestamp: 1455045076
Build timestamp as int: 1455045076

Am I doing something wrong?

—
Reply to this email directly or view it on GitHub
#216 (comment).

[guoshimin]

I went ahead and filed an issue: #964

[htuch]

I think we just hit a case in Envoy where we want this (@mattklein123). We have a Bazel project A that consumes another Bazel project B as a local_repository in a git submodule. We currently use a genrule in B to compute the git SHA, but that genrule has as its sources glob([".git/**"]), which doesn't make any sense with git submodules which set .git to a redirect file instead of directory. Instead, we need to get git status information relative to A's .git. I think if we can just use the workspace version provided by Bazel it would be much simpler.

[htuch]

@ulfjack can we get this reopened? We're not seeing any reasonable way (short of resorting to a repository_rule and sticking the git SHA in an environment variable pre-build) to embed the git SHA1 in C++ source with Bazel 0.4.5? See envoyproxy/envoy#841.

[ittaiz]

Isn't there a build stamping mechanism? I remember there was a blog post about it. Might be old news to you.

…

On Wed, 26 Apr 2017 at 16:57 htuch ***@***.***> wrote: @ulfjack <https://github.com/ulfjack> can we get this reopened? We're not seeing any reasonable way (short of resorting to a repository_rule and sticking the git SHA in an environment variable pre-build) to embed the git SHA1 in C++ source with Bazel 0.4.5? See envoyproxy/envoy#841 <envoyproxy/envoy#841>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#216 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF-N3jbzINMb63T_8NZ6ulYRPsRFrks5rz01JgaJpZM4Et9A6> .

[htuch]

As I look into this, it seems we have the mechanisms, but no documentation on how to use it in the Build Encylopedia. Can we get some official docs on this? It's a huge time sink to have to reinvent things that are not properly documented. E.g. there is https://groups.google.com/forum/#!topic/bazel-discuss/hqyMFjEpJgM.

[kchodorow]

@michelleirvine, who's on a mission to improve the docs.

Michelle, I wrote a thing about stamping here, but the docs are a bit light on it.

[htuch]

Thanks @kchodorow. There's also https://groups.google.com/forum/m/#!topic/bazel-discuss/9kqLmOq_m_U which describes how to tie this into the C++ as a source artifact.

[htuch]

For the record, the steps described by Brian Silverman in https://groups.google.com/forum/m/#!topic/bazel-discuss/9kqLmOq_m_U seem to work, but there's nothing in the public docs that would lead a developer to believe that. For example linkstamp for cc_library is not documented, nor is --workspace_status_command or the reference workspace status command tools/buildstamp/get_workspace_status (which is not the default).

I've created #2893 to track, we can continue the discussion there.

[jayeye]

Here is my solution, at least for python: https://gist.github.com/jayeye/14c91816d10d5b899e1baaaaa9ba4848

[promiseofcake]

@jayeye, but that doesn't update the commit stamp when the bi.py binary changes right?

$ git rev-parse head
1366b5afa643b5dd0c93c7a0ca899aed22075ef8

$ cat bazel-out/volatile-status.txt
BUILD_SCM_REVISION 1366b5afa643b5dd0c93c7a0ca899aed22075ef8
BUILD_SCM_STATUS Modified
BUILD_TIMESTAMP 1513355460235

$ bazel run //:bi
INFO: Analysed target //:bi.
INFO: Found 1 target...
Target //:bi up-to-date:
  bazel-bin/bi
INFO: Elapsed time: 0.115s, Critical Path: 0.02s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/bi
{'BUILD_HOST': 'foo',
 'BUILD_SCM_REVISION': '3abe596830f085470308c5c2fd49e4a4743dad27',
 'BUILD_SCM_STATUS': 'Modified',
 'BUILD_TIMESTAMP': '1513354916425',
 'BUILD_USER': 'lucas'}