Make Handling of the mbedTLS Make File Non-Unique

[gerickson]

Problem

Generally, all project GNU make files are "active" and configured and employ conditionals within the make file itself.

However, looking at configure.ac:

if test "${nl_with_mbedtls}" = "internal"; then
AC_CONFIG_FILES([third_party/mbedtls/Makefile])
fi

the GNU make file for mbedTLS is a bit of a special snowflake.

Summary of Changes

Move conditionality of mbedTLS from configure.ac to the mbedTLS wrapper make file itself.
Ensure that the mbedTLS wrapper make file is always in the distribution so that make distcheck works.

Fixes #482

[woody-apple]

@gerickson Build failure here: https://travis-ci.com/github/project-chip/connectedhomeip/jobs/322638692

[gerickson]

@gerickson Build failure here: https://travis-ci.com/github/project-chip/connectedhomeip/jobs/322638692

I am unable to replicate this on either:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% scripts/build/bootstrap.sh
...
% scripts/build/distribution_check.sh
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================
make: Leaving directory '/Users/gerickson/Source/github.com/gerickson/connectedhomeip/build/default'

or:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% cd ../build/connectedhomeip
% ../../connectedhomeip/configure
% make distcheck
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

[gerickson]

@gerickson Build failure here: https://travis-ci.com/github/project-chip/connectedhomeip/jobs/322638692

I am unable to replicate this on either:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% scripts/build/bootstrap.sh
...
% scripts/build/distribution_check.sh
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================
make: Leaving directory '/Users/gerickson/Source/github.com/gerickson/connectedhomeip/build/default'

or:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% cd ../build/connectedhomeip
% ../../connectedhomeip/configure
% make distcheck
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

I also ran:

% integrations/docker/images/chip-build/run.sh bash -c "mkdir -p build/default && cd build/default && ../../bootstrap-configure && make distcheck" | & tee -a really-docker-bootstrap-distcheck.log
...
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

successfully.

[gerickson]

@gerickson Build failure here: https://travis-ci.com/github/project-chip/connectedhomeip/jobs/322638692

I am unable to replicate this on either:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% scripts/build/bootstrap.sh
...
% scripts/build/distribution_check.sh
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================
make: Leaving directory '/Users/gerickson/Source/github.com/gerickson/connectedhomeip/build/default'

or:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% cd ../build/connectedhomeip
% ../../connectedhomeip/configure
% make distcheck
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

I also ran:

% integrations/docker/images/chip-build/run.sh bash -c "mkdir -p build/default && cd build/default && ../../bootstrap-configure && make distcheck" | & tee -a really-docker-bootstrap-distcheck.log
...
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

successfully.

Replicated the failure with make -f Makefile-bootstrap clean-repos first. A prefetched copy of mbedTLS was causing my local builds to succeed where they should not have been.

[gerickson]

@gerickson Build failure here: https://travis-ci.com/github/project-chip/connectedhomeip/jobs/322638692

I am unable to replicate this on either:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% scripts/build/bootstrap.sh
...
% scripts/build/distribution_check.sh
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================
make: Leaving directory '/Users/gerickson/Source/github.com/gerickson/connectedhomeip/build/default'

or:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% cd ../build/connectedhomeip
% ../../connectedhomeip/configure
% make distcheck
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

I also ran:

% integrations/docker/images/chip-build/run.sh bash -c "mkdir -p build/default && cd build/default && ../../bootstrap-configure && make distcheck" | & tee -a really-docker-bootstrap-distcheck.log
...
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

successfully.

Replicated the failure with make -f Makefile-bootstrap clean-repos first. A prefetched copy of mbedTLS was causing my local builds to succeed where they should not have been.

OK, so the problem is that even thought mbedTLS make file is now conditioned with if CHIP_WITH_MBEDTLS_INTERNAL ... endif, the following still gets emitted in the make file:

distdir: $(DISTFILES)

DISTFILES = ... $(DIST_SOURCES) ...

DIST_SOURCES = $(am__libmbedtls_a_SOURCES_DIST)

am__libmbedtls_a_SOURCES_DIST = repo/library/aes.c ...

KABOOM. So, it would appear that mbedtls really is a special snowflake unlike any other. We have:

1. LwIP, but that's built directly with a project-local make file (ignoring mbedTLS's cmake) and checked in.
2. nlunit-test, nlio, nlfaultinjection, and nlassert, but they are not checked in and are not built with a project-local make file.
3. In openweave-core, we have openssl which is built indirectly with a project-local make file (using OpenSSL's configure and make files) and checked in.

@pan-apple, was there a reason we didn't follow approach (3) above and instead directly built mbedTLS sources with our own project-local make file?

I think I can work around this but curious on our initial approach.

[gerickson]

@gerickson Build failure here: https://travis-ci.com/github/project-chip/connectedhomeip/jobs/322638692

I am unable to replicate this on either:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% scripts/build/bootstrap.sh
...
% scripts/build/distribution_check.sh
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================
make: Leaving directory '/Users/gerickson/Source/github.com/gerickson/connectedhomeip/build/default'

or:

% git checkout user/gerickson/github-issue-482
% ./bootstrap
% cd ../build/connectedhomeip
% ../../connectedhomeip/configure
% make distcheck
...
rm -f Makefile
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

I also ran:

% integrations/docker/images/chip-build/run.sh bash -c "mkdir -p build/default && cd build/default && ../../bootstrap-configure && make distcheck" | & tee -a really-docker-bootstrap-distcheck.log
...
if test -d "chip-0.1.0"; then find "chip-0.1.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "chip-0.1.0" || { sleep 5 && rm -rf "chip-0.1.0"; }; else :; fi
============================================
chip-0.1.0 archives ready for distribution: 
chip-0.1.0.tar.gz
============================================

successfully.

Replicated the failure with make -f Makefile-bootstrap clean-repos first. A prefetched copy of mbedTLS was causing my local builds to succeed where they should not have been.

OK, so the problem is that even thought mbedTLS make file is now conditioned with if CHIP_WITH_MBEDTLS_INTERNAL ... endif, the following still gets emitted in the make file:

distdir: $(DISTFILES)

DISTFILES = ... $(DIST_SOURCES) ...

DIST_SOURCES = $(am__libmbedtls_a_SOURCES_DIST)

am__libmbedtls_a_SOURCES_DIST = repo/library/aes.c ...

KABOOM. So, it would appear that mbedtls really is a special snowflake unlike any other. We have:

1. LwIP, but that's built directly with a project-local make file (ignoring mbedTLS's cmake) and checked in.
2. nlunit-test, nlio, nlfaultinjection, and nlassert, but they are not checked in and are not built with a project-local make file.
3. In openweave-core, we have openssl which is built indirectly with a project-local make file (using OpenSSL's configure and make files) and checked in.

@pan-apple, was there a reason we didn't follow approach (3) above and instead directly built mbedTLS sources with our own project-local make file?

I think I can work around this but curious on our initial approach.

Got 'em:

diff --git a/third_party/mbedtls/Makefile.am b/third_party/mbedtls/Makefile.am
index 4f095e2..f9e8aec 100644
--- a/third_party/mbedtls/Makefile.am
+++ b/third_party/mbedtls/Makefile.am
@@ -27,7 +27,10 @@ if CHIP_WITH_MBEDTLS_INTERNAL
 
 lib_LIBRARIES                                                                  = libmbedtls.a
 
-libmbedtls_a_SOURCES                                                           = \
+# Never include the mbedTLS sources in a distribution; they always will get pulled from upstream
+# via git operations, hence the 'nodist_' qualifier.
+
+nodist_libmbedtls_a_SOURCES                                                    = \
     repo/library/aes.c                      \
     repo/library/pem.c                      \
     repo/library/aesni.c                    \
@@ -106,7 +109,7 @@ libmbedtls_a_SOURCES                                                           =
     $(NULL)
 
 if !CONFIG_DEVICE_LAYER
-libmbedtls_a_SOURCES                                                          += \
+nodist_libmbedtls_a_SOURCES                                                   += \
     repo/library/entropy_poll.c             \
     repo/library/timing.c                   \
     $(NULL)