gh-99108: Add HACL* Blake2 implementation to hashlib #119316

msprotz · 2024-05-21T15:52:23Z

This replaces the existing hashlib module with a single implementation that uses HACL*'s Blake2b/Blake2s implementations. We added support for all the modes exposed by the Python API, including tree hashing, leaf nodes, and so on. We ported and merged all of these changes upstream in HACL*, added test vectors based on Python's existing implementation, and exposed everything needed for hashlib.

This is a preliminary PR but I wanted to gather early feedback from @gpshead.

What remains to be done on this PR:

add support for HACL* 128-bit/256-bit vectorized implementations
benchmark on e.g. a recent Apple ARM to measure perf improvement relative to libb2 (which only has Intel AVX2 support)
make a call as to whether I should restore libb2 support or whether Python would be happy with simply bundling HACL*'s vectorized implementations and drop support for building against libb2

This is joint work with @R1kM

Issue: Replace built-in hashlib with verified implementations from HACL* #99108

cpython-cla-bot · 2024-05-21T15:52:26Z

All commit authors signed the Contributor License Agreement.

msprotz · 2024-05-21T15:57:39Z

@gpshead if you can trigger a round of CI that would be most helpful. Thanks!

msprotz · 2024-06-20T23:32:56Z

So I computed some benchmarks on a variety of machines, comparing HACL's performance with libb2. This is using the benchmarking infrastructure from HACL-packages: https://github.com/cryspen/hacl-packages/blob/main/benchmarks/blake.cc

To read these benchmarks, bear in mind that:

libb2 automatically picks the "best" implementation (via CPUID)
libb2 has portable C code (which is what gets picked on ARM), and AVX2 code (which is what gets picked for the benchmarks below on Intel machines)
HACL* never does runtime CPU detection
HACL's Blake2s comes in regular (a.k.a. "32") version, and a 128-bit version (that runs on either NEON or AVX)
HACL's Blake2b comes in regular (a.k.a. "32") version, and a 256-bit version (that runs on AVX2)
we benchmark various input sizes

M3 Max (ARM)

HACL_blake2b_32_oneshot/0                   121 ns          120 ns      5839173
HACL_blake2b_32_oneshot/16                  121 ns          120 ns      5758236
HACL_blake2b_32_oneshot/256                 226 ns          225 ns      3098757
HACL_blake2b_32_oneshot/4096               3413 ns         3396 ns       206182
HACL_blake2b_32_oneshot/65536             54123 ns        53818 ns        12987
HACL_blake2b_32_oneshot/1048576          867868 ns       863094 ns          815
HACL_blake2b_32_oneshot/16777216       13993602 ns     13929400 ns           50
libb2_blake2b_oneshot/0                     148 ns          147 ns      4757374
libb2_blake2b_oneshot/16                    146 ns          145 ns      4812849
libb2_blake2b_oneshot/256                   276 ns          275 ns      2551783
libb2_blake2b_oneshot/4096                 4224 ns         4202 ns       167284
libb2_blake2b_oneshot/65536               67478 ns        67179 ns        10367
libb2_blake2b_oneshot/1048576           1079958 ns      1073158 ns          644
libb2_blake2b_oneshot/16777216         17322373 ns     17220756 ns           41
HACL_blake2s_32_oneshot/0                   101 ns          100 ns      6901311
HACL_blake2s_32_oneshot/16                  101 ns          101 ns      6980107
HACL_blake2s_32_oneshot/256                 370 ns          368 ns      1894196
HACL_blake2s_32_oneshot/4096               5771 ns         5740 ns       121353
HACL_blake2s_32_oneshot/65536             91417 ns        90835 ns         7762
HACL_blake2s_32_oneshot/1048576         1478896 ns      1469956 ns          475
HACL_blake2s_32_oneshot/16777216       23471740 ns     23337633 ns           30
HACL_blake2s_vec128_oneshot/0               177 ns          176 ns      4077567
HACL_blake2s_vec128_oneshot/16              186 ns          185 ns      3864926
HACL_blake2s_vec128_oneshot/256             685 ns          682 ns      1044433
HACL_blake2s_vec128_oneshot/4096          10868 ns        10819 ns        65872
HACL_blake2s_vec128_oneshot/65536        174311 ns       173302 ns         4041
HACL_blake2s_vec128_oneshot/1048576     2795034 ns      2779949 ns          255
HACL_blake2s_vec128_oneshot/16777216   44968086 ns     44750562 ns           16
libb2_blake2s_oneshot/0                     177 ns          175 ns      3965174
libb2_blake2s_oneshot/16                    176 ns          175 ns      3977815
libb2_blake2s_oneshot/256                   689 ns          684 ns      1034218
libb2_blake2s_oneshot/4096                10729 ns        10686 ns        65111
libb2_blake2s_oneshot/65536              171900 ns       170899 ns         4119
libb2_blake2s_oneshot/1048576           2762123 ns      2746686 ns          255
libb2_blake2s_oneshot/16777216         44150450 ns     43912125 ns           16

interpretation:

HACL* 20% faster than libb2 (Blake2b)
HACL* 47% faster than libb2 (Blake2s) -- as long as you don't use HACL's NEON version (which suffers from bad latency for vector shift instructions, see BLAKE2b NEON suffers poor performance on ARMv8/Aarch64 with Cortex-A57 weidai11/cryptopp#367)

Intel AVX2 (8-Core Intel Core i9, Macbook Pro, 2019)

HACL_blake2b_32_oneshot/0                   189 ns          188 ns      3681808
HACL_blake2b_32_oneshot/16                  192 ns          192 ns      3721267
HACL_blake2b_32_oneshot/256                 334 ns          334 ns      2107862
HACL_blake2b_32_oneshot/4096               4774 ns         4772 ns       151013
HACL_blake2b_32_oneshot/65536             76518 ns        76485 ns         9405
HACL_blake2b_32_oneshot/1048576         1180256 ns      1180031 ns          609
HACL_blake2b_32_oneshot/16777216       18999771 ns     18996081 ns           37
HACL_blake2b_vec256_oneshot/0               144 ns          144 ns      5012424
HACL_blake2b_vec256_oneshot/16              142 ns          142 ns      4864185
HACL_blake2b_vec256_oneshot/256             247 ns          246 ns      2801502
HACL_blake2b_vec256_oneshot/4096           3457 ns         3456 ns       202513
HACL_blake2b_vec256_oneshot/65536         54684 ns        54660 ns        12402
HACL_blake2b_vec256_oneshot/1048576      878910 ns       878574 ns          793
HACL_blake2b_vec256_oneshot/16777216   14755193 ns     14751612 ns           49
libb2_blake2b_oneshot/0                     162 ns          162 ns      4324084
libb2_blake2b_oneshot/16                    175 ns          175 ns      4303296
libb2_blake2b_oneshot/256                   282 ns          282 ns      2425427
libb2_blake2b_oneshot/4096                 4318 ns         4313 ns       176491
libb2_blake2b_oneshot/65536               63438 ns        63411 ns        10497
libb2_blake2b_oneshot/1048576            987715 ns       987456 ns          708
libb2_blake2b_oneshot/16777216         16262424 ns     16262024 ns           41
HACL_blake2s_32_oneshot/0                   137 ns          136 ns      5224504
HACL_blake2s_32_oneshot/16                  133 ns          133 ns      5199706
HACL_blake2s_32_oneshot/256                 447 ns          446 ns      1590439
HACL_blake2s_32_oneshot/4096               6792 ns         6789 ns       104370
HACL_blake2s_32_oneshot/65536            103446 ns       103393 ns         6687
HACL_blake2s_32_oneshot/1048576         1770925 ns      1769228 ns          403
HACL_blake2s_32_oneshot/16777216       27090139 ns     27082875 ns           24
HACL_blake2s_vec128_oneshot/0               121 ns          121 ns      5760132
HACL_blake2s_vec128_oneshot/16              127 ns          127 ns      5784359
HACL_blake2s_vec128_oneshot/256             378 ns          378 ns      1834978
HACL_blake2s_vec128_oneshot/4096           5727 ns         5725 ns       119742
HACL_blake2s_vec128_oneshot/65536         88395 ns        88384 ns         7822
HACL_blake2s_vec128_oneshot/1048576     1462898 ns      1462569 ns          490
HACL_blake2s_vec128_oneshot/16777216   24048063 ns     24040862 ns           29
libb2_blake2s_oneshot/0                     102 ns          102 ns      6673913
libb2_blake2s_oneshot/16                    105 ns          105 ns      6419663
libb2_blake2s_oneshot/256                   367 ns          367 ns      1891554
libb2_blake2s_oneshot/4096                 5540 ns         5537 ns       121671
libb2_blake2s_oneshot/65536               88250 ns        88211 ns         7868
libb2_blake2s_oneshot/1048576           1326051 ns      1325750 ns          519
libb2_blake2s_oneshot/16777216         22597565 ns     22584281 ns           32

interpretation:

for Blake2b, HACL*/AVX2 10% faster than libb2, HACL*/portable C 16% slower than libb2
for Blake2s, HACL*/AVX2 10% slower than libb2, HACL*/portable C 20% slower than libb2 (this is surprising, I'm seeing equal performance using another API, so this may simply be some missing static inlines on hot paths)

AVX2 desktop machine, Haswell

HACL_blake2b_32_oneshot/0                   211 ns          211 ns      3096750
HACL_blake2b_32_oneshot/16                  212 ns          212 ns      3300382
HACL_blake2b_32_oneshot/256                 388 ns          388 ns      1803603
HACL_blake2b_32_oneshot/4096               5638 ns         5638 ns       124313
HACL_blake2b_32_oneshot/65536             89905 ns        89903 ns         7774
HACL_blake2b_32_oneshot/1048576         1438161 ns      1438056 ns          488
HACL_blake2b_32_oneshot/16777216       22998077 ns     22997469 ns           30
HACL_blake2b_vec256_oneshot/0               168 ns          168 ns      4145151
HACL_blake2b_vec256_oneshot/16              168 ns          168 ns      4172085
HACL_blake2b_vec256_oneshot/256             309 ns          309 ns      2263628
HACL_blake2b_vec256_oneshot/4096           4507 ns         4507 ns       155137
HACL_blake2b_vec256_oneshot/65536         72069 ns        72068 ns         9748
HACL_blake2b_vec256_oneshot/1048576     1149298 ns      1149303 ns          610
HACL_blake2b_vec256_oneshot/16777216   18438313 ns     18436309 ns           38
libb2_blake2b_oneshot/0                     190 ns          190 ns      3670823
libb2_blake2b_oneshot/16                    194 ns          194 ns      3618244
libb2_blake2b_oneshot/256                   353 ns          353 ns      1984245
libb2_blake2b_oneshot/4096                 5225 ns         5225 ns       134636
libb2_blake2b_oneshot/65536               82645 ns        82642 ns         8502
libb2_blake2b_oneshot/1048576           1328695 ns      1328701 ns          529
libb2_blake2b_oneshot/16777216         21131069 ns     21130208 ns           33
HACL_blake2s_32_oneshot/0                   172 ns          172 ns      4080561
HACL_blake2s_32_oneshot/16                  174 ns          174 ns      4024023
HACL_blake2s_32_oneshot/256                 605 ns          605 ns      1154699
HACL_blake2s_32_oneshot/4096               9210 ns         9210 ns        76047
HACL_blake2s_32_oneshot/65536            147054 ns       147054 ns         4744
HACL_blake2s_32_oneshot/1048576         2354038 ns      2354016 ns          296
HACL_blake2s_32_oneshot/16777216       37686817 ns     37686992 ns           19
HACL_blake2s_vec128_oneshot/0               150 ns          150 ns      4686347
HACL_blake2s_vec128_oneshot/16              151 ns          151 ns      4628570
HACL_blake2s_vec128_oneshot/256             513 ns          513 ns      1367690
HACL_blake2s_vec128_oneshot/4096           7748 ns         7748 ns        90207
HACL_blake2s_vec128_oneshot/65536        123277 ns       123276 ns         5676
HACL_blake2s_vec128_oneshot/1048576     1971694 ns      1971703 ns          355
HACL_blake2s_vec128_oneshot/16777216   31514541 ns     31512743 ns           22
libb2_blake2s_oneshot/0                     151 ns          151 ns      4651126
libb2_blake2s_oneshot/16                    154 ns          154 ns      4542519
libb2_blake2s_oneshot/256                   542 ns          542 ns      1293258
libb2_blake2s_oneshot/4096                 8381 ns         8381 ns        83415
libb2_blake2s_oneshot/65536              133872 ns       133869 ns         5217
libb2_blake2s_oneshot/1048576           2140551 ns      2140560 ns          327
libb2_blake2s_oneshot/16777216         34272522 ns     34271712 ns           20

interpretation:

HACL*/AVX2 13% faster than libb2, HACL*/portable C 8% slower than libb2 (Blake2b)
HACL*/AVX2 8% faster than libb2, HACL*/portable C 19% slower than libb2 (Blake2s)

This leaves us with several options. I would love to have your opinion @gpshead

CPython loses the ability to build against libb2, and simply packages HACL's portable versions, which offer a significant performance boost on ARM but are slightly slower on Intel.

Pros: CPython build is simplified, drops an external dependency, compelling story with simple C code everywhere
Cons: slight performance impact on Intel

CPython loses the ability to build against libb2, and packages HACL's portable versions and vectorized (128-bit and 256-bit) versions, to be enabled on Intel and AVX (but not on NEON, see issue with high-latency vector shift, above)

Pros: CPython increases Blake2 performance across the board (Intel and ARM), + build simplification and loss of an extra dependency
Cons: I need to author a CPU detection layer and fiddle with the CPython build (not a big deal, happy to do so)

CPython maintains HACL and libb2 side by side

Pros: none
Cons: duplication of code, build nightmare, and need to deal with two different APIs for the bindings from C to the Python module.

Please share thoughts. Thanks!

CC @R1kM who provided considerable help with libb2 benchmarking

gpshead · 2024-06-21T05:00:39Z

I'd personally go for "2. CPython loses the ability to build against libb2, and packages HACL's portable versions and vectorized".

But even if we only get to option 1. without the cpu detection and just using the portable version, it still seems reasonable. The differences are not huge. And moving from there to option 2 would still be a later followup possibility.

Both of those reduce our dependencies.

msprotz · 2024-06-25T11:12:29Z

Happy to do option 2, however, could you help me out a little bit with the build? I've pushed the new files to my branch, but I need to encode in the build system that:

Hacl_Hash_Blake2s_Simd128.c should be built with -mavx, and only if the toolchain supports it (as an approximation, this criterion should be "toolchain is Intel") -- otherwise, just don't build it
Hacl_Hash_Blake2b_Simd256.c should be built with -mavx2, and only if the toolchain supports it (same) -- otherwise, just don't build it

I don't know if Setup.stdlib.in supports this, and if so, how to encode it. Any pointers appreciated. Thanks!

eli-schwartz · 2024-06-25T20:07:21Z

cpython/Modules/Setup.stdlib.in

Lines 11 to 13 in 769aea3

    
           # * @MODULE_{NAME}_TRUE@ is removed when configure detects all build 
        
           #   dependencies for a module. Otherwise the template variable is replaced 
        
           #   by a comment "#" and the module is skipped.

This is a generalization of a simpler fact: Setup.stdlib.in is processed as an AC_CONFIG_FILES, which means you can do any @SOMETHING@ replacement you could do in the Makefile.

cpython/Modules/makesetup

Lines 156 to 165 in 769aea3

    
           case $arg in 
        
           -framework)	libs="$libs $arg"; skip=libs; 
        
           	        # OSX/OSXS/Darwin framework link cmd 
        
           		;; 
        
           -[IDUCfF]*)	cpps="$cpps $arg";; 
        
           -Xcompiler)	skip=cpps;; 
        
           -Xlinker)	libs="$libs $arg"; skip=libs;; 
        
           -rpath)		libs="$libs $arg"; skip=libs;; 
        
           --rpath)	libs="$libs $arg"; skip=libs;; 
        
           -[A-Zl]*)	libs="$libs $arg";;

indicates you need to update a match for -m*) before you can use -mavx in a Setup file. On the other hand, you can add arbitrary flags by reference:

cpython/Modules/makesetup

Lines 178 to 182 in 769aea3

    
           \$\(*_CFLAGS\))		cpps="$cpps $arg";; 
        
           \$\(*_INCLUDES\))	cpps="$cpps $arg";; 
        
           \$\(*_LIBS\))		libs="$libs $arg";; 
        
           \$\(*_LDFLAGS\))	libs="$libs $arg";; 
        
           \$\(*_RPATH\))		libs="$libs $arg";;

so e.g. $(AVX_CFLAGS).

There doesn't currently seem to be a way to build a module where some of its files are built with certain CFLAGS and some are built with other CFLAGS, which indicates you need to pass -mavx* to all files in the blake module, which may be a problem

In commit d777790, @gpshead updated the sha* code from HACL* to build as a standalone static library instead, which then gets linked to in Setup.stdlib.in by specifying a filepath to a *.a file, which makesetup uses to assume it needs to depend on a rule defined in Makefile.pre.in

I think that last approach is what you want to do as that gives you the freedom to define HACL* to build however you need it to, and then Setup.stdlib.in just needs to link to HACL* rather than define the fiddly bits of how to compile it.

eli-schwartz

At

cpython/Makefile.pre.in

Lines 216 to 219 in 769aea3

    
           # Internal static libraries 
        
           LIBMPDEC_A= Modules/_decimal/libmpdec/libmpdec.a 
        
           LIBEXPAT_A= Modules/expat/libexpat.a 
        
           LIBHACL_SHA2_A= Modules/_hacl/libHacl_Hash_SHA2.a

Add:

LIBHACL_BLAKE2_A= Modules/_hacl/libHacl_Hash_BLAKE2.a

After:

cpython/Makefile.pre.in

Lines 638 to 657 in 769aea3

    
           ########################################################################## 
        
           # hashlib's HACL* library 
        
           LIBHACL_SHA2_OBJS= \ 
        
                           Modules/_hacl/Hacl_Hash_SHA2.o 
        
           LIBHACL_HEADERS= \ 
        
                           Modules/_hacl/include/krml/FStar_UInt128_Verified.h \ 
        
                           Modules/_hacl/include/krml/FStar_UInt_8_16_32_64.h \ 
        
                           Modules/_hacl/include/krml/fstar_uint128_struct_endianness.h \ 
        
                           Modules/_hacl/include/krml/internal/target.h \ 
        
                           Modules/_hacl/include/krml/lowstar_endianness.h \ 
        
                           Modules/_hacl/include/krml/types.h \ 
        
           		Modules/_hacl/Hacl_Streaming_Types.h \ 
        
                           Modules/_hacl/python_hacl_namespaces.h 
        
           LIBHACL_SHA2_HEADERS= \ 
        
                           Modules/_hacl/Hacl_Hash_SHA2.h \ 
        
                           Modules/_hacl/internal/Hacl_Hash_SHA2.h \ 
        
           		$(LIBHACL_HEADERS)

Add:

LIBHACL_BLAKE2_AVX_OBJ = Modules/_hacl/Hacl_Hash_Blake2s_Simd128.o
LIBHACL_BLAKE2_AVX2_OBJ = Modules/_hacl/Hacl_Hash_Blake2s_Simd256.o
LIBHACL_BLAKE2_OBJS= \
                Modules/_hacl/Hacl_Hash_Blake2s.o \
                Modules/_hacl/Hacl_Hash_Blake2b.o \
                @LIBHACL_BLAKE2_AVX_OBJ@ \
                @LIBHACL_BLAKE2_AVX2_OBJ@ \
                Modules/_hacl/Lib_Memzero0.o

(and teach configure.ac to replace these @XXX@ with $(XXX) if and only if they are supported by the compiler)

at:

cpython/Makefile.pre.in

Lines 1332 to 1340 in 769aea3

    
           # Build HACL* static libraries for hashlib: libHacl_Hash_SHA2.a 
        
           LIBHACL_CFLAGS=-I$(srcdir)/Modules/_hacl/include -D_BSD_SOURCE -D_DEFAULT_SOURCE $(PY_STDMODULE_CFLAGS) $(CCSHARED) 
        
           Modules/_hacl/Hacl_Hash_SHA2.o: $(srcdir)/Modules/_hacl/Hacl_Hash_SHA2.c $(LIBHACL_SHA2_HEADERS) 
        
           	$(CC) -c $(LIBHACL_CFLAGS) -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_SHA2.c 
        
           $(LIBHACL_SHA2_A): $(LIBHACL_SHA2_OBJS) 
        
           	-rm -f $@ 
        
           	$(AR) $(ARFLAGS) $@ $(LIBHACL_SHA2_OBJS)

Add:

LIBHACL_BLAKE2_CFLAGS=-I$(srcdir)/Modules/_hacl/ -I$(srcdir)/Modules/_hacl/include -D_BSD_SOURCE -D_DEFAULT_SOURCE

Modules/_hacl/Hacl_Hash_Blake2s.o: $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2s.c $(LIBHACL_BLAKE2_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2s.c

Modules/_hacl/Hacl_Hash_Blake2b.o: $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2b.c $(LIBHACL_BLAKE2_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2b.c

Modules/_hacl/Hacl_Hash_Blake2s_Simd128.o: $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2s_Simd128.c $(LIBHACL_BLAKE2_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -mavx -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2s_Simd128.c

Modules/_hacl/Hacl_Hash_Blake2s_Simd256.o: $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2s_Simd256.c $(LIBHACL_BLAKE2_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -mavx2 -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_Blake2s_Simd256.c

Modules/_hacl/Lib_Memzero0.o: $(srcdir)/Modules/_hacl/Lib_Memzero0.c $(LIBHACL_BLAKE2_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -o $@ $(srcdir)/Modules/_hacl/Lib_Memzero0.c

$(LIBHACL_BLAKE2_A): $(LIBHACL_BLAKE2_OBJS)
	-rm -f $@
	$(AR) $(ARFLAGS) $@ $(LIBHACL_BLAKE2_OBJS)

Modules/Setup.stdlib.in

msprotz · 2024-06-25T22:28:29Z

Thanks @eli-schwartz that's super helpful. In another issue, someone mentioned that the .a file was causing complications #116043 so I have a pending PR #119320 to remove that. But I have a hunch you might be able to shed some light on the matter :-). Thanks!

eli-schwartz · 2024-06-25T23:03:09Z

I've never used freeze.py myself. I would guess that it's collecting compiled objects used to produce python and its extensions and thinks it needs the *.a files for that purpose, even though it's used to build extensions, and is simply a matter of it not being adapted as the Makefile evolves.

...

You might think it would be simple to just add $(LIBHACL_BLAKE2_OBJS) to Setup.stdlib.in instead -- makesetup supports object files in addition to static archives. Unfortunately it appears that makesetup supports it by replacing the .o at the end with a .c, and continuing as though you had originally specified .c files. Then it generates another compile rule.

We really want to use the compile rule from Makefile.pre.in since that allows specifying -mavx per-file...

…tdlib.in

msprotz · 2024-07-18T17:26:26Z

silly question, why expand every single object file into its own recipe instead of relying on a pattern rule?

…ASHLIB_HASHES appears twice as a #define in confdefs.h -- try to fix this

…erly by distinguishing simd128 from simd256

eli-schwartz · 2024-07-18T22:23:33Z

silly question, why expand every single object file into its own recipe instead of relying on a pattern rule?

The traditional reason for this is the necessity of supporting Make implementations that don't support the GNU pattern rule extension. Opinions vary about whether to support other Make implementations but the fact of the matter is that you can't even use modern 2024 editions of BSD make in bleeding edge BSD platforms and have those work; you're forced to install the GNU make port and execute gmake instead of make to build such projects. If depending on GNU specific features it is advised to rename Makefile to GNUmakefile to avoid confusing errors with BSD make and force the use of gmake.

Making such a change requires a bit more discussion than fits this PR, I bet.

msprotz · 2024-07-19T01:43:38Z

No worries, that's what I suspected but wanted to double-check. Thanks for your prompt response.

I'm putting the final touches for this, and then you'll have to double-check I haven't used any GNUisms -- all of the Makefiles I've written in recent years assume GNU make > 3.81.

bedevere-bot · 2024-08-13T05:40:24Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 53dfa81 🤖

The command will test the builders whose names match following regular expression: FreeBSD

The builders matched are:

AMD64 FreeBSD PR
AMD64 FreeBSD14 PR
AMD64 FreeBSD15 PR

msprotz · 2024-08-13T14:10:56Z

Thanks. FreeBSD is good now, and MacOS failures seem unrelated. Do you see anything else suspicious / worth investigating? (I don't.)

gpshead · 2024-08-13T21:10:53Z

I don't see any other issues. It won't totally surprise me if some more compiler flag tweaks pop up as necessary in the future on this under various people's configs given those freebsd bots (which should just be using a modern clang or gcc?) didn't like -mavx. they might also not like -mavx2 if run on VMs exposing a modern virtual hardware type but lets wait to find out from the wider world given our buildbots don't show it. I'm merging.

msprotz · 2024-08-13T21:46:32Z

Thanks everyone! I'll follow up with the validation / cosmetic changes soon.

bedevere-bot · 2024-08-13T22:00:53Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 3.x has failed when building commit 325e9b8.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#builders/15/builds/8331) and take a look at the build logs.
Check if the failure is related to this commit (325e9b8) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#builders/15/builds/8331

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 55, done.        
remote: Counting objects:   2% (1/45)        
remote: Counting objects:   4% (2/45)        
remote: Counting objects:   6% (3/45)        
remote: Counting objects:   8% (4/45)        
remote: Counting objects:  11% (5/45)        
remote: Counting objects:  13% (6/45)        
remote: Counting objects:  15% (7/45)        
remote: Counting objects:  17% (8/45)        
remote: Counting objects:  20% (9/45)        
remote: Counting objects:  22% (10/45)        
remote: Counting objects:  24% (11/45)        
remote: Counting objects:  26% (12/45)        
remote: Counting objects:  28% (13/45)        
remote: Counting objects:  31% (14/45)        
remote: Counting objects:  33% (15/45)        
remote: Counting objects:  35% (16/45)        
remote: Counting objects:  37% (17/45)        
remote: Counting objects:  40% (18/45)        
remote: Counting objects:  42% (19/45)        
remote: Counting objects:  44% (20/45)        
remote: Counting objects:  46% (21/45)        
remote: Counting objects:  48% (22/45)        
remote: Counting objects:  51% (23/45)        
remote: Counting objects:  53% (24/45)        
remote: Counting objects:  55% (25/45)        
remote: Counting objects:  57% (26/45)        
remote: Counting objects:  60% (27/45)        
remote: Counting objects:  62% (28/45)        
remote: Counting objects:  64% (29/45)        
remote: Counting objects:  66% (30/45)        
remote: Counting objects:  68% (31/45)        
remote: Counting objects:  71% (32/45)        
remote: Counting objects:  73% (33/45)        
remote: Counting objects:  75% (34/45)        
remote: Counting objects:  77% (35/45)        
remote: Counting objects:  80% (36/45)        
remote: Counting objects:  82% (37/45)        
remote: Counting objects:  84% (38/45)        
remote: Counting objects:  86% (39/45)        
remote: Counting objects:  88% (40/45)        
remote: Counting objects:  91% (41/45)        
remote: Counting objects:  93% (42/45)        
remote: Counting objects:  95% (43/45)        
remote: Counting objects:  97% (44/45)        
remote: Counting objects: 100% (45/45)        
remote: Counting objects: 100% (45/45), done.        
remote: Compressing objects:   3% (1/33)        
remote: Compressing objects:   6% (2/33)        
remote: Compressing objects:   9% (3/33)        
remote: Compressing objects:  12% (4/33)        
remote: Compressing objects:  15% (5/33)        
remote: Compressing objects:  18% (6/33)        
remote: Compressing objects:  21% (7/33)        
remote: Compressing objects:  24% (8/33)        
remote: Compressing objects:  27% (9/33)        
remote: Compressing objects:  30% (10/33)        
remote: Compressing objects:  33% (11/33)        
remote: Compressing objects:  36% (12/33)        
remote: Compressing objects:  39% (13/33)        
remote: Compressing objects:  42% (14/33)        
remote: Compressing objects:  45% (15/33)        
remote: Compressing objects:  48% (16/33)        
remote: Compressing objects:  51% (17/33)        
remote: Compressing objects:  54% (18/33)        
remote: Compressing objects:  57% (19/33)        
remote: Compressing objects:  60% (20/33)        
remote: Compressing objects:  63% (21/33)        
remote: Compressing objects:  66% (22/33)        
remote: Compressing objects:  69% (23/33)        
remote: Compressing objects:  72% (24/33)        
remote: Compressing objects:  75% (25/33)        
remote: Compressing objects:  78% (26/33)        
remote: Compressing objects:  81% (27/33)        
remote: Compressing objects:  84% (28/33)        
remote: Compressing objects:  87% (29/33)        
remote: Compressing objects:  90% (30/33)        
remote: Compressing objects:  93% (31/33)        
remote: Compressing objects:  96% (32/33)        
remote: Compressing objects: 100% (33/33)        
remote: Compressing objects: 100% (33/33), done.        
remote: Total 55 (delta 19), reused 13 (delta 12), pack-reused 10 (from 1)        
From https://github.com/python/cpython
 * branch            main       -> FETCH_HEAD
Note: checking out '325e9b8ef400b86fb077aa40d5cb8cec6e4df7bb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 325e9b8... gh-99108: Add HACL* Blake2 implementation to hashlib (GH-119316)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdecimal found; falling back to bundled libmpdecimal (deprecated and scheduled for removal in Python 3.15)

../Modules/_hacl/Lib_Memzero0.c: In function ‘Lib_Memzero0_memzero0’:
../Modules/_hacl/Lib_Memzero0.c:43:5: error: implicit declaration of function ‘explicit_bzero’ [-Werror=implicit-function-declaration]
     explicit_bzero(dst, len_);
     ^~~~~~~~~~~~~~
cc1: some warnings being treated as errors
make: *** [Makefile:1531: Modules/_hacl/Lib_Memzero0.o] Error 1
make: *** Waiting for unfinished jobs....

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:3180: clean-retain-profile] Error 1 (ignored)

bedevere-bot · 2024-08-13T22:01:55Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO + PGO 3.x has failed when building commit 325e9b8.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#builders/96/builds/8086) and take a look at the build logs.
Check if the failure is related to this commit (325e9b8) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#builders/96/builds/8086

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 55, done.        
remote: Counting objects:   2% (1/44)        
remote: Counting objects:   4% (2/44)        
remote: Counting objects:   6% (3/44)        
remote: Counting objects:   9% (4/44)        
remote: Counting objects:  11% (5/44)        
remote: Counting objects:  13% (6/44)        
remote: Counting objects:  15% (7/44)        
remote: Counting objects:  18% (8/44)        
remote: Counting objects:  20% (9/44)        
remote: Counting objects:  22% (10/44)        
remote: Counting objects:  25% (11/44)        
remote: Counting objects:  27% (12/44)        
remote: Counting objects:  29% (13/44)        
remote: Counting objects:  31% (14/44)        
remote: Counting objects:  34% (15/44)        
remote: Counting objects:  36% (16/44)        
remote: Counting objects:  38% (17/44)        
remote: Counting objects:  40% (18/44)        
remote: Counting objects:  43% (19/44)        
remote: Counting objects:  45% (20/44)        
remote: Counting objects:  47% (21/44)        
remote: Counting objects:  50% (22/44)        
remote: Counting objects:  52% (23/44)        
remote: Counting objects:  54% (24/44)        
remote: Counting objects:  56% (25/44)        
remote: Counting objects:  59% (26/44)        
remote: Counting objects:  61% (27/44)        
remote: Counting objects:  63% (28/44)        
remote: Counting objects:  65% (29/44)        
remote: Counting objects:  68% (30/44)        
remote: Counting objects:  70% (31/44)        
remote: Counting objects:  72% (32/44)        
remote: Counting objects:  75% (33/44)        
remote: Counting objects:  77% (34/44)        
remote: Counting objects:  79% (35/44)        
remote: Counting objects:  81% (36/44)        
remote: Counting objects:  84% (37/44)        
remote: Counting objects:  86% (38/44)        
remote: Counting objects:  88% (39/44)        
remote: Counting objects:  90% (40/44)        
remote: Counting objects:  93% (41/44)        
remote: Counting objects:  95% (42/44)        
remote: Counting objects:  97% (43/44)        
remote: Counting objects: 100% (44/44)        
remote: Counting objects: 100% (44/44), done.        
remote: Compressing objects:   3% (1/33)        
remote: Compressing objects:   6% (2/33)        
remote: Compressing objects:   9% (3/33)        
remote: Compressing objects:  12% (4/33)        
remote: Compressing objects:  15% (5/33)        
remote: Compressing objects:  18% (6/33)        
remote: Compressing objects:  21% (7/33)        
remote: Compressing objects:  24% (8/33)        
remote: Compressing objects:  27% (9/33)        
remote: Compressing objects:  30% (10/33)        
remote: Compressing objects:  33% (11/33)        
remote: Compressing objects:  36% (12/33)        
remote: Compressing objects:  39% (13/33)        
remote: Compressing objects:  42% (14/33)        
remote: Compressing objects:  45% (15/33)        
remote: Compressing objects:  48% (16/33)        
remote: Compressing objects:  51% (17/33)        
remote: Compressing objects:  54% (18/33)        
remote: Compressing objects:  57% (19/33)        
remote: Compressing objects:  60% (20/33)        
remote: Compressing objects:  63% (21/33)        
remote: Compressing objects:  66% (22/33)        
remote: Compressing objects:  69% (23/33)        
remote: Compressing objects:  72% (24/33)        
remote: Compressing objects:  75% (25/33)        
remote: Compressing objects:  78% (26/33)        
remote: Compressing objects:  81% (27/33)        
remote: Compressing objects:  84% (28/33)        
remote: Compressing objects:  87% (29/33)        
remote: Compressing objects:  90% (30/33)        
remote: Compressing objects:  93% (31/33)        
remote: Compressing objects:  96% (32/33)        
remote: Compressing objects: 100% (33/33)        
remote: Compressing objects: 100% (33/33), done.        
remote: Total 55 (delta 18), reused 12 (delta 11), pack-reused 11 (from 1)        
From https://github.com/python/cpython
 * branch            main       -> FETCH_HEAD
Note: checking out '325e9b8ef400b86fb077aa40d5cb8cec6e4df7bb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 325e9b8... gh-99108: Add HACL* Blake2 implementation to hashlib (GH-119316)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdecimal found; falling back to bundled libmpdecimal (deprecated and scheduled for removal in Python 3.15)

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:3180: clean-retain-profile] Error 1 (ignored)
./Modules/_hacl/Lib_Memzero0.c: In function ‘Lib_Memzero0_memzero0’:
./Modules/_hacl/Lib_Memzero0.c:43:5: error: implicit declaration of function ‘explicit_bzero’ [-Werror=implicit-function-declaration]
     explicit_bzero(dst, len_);
     ^~~~~~~~~~~~~~
cc1: some warnings being treated as errors
make[2]: *** [Makefile:1531: Modules/_hacl/Lib_Memzero0.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:903: profile-gen-stamp] Error 2
make: *** [Makefile:915: profile-run-stamp] Error 2

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:3180: clean-retain-profile] Error 1 (ignored)

bedevere-bot · 2024-08-13T22:02:56Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO 3.x has failed when building commit 325e9b8.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#builders/507/builds/8024) and take a look at the build logs.
Check if the failure is related to this commit (325e9b8) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#builders/507/builds/8024

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 55, done.        
remote: Counting objects:   2% (1/45)        
remote: Counting objects:   4% (2/45)        
remote: Counting objects:   6% (3/45)        
remote: Counting objects:   8% (4/45)        
remote: Counting objects:  11% (5/45)        
remote: Counting objects:  13% (6/45)        
remote: Counting objects:  15% (7/45)        
remote: Counting objects:  17% (8/45)        
remote: Counting objects:  20% (9/45)        
remote: Counting objects:  22% (10/45)        
remote: Counting objects:  24% (11/45)        
remote: Counting objects:  26% (12/45)        
remote: Counting objects:  28% (13/45)        
remote: Counting objects:  31% (14/45)        
remote: Counting objects:  33% (15/45)        
remote: Counting objects:  35% (16/45)        
remote: Counting objects:  37% (17/45)        
remote: Counting objects:  40% (18/45)        
remote: Counting objects:  42% (19/45)        
remote: Counting objects:  44% (20/45)        
remote: Counting objects:  46% (21/45)        
remote: Counting objects:  48% (22/45)        
remote: Counting objects:  51% (23/45)        
remote: Counting objects:  53% (24/45)        
remote: Counting objects:  55% (25/45)        
remote: Counting objects:  57% (26/45)        
remote: Counting objects:  60% (27/45)        
remote: Counting objects:  62% (28/45)        
remote: Counting objects:  64% (29/45)        
remote: Counting objects:  66% (30/45)        
remote: Counting objects:  68% (31/45)        
remote: Counting objects:  71% (32/45)        
remote: Counting objects:  73% (33/45)        
remote: Counting objects:  75% (34/45)        
remote: Counting objects:  77% (35/45)        
remote: Counting objects:  80% (36/45)        
remote: Counting objects:  82% (37/45)        
remote: Counting objects:  84% (38/45)        
remote: Counting objects:  86% (39/45)        
remote: Counting objects:  88% (40/45)        
remote: Counting objects:  91% (41/45)        
remote: Counting objects:  93% (42/45)        
remote: Counting objects:  95% (43/45)        
remote: Counting objects:  97% (44/45)        
remote: Counting objects: 100% (45/45)        
remote: Counting objects: 100% (45/45), done.        
remote: Compressing objects:   3% (1/33)        
remote: Compressing objects:   6% (2/33)        
remote: Compressing objects:   9% (3/33)        
remote: Compressing objects:  12% (4/33)        
remote: Compressing objects:  15% (5/33)        
remote: Compressing objects:  18% (6/33)        
remote: Compressing objects:  21% (7/33)        
remote: Compressing objects:  24% (8/33)        
remote: Compressing objects:  27% (9/33)        
remote: Compressing objects:  30% (10/33)        
remote: Compressing objects:  33% (11/33)        
remote: Compressing objects:  36% (12/33)        
remote: Compressing objects:  39% (13/33)        
remote: Compressing objects:  42% (14/33)        
remote: Compressing objects:  45% (15/33)        
remote: Compressing objects:  48% (16/33)        
remote: Compressing objects:  51% (17/33)        
remote: Compressing objects:  54% (18/33)        
remote: Compressing objects:  57% (19/33)        
remote: Compressing objects:  60% (20/33)        
remote: Compressing objects:  63% (21/33)        
remote: Compressing objects:  66% (22/33)        
remote: Compressing objects:  69% (23/33)        
remote: Compressing objects:  72% (24/33)        
remote: Compressing objects:  75% (25/33)        
remote: Compressing objects:  78% (26/33)        
remote: Compressing objects:  81% (27/33)        
remote: Compressing objects:  84% (28/33)        
remote: Compressing objects:  87% (29/33)        
remote: Compressing objects:  90% (30/33)        
remote: Compressing objects:  93% (31/33)        
remote: Compressing objects:  96% (32/33)        
remote: Compressing objects: 100% (33/33)        
remote: Compressing objects: 100% (33/33), done.        
remote: Total 55 (delta 19), reused 13 (delta 12), pack-reused 10 (from 1)        
From https://github.com/python/cpython
 * branch            main       -> FETCH_HEAD
Note: checking out '325e9b8ef400b86fb077aa40d5cb8cec6e4df7bb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 325e9b8... gh-99108: Add HACL* Blake2 implementation to hashlib (GH-119316)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdecimal found; falling back to bundled libmpdecimal (deprecated and scheduled for removal in Python 3.15)

./Modules/_hacl/Lib_Memzero0.c: In function ‘Lib_Memzero0_memzero0’:
./Modules/_hacl/Lib_Memzero0.c:43:5: error: implicit declaration of function ‘explicit_bzero’ [-Werror=implicit-function-declaration]
     explicit_bzero(dst, len_);
     ^~~~~~~~~~~~~~
cc1: some warnings being treated as errors
make: *** [Makefile:1531: Modules/_hacl/Lib_Memzero0.o] Error 1
make: *** Waiting for unfinished jobs....

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:3180: clean-retain-profile] Error 1 (ignored)

msprotz · 2024-08-13T22:32:56Z

Oh and yes please keep me posted on RHEL7 and whether it matters or not. If yes, I'll do some more autoconf wizardry, though I can't say this is something I look forward to :-D

mhsmith · 2024-08-13T23:23:18Z

This build failure also happened on Android – #121595 (comment). It looks like it would be sufficient to define LINUX_NO_EXPLICIT_BZERO on that platform. Though I guess it wouldn't be too hard to add a generic autoconf check for whether the function exists.

picnixz · 2024-08-14T06:44:00Z

Thanks everyone! I'll follow up with the validation / cosmetic changes soon.

@msprotz Could you ping me when the PR is ready please? (or just bluntly ask for a review from me?) Thank you in advance.

encukou · 2024-08-14T08:15:24Z

Oh and yes please keep me posted on RHEL7 and whether it matters or not. I'll do some more autoconf wizardry, though I can't say this is something I look forward to :-D

RHEL7 doesn't matter, but if you don't do the icky wizardry patch, you're effectively leaving it to the maintainer of Android (or if we didn't have Android: some niche BSD that'll struggle to update to 3.14 in a few years).

mhsmith · 2024-08-14T12:52:30Z

This is partly my fault for not setting up the Android buildbot soon enough, so I'll submit a fix today – #99108 (comment).

asottile · 2024-08-17T19:18:14Z

👋 it looks like Modules/Setup was not updated as part of this patch (and so the deadsnakes builds are failing!)

the line looks like this:

Modules/Setup:#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c

I assume they should look similar to the hashing modules below them?

msprotz · 2024-08-19T02:18:53Z

What is Modules/Setup used for?

The build of blake2 is more complicated because some files can only be compiled if the toolchain supported them (see this PR, and notably the changes in configure.ac and Makefile.pre.in).

The logic should be pretty self-explanatory so hopefully you'll know how to fix this? Otherwise, happy to explain in more detail what's happening for the build of the new blake2 module. Thanks!

eli-schwartz · 2024-08-19T02:44:58Z

@msprotz the file Modules/Setup.stdlib.in was updated in this PR. It actually has a comment that says it isn't used by default "yet". :/

I think that Modules/Setup maybe just has to be updated exactly the same way as the former file?

msprotz · 2024-08-19T14:59:13Z

Strange! My local testing picked up my changes in Setup.stdlib.in so I'm not sure in what scenario Setup.stdlib.in is ignored in favor of Setup. But it's a small change so happy to submit a followup PR. See #123146

@R1kM

…119316) This replaces the existing hashlib Blake2 module with a single implementation that uses HACL\*'s Blake2b/Blake2s implementations. We added support for all the modes exposed by the Python API, including tree hashing, leaf nodes, and so on. We ported and merged all of these changes upstream in HACL\*, added test vectors based on Python's existing implementation, and exposed everything needed for hashlib. This was joint work done with @R1kM. See the PR for much discussion and benchmarking details. TL;DR: On many systems, 8-50% faster (!) than `libb2`, on some systems it appeared 10-20% slower than `libb2`.

erlend-aasland · 2024-08-23T06:36:25Z

Modules/Setup is not used by CPython, and it never will. Christian's plan was to replace Modules/Setup with the new Modules/Setup.*.in files. The *.in files are now used in the CPython build, the unfortunately, Modules/Setup still remains. I'd prefer it removed.

msprotz added 3 commits March 25, 2024 15:11

Refresh HACL*; update modules accordingly; fix namespacing

81b42f2

make regen-sbom

9514fb4

wip

b6b8ea9

bedevere-app bot mentioned this pull request May 21, 2024

Replace built-in hashlib with verified implementations from HACL* #99108

Open

msprotz added 2 commits May 21, 2024 08:56

Initial prototype of hashlib running atop HACL*'s blake2

94bb7aa

All hashlib tests pass

e8a3060

msprotz force-pushed the hacl_blake2 branch from 90532c0 to e8a3060 Compare May 21, 2024 15:56

msprotz added 2 commits June 24, 2024 13:09

Merge remote-tracking branch 'upstream/main' into hacl_blake2

3896560

Import more files, now to figure out the build system...

d9e5568

eli-schwartz reviewed Jun 25, 2024

View reviewed changes

Modules/Setup.stdlib.in Outdated Show resolved Hide resolved

No need to repeat those flags in configure -- they are set in Setup.s…

0d7a2ce

…tdlib.in

msprotz added 8 commits July 18, 2024 10:47

Try some AC stuff

b00af13

Merge remote-tracking branch 'upstream/main' into hacl_blake2

cd3782a

Every single AX_CHECK_COMPILE_FLAG returns false because PY_BUILTIN_H…

36f7193

…ASHLIB_HASHES appears twice as a #define in confdefs.h -- try to fix this

And another pyconfig.h flag for the second variant -- doing this prop…

102dbbf

…erly by distinguishing simd128 from simd256

Getting there with the build

a61aa04

Multiplex everything

11d7c76

Defeat unused variable warning

9e0df3c

We seem to be down to a linking problem

d959d6d

gpshead approved these changes Aug 13, 2024

View reviewed changes

gpshead enabled auto-merge (squash) August 13, 2024 21:15

Merge branch 'main' into hacl_blake2

40374ac

gpshead merged commit 325e9b8 into python:main Aug 13, 2024
45 checks passed

bedevere-app bot removed the awaiting merge label Aug 13, 2024

msprotz deleted the hacl_blake2 branch August 13, 2024 21:45

freakboy3742 mentioned this pull request Aug 13, 2024

gh-116622: Add Android test script #121595

Merged

msprotz mentioned this pull request Aug 19, 2024

GH-99108: Fix Setup in addition to Setup.stdlib.in #123146

Merged

This was referenced Sep 5, 2024

Error compiling HACL* Blake2 support for macOS universal binaries #123748

Closed

gh-123748: Disable HACL SIMD256 on universal builds. #123927

Closed

gh-123748: Add conditional compilation rules to allow HACL SIMD256 to be compiled on universal #123989

Merged

picnixz mentioned this pull request Nov 1, 2024

gh-99108: Cleanup references to inexisting Modules/_blake2. #126270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-99108: Add HACL* Blake2 implementation to hashlib #119316

gh-99108: Add HACL* Blake2 implementation to hashlib #119316

msprotz commented May 21, 2024 •

edited by bedevere-app bot

Loading

cpython-cla-bot bot commented May 21, 2024 •

edited

Loading

msprotz commented May 21, 2024

msprotz commented Jun 20, 2024

gpshead commented Jun 21, 2024

msprotz commented Jun 25, 2024

eli-schwartz commented Jun 25, 2024

eli-schwartz left a comment

msprotz commented Jun 25, 2024

eli-schwartz commented Jun 25, 2024

msprotz commented Jul 18, 2024 •

edited

Loading

eli-schwartz commented Jul 18, 2024

msprotz commented Jul 19, 2024

bedevere-bot commented Aug 13, 2024

msprotz commented Aug 13, 2024

gpshead commented Aug 13, 2024

msprotz commented Aug 13, 2024

bedevere-bot commented Aug 13, 2024

bedevere-bot commented Aug 13, 2024

bedevere-bot commented Aug 13, 2024

msprotz commented Aug 13, 2024

mhsmith commented Aug 13, 2024 •

edited

Loading

picnixz commented Aug 14, 2024

encukou commented Aug 14, 2024

mhsmith commented Aug 14, 2024 •

edited

Loading

asottile commented Aug 17, 2024

msprotz commented Aug 19, 2024

eli-schwartz commented Aug 19, 2024

msprotz commented Aug 19, 2024

erlend-aasland commented Aug 23, 2024

	# Internal static libraries
	LIBMPDEC_A= Modules/_decimal/libmpdec/libmpdec.a
	LIBEXPAT_A= Modules/expat/libexpat.a
	LIBHACL_SHA2_A= Modules/_hacl/libHacl_Hash_SHA2.a

	##########################################################################
	# hashlib's HACL* library

	LIBHACL_SHA2_OBJS= \
	Modules/_hacl/Hacl_Hash_SHA2.o

	LIBHACL_HEADERS= \
	Modules/_hacl/include/krml/FStar_UInt128_Verified.h \
	Modules/_hacl/include/krml/FStar_UInt_8_16_32_64.h \
	Modules/_hacl/include/krml/fstar_uint128_struct_endianness.h \
	Modules/_hacl/include/krml/internal/target.h \
	Modules/_hacl/include/krml/lowstar_endianness.h \
	Modules/_hacl/include/krml/types.h \
	Modules/_hacl/Hacl_Streaming_Types.h \
	Modules/_hacl/python_hacl_namespaces.h

	LIBHACL_SHA2_HEADERS= \
	Modules/_hacl/Hacl_Hash_SHA2.h \
	Modules/_hacl/internal/Hacl_Hash_SHA2.h \
	$(LIBHACL_HEADERS)

	# Build HACL* static libraries for hashlib: libHacl_Hash_SHA2.a
	LIBHACL_CFLAGS=-I$(srcdir)/Modules/_hacl/include -D_BSD_SOURCE -D_DEFAULT_SOURCE $(PY_STDMODULE_CFLAGS) $(CCSHARED)

	Modules/_hacl/Hacl_Hash_SHA2.o: $(srcdir)/Modules/_hacl/Hacl_Hash_SHA2.c $(LIBHACL_SHA2_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_SHA2.c

	$(LIBHACL_SHA2_A): $(LIBHACL_SHA2_OBJS)
	-rm -f $@
	$(AR) $(ARFLAGS) $@ $(LIBHACL_SHA2_OBJS)

gh-99108: Add HACL* Blake2 implementation to hashlib #119316

gh-99108: Add HACL* Blake2 implementation to hashlib #119316

Conversation

msprotz commented May 21, 2024 • edited by bedevere-app bot Loading

cpython-cla-bot bot commented May 21, 2024 • edited Loading

msprotz commented May 21, 2024

msprotz commented Jun 20, 2024

gpshead commented Jun 21, 2024

msprotz commented Jun 25, 2024

eli-schwartz commented Jun 25, 2024

eli-schwartz left a comment

Choose a reason for hiding this comment

msprotz commented Jun 25, 2024

eli-schwartz commented Jun 25, 2024

msprotz commented Jul 18, 2024 • edited Loading

eli-schwartz commented Jul 18, 2024

msprotz commented Jul 19, 2024

bedevere-bot commented Aug 13, 2024

msprotz commented Aug 13, 2024

gpshead commented Aug 13, 2024

msprotz commented Aug 13, 2024

bedevere-bot commented Aug 13, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Aug 13, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Aug 13, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

msprotz commented Aug 13, 2024

mhsmith commented Aug 13, 2024 • edited Loading

picnixz commented Aug 14, 2024

encukou commented Aug 14, 2024

mhsmith commented Aug 14, 2024 • edited Loading

asottile commented Aug 17, 2024

msprotz commented Aug 19, 2024

eli-schwartz commented Aug 19, 2024

msprotz commented Aug 19, 2024

erlend-aasland commented Aug 23, 2024

msprotz commented May 21, 2024 •

edited by bedevere-app bot

Loading

cpython-cla-bot bot commented May 21, 2024 •

edited

Loading

msprotz commented Jul 18, 2024 •

edited

Loading

mhsmith commented Aug 13, 2024 •

edited

Loading

mhsmith commented Aug 14, 2024 •

edited

Loading