cSHAKE results are incorrect for extremely large S (and probably N)

[FiloSottile]

We got an equivalent report over at golang/go#66232, so I used @noble/hashes to generate a test vector after fixing the issue, but while the fixed Go implementation, pycryptodome, and @noble/hashes agree for smaller S values, @noble/hashes disagrees for len(s) = 536871912 which makes the size in bits overflow a 32-bit integer.

const s = (1<<32)/8 + 1000 // s * 8 > 2^32
S := make([]byte, s)
rnd := NewShake128()
rnd.Read(S)
c := NewCShake128(nil, S)
io.CopyN(c, rnd, 1000)

// Generated with [email protected]
//
//    from Crypto.Hash import cSHAKE128
//    rng = cSHAKE128.new()
//    S = rng.read(536871912)
//    c = cSHAKE128.new(custom=S)
//    c.update(rng.read(1000))
//    print(c.read(32).hex())
//
exp := "2cb9f237767e98f2614b8779cf096a52da9b3a849280bbddec820771ae529cf0"
if got := hex.EncodeToString(c.Sum(nil)); got != exp {
	t.Errorf("got %s, want %s", got, exp)
}

[paulmillr]

Previously, Uint8Array larger than 4GB crashed due to JS engine limitations.

We should just place a limit for now and figure out how to make it fast with >u32 numbers later (it would be slow in naive impl).

[paulmillr]

FYI node.js built-ins just crash on >4gb buffers

[FiloSottile]

BTW, note that (1<<32)/8 + 1000 is more than 512MiB, not more than 4GiB. It's its size in bits that overflows a uint32.

[paulmillr]

We are now testing big buffers, also scrypt with high values 0a67c7c