Compilation response exceeds the Lambda function response limits

[kuzdogan]

We had a contract case reported that returns with an uninformative "Compiler error". We first thought this was a session issue as i n #1289 but turns out it's something else.

Method: Import from Etherscan
Address: 0xec713053c5c07A2A57185f6ED60FEf98094d65c9
Chain ID: 1 (Mainnet)

The contract was verifying fine locally but failing on staging and prod. so the AWS Lambda is to blame.

I tried switching the compilation method to lambda locally and sure enough, I could reproduce it. Interestingly the first compilation (that is used for generating the metadata files from Etherscan reqs) works fine. The second one (for verifying) fails. This is because the compiler output is exceeding the hard limit of AWS Lambda responses (6 MB). The first compilation only outputs the metadata whereas the second compilation is a full output.

Lambda logs: https://eu-north-1.console.aws.amazon.com/cloudwatch/home?region=eu-north-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fcompile/log-events/2024$252F04$252F05$252F$255B$2524LATEST$255Dc658e48fd88c415194681199b19ce7e4

2024-04-05T12:34:08.874+02:00	START RequestId: e4ea372e-c8cd-4158-87b7-fd035e90ee0d Version: $LATEST
2024-04-05T12:34:08.888+02:00	2024-04-05T10:34:08.888Z e4ea372e-c8cd-4158-87b7-fd035e90ee0d DEBUG Found solc 0.8.18+commit.87f61d96 with platform linux-amd64 at /tmp/solc-repo/solc-linux-amd64-v0.8.18+commit.87f61d96
2024-04-05T12:34:08.888+02:00	2024-04-05T10:34:08.888Z e4ea372e-c8cd-4158-87b7-fd035e90ee0d DEBUG Compiling with solc binary 0.8.18+commit.87f61d96 at /tmp/solc-repo/solc-linux-amd64-v0.8.18+commit.87f61d96
2024-04-05T12:34:15.519+02:00	2024-04-05T10:34:15.519Z e4ea372e-c8cd-4158-87b7-fd035e90ee0d INFO Compilation time : 6631 ms
2024-04-05T12:34:15.547+02:00	END RequestId: e4ea372e-c8cd-4158-87b7-fd035e90ee0d
2024-04-05T12:34:15.547+02:00	REPORT RequestId: e4ea372e-c8cd-4158-87b7-fd035e90ee0d Duration: 6673.09 ms Billed Duration: 6674 ms Memory Size: 3008 MB Max Memory Used: 532 MB
2024-04-05T12:34:23.601+02:00	START RequestId: b016dafd-8950-47b0-8b47-7beaff879521 Version: $LATEST
2024-04-05T12:34:23.622+02:00	2024-04-05T10:34:23.622Z b016dafd-8950-47b0-8b47-7beaff879521 DEBUG Found solc 0.8.18+commit.87f61d96 with platform linux-amd64 at /tmp/solc-repo/solc-linux-amd64-v0.8.18+commit.87f61d96
2024-04-05T12:34:23.622+02:00	2024-04-05T10:34:23.622Z b016dafd-8950-47b0-8b47-7beaff879521 DEBUG Compiling with solc binary 0.8.18+commit.87f61d96 at /tmp/solc-repo/solc-linux-amd64-v0.8.18+commit.87f61d96
2024-04-05T12:34:31.031+02:00	2024-04-05T10:34:31.031Z b016dafd-8950-47b0-8b47-7beaff879521 INFO Compilation time : 7409 ms
2024-04-05T12:34:31.299+02:00	[ERROR] [1712313271299] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413.

This likely won't be the only case and there probably have been cases we haven't noticed and we should fix this for all cases.

With a quick research I see a solution is to switch to streaming responses. Another possibility is to paginate the responses. We can do whichever is easier but I feel streaming would work fine.

I've added some proper error handling here: 0f78b4b

[manuelwedler]

Fixed with #1340