Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in TLS context - allow incoming write buffer to be relocated #5482

Merged
merged 3 commits into from
Jul 31, 2023

Conversation

eddyashton
Copy link
Member

Spotted some odd TLS errors during multi-threaded perf tests, where clients were being disconnected. Tracked it down to this. There were cases where we got a WANTS_WRITE/WANTS_READ response from an attempted write, which means a handshake is still in-progress or similar, and we need to retry the write later. The default semantics are very strict, and we must "retry the write" with exactly the same pointer and size. Our source for this is a pending_writes buffer, so if it got resized and reallocated, we'd pass a new pointer and this would blow up. There's a magic flag to make it less strict, which it seems we want. This is well explained in this StackOverflow answer, once you've convinced OpenSSL to log its errors correctly.

@eddyashton eddyashton requested a review from a team July 28, 2023 13:58
src/enclave/tls_session.h Outdated Show resolved Hide resolved
@eddyashton eddyashton added auto-backport Automatically backport this PR to LTS branch 4.x-todo PRs which should be backported to 4.x labels Jul 28, 2023
@ghost
Copy link

ghost commented Jul 28, 2023

ssl_debug@73621 aka 20230731.6 vs main ewma over 20 builds from 73098 to 73609

Click to see table

main

build_id build_number Commit latency factor pi_basic_mt_virtual_cft^ ls_virtual_cft^ pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem pi_ls_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_basic_virtual_cft^ pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem hist_sgx_cft^ ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
73098 20230714.1 0.790865 64627 45877.5 35632.3 2.51822e+07 47559.3 19988.6 1.88908e+07 54965.2 20230 1.25993e+07 12609.9 12662.7 22972.5 1.25993e+07 6374.19 1.67936e+07 4538.04 6511.7 6.30784e+06 3556.02 3275.15 1745.51 1.05021e+07 1432.05 1.05021e+07 48445.8 1425.52 1.05021e+07 843704 1.18342e+06 8.15209e+06 3.16739e+07
73119 20230714.10 0.819362 69624.9 45837.8 36055.4 2.51822e+07 47413.6 20021.7 1.88908e+07 55492.5 20101.8 1.25993e+07 12496 12661.6 22955.1 1.25993e+07 6359.6 1.67936e+07 4370.56 6473.5 6.30784e+06 3530.65 3285.52 1736.18 1.05021e+07 1435.26 1.05021e+07 48956 1421.37 1.05021e+07 830236 1.18254e+06 8.13234e+06 2.99709e+07
73140 20230714.16 0.775764 64726.9 45939.2 35787.7 2.51822e+07 47508.2 19982.8 1.88908e+07 56422.4 20136 1.25993e+07 12443.3 12725.2 23086 1.25993e+07 6702.03 1.67936e+07 4410.45 6473.1 6.30784e+06 3556.53 3276.78 1740.25 1.05021e+07 1437.95 1.05021e+07 46529 1428 1.05021e+07 834004 1.18232e+06 8.02426e+06 3.15227e+07
73180 20230717.1 0.800222 62847.2 45834.3 35716.3 2.51822e+07 46216.7 19924.6 1.88908e+07 56263.1 20159.7 1.25993e+07 12379.3 12902.1 22998.5 1.25993e+07 6718.37 1.67936e+07 4450.21 6519.5 6.30784e+06 3464.92 3276.36 1745.61 1.05021e+07 1444.06 1.05021e+07 43251.5 1434.21 1.05021e+07 835650 1.18058e+06 8.15572e+06 3.10261e+07
73193 20230717.7 0.78772 69751.7 43736.8 36011.4 2.51822e+07 46336 19894.2 1.88908e+07 55360.9 20159.3 1.25993e+07 12395.1 12427.6 22907.9 1.25993e+07 6307.23 1.67936e+07 4489.52 6514.9 6.30784e+06 3540.58 3265.45 1744.85 1.05021e+07 1440.41 1.05021e+07 47368.6 1422.28 1.05021e+07 838900 1.1819e+06 8.15066e+06 3.07097e+07
73203 20230718.1 0.793044 62165.1 45920.6 35794.8 2.51822e+07 42674.3 19822.2 1.88908e+07 56150.9 20161.2 1.25993e+07 12464.6 12757.4 22927.2 1.25993e+07 6704.73 1.67936e+07 4507.92 6511.9 6.30784e+06 3532 3266.4 1746.77 1.05021e+07 1441.47 1.05021e+07 45151 1435.35 1.05021e+07 832649 1.17671e+06 8.15647e+06 3.08346e+07
73222 20230718.8 0.805911 60457.5 45753.2 35990.6 2.51822e+07 47295.4 19907.3 1.88908e+07 55714.2 20202.7 1.25993e+07 12363.8 12886.6 22911.9 1.25993e+07 6700.6 1.67936e+07 4419.88 6562.7 6.30784e+06 3524.46 3302.35 1745.25 1.05021e+07 1443.47 1.05021e+07 46263.6 1433.57 1.05021e+07 840209 1.17318e+06 8.15423e+06 3.02887e+07
73253 20230718.14 0.777385 58267.4 46099.1 35660.2 2.51822e+07 46979.9 19676.5 1.88908e+07 56428.6 20055.2 1.25993e+07 12513.6 12701.8 22817.8 1.25993e+07 6361 1.67936e+07 4478.33 6428.3 6.30784e+06 3561.56 3194.57 1736.39 1.05021e+07 1436.52 1.05021e+07 44172.2 1426 1.05021e+07 834585 1.17384e+06 8.17023e+06 3.07683e+07
73282 20230718.23 0.796516 57369.1 45505.1 35647.1 2.51822e+07 46435.2 19967.8 1.88908e+07 56228.4 20190.2 1.05021e+07 12487.1 12915 22909.1 1.25993e+07 6763.57 1.67936e+07 4374.66 6519.8 6.30784e+06 3541.61 3299.13 1744.54 1.05021e+07 1439.03 1.05021e+07 46736.7 1427.97 1.05021e+07 835153 1.18022e+06 8.15254e+06 3.07143e+07
73295 20230719.3 0.813958 74193.9 45809.8 35703.7 2.30851e+07 42000.2 19855.4 1.88908e+07 56193.1 20100.3 1.25993e+07 12611.1 12906.4 23035 1.25993e+07 6684.01 1.67936e+07 4373.36 6518.1 6.30784e+06 3552.29 3313.49 1748.49 1.05021e+07 1442.45 1.05021e+07 48761 1427.62 1.05021e+07 836495 1.18362e+06 8.15293e+06 3.08736e+07
73334 20230720.1 0.837609 62884 45559.8 35932.5 2.51822e+07 47568.3 19828.5 1.88908e+07 56224.2 20042.3 1.25993e+07 12314.8 12825.3 23024 1.25993e+07 6710.13 1.67936e+07 4417.06 6511.8 6.30784e+06 3564.71 3294.75 1764.63 1.05021e+07 1438.09 1.05021e+07 46722.6 1429.68 1.05021e+07 838421 1.17085e+06 8.15274e+06 3.14583e+07
73390 20230720.18 0.787852 63739.9 43770.3 35663.2 2.51822e+07 46165.5 19996 1.88908e+07 55637.4 20256.2 1.25993e+07 12476.6 12778.2 22984.5 1.25993e+07 6722.76 1.67936e+07 4470.6 6520 6.30784e+06 3535.32 3301.1 1740.43 1.05021e+07 1435.12 1.05021e+07 46172.8 1429.11 1.05021e+07 836516 1.17791e+06 8.14807e+06 3.10237e+07
73419 20230721.1 0.775685 64508.4 43987.7 36069.5 2.51822e+07 46809.1 19991.9 1.88908e+07 57059 20233 1.25993e+07 12504.2 12842.9 22938.2 1.25993e+07 6692.61 1.67936e+07 4537.12 6567.4 6.30784e+06 3572.1 3269.58 1745.07 1.05021e+07 1439.02 1.05021e+07 48845 1432.44 1.05021e+07 826915 1.18017e+06 8.15319e+06 3.08615e+07
73478 20230724.1 0.762478 70095.8 45539.9 35978.9 2.51822e+07 46741.4 19963 1.67936e+07 56502.8 20224.7 1.25993e+07 12405.4 12854 22913.7 1.25993e+07 6409.59 1.67936e+07 4447.46 6527.5 6.30784e+06 3550.35 3312.76 1741.61 1.05021e+07 1436.29 1.05021e+07 48414.1 1430.05 1.05021e+07 816546 1.18442e+06 8.17164e+06 3.0724e+07
73522 20230726.3 0.774701 62106.4 43593.5 35665.2 2.51822e+07 46399.6 20025.5 1.88908e+07 56351 20231.4 1.25993e+07 12430 12793.8 22951.9 1.25993e+07 6746.02 1.67936e+07 4493.21 6516.4 6.30784e+06 3542.28 3327.47 1739.84 1.05021e+07 1426.07 1.05021e+07 45817.8 1427.23 1.05021e+07 835763 1.18085e+06 8.15352e+06 3.07762e+07
73535 20230727.1 0.789798 74853.9 45726.3 35786.2 2.30851e+07 40488.9 20098.9 1.88908e+07 56866.5 20256.3 1.25993e+07 12443.7 12615.1 22886.7 1.25993e+07 6729.54 1.67936e+07 4381.25 6520.5 6.30784e+06 3547.19 3304.08 1744.16 1.05021e+07 1433.26 1.05021e+07 46569.8 1427.65 1.05021e+07 828842 1.17948e+06 8.1552e+06 3.07863e+07
73557 20230728.1 0.836629 59443.4 45820.3 35671.9 2.51822e+07 46580.1 20086.9 1.88908e+07 56332 20271.6 1.25993e+07 12411.2 12546.4 23088.8 1.25993e+07 6396.5 1.67936e+07 4438.38 6521.7 6.30784e+06 3569.17 3310.11 1742.31 1.05021e+07 1436.82 1.05021e+07 45127.1 1429.11 1.05021e+07 834242 1.18091e+06 8.15416e+06 3.12505e+07
73574 20230728.9 0.812198 63710.8 43649.2 35481.2 2.30851e+07 46556.5 20009 1.88908e+07 55794.3 20152.6 1.25993e+07 12398.9 12690.1 22991 1.25993e+07 6379.23 1.67936e+07 4522.53 6518.2 6.30784e+06 3537.9 3302.15 1744.03 1.05021e+07 1437.8 1.05021e+07 45050.2 1431.48 1.05021e+07 826253 1.17636e+06 8.15345e+06 3.15261e+07
73602 20230728.19 0.803596 72391.8 45627.2 35902 2.51822e+07 46837.7 20011.2 1.88908e+07 56194.2 20232.3 1.25993e+07 12498.4 12678 22934.1 1.25993e+07 6335.6 1.67936e+07 4548.67 6517.7 6.30784e+06 3557.51 3297.85 1738.58 1.05021e+07 1438.01 1.05021e+07 49016 1426.15 1.05021e+07 838863 1.18103e+06 8.15631e+06 3.08174e+07
73609 20230731.1 0.774073 59683.3 44147.5 35860.5 2.51822e+07 46769.4 20054.8 1.88908e+07 55931 20237.5 1.25993e+07 12407.8 12973.2 23012.6 1.25993e+07 6731.81 1.67936e+07 4381.07 6520.6 6.30784e+06 3560.03 3266.55 1744.4 1.05021e+07 1434.11 1.05021e+07 46892 1429.74 1.05021e+07 835985 1.17981e+06 8.15342e+06 3.14661e+07

ssl_debug

build_id build_number Commit latency factor ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem pi_basic_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_virtual_cft^ pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem ls_js_jwt_virtual_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ pi_basic_mt_virtual_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
73592 20230728.14 0.802474 20039.9 1.88908e+07 20170.4 1.25993e+07 45849.9 46735.5 23006 1.25993e+07 6361.35 1.67936e+07 6511.4 6.30784e+06 56050.3 12492.7 12649.6 4469.48 1737.7 1.05021e+07 3549.72 35023.7 2.30851e+07 3304.09 1434.24 1.05021e+07 1424.13 1.05021e+07 46489.1 62753.6 836327 1.18155e+06 8.15306e+06 3.04576e+07
73621 20230731.6 0.79948 20015.7 1.88908e+07 20224.5 1.25993e+07 45698.8 46832.9 23029.3 1.25993e+07 6736.8 1.67936e+07 6519.3 6.30784e+06 55751.6 12489.2 12924.9 4454.41 1742.5 1.05021e+07 3458.89 35967.8 2.51822e+07 3313.02 1435.28 1.05021e+07 1429.96 1.05021e+07 43481 64193.6 832874 1.1798e+06 8.148e+06 3.06941e+07

images

@achamayou achamayou enabled auto-merge (squash) July 31, 2023 09:33
@achamayou achamayou merged commit e37d2f6 into microsoft:main Jul 31, 2023
19 checks passed
ghost pushed a commit that referenced this pull request Jul 31, 2023
@ghost ghost added the backported This PR was successfully backported to LTS branch label Jul 31, 2023
eddyashton pushed a commit that referenced this pull request Jul 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
4.x-todo PRs which should be backported to 4.x auto-backport Automatically backport this PR to LTS branch backported This PR was successfully backported to LTS branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants