-H:+ParseOnce singificantly increases the memory footprint of native-image in Quarkus' integration-tests/main

[zakkak]

Description

As described in #304 (comment) -H:+ParseOnce significantly increases the memory footprint of native-image in Quarkus' integration-tests/main.

We need to better understand why this is happening while other integration tests are not affected to such an extend.

My plan is to:

• generate reports for all integration tests and see what makes main special (my guess is that it is pulling in more methods/classes resulting in a larger graph).
• enable gc logging for all integration tests and see what their live data set appears to be (both with parse once and without it),
  Add RSS reporting for Linux oracle/graal#3913 will probably be helpful as well.
• correlate statistics from the reports with memory usage and see if there is some pattern

How To Reproduce

git clone https://github.com/quarkusio/quarkus.git
cd quarkus
./mvnw -Dnative \
  -Dquarkus.native.additional-build-args="-H:+ParseOnce,-J-Xlog:gc*" \
  -Dquarkus.native.native-image-xmx=5g \
  -pl integration-tests/main package

[github-actions]

This issue appears to be stale because it has been open 30 days with no activity. This issue will be closed in 7 days unless Stale label is removed, a new comment is made, or not-Stale label is added.

[zakkak]

After gathering the data it's clear that main is the test with the most reachable Classes, Fields, and Methods, and as expected has the highest Peak RSS. Another interesting observation is that the top "offenders" (in terms of Peak RSS) all have a high number of classes/fields/methods registered for JNI access, so there seems to be some correlation between those JNI accesses and Peak RSS as well when ParseOnce is enabled (with awt and jaxb being the exceptions)

I am attaching the results sorted by Peak RSS. The measurements where gathered using GraalVM CE 22.0-dev build without libgraal from source based on e00d063 and Quarkus quarkusio/quarkus@af3db6daf5

Benchmark	Reach. Classes	Reach. Fields	Reach. Methods	Refl. Classes	Refl. Fields	Refl. Methods	JNI Classes	JNI Fields	JNI Methods	-ParseOnce	ParseOnce	Heap Classes	Heap Objects	Compilation Units
main	25,130	42,548	131,695	994	953	3,690	119	840	1,095	8.99	9.45	17,525	761,557	91,419
liquibase	17,824	29,501	104,189	915	188	5,875	119	840	1,095	8.01	9.39	12,588	640,144	72,731
spring-web	16,727	29,622	91,652	578	289	1,761	119	840	1,095	7.6	9.1	11,634	482,136	64,140
hibernate-orm-panache	19,567	33,188	108,222	688	208	3,209	119	840	1,095	7.93	8.89	13,393	684,018	75,783
tika	17,838	30,300	94,380	443	22	752	119	840	1,095	8.09	8.87	12,557	555,447	63,501
jpa	15,487	25,942	78,824	541	44	1,015	69	91	54	7.7	8.84	10,485	386,700	52,498
hibernate-search-orm-elasticsearch-aws	17,512	27,069	84,509	623	149	1,529	68	89	54	6.89	8.78	11,729	449,096	55,883
liquibase-mongodb	16,488	29,874	96,929	1,015	500	5,953	120	842	1,095	7.72	8.75	11,682	644,305	68,841
hibernate-validator	18,280	30,147	95,131	629	57	1,175	119	840	1,095	7.21	8.7	12,456	470,226	65,389
hibernate-search-orm-elasticsearch	17,792	27,419	85,396	615	90	1,298	68	89	54	7.65	8.61	11,937	453,074	56,251
reactive-messaging-kafka	14,970	22,755	74,938	561	180	1,351	68	89	54	6.78	8.16	10,932	376,645	49,640
kafka-avro	16,356	26,883	85,292	642	893	2,348	68	89	54	7.56	8.07	11,656	453,478	57,386
keycloak-authorization	16,008	27,216	86,314	721	1,079	3,934	68	89	54	6.77	7.8	11,246	558,969	58,226
kafka	15,872	26,405	83,651	650	1,106	2,523	68	89	54	7.48	7.79	11,431	448,826	56,114
jpa-db2	13,387	20,577	66,368	529	24	912	68	89	54	6.99	7.77	8,954	335,329	43,711
kubernetes-client	13,661	22,251	81,274	848	43	7,946	68	89	54	7.09	7.75	9,378	705,436	52,777
hibernate-orm-rest-data-panache	15,698	23,183	78,281	588	47	1,273	68	89	54	7.03	7.73	10,583	411,763	52,083
micrometer-prometheus	15,853	23,862	79,521	587	26	1,178	68	89	54	7.4	7.72	10,814	409,563	53,042
jgit	12,391	21,607	62,274	512	1,025	1,003	68	89	55	6.91	7.66	8,729	309,221	41,201
openshift-client	13,320	21,638	79,411	848	32	7,991	68	89	54	7.22	7.63	9,156	698,926	51,487
oidc-client-wiremock	12,388	18,800	62,881	460	73	1,004	68	89	54	6.75	7.61	8,682	318,392	42,333
spring-data-rest	15,728	23,195	78,421	596	48	1,372	68	89	54	7.33	7.61	10,605	412,219	52,240
jpa-mysql	16,435	27,669	82,916	552	999	1,164	75	112	55	7.72	7.6	11,300	422,590	55,913
jpa-h2	13,518	20,810	66,411	530	122	1,001	68	89	54	6.89	7.54	9,070	342,594	43,740
jpa-oracle	14,002	20,965	67,446	537	44	1,004	68	89	54	7.17	7.52	9,327	354,639	44,366
kafka-streams	14,236	21,778	72,098	556	168	1,284	68	89	54	7.09	7.52	10,282	358,313	48,041
hibernate-reactive-panache	16,820	24,343	81,610	576	149	2,240	68	89	54	7.13	7.5	11,661	523,774	53,603
cache	15,684	23,361	76,322	610	38	1,257	68	89	54	6.98	7.47	10,641	404,399	50,341
opentelemetry	12,246	17,933	60,864	484	66	1,138	68	89	54	6.69	7.45	8,377	314,034	40,803
reactive-messaging-amqp	12,997	18,729	64,320	454	36	801	68	89	54	6.69	7.45	9,200	306,849	42,594
rest-client-reactive-multipart	12,255	18,451	60,806	427	22	777	68	89	54	6.52	7.45	8,414	294,854	40,522
vault-app	15,938	24,306	78,926	658	220	1,075	68	89	54	7.08	7.45	10,901	405,766	52,379
jaxb	12,729	23,532	70,866	473	19	1,088	119	840	1,095	6.91	7.41	8,761	374,910	50,280
jpa-mariadb	14,940	24,189	73,629	539	24	922	68	89	54	6.91	7.38	10,137	376,686	48,442
jpa-postgresql-withxml	14,369	21,419	70,450	576	99	1,242	68	89	54	7.23	7.38	9,621	373,746	46,962
jpa-mssql	15,194	26,355	79,351	585	27	2,051	68	89	54	7.16	7.36	10,313	446,458	53,398
hibernate-orm-envers	13,699	20,559	66,978	556	20	954	68	89	54	7.09	7.33	9,180	341,553	44,308
mongodb-panache	14,400	21,253	72,949	571	215	2,704	68	89	54	6.74	7.32	10,385	522,034	48,179
vertx-http	12,324	19,891	63,345	474	539	1,026	68	89	54	6.84	7.32	8,545	320,382	42,810
flyway	14,824	21,870	72,320	575	33	1,004	68	89	54	7.05	7.3	10,111	374,341	47,963
smallrye-jwt-token-propagation	12,909	19,709	65,184	475	119	1,050	68	89	54	6.73	7.29	8,946	322,477	43,736
qute	15,248	22,747	76,179	599	37	1,129	68	89	54	6.66	7.28	10,351	390,062	50,662
grpc-plain-text-gzip	13,653	22,661	70,175	481	49	1,069	68	89	54	6.78	7.26	9,338	337,352	45,820
consul-config	12,074	19,502	62,219	442	19	783	68	89	54	6.91	7.25	8,370	296,551	41,757
jpa-derby	13,935	20,942	68,460	554	122	1,315	68	89	54	6.76	7.25	9,287	355,249	45,442
kafka-avro-apicurio2	13,962	21,705	70,805	534	164	1,275	68	89	54	6.7	7.25	10,110	367,605	47,137
smallrye-config	13,479	21,095	68,847	520	45	1,456	68	89	55	7.04	7.25	9,327	365,879	46,350
jpa-without-entity	10,935	15,669	51,838	496	22	809	68	89	54	6.52	7.24	7,437	272,516	34,017
spring-di	11,255	18,475	56,912	445	26	847	68	89	54	6.66	7.24	7,735	288,762	37,846
jpa-postgresql	15,018	23,841	73,254	540	25	948	68	89	54	7.16	7.2	10,024	374,836	48,222
artemis-core	12,036	19,451	60,079	457	36	776	68	89	54	6.92	7.19	8,232	302,112	39,981
elasticsearch-rest-high-level-client	12,905	18,568	62,938	441	47	878	68	89	54	6.77	7.19	8,964	317,352	42,450
neo4j	13,225	19,062	65,643	463	45	849	68	89	54	6.94	7.19	9,204	310,801	44,174
oidc	11,490	17,520	58,073	433	49	726	68	89	54	5.79	7.18	8,021	284,425	39,178
amazon-lambda	13,437	21,595	69,113	478	80	1,114	68	89	54	6.63	7.17	9,328	344,622	46,144
bouncycastle	13,547	21,550	65,561	446	40	788	68	89	54	7.09	7.17	9,672	327,489	43,812
elytron-security	11,797	19,180	60,163	468	23	841	68	89	54	6.75	7.17	8,124	301,873	40,458
grpc-tls	12,387	20,439	63,842	460	39	1,001	68	89	54	6.82	7.17	8,483	311,546	41,898
kafka-oauth-keycloak	14,069	21,316	70,892	545	149	1,243	68	89	54	6.86	7.17	10,140	351,664	47,294
spring-cloud-config-client	12,357	19,980	63,537	447	19	788	68	89	54	6.83	7.16	8,595	302,302	42,532
grpc-mutual-auth	12,400	20,456	63,884	462	39	1,003	68	89	54	6.42	7.15	8,494	311,961	41,990
hibernate-reactive-postgresql	15,061	22,237	71,837	556	58	1,086	68	89	54	7.02	7.12	10,415	367,962	47,293
artemis-jms	12,180	19,666	60,796	452	36	771	68	89	54	6.88	7.1	8,309	303,817	40,284
reactive-messaging-rabbitmq	13,031	18,893	63,581	461	50	802	68	89	54	6.57	7.1	9,203	310,529	42,141
grpc-plain-text-mutiny	12,518	18,959	64,573	453	44	1,093	68	89	54	6.53	7.09	8,573	313,864	42,762
kafka-ssl	14,722	23,119	74,884	550	164	1,351	71	91	59	7.37	7.09	10,620	374,039	49,794
smallrye-graphql	14,672	23,011	74,231	508	157	1,588	68	89	54	7.2	7.09	10,405	382,079	49,100
elytron-undertow	13,020	20,862	65,838	456	23	791	68	89	54	6.57	7.08	8,992	314,977	43,633
mongodb-rest-data-panache	12,309	17,919	63,043	444	36	925	68	89	54	6.53	7.08	8,572	322,818	42,398
oidc-code-flow	11,974	18,074	60,750	447	73	908	68	89	54	6.42	7.07	8,355	302,532	40,907
kafka-sasl	14,281	21,823	72,454	557	183	1,322	68	89	54	7.15	7.06	10,316	360,127	48,336
mongodb-client	12,061	17,846	59,048	438	57	799	68	89	54	6.69	7.06	8,507	299,999	39,279
oidc-token-propagation	11,906	17,988	60,546	449	48	788	68	89	54	6.58	7.06	8,299	296,656	40,900
kafka-snappy	14,069	21,316	70,890	545	149	1,242	68	89	54	6.95	7.04	10,140	351,588	47,292
oidc-client-reactive	12,491	20,600	62,804	474	61	899	68	89	54	6.8	7.02	8,620	309,420	41,703
reactive-messaging-http	13,052	20,841	65,897	456	33	784	68	89	54	6.65	7.01	9,176	312,091	43,982
oidc-tenancy	13,685	22,425	69,770	480	72	900	68	89	54	6.87	7	9,512	331,453	46,760
elytron-resteasy	10,840	15,819	55,091	435	18	766	68	89	54	6.34	6.99	7,489	278,009	37,378
micrometer-mp-metrics	11,514	16,772	57,937	453	24	981	68	89	54	6.35	6.99	7,958	292,293	39,032
reactive-mysql-client	11,438	18,561	57,667	447	33	775	68	89	54	6.57	6.99	7,899	293,173	38,699
reactive-pg-client	11,501	18,614	57,653	447	33	775	68	89	54	6.59	6.99	7,954	292,380	38,546
hibernate-reactive-mysql	14,200	20,963	67,402	541	15	898	68	89	54	6.67	6.98	9,699	347,955	43,952
elytron-security-jdbc	11,382	18,444	57,460	449	25	793	68	89	54	6.53	6.97	7,861	287,446	38,402
hibernate-reactive-db2	14,133	20,962	67,225	541	15	898	68	89	54	6.73	6.96	9,650	346,129	43,767
reactive-db2-client	11,385	18,564	57,547	446	34	780	68	89	54	6.59	6.96	7,857	292,016	38,507
redis-client	11,469	18,794	58,093	461	40	812	68	89	54	6.64	6.96	7,984	296,122	39,104
oidc-wiremock	11,974	18,062	60,624	450	66	873	68	89	54	6.25	6.94	8,359	297,729	40,809
devtools-registry-client	11,012	16,104	56,049	491	110	1,276	68	89	54	6.59	6.91	7,573	293,158	37,716
rest-client	11,009	16,521	55,920	434	17	751	68	89	54	6.57	6.91	7,624	279,114	37,765
smallrye-opentracing	13,415	20,130	66,458	501	56	1,165	68	89	55	6.88	6.91	9,219	346,530	44,365
quartz	11,180	17,209	57,252	459	29	1,192	68	89	54	5.93	6.9	7,716	309,946	38,420
spring-boot-properties	10,679	17,259	53,894	440	22	769	68	89	54	6.37	6.86	7,342	275,092	35,949
grpc-proto-v2	12,390	20,449	63,870	460	46	1,011	68	89	54	6.9	6.84	8,485	311,936	41,914
bouncycastle-fips	12,152	19,509	62,362	456	82	837	68	89	54	6.64	6.83	8,336	333,312	42,136
infinispan-client	12,100	19,657	60,876	458	27	809	68	89	54	6.72	6.83	8,311	302,941	40,325
elasticsearch-rest-client	11,588	17,088	58,028	439	37	850	68	89	54	6.35	6.81	7,964	293,607	39,090
oidc-client	13,673	22,403	69,683	477	65	892	68	89	54	6.77	6.81	9,503	330,627	46,683
rest-client-reactive-stork	13,346	21,822	66,242	520	593	1,242	68	89	54	6.92	6.81	9,204	345,063	44,255
resteasy-jackson	11,405	16,745	57,596	444	107	1,205	68	89	54	6.44	6.81	7,858	302,632	38,528
spring-data-jpa	16,245	23,702	82,050	657	215	2,144	68	89	54	7.45	6.8	11,076	461,841	55,560
vertx-graphql	10,684	17,243	53,905	439	22	753	68	89	54	6.33	6.8	7,343	274,576	35,973
infinispan-cache-jpa	13,851	20,949	68,770	566	70	1,160	68	89	54	6.81	6.79	9,299	362,117	45,710
jaxp	10,963	17,681	55,354	439	18	807	68	89	54	6.42	6.78	7,536	281,591	37,075
bouncycastle-jsse	11,735	18,681	59,485	454	71	820	68	89	54	6.58	6.77	8,092	321,179	40,113
narayana-jta	9,923	14,821	49,044	430	19	740	68	89	54	6.43	6.76	6,793	256,260	32,511
jsch	10,810	17,812	54,817	484	169	984	68	89	54	6.53	6.74	7,432	284,173	36,580
logging-gelf	10,749	17,372	54,203	436	19	743	68	89	54	6.58	6.73	7,377	274,762	36,122
rest-client-reactive	10,918	16,340	55,493	430	8	725	68	89	54	6.59	6.73	7,551	275,876	37,420
vertx	10,816	16,346	53,674	386	13	680	68	89	54	6.41	6.72	7,708	259,124	35,060
logging-json	9,636	14,349	47,765	421	19	726	68	89	54	5.89	6.71	6,607	250,640	31,737
smallrye-metrics	10,914	15,744	52,910	444	25	839	68	89	54	6.41	6.69	7,672	280,728	35,549
elytron-resteasy-reactive	11,165	17,957	56,354	447	24	787	68	89	54	6.6	6.67	7,708	284,431	37,674
logging-min-level-set	9,620	14,328	47,710	425	24	739	68	89	54	5.96	6.64	6,606	251,630	31,715
native-config-profile	9,664	14,450	48,303	421	19	746	68	89	54	6.1	6.62	6,660	251,773	32,311
elytron-security-oauth2	10,741	15,534	52,733	446	42	814	68	89	54	6.26	6.6	7,480	272,667	35,207
mailer	11,234	16,745	54,839	467	29	801	68	89	54	6.35	6.56	7,925	282,234	36,488
vertx-web-jackson	9,937	14,705	51,152	395	11	697	68	89	54	6.31	6.5	6,865	259,146	34,607
smallrye-graphql-client	11,858	17,726	59,627	439	675	1,553	68	89	54	6.69	6.48	8,432	339,627	39,513
elytron-security-ldap	10,819	15,691	52,943	442	27	775	68	89	54	6.51	6.43	7,535	271,161	35,316
jackson	10,638	15,642	53,998	444	45	828	68	89	54	6.36	6.37	7,337	271,040	36,245
jsonb	10,213	14,830	49,910	429	21	758	68	89	54	6.22	6.37	7,087	260,343	33,192
vertx-web	9,862	14,461	50,422	390	10	686	68	89	54	6.32	6.24	6,801	249,020	31,928
awt	6,780	10,966	37,661	342	0	591	117	836	1,094	5.66	6.22	4,632	212,244	27,379
test-extension	9,381	14,235	46,775	392	105	847	68	89	54	5.84	6.21	6,423	241,359	30,948
grpc-interceptors	11,490	17,595	58,027	435	27	1,011	68	89	54	6.58	6.14	7,861	282,964	37,824
amazon-lambda-http	7,859	10,983	39,023	372	118	934	66	85	53	6.22	6.08	5,378	250,104	26,564
logging-min-level-unset	9,611	14,320	47,658	422	21	735	68	89	54	6.11	6.07	6,597	250,930	31,669
picocli-native	6,587	9,493	33,737	529	590	2,671	66	85	53	5.88	5.5	4,547	287,893	22,986

It's worth noting that after the merge of quarkusio/quarkus#20850 Peak RSS seams to have reduced significantly for main (dropped by ~1GB), as well as the classes/fields/methods registered for JNI access.

Compiling main of Quarkus quarkusio/quarkus@07e923aed3 (which includes quarkusio/quarkus#20850) gives:

Benchmark	Reach. Classes	Reach. Fields	Reach. Methods	Refl. Classes	Refl. Fields	Refl. Methods	JNI Classes	JNI Fields	JNI Methods	Peak RSS -ParseOnce	PeakRSS +ParseOnce	Heap Classes	Heap Objects	Compilation Units
main	26,262	53,493	208,112	906	867	3587	69	89	57	8.57	8.43	16425	705470	79303

[zakkak]

Pasting the results (sorted by Peak RSS with ParseOnce) from Quarkus quarkusio/quarkus@963e380 (i.e. after quarkusio/quarkus#20850).

As expected, the pattern that the tests with the most reachable Classes, Fields, and Methods have the highest Peak RSS seems to persist. What's interesting though is that if we sort the data by the Peak RSS difference between the -ParseOnce and the +ParseOnce runs we see that there is no clear pattern. For instance main, despite the high number of fields, methods and classes, doesn't come high in terms of Peak RSS diff.

Note that the above also negates my earlier comment:

... other integration tests are not affected to such an extend.

There are tests that are much more affected but their Peak RSS was low enough to not push the limits of the GH runner.

Benchmark	Reach. Classes	Reach. Fields	Reach. Methods	Refl. Classes	Refl. Fields	Refl. Methods	JNI Classes	JNI Fields	JNI Methods	-ParseOnce	ParseOnce	Heap Classes	Heap Objects	Compilation Units
hibernate-search-orm-elasticsearch	17,493	26,526	84,215	623	149	1,529	68	89	54	7.22	8.82	11,730	449,078	55,887
main	23,822	38,035	118,890	906	867	3,587	69	89	57	8.51	8.64	16,517	707,874	80,577
kafka-avro	16,370	26,424	85,073	642	893	2,348	68	89	54	7.67	8.21	11,657	453,454	57,389
jpa-mysql	16,432	27,313	82,758	552	999	1,164	75	112	55	7.58	7.89	11,305	422,851	55,973
jpa	15,469	25,602	78,578	541	44	1,015	69	91	54	7.48	7.87	10,486	386,727	52,502
keycloak-authorization	16,010	26,863	86,131	721	1,079	3,934	68	89	54	7.16	7.85	11,248	559,057	58,254
kafka	15,889	25,951	83,487	650	1,106	2,523	68	89	54	7.7	7.83	11,432	448,783	56,117
bouncycastle	13,530	21,273	65,300	446	40	788	68	89	54	6.75	7.7	9,673	327,563	43,815
kubernetes-client	14,102	22,624	83,131	848	43	7,946	68	89	54	7.26	7.69	9,679	712,103	54,151
spring-data-rest	15,731	22,824	78,202	596	48	1,372	68	89	54	7.17	7.69	10,606	412,297	52,244
liquibase-mongodb	15,466	26,272	85,809	1,013	500	5,951	69	91	54	7.49	7.65	10,898	608,973	59,204
qute	15,238	22,363	75,950	599	37	1,129	68	89	54	7.09	7.64	10,354	390,180	50,685
smallrye-graphql	14,691	22,544	74,152	508	157	1,588	68	89	54	7.34	7.62	10,432	382,661	49,171
openshift-client	13,313	21,314	79,188	848	32	7,991	68	89	54	6.75	7.59	9,157	698,965	51,490
grpc-plain-text-gzip	13,640	22,364	69,852	482	49	1,070	68	89	54	6.79	7.56	9,347	337,515	45,856
liquibase	16,833	25,923	93,136	913	188	5,873	68	89	54	7.5	7.56	11,807	604,998	63,111
hibernate-orm-panache	18,461	29,088	96,080	686	208	3,207	69	89	57	7.64	7.55	12,547	646,220	65,359
hibernate-orm-rest-data-panache	15,701	22,813	78,054	588	47	1,273	68	89	54	7.18	7.55	10,584	411,837	52,087
jpa-mssql	15,176	25,971	79,076	585	27	2,051	68	89	54	7.3	7.55	10,314	446,478	53,402
hibernate-reactive-panache	16,803	23,742	81,355	576	149	2,240	68	89	54	6.72	7.52	11,654	527,508	53,592
hibernate-validator	17,156	26,099	82,970	627	57	1,173	69	89	57	7.14	7.52	11,610	432,477	54,965
cache	15,666	22,934	76,059	610	38	1,257	68	89	54	7.15	7.5	10,642	404,436	50,344
jpa-postgresql-withxml	14,998	23,503	72,951	576	99	1,242	68	89	54	7.06	7.5	9,622	373,765	46,966
spring-web	15,634	25,599	79,520	576	289	1,759	69	89	57	7.31	7.49	10,788	444,545	53,688
jaxb	11,725	19,851	59,438	481	26	1,102	69	89	57	6.64	7.45	8,016	340,251	40,212
flyway	14,805	21,465	72,067	575	33	1,004	68	89	54	6.7	7.44	10,112	374,356	47,968
hibernate-reactive-db2	14,116	20,542	66,988	541	15	898	68	89	54	6.91	7.42	9,651	346,143	43,770
jpa-mariadb	14,919	23,835	73,347	539	24	922	68	89	54	7.03	7.42	10,138	376,670	48,446
reactive-messaging-kafka	14,992	22,249	74,810	571	182	1,365	68	89	54	7.02	7.42	10,952	377,671	49,703
kafka-oauth-keycloak	14,239	21,302	71,890	545	149	1,243	68	89	54	7.01	7.41	10,141	351,696	47,294
micrometer-prometheus	15,862	23,401	79,390	587	26	1,178	68	89	54	7.02	7.4	10,830	409,712	53,085
reactive-messaging-rabbitmq-dyn	13,026	18,474	63,353	462	50	803	68	89	54	6.72	7.4	9,204	310,588	42,145
jpa-db2	13,369	20,248	66,111	529	24	912	68	89	54	7.04	7.32	8,955	335,344	43,715
jpa-derby	13,919	20,607	68,202	554	122	1,315	68	89	54	6.97	7.29	9,288	355,262	45,446
kafka-snappy	14,284	21,347	72,247	545	149	1,242	68	89	54	6.59	7.29	10,141	351,666	47,292
kafka-streams	14,738	22,506	74,692	556	168	1,284	68	89	54	7.07	7.29	10,283	358,324	48,044
kafka-avro-apicurio2	13,965	21,246	70,588	534	164	1,275	68	89	54	7.05	7.27	10,111	367,678	47,140
hibernate-orm-envers	13,679	20,215	66,734	557	20	955	68	89	54	6.91	7.26	9,181	341,746	44,329
jpa-without-entity	10,922	15,372	51,608	496	22	809	68	89	54	6.48	7.24	7,438	272,535	34,020
kafka-sasl	14,072	20,847	70,684	557	183	1,322	68	89	54	7.12	7.24	10,317	360,143	48,339
mongodb-rest-data-panache	12,306	17,636	62,833	444	36	925	68	89	54	5.45	7.24	8,573	322,826	42,401
amazon-lambda	13,428	21,227	68,821	478	80	1,114	68	89	54	6.67	7.23	9,329	344,578	46,147
oidc-wiremock	11,983	17,790	60,500	450	66	873	68	89	54	6.62	7.21	8,361	297,827	40,837
kafka-ssl	14,072	20,847	70,682	550	164	1,351	71	91	59	6.99	7.2	10,621	374,061	49,797
elytron-undertow	13,012	20,512	65,587	456	23	791	68	89	54	6.8	7.19	8,993	315,011	43,636
smallrye-config	13,485	20,802	68,628	520	45	1,456	68	89	55	6.78	7.18	9,328	365,949	46,353
smallrye-jwt-token-propagation	12,917	19,401	65,024	475	119	1,050	68	89	54	6.76	7.18	8,947	322,476	43,749
jpa-h2	13,501	20,468	66,156	530	122	1,001	68	89	54	6.71	7.15	9,071	342,659	43,744
reactive-messaging-amqp	12,995	18,321	64,117	454	36	801	68	89	54	6.72	7.15	9,201	306,934	42,598
reactive-messaging-http	13,049	20,479	65,687	456	33	784	68	89	54	6.81	7.15	9,177	312,064	43,985
jpa-oracle	14,351	21,038	70,194	537	44	1,003	68	89	54	7.04	7.14	9,328	354,637	44,369
hibernate-reactive-mysql	14,183	20,542	67,159	541	15	898	68	89	54	6.88	7.1	9,700	348,024	43,954
vertx-http	12,312	19,595	63,082	474	539	1,026	68	89	54	6.75	7.1	8,546	320,481	42,813
elytron-resteasy	11,151	17,686	56,109	435	18	766	68	89	54	6.38	7.07	7,490	277,976	37,381
rest-client-reactive-multipart	12,782	19,331	63,692	427	557	955	68	89	54	6.65	7.07	8,820	319,051	42,800
awt	11,220	19,199	67,688	446	44	792	183	1,568	2,102	6.92	7.05	7,853	320,169	50,060
jaxp	10,945	17,406	55,087	439	18	807	68	89	54	6.46	7.04	7,537	281,615	37,078
grpc-proto-v2	12,389	20,179	63,675	461	46	1,012	68	89	54	6.92	7.02	8,494	312,104	41,950
reactive-db2-client	11,372	18,276	57,302	446	34	780	68	89	54	6.53	7	7,858	291,962	38,510
reactive-mysql-client	11,425	18,271	57,416	447	33	775	68	89	54	6.66	7	7,900	293,172	38,702
hibernate-reactive-postgresql	15,046	21,782	71,608	556	58	1,086	68	89	54	7.02	6.99	10,416	367,932	47,296
jpa-postgresql	13,983	20,633	67,161	540	25	948	68	89	54	7.35	6.99	10,025	374,850	48,226
oidc-tenancy	11,983	17,798	60,617	480	72	900	68	89	54	6.73	6.99	9,514	331,564	46,798
spring-di	11,241	18,209	56,653	445	26	847	68	89	54	6.6	6.99	7,736	288,780	37,849
mongodb-client	12,043	17,504	58,822	438	57	799	68	89	54	6.66	6.96	8,508	300,089	39,282
infinispan-client	12,083	19,350	60,490	458	27	809	68	89	54	6.83	6.95	8,312	302,889	40,328
elasticsearch-rest-high-level-client	12,915	18,187	62,715	441	47	878	68	89	54	6.71	6.94	8,965	317,362	42,453
smallrye-graphql-client	11,848	17,335	59,433	439	675	1,553	68	89	54	6.69	6.94	8,441	340,053	39,546
bouncycastle-fips	12,136	19,238	62,105	456	82	837	68	89	54	6.71	6.91	8,337	333,325	42,139
bouncycastle-jsse	11,719	18,402	59,219	454	71	820	68	89	54	6.59	6.91	8,093	321,259	40,116
elytron-security	11,783	18,899	59,922	468	23	841	68	89	54	6.53	6.91	8,125	301,942	40,462
oidc-token-propagation	13,679	22,110	69,592	449	48	788	68	89	54	6.68	6.91	8,301	296,799	40,928
reactive-messaging-rabbitmq	13,039	18,496	63,430	463	52	802	68	89	54	6.56	6.91	9,215	310,861	42,228
redis-client	11,456	18,485	57,849	461	40	812	68	89	54	6.79	6.9	7,985	296,076	39,107
grpc-plain-text-mutiny	12,537	18,695	64,417	454	44	1,094	68	89	54	6.66	6.89	8,581	314,015	42,795
grpc-mutual-auth	12,399	20,181	63,689	463	39	1,004	68	89	54	6.69	6.88	8,503	312,084	42,025
infinispan-cache-jpa	13,831	20,587	68,513	566	70	1,160	68	89	54	6.92	6.88	9,300	362,121	45,714
micrometer-mp-metrics	11,523	16,426	57,734	453	24	981	68	89	54	6.58	6.88	7,959	292,352	39,035
oidc-client-wiremock	12,397	18,484	62,767	460	73	1,004	68	89	54	6.73	6.88	8,684	318,529	42,369
devtools-registry-client	11,017	15,839	55,836	491	110	1,276	68	89	54	6.11	6.86	7,574	293,126	37,719
oidc-client-reactive	12,477	20,286	62,612	474	62	901	68	89	54	6.74	6.85	8,621	309,517	41,717
opentelemetry	12,319	17,658	60,986	484	62	1,139	68	89	54	6.58	6.84	8,427	315,867	41,003
mongodb-panache	14,396	20,748	72,756	571	215	2,704	68	89	54	7	6.83	10,386	522,091	48,183
oidc-code-flow	11,901	17,714	60,397	447	74	910	68	89	54	6.26	6.83	8,357	302,868	40,937
reactive-pg-client	11,488	18,321	57,413	447	33	775	68	89	54	6.61	6.83	7,955	292,403	38,548
rest-client	13,327	21,433	65,975	434	17	751	68	89	54	6.48	6.83	7,668	280,087	37,937
spring-cloud-config-client	12,352	19,692	63,333	447	19	788	68	89	54	6.8	6.83	8,596	302,377	42,535
elasticsearch-rest-client	11,595	16,819	57,814	439	37	850	68	89	54	6.57	6.82	7,965	293,625	39,093
oidc-client	11,508	17,301	58,061	477	65	892	68	89	54	6.73	6.79	9,505	330,735	46,719
no-awt	10,795	17,238	53,914	436	19	744	69	89	57	6.44	6.78	7,418	275,637	36,075
narayana-jta	9,908	14,555	48,811	430	19	740	68	89	54	6.39	6.76	6,794	256,269	32,514
resteasy-jackson	11,412	16,405	57,383	444	107	1,205	68	89	54	6.48	6.76	7,859	302,655	38,531
spring-boot-properties	10,661	16,988	53,629	440	22	769	68	89	54	6.36	6.76	7,343	275,105	35,952
elytron-resteasy-reactive	10,834	15,560	54,870	447	24	787	68	89	54	6.58	6.75	7,709	284,445	37,677
logging-gelf	10,731	17,097	53,942	436	19	743	68	89	54	6.47	6.75	7,378	274,783	36,125
logging-min-level-set	9,603	14,067	47,477	425	24	739	68	89	54	6.05	6.75	6,607	251,648	31,718
quartz	11,164	16,916	57,015	459	29	1,192	68	89	54	6.64	6.75	7,717	310,021	38,424
vertx-graphql	10,658	16,961	53,559	439	22	753	68	89	54	6.54	6.75	7,344	274,599	35,976
smallrye-metrics	10,901	15,408	52,676	444	25	839	68	89	54	6.37	6.67	7,673	280,709	35,552
elytron-security-jdbc	11,367	18,171	57,223	449	25	793	68	89	54	6.57	6.66	7,862	287,516	38,405
grpc-tls	12,386	20,167	63,647	461	39	1,002	68	89	54	6.76	6.66	8,492	311,723	41,934
logging-min-level-unset	9,594	14,056	47,417	422	21	735	68	89	54	5.97	6.66	6,598	250,953	31,673
rest-client-reactive	11,052	16,313	55,922	430	8	725	68	89	54	6.43	6.66	7,567	276,150	37,462
smallrye-opentracing	13,422	19,820	66,256	501	56	1,165	68	89	55	7.08	6.63	9,220	346,551	44,368
jsonb	10,200	14,515	49,672	429	21	758	68	89	54	6.23	6.61	7,088	260,349	33,195
rest-client-reactive-stork	10,932	16,088	55,377	520	593	1,242	68	89	54	6.77	6.55	9,205	345,229	44,258
elytron-security-ldap	10,806	15,398	52,731	442	27	775	68	89	54	6.26	6.54	7,536	271,243	35,319
elytron-security-oauth2	10,727	15,231	52,511	446	42	814	68	89	54	6.47	6.52	7,481	272,748	35,210
native-config-profile	9,649	14,186	48,065	421	19	746	68	89	54	6.08	6.5	6,661	251,741	32,314
jackson	10,629	15,368	53,757	444	45	828	68	89	54	6.45	6.44	7,338	271,083	36,248
vertx	10,799	15,805	53,427	386	13	680	68	89	54	6.25	6.44	7,709	259,154	35,063
vertx-web-jackson	9,928	14,442	50,902	395	11	697	68	89	54	6.15	6.41	6,866	259,131	34,610
mailer	11,227	16,398	54,640	467	29	801	68	89	54	6.41	6.38	7,928	282,306	36,510
logging-json	9,619	14,085	47,524	421	19	726	68	89	54	6.25	6.37	6,608	250,660	31,740
oidc	13,667	22,091	69,512	433	49	726	68	89	54	6.62	6.35	8,043	284,749	39,292
vertx-web	9,853	14,200	50,166	390	10	686	68	89	54	6.03	6.27	6,802	249,037	31,931
amazon-lambda-http	7,854	10,744	38,894	370	118	932	66	85	53	6.02	6.01	5,372	239,599	26,467
test-extension	9,368	13,996	46,534	392	105	847	68	89	54	6.01	5.99	6,424	241,369	30,950
spring-data-jpa	16,233	23,278	81,797	657	215	2,144	68	89	54	7.27	5.96	11,077	461,813	55,564
grpc-interceptors	11,491	17,275	57,844	436	27	1,012	68	89	54	6.43	5.83	7,869	283,125	37,853
picocli-native	6,583	9,288	33,710	529	590	2,671	66	85	53	5.92	5.59	4,548	287,917	22,990

[zakkak]

Depending on the case using ParseOnce results in increasing Peak RSS by up to ~30% while there are cases where Peak RSS is reduced by ~18%, but in most cases we see an increase (as expected) ranging from 1% to 15%.

[jerboaa]

Very interesting results. Do you have numbers comparing mandrel vs Graal VM CE? It would certainly be useful to monitor those results continuously. Especially for Quarkus as we see a significant change pre/post AWT extension integration.

[zakkak]

Do you have numbers comparing mandrel vs Graal VM CE?

No, I might schedule some runs for the weekend.

It would certainly be useful to monitor those results continuously.
Especially for Quarkus as we see a significant change pre/post AWT extension integration.

I agree, the integration-tests are already being built regularly, so what I think we need to do is to gather the data and raise some alarm in case of big differences between runs.

[zakkak]

Update 2022-08-17

After running similar tests with GraalVM CE 22.2.0 I am still observing similar behavior.

To make it easier to parse I have created a plot form the data in #316 (comment)

and a plot with the new data using GraalVM CE 22.2.0

Comparing the two plots we see that there is a significant reduction in Peak RSS in 22.2.0 and the gap between +ParseOnce and -ParseOnce seems to be shrinking. However, the difference between the two still ranges between -12% and 19% with the majority of the integration tests ending up needing more Peak RSS, see next plot:

[jerboaa]

Nice work! So it seems -H:-ParseOnce still makes sense.

[zakkak]

Update 2022-12-15

After running similar tests with Mandrel 22.3.0 I am still observing similar behavior.

Note: Some tests didn't compile in one configuration but compiled in the other. Not sure why that is (they should all compile) but the trend is still clear.

[zakkak]

Closing this as ParseOnce has been the default for quite some time in Quarkus and it's also now deprecated (and has no longer any effect).