Make google cloud output split requests if they are too large

[djaglowski]

resolves #257

[codecov]

Codecov Report

Merging #263 (94da7b4) into master (6749dc9) will increase coverage by 0.03%.
The diff coverage is 95.00%.

@@            Coverage Diff             @@
##           master     #263      +/-   ##
==========================================
+ Coverage   71.67%   71.70%   +0.03%     
==========================================
  Files         103      103              
  Lines        5676     5689      +13     
==========================================
+ Hits         4068     4079      +11     
- Misses       1172     1174       +2     
  Partials      436      436              

Impacted Files	Coverage Δ
operator/buffer/memory.go	76.52% <0.00%> (ø)
...perator/builtin/output/googlecloud/google_cloud.go	54.55% <100.00%> (+5.20%)	⬆️
operator/flusher/flusher.go	84.00% <0.00%> (-4.00%)	⬇️
operator/builtin/output/forward/forward.go	56.52% <0.00%> (-2.90%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6749dc9...94da7b4. Read the comment docs.

[djaglowski]

Log Files	Logs / Second	CPU Avg (%)	CPU Avg Δ (%)	Memory Avg (MB)	Memory Avg Δ (MB)
1	1000	1.8793484	+0.15516508	124.44774	+0.32569885
1	5000	6.569074	+0.4310422	132.93803	-0.7412567
1	10000	12.482988	+0.15513515	139.85965	-0.36206055
1	50000	59.757153	-4.1068764	176.33473	+0.53556824
1	100000	118.36555	+2.1602554	230.69559	+6.6095123
10	100	2.3449147	+0.01728487	128.44571	+1.4744034
10	500	7.500286	+0.1380024	138.7065	+2.5995483
10	1000	13.396839	-0.36199284	138.35184	-3.577179
10	5000	64.7081	+0.97166824	185.57382	+6.9484253
10	10000	136.8979	+9.948463	237.59227	+9.695587

[jsirianni]

Looks great, and resolves the issue I was seeing in my environment.

[djaglowski]

Log Files	Logs / Second	CPU Avg (%)	CPU Avg Δ (%)	Memory Avg (MB)	Memory Avg Δ (MB)
1	1000	1.9310892	+0.20690584	123.20407	-0.91796875
1	5000	6.3277926	+0.18976068	135.1906	+1.5113068
1	10000	13.15556	+0.82770634	140.23007	+0.008361816
1	50000	59.673717	-4.1903114	177.9437	+2.1445312
1	100000	123.60719	+7.4019012	232.4363	+8.35022
10	100	2.5345833	+0.20695353	128.32045	+1.349144
10	500	7.000205	-0.36207867	137.01671	+0.9097595
10	1000	14.293792	+0.5349598	140.04243	-1.8865814
10	5000	65.67377	+1.937336	182.38052	+3.755127
10	10000	127.6933	+0.74385834	215.9154	-11.981277

[KonradSchieban]

👍 thanks, looks good!

[KonradSchieban]

What is the message that is being logged in the case a larger message has been split? Is there a way to find that out? @djaglowski @jsirianni

[jsirianni]

Large single messages are not being split. This is what we are doing:

• If bulk submission (multiple log entries) too big, split the bulk entries in half and retry. Continue doing this until all messages are submitted.
• If a single message is too big, replace it's record with the error given by the Google API.

For example, I submitted a log entry that was well over the limit, the result in Cloud Logging looks like this:

[KonradSchieban]

Thanks @jsirianni , this is what I was looking for!