Statistics

Day

Script speed, input and output size was calculated for 2017-02-01.

data_collection.py statistics:

Stream	Input size	Output size	Time
AAA	17.5 G	633.8 M	8min 14s
EOS	110.3 G	1.3 G	20min 40s
CMSSW	663.0 M	540.0 M	3min 18s
CRAB	292.2 M	294.1 M	4min 33s
Total		2.7 G	37min 46s*

data_aggregation.py statistics:

Stream	Input size	Output size	Input records	Output records	Time
AAA	17.5 G	541 K	11638431	4557	10min 30s
EOS	110.3 G	568.4 K	119512504	3902	19min 9s
CMSSW	663.0 M	1.2 M	2378179	9580	3min 58s
CRAB	292.2 M	965 K	3502211	8134	6min 58s
Total		3.2 M	137031325	26173	52min 35s*

*Total time includes DBS tables loading, file_block_size table generation (usually takes 5-6min in aggregation script), data output and some other functions

Verbose logging was turned on.

Month

data_aggregation.py took 11 hours and 54 minutes to complete for 31 days (August, 2017). This averages to under 24 minutes per day.

Verbose logging was turned off.