Create spark job to get user file acc times using wmarchive lfnarray belonging crab jobs

[mrceyhun]

This PR includes calculation of last/first access times of datasets accessed by user jobs, using WMArchive data by filtering only CRAB* jobtype.

There are so many lines for extracting DBS additional information but main logic that extract LFN files and their information from WMArchive data can be found in udf_lfn_extract function.

@drkovalskyi WMArchive HDFS data goes back to 18 months, so to 2021-03. It means that if a file was accessed before 2021-03-01, we will not have this information. I'm running the spark job on full data and send it to ES. I'll inform you when it is visible in Kibana.

@vkuznet you're the WMArchive expert and creator of its producer. If you've time, a review would be great.

Bash script is long but it's our general script that we use for other cron jobs, so it's trivial on our side.

[mrceyhun]

Changed access time from data.wmats to data.meta_data.ts

[brij01]

Thanks @mrceyhun.
changes look good to me, description in the script file is very clear as well.
awaiting review by WMArchive expert @vkuznet

[vkuznet]

this is very dangerous construct which hides the dependency that this script requires specific data structure with python source. I rather prefer to put it on top of the script and perform check of loading some python module which this script requires. If module import fail you can through an error asking user to setup proper PYTHONPATH environment.

[vkuznet]

The script should print all environment variable it uses, e.g.

echo "LOG_DIR=$LOGDIR"
...

This will help you later in debug process.

[vkuznet]

please add author part into the script.

[vkuznet]

I suggest to add dump of all global variable to the stdout to help debugging process

[vkuznet]

please add docstring to describe format of input file.

[mrceyhun]

Thanks @vkuznet , I applied all changes.

@drkovalskyi Spark job is ready to test. Since it takes so much time, I could not fully test it, I'll do tomorrow. And also there were some problems in swan which slowed me a bit.

In general:

• dataset creation time will come from max(CREATION_DATE) of files.
• Instead of filtering only CRAB3 wma jobs, I included all of them in case we need.
• Joined them with previous access times that come from Rucio Replicas. There will be separate columns for dataset last access times, such as RucioLastAccess and WmaLastAccess.
• Only datasets on DISK and prod RSEs(not test/temp) are used.
• There are so many logical operations and joins of 10 tables, hope I'm not missing anything.

[mrceyhun]

This spark job requires proper documentation when it is completely done. Let me explain latest changes here:

As requested by Dima, we need to get last access times and access counts of user jobs to datasets. WMArchive provides user job information in jobtype:CRAB3. Without using any jobstatus filter, WmaAccessCnt is calculated for each dataset. Without giving into more detail, this information is extracted from files the lots of joins and aggregations with files of LFNArray in WMArchive, Rucio tables and DBS tables.
To be able to calculate access count properly, we need to calculate accesses to child datasets too. DBS dataset_parents table provide the parent/child relationship between datasets. Because a dataset can have multiple parents, I used a custom logic to get parent datasets count including their child datasets counts. This gist link gives a hint how it is calculated: https://gist.github.com/mrceyhun/b2b13dab8bc401d7f6b0e6035866d154 and you can check related functions. Said that WmaTotalAccessCnt provides total USER accesses to dataset and WmaAccessCnt provides USER access to dataset itself only. If we can say how deep parent:child dataset relationship is AS hierarchy level, to make things efficient, I did not include calculations after 5 level, though we can include. For example, NANOAOD->MINIAOD->AOD->RAW is 3 level: child->parent->grandparent->grandgrandparent .

When we come to access times, LastAccess provides latest access to dataset which include Rucio REPLICAS values. However, WmaLastAccess and WmaFirstAccess come from WMArchive USER jobs. As it can be understood by the naming, any field starts with Wma means only CRAB3 job results of WMArchive.

Last but not least, there 2 critical filters: RseType: DISK and RseKind:prod which means no test/temp RSE files(replicas) are included in the calculations that requires Rucio REPLICAS table, like LastAccess.

fyi @drkovalskyi