Merge pull request #63 from lucalavezzo/main

CMS Production & Reprocessing Projects

project_metadata.yml

@@ -6,6 +6,7 @@ categories:
   - ML/AI
   - Analytics
   - Networking
+  - Computing
 
 durations:
   - 3 months

projects/pnr-cicd-automation.yml

@@ -0,0 +1,38 @@
+---
+name: CI/CD and Automation of Manual Operations
+postdate: 2024-01-30
+categories:
+  - Computing
+durations:
+  - 3 months
+experiments:
+  - CMS
+skillset:
+  - Python
+  - CI/CD
+status:
+  - Available
+project:
+  - IRIS-HEP
+location:
+  - Any
+commitment:
+  - Full time
+program:
+  - IRIS-HEP fellow
+shortdescription: Automate manual operations and implement CI/CD for CMS Production & Reprocessing group.
+description: >
+  The Production & Reprocessing (P&R) group is responsible for maintaining and operating the CMS central workload management system, which processes hundreds of physics workflows daily which produce the data that physicists use in their analyses. The requests which have a similar physics goal are grouped as ‘campaigns’. P&R manages the lifecycle of hundreds of campaigns, each with its unique needs. The objective of this project is to automate the checks that are performed manually on these campaigns. This involves creating a system to automatically set up, verify, and activate new campaigns, along with managing data storage and allocation. The second part of the project focuses on implementing a Continuous Integration and Continuous Deployment (CI/CD) pipeline for efficiently deploying and maintaining software services. This will include converting manual testing procedures into automated ones, improving overall efficiency and reducing errors. Tools such as Gitlab Pipelines for CI/CD, Python for scripting, and various automated testing frameworks will be employed. This initiative is designed to streamline operations, making them more efficient and effective.
+contacts:
+  - name: Hassan Ahmed
+    email: [email protected]
+  - name: Hasan Ozturk
+    email: [email protected]
+  - name: Luca Lavezzo
+    email: [email protected]
+  - name: Jennifer Adelman McCarthy
+    email: [email protected]
+  - name: Zhangqier Wang
+    email: [email protected]
+  - name: Dmytro Kovalskyi
+    email: [email protected]

projects/pnr-smart-job-retries.yml

@@ -0,0 +1,37 @@
+---
+name: Smart job retries for CMS workload management system
+postdate: 2024-01-30
+categories:
+  - Computing
+durations:
+  - 3 months
+experiments:
+  - CMS
+skillset:
+  - Python
+status:
+  - Available
+project:
+  - IRIS-HEP
+location:
+  - Any
+commitment:
+  - Full time
+program:
+  - IRIS-HEP fellow
+shortdescription: Develop a tool to monitor and make smart decisions on how to retry CMS grid jobs.
+description: >
+  The CMS experiment runs its data processing and simulation jobs on the Worldwide LHC Computing Grid in the scale of ~100k jobs in parallel. It’s inevitable to avoid job failures on this scale, and thus it’s crucial to have an effective failure recovery system. The existing algorithm is agnostic to the information of other jobs which run at the same site or belong to the same physics class. The objective of this project is to develop a tool which will monitor all the CMS grid jobs and make smart decisions on how to retry them by aggregating the data coming from different jobs across the globe. Such decisions can potentially be: reducing the job submission to computing sites experiencing particular failures, changing the job configuration in case of inaccurate configurations, and not retrying potentially ill-configured jobs. This project has the potential to significantly improve efficiency of the whole CMS computing grid, reducing the wasted cpu cycles and increasing the overall throughput.
+contacts:
+  - name: Hassan Ahmed
+    email: [email protected]
+  - name: Hasan Ozturk
+    email: [email protected]
+  - name: Luca Lavezzo
+    email: [email protected]
+  - name: Jennifer Adelman McCarthy
+    email: [email protected]
+  - name: Zhangqier Wang
+    email: [email protected]
+  - name: Dmytro Kovalskyi
+    email: [email protected]