Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix
Browse files Browse the repository at this point in the history
fstagni committed Oct 4, 2024
1 parent 6a5c986 commit ee5ed6f
Showing 5 changed files with 155 additions and 68 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/create.yml
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ jobs:

- name: Install dependencies and build
run: |
npm install @iconify-json/clarity @iconify-json/devicon @slidev/cli @slidev/theme-default slidev-theme-hep slidev-theme-neversink
npm install @iconify-json/clarity @iconify-json/devicon @slidev/cli @slidev/theme-default slidev-theme-hep slidev-theme-neversink playwright-chromium
npm run build -- --base /${{ github.event.repository.name }}/
npm list
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
"version": "1.0.0",
"main": "index.js",
"scripts": {
"build": "slidev build",
"build": "slidev build --download",
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
Binary file added public/images/WebApp.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added public/images/fcc.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
219 changes: 153 additions & 66 deletions slides.md
Original file line number Diff line number Diff line change
@@ -55,26 +55,57 @@ color: gray-light
---

<div style="display: flex; flex-wrap: wrap; justify-content: space-around; gap: 20px;">
<img id="DIRAC" src="/public/images/Juno.jpeg" alt="Juno" style="width: 80px;">
<img id="DIRAC" src="/public/images/Belle2_logo.png" alt="Belle2" style="width: 80px;">
<img id="DIRAC" src="/public/images/CTA.jpg" alt="CTA" style="width: 80px;">
<img id="DIRAC" src="/public/images/ilc.png" alt="ILC" style="width: 80px;">
<img id="DIRAC" src="/public/images/LHCb.png" alt="LHCb" style="width: 80px;">
<img id="DIRAC" src="/public/images/nica.jpg" alt="nica" style="width: 80px;">
<img id="DIRAC" src="/public/images/BES3.png" alt="BES3" style="width: 80px;">
<img id="DIRAC" src="/public/images/GridPP-Logo.png" alt="gridPP" style="width: 80px;">
<img id="DIRAC" src="/public/images/pierre-auger-logo.png" alt="pierre-auger" style="width: 80px;">
<img id="DIRAC" src="/public/images/EGI.png" alt="EGI" style="width: 80px;">
<img id="DIRAC" src="/public/images/cepc.png" alt="cepc" style="width: 80px;">
<img id="DIRAC" src="/public/images/na62.jpeg" alt="na62" style="width: 80px;">
<img id="DIRAC" src="/public/images/t2k.png" alt="t2k" style="width: 80px;">
<img id="DIRAC" src="/public/images/WeNMR-logo.png" alt="weNMR" style="width: 80px;">
<img id="DIRAC" src="/public/images/LogoHyperK.png" alt="hyperk" style="width: 80px;">
<img id="DIRAC" src="/public/images/euclid.jpeg" alt="euclid" style="width: 80px;">
<img id="DIRAC" src="/public/images/LZ.png" alt="lz" style="width: 80px;">
<img id="Juno" src="/public/images/Juno.jpeg" alt="Juno" style="width: 90px;">
<img id="Belle2" src="/public/images/Belle2_logo.png" alt="Belle2" style="width: 90px;">
<img id="CTA" src="/public/images/CTA.jpg" alt="CTA" style="width: 90px;">
<img id="ilc" src="/public/images/ilc.png" alt="ILC" style="width: 90px;">
<img id="LHCb" src="/public/images/LHCb.png" alt="LHCb" style="width: 90px;">
<img id="nica" src="/public/images/nica.jpg" alt="nica" style="width: 90px;">
<img id="BES3" src="/public/images/BES3.png" alt="BES3" style="width: 90px;">
<img id="GriPP" src="/public/images/GridPP-Logo.png" alt="gridPP" style="width: 90px;">
<img id="PA" src="/public/images/pierre-auger-logo.png" alt="pierre-auger" style="width: 90px;">
<img id="EGI" src="/public/images/EGI.png" alt="EGI" style="width: 90px;">
<img id="cepc" src="/public/images/cepc.png" alt="cepc" style="width: 90px;">
<img id="na62" src="/public/images/na62.jpeg" alt="na62" style="width: 90px;">
<img id="t2k" src="/public/images/t2k.png" alt="t2k" style="width: 90px;">
<img id="weNMR" src="/public/images/WeNMR-logo.png" alt="weNMR" style="width: 90px;">
<img id="hyperk" src="/public/images/LogoHyperK.png" alt="hyperk" style="width: 90px;">
<img id="euclid" src="/public/images/euclid.jpeg" alt="euclid" style="width: 90px;">
<img id="LZ" src="/public/images/LZ.png" alt="lz" style="width: 90px;">
<img id="fcc" src="/public/images/fcc.png" alt="lz" style="width: 90px;">
</div>


---
layout: top-title
color: gray-light
align: l
title: history
---

:: title ::

# DIRAC timeline

:: content ::

```mermaid
timeline
section LHCb software
around 2000 : MC production system: bash scripts running at production sites
2002 : DIRAC2 <br> Rewritten in Python, using xml-rpc, interfacing to EDG
Data Challenge 04 : First successful grid usage ever.
: First use of pilot jobs based WMS
2006-2007 : DIRAC3<br> Full rewriting, development of the DISET protocol -- still in use today!
: the current DIRAC framework is still based on this work
section Open sourced, wider adoption
2008 : Large-ish reshuffling to become multi-VO
: LHCbDIRAC extension separated from Vanilla DIRAC code
2009 : CLIC community adopts DIRAC
2011 : France-Grilles is the first multi-VO DIRAC installation
2012 : Belle2, BES3, CTA adopt DIRAC
2021 : Python3 full support
```


---
@@ -86,7 +117,7 @@ title: WMS

:: title ::

# Workload Management System
# WMS: Workload Management System
- Pull model based on Pilot jobs
- Also "Push" solution for HPCs that do not support pilots (because of limited internet access).
- Will integrate [CWL (Common Workflow Language)](https://www.commonwl.org) as a way of defining jobs (replacing JDL)
@@ -126,15 +157,14 @@ titlewidth: is-5

:: title ::

# Data Management System
# DMS: Data Management System
It’s about **files**:​ placing, replicating, removing files​

- there are **LFNs** (logical file names) → and users ONLY work with these​
- **LFNs** are registered in *catalog(s)​*
- where are the LFNs? (in the DIRAC File Catalog (DFC), or in Rucio)​
- what are their metadata? (in the DFC, or in the LHCb Bookkeeping, or in AMGA)​
- LFNs *may* have **PFNs**, stored in **SEs**.​
- You can access those PFNs with several protocols.​
- LFNs *may* have **PFNs**, stored in **SEs**, that can accessed with several protocols.​

:: content ::

@@ -166,12 +196,12 @@ title: Productions

- A `production` is a *Data Processing transformation* (e.g. Simulation, Merge, DataReconstruction…). A Production creates jobs in the WMS (and re-sumbit them, and destroy them).​

- (Unless you are using Rucio) a *Data Manipulation transformation* replicates, or remove, data from storage elements.
- (Unless you are using Rucio) a *Data Manipulation* `transformation` replicates, or remove, data from storage elements.

:: content ::

<span class="bg-cyan-100 text-cyan-600 p-4 border-l-6 border-2 border-cyan-400 rounded-lg pl-8 pr-8 w-full block">
The **Transformation System (TS)** is used to automate common tasks related to production activities
The Transformation System (TS) is used to automate common tasks related to production activities
</span>

&nbsp;
@@ -193,35 +223,34 @@ TransformationSystem((TransformationSystem))
```

---
layout: top-title
layout: top-title-two-cols
align: c-lm-lm
color: gray-light
align: l
title: history
title: webapp
---


:: title ::

# Brief history of DIRAC
# Tech stack

:: content ::
:: left ::

- DIRAC is written in python3
- with the exception of the DIRAC Pilot, which needs to still be able to run with Python2
- Services are exposed at urls like `dips://box.some.where:9132/WorloadManagement/`
- `dips` stands for "DIRAC protocol"
- The DIRAC framework provides also "Agents" and "Executors" to animate the system
- As backends, MySQL and OpenSearch are supported (for different purposes)

:: right ::

DIRAC also provides a WebApp

<img id="WebApp" src="/public/images/WebApp.png">

Implemented using `ExtJS`, and fully custom Python "bindings"

```mermaid
timeline
section LHCb software
around 2000 : MC production system: bash scripts running at production sites
2002 : DIRAC2 <br> Rewritten in Python, using xml-rpc, interfacing to EDG
Data Challenge 04 : First successful grid usage ever.
: First use of pilot jobs based WMS
2006-2007 : DIRAC3<br> Full rewriting, development of the DISET protocol -- still in use today!
: the current DIRAC framework is still based on this work
section Open sourced, wider adoption
2008 : Large-ish reshuffling to become multi-VO
: LHCbDIRAC extension separated from Vanilla DIRAC code
2009 : CLIC community adopts DIRAC
2011 : France-Grilles is the first multi-VO DIRAC installation
2012 : Belle2, BES3, CTA adopt DIRAC
2021 : Python3 full support
```

---
layout: side-title
@@ -237,7 +266,7 @@ title: issues

:: content ::

<ul class="text-sm mx-auto">
<ul class="list-image-none font-sans text-sm mx-auto">
<li> complex, with high entrance bar</li>
<li> somewhat cumbersome deployment</li>
<li> late on “standards”
@@ -253,14 +282,53 @@ title: issues
<li> no clear interface to a running DIRAC instance</li>
</ul>


---
layout: section
color: cyan-light
---
---

## DiracX, the neXt DIRAC incarnation


---
layout: top-title-two-cols
color: gray-light
align: c-lm-lm
title: requirements
---

## Developers and maintainers requirements

- pippo
- pippo
- pippo
- pippo


:: title ::

# Requirements

:: left ::

## Communities/Users requirements

- pippo
- pippo
- pippo
- pippo


:: right ::

## Administrator requirements

- pippo
- pippo
- pippo
- pippo

---
layout: side-title
side: left
@@ -293,11 +361,11 @@ color: gray-light

# DiracX Web API

- DIRAC Web APIs are developed using FastAPI
- Nicely documented in Swagger (or Redoc), this what you see on the right
- DIRAC Web APIs are developed using FastAPI <span class="iconify" data-icon="devicon:fastapi-wordmark" data-inline="false"></span>
- Nicely documented in Swagger (or Redoc), this what you see on the right <span class="iconify" data-icon="devicon:swagger-wordmark" data-inline="false"></span>

<AdmonitionType type='caution' >
What is on the right is the certification WebApp, loaded live. Use with caution!
What is on the right is the certification Web API, loaded live. Use with caution!
</AdmonitionType>

---
@@ -378,8 +446,27 @@ https://github.com/DIRACGrid/diracx-web
What is on the left is the certification WebApp, loaded live. Use with caution!
</AdmonitionType>

---
layout: default
color: gray-light
---

# Architecture diagram



---
layout: quote
color: sky-light
quotesize: text-m
authorsize: text-s
author: 'Some of you out there'
---

"OK, but the Grid still uses proxies.

VOMS is alive!"

---
layout: top-title
color: gray-light
@@ -395,13 +482,19 @@ title: tokens

DIRAC uses X509 certificates and proxies to verify identities. DiracX uses tokens. For a (long) while, **users will have both a token and proxy**.



---
layout: default
color: gray-light
layout: quote
color: sky-light
quotesize: text-m
authorsize: text-s
author: 'Again, some of you out there'
---

# Architecture diagram
"OK, but there are several communities using DIRAC right now.

What should they do?"

---
layout: side-title
@@ -487,9 +580,14 @@ title: Extensions

:: content ::

The ol'good DIRAC is extendable: from the main code (DIRAC) to the Web and the Pilot, communites can create/run extensions.

DiracX follows the same path, including DiracX-Web. We also provide a reference extension, dubbed "gubbins"
Dirac has to support different Communities with different workflows and requirements. For example:
- LHCb is a HEP experiment. Stores the metadata and provenance of every produced files in a LHCb-specific database (with an Oracle backend)
- Belle2 is a HEP experiment. Uses Rucio as data management solution.
- CTAO is an Astrophysics experiment with radically different requirements (wrt to HEP exp.) on how to process its data
- WeNMR brings together complementary research teams in the structural biology and life science area

It is sometimes necessary to extend all Dirac(X) components (including Web and Pilot). DiracX aims to provide an easy way to do so.


---
@@ -551,7 +649,6 @@ title: summary

# Summary


:: left ::

<img id="DiracX" src="https://raw.githubusercontent.com/DIRACGrid/management/master/branding/diracx/svg/diracx-logo-square.svg" class="mx-auto w-4/5"> </img>
@@ -603,16 +700,6 @@ speed: 0.4
Federico Stagni <i>CERN, LHCb</i><br/>
Andrei Tsaregorotsev <i>CPPM (FR), EGI and LHCb</i>
</div>
<!-- <div class="grid-item text-right mr-4 col-span-1">
<strong>Software (very limited list)</strong>
</div>
<div class="grid-item col-span-2">
FastAPI<br/>
MySQL/PostGres<br/>
OpenSearch<br/>
MinIO<br/>
K8 and Helm
</div> -->
</div>

&nbsp;

0 comments on commit ee5ed6f

Please sign in to comment.