Skip to content

Commit

Permalink
fix
Browse files Browse the repository at this point in the history
  • Loading branch information
fstagni committed Oct 4, 2024
1 parent 6a5c986 commit 0d4f221
Show file tree
Hide file tree
Showing 5 changed files with 126 additions and 65 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/create.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:

- name: Install dependencies and build
run: |
npm install @iconify-json/clarity @iconify-json/devicon @slidev/cli @slidev/theme-default slidev-theme-hep slidev-theme-neversink
npm install @iconify-json/clarity @iconify-json/devicon @slidev/cli @slidev/theme-default slidev-theme-hep slidev-theme-neversink playwright-chromium
npm run build -- --base /${{ github.event.repository.name }}/
npm list
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"version": "1.0.0",
"main": "index.js",
"scripts": {
"build": "slidev build",
"build": "slidev build --download",
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
Expand Down
Binary file added public/images/WebApp.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added public/images/fcc.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
187 changes: 124 additions & 63 deletions slides.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,26 +55,57 @@ color: gray-light
---

<div style="display: flex; flex-wrap: wrap; justify-content: space-around; gap: 20px;">
<img id="DIRAC" src="/public/images/Juno.jpeg" alt="Juno" style="width: 80px;">
<img id="DIRAC" src="/public/images/Belle2_logo.png" alt="Belle2" style="width: 80px;">
<img id="DIRAC" src="/public/images/CTA.jpg" alt="CTA" style="width: 80px;">
<img id="DIRAC" src="/public/images/ilc.png" alt="ILC" style="width: 80px;">
<img id="DIRAC" src="/public/images/LHCb.png" alt="LHCb" style="width: 80px;">
<img id="DIRAC" src="/public/images/nica.jpg" alt="nica" style="width: 80px;">
<img id="DIRAC" src="/public/images/BES3.png" alt="BES3" style="width: 80px;">
<img id="DIRAC" src="/public/images/GridPP-Logo.png" alt="gridPP" style="width: 80px;">
<img id="DIRAC" src="/public/images/pierre-auger-logo.png" alt="pierre-auger" style="width: 80px;">
<img id="DIRAC" src="/public/images/EGI.png" alt="EGI" style="width: 80px;">
<img id="DIRAC" src="/public/images/cepc.png" alt="cepc" style="width: 80px;">
<img id="DIRAC" src="/public/images/na62.jpeg" alt="na62" style="width: 80px;">
<img id="DIRAC" src="/public/images/t2k.png" alt="t2k" style="width: 80px;">
<img id="DIRAC" src="/public/images/WeNMR-logo.png" alt="weNMR" style="width: 80px;">
<img id="DIRAC" src="/public/images/LogoHyperK.png" alt="hyperk" style="width: 80px;">
<img id="DIRAC" src="/public/images/euclid.jpeg" alt="euclid" style="width: 80px;">
<img id="DIRAC" src="/public/images/LZ.png" alt="lz" style="width: 80px;">
<img id="Juno" src="/public/images/Juno.jpeg" alt="Juno" style="width: 90px;">
<img id="Belle2" src="/public/images/Belle2_logo.png" alt="Belle2" style="width: 90px;">
<img id="CTA" src="/public/images/CTA.jpg" alt="CTA" style="width: 90px;">
<img id="ilc" src="/public/images/ilc.png" alt="ILC" style="width: 90px;">
<img id="LHCb" src="/public/images/LHCb.png" alt="LHCb" style="width: 90px;">
<img id="nica" src="/public/images/nica.jpg" alt="nica" style="width: 90px;">
<img id="BES3" src="/public/images/BES3.png" alt="BES3" style="width: 90px;">
<img id="GriPP" src="/public/images/GridPP-Logo.png" alt="gridPP" style="width: 90px;">
<img id="PA" src="/public/images/pierre-auger-logo.png" alt="pierre-auger" style="width: 90px;">
<img id="EGI" src="/public/images/EGI.png" alt="EGI" style="width: 90px;">
<img id="cepc" src="/public/images/cepc.png" alt="cepc" style="width: 90px;">
<img id="na62" src="/public/images/na62.jpeg" alt="na62" style="width: 90px;">
<img id="t2k" src="/public/images/t2k.png" alt="t2k" style="width: 90px;">
<img id="weNMR" src="/public/images/WeNMR-logo.png" alt="weNMR" style="width: 90px;">
<img id="hyperk" src="/public/images/LogoHyperK.png" alt="hyperk" style="width: 90px;">
<img id="euclid" src="/public/images/euclid.jpeg" alt="euclid" style="width: 90px;">
<img id="LZ" src="/public/images/LZ.png" alt="lz" style="width: 90px;">
<img id="fcc" src="/public/images/fcc.png" alt="lz" style="width: 90px;">
</div>


---
layout: top-title
color: gray-light
align: l
title: history
---

:: title ::

# DIRAC timeline

:: content ::

```mermaid
timeline
section LHCb software
around 2000 : MC production system: bash scripts running at production sites
2002 : DIRAC2 <br> Rewritten in Python, using xml-rpc, interfacing to EDG
Data Challenge 04 : First successful grid usage ever.
: First use of pilot jobs based WMS
2006-2007 : DIRAC3<br> Full rewriting, development of the DISET protocol -- still in use today!
: the current DIRAC framework is still based on this work
section Open sourced, wider adoption
2008 : Large-ish reshuffling to become multi-VO
: LHCbDIRAC extension separated from Vanilla DIRAC code
2009 : CLIC community adopts DIRAC
2011 : France-Grilles is the first multi-VO DIRAC installation
2012 : Belle2, BES3, CTA adopt DIRAC
2021 : Python3 full support
```


---
Expand All @@ -86,7 +117,7 @@ title: WMS

:: title ::

# Workload Management System
# WMS: Workload Management System
- Pull model based on Pilot jobs
- Also "Push" solution for HPCs that do not support pilots (because of limited internet access).
- Will integrate [CWL (Common Workflow Language)](https://www.commonwl.org) as a way of defining jobs (replacing JDL)
Expand Down Expand Up @@ -126,15 +157,14 @@ titlewidth: is-5

:: title ::

# Data Management System
# DMS: Data Management System
It’s about **files**:​ placing, replicating, removing files​

- there are **LFNs** (logical file names) → and users ONLY work with these​
- **LFNs** are registered in *catalog(s)​*
- where are the LFNs? (in the DIRAC File Catalog (DFC), or in Rucio)​
- what are their metadata? (in the DFC, or in the LHCb Bookkeeping, or in AMGA)​
- LFNs *may* have **PFNs**, stored in **SEs**.​
- You can access those PFNs with several protocols.​
- LFNs *may* have **PFNs**, stored in **SEs**, that can accessed with several protocols.​

:: content ::

Expand Down Expand Up @@ -166,12 +196,12 @@ title: Productions

- A `production` is a *Data Processing transformation* (e.g. Simulation, Merge, DataReconstruction…). A Production creates jobs in the WMS (and re-sumbit them, and destroy them).​

- (Unless you are using Rucio) a *Data Manipulation transformation* replicates, or remove, data from storage elements.
- (Unless you are using Rucio) a *Data Manipulation* `transformation` replicates, or remove, data from storage elements.

:: content ::

<span class="bg-cyan-100 text-cyan-600 p-4 border-l-6 border-2 border-cyan-400 rounded-lg pl-8 pr-8 w-full block">
The **Transformation System (TS)** is used to automate common tasks related to production activities
The Transformation System (TS) is used to automate common tasks related to production activities
</span>

&nbsp;
Expand All @@ -193,35 +223,34 @@ TransformationSystem((TransformationSystem))
```

---
layout: top-title
layout: top-title-two-cols
align: c-lm-lm
color: gray-light
align: l
title: history
title: webapp
---


:: title ::

# Brief history of DIRAC
# Tech stack

:: content ::
:: left ::

- DIRAC is written in python3
- with the exception of the DIRAC Pilot, which needs to still be able to run with Python2
- Services are exposed at urls like `dips://box.some.where:9132/WorloadManagement/`
- `dips` stands for "DIRAC protocol"
- The DIRAC framework provides also "Agents" and "Executors" to animate the system
- As backends, MySQL and OpenSearch are supported (for different purposes)

:: right ::

DIRAC also provides a WebApp

<img id="WebApp" src="/public/images/WebApp.png">

Implemented using `ExtJS`, and fully custom Python "bindings"

```mermaid
timeline
section LHCb software
around 2000 : MC production system: bash scripts running at production sites
2002 : DIRAC2 <br> Rewritten in Python, using xml-rpc, interfacing to EDG
Data Challenge 04 : First successful grid usage ever.
: First use of pilot jobs based WMS
2006-2007 : DIRAC3<br> Full rewriting, development of the DISET protocol -- still in use today!
: the current DIRAC framework is still based on this work
section Open sourced, wider adoption
2008 : Large-ish reshuffling to become multi-VO
: LHCbDIRAC extension separated from Vanilla DIRAC code
2009 : CLIC community adopts DIRAC
2011 : France-Grilles is the first multi-VO DIRAC installation
2012 : Belle2, BES3, CTA adopt DIRAC
2021 : Python3 full support
```

---
layout: side-title
Expand All @@ -237,7 +266,7 @@ title: issues

:: content ::

<ul class="text-sm mx-auto">
<ul class="list-image-none font-sans text-sm mx-auto">
<li> complex, with high entrance bar</li>
<li> somewhat cumbersome deployment</li>
<li> late on “standards”
Expand All @@ -253,14 +282,52 @@ title: issues
<li> no clear interface to a running DIRAC instance</li>
</ul>


---
layout: section
color: cyan-light
---
---

## DiracX, the neXt DIRAC incarnation


---
layout: top-title-two-cols
color: gray-light
align: c-lr-lr
---

## Developers and maintainers requirements

- pippo
- pippo
- pippo
- pippo


:: title ::

# Requirements

:: left ::

## Communities/Users requirements

- pippo
- pippo
- pippo
- pippo


:: right ::

## Administrator requirements

- pippo
- pippo
- pippo
- pippo

---
layout: side-title
side: left
Expand Down Expand Up @@ -293,11 +360,11 @@ color: gray-light

# DiracX Web API

- DIRAC Web APIs are developed using FastAPI
- Nicely documented in Swagger (or Redoc), this what you see on the right
- DIRAC Web APIs are developed using FastAPI <span class="iconify" data-icon="devicon:fastapi-wordmark" data-inline="false"></span>
- Nicely documented in Swagger (or Redoc), this what you see on the right <span class="iconify" data-icon="devicon:swagger-wordmark" data-inline="false"></span>

<AdmonitionType type='caution' >
What is on the right is the certification WebApp, loaded live. Use with caution!
What is on the right is the certification Web API, loaded live. Use with caution!
</AdmonitionType>

---
Expand Down Expand Up @@ -487,9 +554,14 @@ title: Extensions

:: content ::

The ol'good DIRAC is extendable: from the main code (DIRAC) to the Web and the Pilot, communites can create/run extensions.

DiracX follows the same path, including DiracX-Web. We also provide a reference extension, dubbed "gubbins"
Dirac has to support different Communities with different workflows and requirements. For example:
- LHCb is a HEP experiment. Stores the metadata and provenance of every produced files in a LHCb-specific database (with an Oracle backend)
- Belle2 is a HEP experiment. Uses Rucio as data management solution.
- CTAO is an Astrophysics experiment with radically different requirements (wrt to HEP exp.) on how to process its data
- WeNMR brings together complementary research teams in the structural biology and life science area

It is sometimes necessary to extend all Dirac(X) components (including Web and Pilot). DiracX aims to provide an easy way to do so.


---
Expand Down Expand Up @@ -551,7 +623,6 @@ title: summary

# Summary


:: left ::

<img id="DiracX" src="https://raw.githubusercontent.com/DIRACGrid/management/master/branding/diracx/svg/diracx-logo-square.svg" class="mx-auto w-4/5"> </img>
Expand Down Expand Up @@ -603,16 +674,6 @@ speed: 0.4
Federico Stagni <i>CERN, LHCb</i><br/>
Andrei Tsaregorotsev <i>CPPM (FR), EGI and LHCb</i>
</div>
<!-- <div class="grid-item text-right mr-4 col-span-1">
<strong>Software (very limited list)</strong>
</div>
<div class="grid-item col-span-2">
FastAPI<br/>
MySQL/PostGres<br/>
OpenSearch<br/>
MinIO<br/>
K8 and Helm
</div> -->
</div>

&nbsp;
Expand Down

0 comments on commit 0d4f221

Please sign in to comment.