Provide BLAS, LAPACK backends and interfaces #772

perazz · 2024-03-11T13:11:24Z

This PR introduces BLAS and LAPACK backends and interfaces.

Modernized BLAS and LAPACK backends as developed at this repo: no implicit typing, all pure functions, kind-agnostic interfaces, old constructs removed, etc.
support for external libraries (e.g. OpenBLAS) via cpp macros.
Quadruple-precision support is provided by the local implementation, and enabled by the WITH_QP macro

The backend will necessarily support linear algebra development for users that don't have access to an external library, and, to enable quadruple precision support. How this was designed:

Macro FPM_DEPLOYMENT guards code that is needed to enable fpm branch to be linked to an external library. This is done with C preprocessor macros, so, they don't have to be present in the CMake build
Macros EXTERNAL_BLAS and EXTERNAL_LAPACK can be defined to make fypp use interfaces instead of the module subroutines. The latter are always available via modules stdlib_linalg_blas and stdlib_linalg_lapack
This PR does not address enabling OpenBLAS or other implementations in CMake yet. Current focus is to implement linear algebra subroutines, that will come next.

cf. #710 #749 #450
cc: @jvdp1 @jalvesz @gnikit @everythingfunctional @fortran-lang/stdlib @henilp105

gnikit · 2024-03-11T18:02:47Z

Any advice @perazz on how to attack this PR? I will have a detailed look on the code outside the src dir. Should we assume that code inside src/ has been reviewed already?

perazz · 2024-03-11T18:14:47Z

Yep, I've limited this PR to BLAS/LAPACK source code hoping to make everything easier to review. The sources in src/ are automated translations of LAPACK 3.10.1 and hence contain no changes to Reference-LAPACK besides the modernization effort (implicitness, intents, consistent TKR, indentation, ...). So I'm tempted to say that the fact that the code builds fine with the Modern fortran compile-time checks is itself a positive statement already.
I'm available to add more tests on the LAPACK/BLAS calls, but I think we'd be better off creating more tests later on the high-level APIs rather than to the raw LAPACK calls, what do you think?

Also: the most important thing to agree upon is the mechanism that allows to switch between internal/external implementation both in the CMake build and the fpm branch. Fpm only supports cpp, so this is how I'm doing it:

FPM_DEPLOYMENT is the fypp guard block that will let cpp directives be printed to the fpm branch
STDLIB_EXTERNAL_BLAS and STDLIB_EXTERNAL_LAPACK are the fypp directives that CMake will set to use an external library
External library? write interface. Otherwise: module procedure (internal implementation)

stdlib/src/stdlib_linalg_blas.fypp

Lines 20 to 37 in a356f14

    
           #:if FPM_DEPLOYMENT 
        
           #ifdef STDLIB_EXTERNAL_BLAS 
        
           #:endif 
        
           #:if FPM_DEPLOYMENT or STDLIB_EXTERNAL_BLAS 
        
                          pure subroutine caxpy(n,ca,cx,incx,cy,incy) 
        
                               import sp,dp,qp,ilp,lk  
        
                               implicit none(type,external)  
        
                               complex(sp), intent(in) :: ca,cx(*) 
        
                               integer(ilp), intent(in) :: incx,incy,n 
        
                               complex(sp), intent(inout) :: cy(*) 
        
                          end subroutine caxpy 
        
           #:endif  
        
           #:if FPM_DEPLOYMENT 
        
           #else  
        
           #:endif 
        
           #:if FPM_DEPLOYMENT or not STDLIB_EXTERNAL_BLAS 
        
                          module procedure stdlib_caxpy 
        
           #:endif

jalvesz · 2024-03-11T19:52:39Z

@perazz I'm intrigued why you needed to branch the build process this way? From what I understood after working in stdlib is that either with CMake or the fpm-deployement script, build keywords are passed and used to have the 2-stage preprocessing. The main difference being that with fpm we are obliged to call this script because of fypp, then the fpm build. With CMake it does both in one go.

I would have expected then that having exactly the same .f90(s) after fypp (script or CMake) and having cpp/fpp preprocessing activated, the build would follow the same rules.

(I'm thinking outloud here so please correct me if I'm missing something)

Ok, I think I'm getting the issue... with CMake, using find_package([...]) it can be possible to affect a variable like STDLIB_EXTERNAL_BLAS having found a BLAS library (OpenBLAS or mkl ?), which would easily then enable to compile with the conditional preprocessor. While with fpm, deciding on using OpenBLAS or mkl would require to modify the toml file which currently does not accept conditional build. Right?

Would it make sense to maybe use a python script instead of a shell script for the fpm preprocessing and probably modify the toml file accordingly for compiling against and external lib or not?

perazz · 2024-03-12T06:58:24Z

You've got the point @jalvesz - in fpm, interface-vs-implementation can only be resolved via cpp macros. With CMake there's more flexibility as you point out. One would be to use fypp+cpp to remove some of the boilerplate, but I didn't find issues/threads that supported introducing cpp as a second preprocessor, so I'm sticking with fypp only (though it's a bit more verbose).

I actually used a python script to generate that interface code, so if we find a better/more elegant alternative, I'm definitely up for putting the effort into it. However, I think that with these constraints standing, there's not many more options

jalvesz · 2024-03-12T07:12:39Z

but I didn't find issues/threads that supported introducing cpp as a second preprocessor, so I'm sticking with fypp only

I think there is at least stdlib_system.F90 that uses a #ifdef _WIN32 for OS dependent build. So the cpp/fpp preprocessing is already there even if not exploited more intensively (maybe this is a good thing?). In this case I'm guessing it is safe considering as a hierarchy of preprocessing instead of one vs the other.

I actually used a python script to generate that interface code, so if we find a better/more elegant alternative, I'm definitely up for putting the effort into it. However, I think that with these constraints standing, there's not many more options

If we find a way to conditionally affect the .toml file (or introduce conditionality within) we might get somewhere in this direction I guess... Maybe for fpm it could be possible to conditionally modify the extra --flag to add link build options, with a .rsp file for instance?

Coming back to

I'm available to add more tests on the LAPACK/BLAS calls, but I think we'd be better off creating more tests later on the high-level APIs rather than to the raw LAPACK calls, what do you think?

This seems like the best approach

perazz · 2024-03-12T13:23:43Z

Agree @jalvesz, here's an update that removes most of the boilerplate. now the files are pre-processed to .F90 instead of .f90, that seems the way using C preprocessing is enabled in stdlib.

jvdp1 · 2024-03-12T20:15:27Z

src/stdlib_linalg_constants.fypp

+     use stdlib_kinds, only: sp, dp, qp, int32, int64, lk
+     use, intrinsic :: ieee_arithmetic, only: ieee_is_nan 
+#if defined(_OPENMP)
+     use omp_lib


Suggested change

use omp_lib

!$ use omp_lib

Using "!$" will make this line compiled conditionnaly on the support of OpenMP by the compiler (and the CPP dir. is therefore not needed)

Thank you @jvdp1, interesting! These #if defined(_OPENMP) statements were in the original LAPACK and I left them unchanged. Is it standard Fortran that they should be always properly interpreted if they start with !$? If so, I could update them all

This is intriguing, what I had understood is that conditional directives require specific signatures such as !$omp or !$dir, etc. But this !$ something is still just a comment... no?

!$ something is a conditional compilation sentinel and part of the OpenMP API (see section 3.3.2 of the OpenMP specs).

@perazz It is indeed part of OpenMP API specs (at least, I think, since OpenMP 2.5). I use it since many years with both Intel and gfortran compilers. I guess that it is also supported by other compilers. However, note that I am not against the cpp directive. I just find using !$ cleaner ;)

Wow, didn't know that!! just tested a small code using this with nvfortran 24.1 and works as expected!

jvdp1 · 2024-03-12T20:29:35Z

Agree @jalvesz, here's an update that removes most of the boilerplate. now the files are pre-processed to .F90 instead of .f90, that seems the way using C preprocessing is enabled in stdlib.

thank you @perazz! I think that your approach of using fypp and fpp is the best way for this case, considering the current support of fypp by fpm. I guess that this approach could be reviewed in the future when fpm will fully support fypp.

gnikit

Great work! I am very excited to see this get merged.
I have some minor comments, but other than that it looks great to me.

Comments

-DWITH_QP would be good to have documentation in README.md
-DSTDLIB_EXTERNAL_BLAS would be good to have documentation in README.md
-DSTDLIB_EXTERNAL_LAPACK would be good to have documentation in README.md
As mentioned in the monthly call, and if easily doable, intend !$ sentinels to be on the same level of nesting as the code they operate on.
Could we write down somewhere the version of BLAS/LAPACK is being used for the reference implementation? I know you mentioned that it is v3.10.1 in the comments, but it would be good if it was explicitly stated somewhere in the docs.

A slightly bigger picture question and related to my last comment. If I understood correctly the modernisation script for BLAS/LAPACK is fully automated. Would it make sense to be incorporated as part of stdlib's CD pipeline to keep stdlib's reference implementation updated?

To be clear I am not suggesting this to be done in this PR, or even as a next step. It's more like a final touch to the whole project.

Great work again.

Revert "indent OpenMP sentinels" This reverts commit 35b721b. indent OpenMP, do not strip EOL

jvdp1

Thank you @perazz . LGTM.
It would be be nice if some specs could be added too, e.g., stdlib_linalg_blas.md and stdlib_linalg_lapack.md, mainly to document how these files were generated, from which version of BLAS/LAPACK, the phylosophy behind it (internal or external implementation,...).
IMO, for API of the different procedures, a link to any BLAS/LAPACK documentation is enough (no need to generate a full description of API of all procedures).

ci/fpm-deployment.sh

test/stats/array3.dat

src/stdlib_linalg_constants.fypp

gnikit · 2024-03-26T23:19:18Z

The other thing that I would suggest and might be helpful is to form a GitHub project (if one doesn't exist), to track Features/Issues specific to the BLAS/LAPACK work of stdlib.

jvdp1 · 2024-03-27T06:37:43Z

Hence, I am in favour of making another minor release of stdlib with v1 of the BLAS/LAPACK backends.

I agree with this. This PR will be quite a milestone, in terms of usage and of code size.

perazz · 2024-03-27T07:54:05Z

keep the kinds in a single module already provided by stdlib_kinds?

Thanks @jalvesz for testing it out! The reason I made a special kinds module for linear algebra is that some implementations (MKL) may opt for 64-bit integer users ("ILP64" version). I already took this into account by defining an ilp integer kind that applies to all integers, so, when we want to support it, we can just define the appropriate integer type here:

stdlib/src/stdlib_linalg_constants.fypp

Lines 11 to 13 in 545b4c3

    
           ! Integer size support for ILP64 builds should be done here 
        
           integer, parameter :: ilp = int32 
        
           private            :: int32, int64

We could move it all back to stdlib_kinds, no big deal if you think it's better. But maybe, down the road there will be more linear-algebra parameters, so this would be the right place to put them in.

there is a kind definition missing

Thanks for catching these! Yes this is definitely a typo.

Would it be the occasion to have a table in the head README.md

Absolutely. I'd suggest to do this as a separate issue though, as the current PR is already huge.

Should PR #774 also be included

Thanks @gnikit, It's very much necessary, as otherwise, we won't be able to start deploying any high-level API. So if you have time, please take a look at that one too! (it's a far smaller PR).

Thank you all @jvdp1 @jalvesz @gnikit for your time, and if you agree, let's try to get this merged soon.

ci/fpm-deployment.sh

src/stdlib_linalg_blas.fypp

jalvesz · 2024-03-26T21:49:36Z

src/stdlib_linalg_lapack.fypp

In this file, there are 8 interfaces with the following signature:
procedure(stdlib_sel<>) :: select
need adding
import sp,dp,qp,ilp,lk, stdlib_sel<> (stdlib_select_? or stdlib_selctg_?)

I think you may be right. Do procedure interfaces need to be imported? gfortran is not complaining for this, but it may be compiler-specific

Added in c12b3c3

I'm double checking with gfortran, both ifort and ifx complained.

Yeah I overlooked that because I'm testing with gfortran, but makes total sense, thanks for checking it!

gfortran accepts building with and without. Since you added it in the latest push I think this is good to go :)

perazz · 2024-03-28T14:08:16Z

With two approvals, I think it makes sense to wait another few days, and if no more comments are made, it should be ready to merge. Thank you all

jvdp1 · 2024-04-02T08:01:16Z

@perazz @gnikit @jalvesz Should we merge this PR to move on? Then a new release of stdlib can be generated and advertised.

perazz · 2024-04-02T08:23:47Z

I agree @jvdp1, I will merge this now.

jvdp1 · 2024-04-02T08:48:52Z

@perazz Thank you. I will prepare a PR for the release v0.5.0 .

perazz · 2024-04-02T08:56:22Z

@jvdp1 can I suggest to try review and merge also PR #774 before releasing v0.5.0? This would allow us to begin working on several high-level APIs in parallel, so people would have time to look and comment and we could iterate on them efficiently.

jvdp1 · 2024-04-02T09:11:59Z

Yes, sure. no problem. I suggest that you already advertise it (even if not being part of a release yet). Le mar. 2 avr. 2024 à 10:56, Federico Perini ***@***.***> a écrit :

…

@jvdp1 <https://github.com/jvdp1> can I suggest to try review and merge also PR #774 <#774> before releasing v0.5.0? This would allow us to begin working on several high-level APIs in parallel, so people would have time to look and comment and we could iterate on them efficiently. — Reply to this email directly, view it on GitHub <#772 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD5RO7D7I2ZXT7SYC5CKJQLY3JXEZAVCNFSM6AAAAABEQJBQHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZRGQ2DEMZRGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

cyrilgandon · 2024-04-05T08:55:59Z

FYI, our build failed on CMAKE following the addition or #ifdef directives in the file stdlib_linalg_blas.f90.
*.f90 files are generally not supposed to be preprocessed.

So we got some errors like

Error: Ambiguous interfaces in generic interface 'nrm2' for ‘snrm2’ at (1) and ‘stdlib_snrm2’ at (2)
/dev/build/_deps/stdlib-src/src/stdlib_linalg_blas.f90:848:35:

Anyway, we fix our build by adding in CMakeLists.txt

set(CMAKE_Fortran_PREPROCESS TRUE)

perazz · 2024-04-05T10:14:12Z

*.f90 files are generally not supposed to be preprocessed

@cyrilgandon what OS/config are you using? those files are preprocessed by fypp and return capitalized .F90 sources:

stdlib/src/CMakeLists.txt

Lines 67 to 71 in f53ad6b

    
           # Preprocessed files to contain preprocessor directives -> .F90  
        
           set(cppFiles 
        
               stdlib_linalg_constants.fypp 
        
               stdlib_linalg_blas.fypp 
        
               stdlib_linalg_blas_aux.fypp

cyrilgandon · 2024-04-05T11:25:06Z

I import the .f90 files of stdlib directly from the stdlib-fpm branch of the project like this:

FetchContent_Declare(
  stdlib
  GIT_REPOSITORY https://github.com/fortran-lang/stdlib.git
  GIT_TAG        stdlib-fpm
)
FetchContent_MakeAvailable(stdlib)

# Get all the f90 files
file(GLOB_RECURSE sources src/*.f90 app/*.f90 ${stdlib_SOURCE_DIR}/src/*.f90)

From my undestanding, the preprocessing of the fypp files are handling command like #:if but not the #ifdef.

jalvesz · 2024-04-05T12:14:24Z

I also pick stdlib from the fpm branch that I preprocess locally with fypp before hand, having only .f90s. Given that most compilers and build tools allow to add the appropiate compile flag to C-preprocess the common formats (*.f90;*.f), I do not see the need for .F90. I might be missing something though.

@cyrilgandon indeed fypp is there as a pre-pre-processor to handle the meta-programming... would be so good if one could debug directly the .fypp files...

gnikit · 2024-04-05T12:46:30Z

We should definitely be using capitalised extensions if files contain cpp preprocessor directives. This is a convention that all modern compiler vendors adhere to and so should the Standard Library.

perazz · 2024-04-05T12:58:33Z

Files are not capitalized in the stdlib-fpm branch because preprocessing is set in the fpm manifest:

stdlib/fpm.toml

Lines 12 to 13 in e958a29

    
           [preprocess] 
        
           [preprocess.cpp]

I hadn't considered that people may just grab the fpm branch because it's not got any fypp files. But, it makes sense to capitalize extensions here too.

zoziha · 2024-05-11T14:49:01Z

This PR does not address enabling OpenBLAS or other implementations in CMake yet. Current focus is to implement linear algebra subroutines, that will come next.

@perazz Excuse me, are there any immediate plans for CMake to call OpenBLAS and other BLAS backends? I am updating the Fortran-stdlib version for MSYS2, and it seems that I have prematurely linked it to OpenBLAS in MSYS2. (I'm about to revert to intrinsic Reference-BLAS of Fortran-stdlib)

msys2/MINGW-packages#20874

perazz · 2024-05-12T14:20:37Z

@zoziha if MSYS2 provides openblas, it makes absolute sense to link against it instead of using the internal implementation. It is probably faster

zoziha · 2024-05-12T14:29:49Z

@perazz, MSYS2 developers seem to suggest implementing CMake updates in the upstream Fortran-stdlib library, rather than adding a CMake patch to Fortran-stdlib in MSYS2. I am currently adding a CMake patch to MSYS2 that uses OpenBLAS, it seems too early to do so?

perazz · 2024-05-12T14:32:08Z

Correct: although the mechanism to link against external BLAS is already in place (via preprocessor macros), we need to find a way to enable it both from CMake and fpm builds. This is not done yet

jvdp1 · 2024-05-12T14:51:14Z

with CMake, we could adapt the following CMake script for BLAS/LAPACK libraries. I believe @awvwgk had a similar one in one of his libraries (I actually adapted this one from stdlib).

Then we should just add in the main CMake file (with an adaptation for the CPP macro) the following:

find_package("BLAS" REQUIRED)

I just found back the CMake script: FindCustomBlas.cmake. There is a similar one for LAPACK

zoziha · 2024-05-12T15:40:34Z

If we utilize the find_package(BLAS) and find_package(LAPACK) that come with CMake, the following diff patches are useful in MSYS2:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index b10e1f7..167efb5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -42,6 +42,26 @@ if(NOT DEFINED CMAKE_MAXIMUM_RANK)
   set(CMAKE_MAXIMUM_RANK 4 CACHE STRING "Maximum array rank for generated procedures")
 endif()

+find_package(BLAS)
+if(BLAS_FOUND)
+  add_compile_definitions(STDLIB_EXTERNAL_BLAS)
+  include_directories(${BLAS_INCLUDE_DIRS})
+  link_directories(${BLAS_LIBRARY_DIRS})
+  message(STATUS "Using external BLAS")
+else()
+  message(STATUS "Using intrinsic reference BLAS")
+endif()
+
+find_package(LAPACK)
+if(LAPACK_FOUND)
+  add_compile_definitions(STDLIB_EXTERNAL_LAPACK)
+  include_directories(${LAPACK_INCLUDE_DIRS})
+  link_directories(${LAPACK_LIBRARY_DIRS})
+  message(STATUS "Using external LAPACK")
+else()
+  message(STATUS "Using intrinsic reference LAPACK")
+endif()
+
 # --- find preprocessor
 find_program(FYPP fypp)
 if(NOT FYPP)
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index a031ab8..91f4020 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -115,6 +115,14 @@ set(SRC

 add_library(${PROJECT_NAME} ${SRC})

+if(BLAS_FOUND)
+  target_link_libraries(${PROJECT_NAME} ${BLAS_LIBRARIES})
+endif()
+
+if(LAPACK_FOUND)
+  target_link_libraries(${PROJECT_NAME} ${LAPACK_LIBRARIES})
+endif()
+
 set_target_properties(
   ${PROJECT_NAME}
   PROPERTIES

If we need to deal with more complex external BLAS, the links mentioned by @jvdp1 are useful.

perazz added 2 commits March 11, 2024 14:01

Upload BLAS/LAPACK, create preprocessor directives

f3ac970

enable fpm deployment

5882ad6

perazz marked this pull request as draft March 11, 2024 13:12

perazz added 8 commits March 11, 2024 14:16

update fpm version in CI

ac65112

add tests

c25e3ff

typo: getri

ce52faf

safeguards for qp procedure templates

3e8c5b6

public qp interfaces

29eeec2

more WITH_QP guards

8be6484

skip unsupported xdp precision

a937382

Delete .test_trapz.fypp.swp

a356f14

perazz marked this pull request as ready for review March 11, 2024 17:40

unify pre-processing of blas/lapack sources

2ae057f

jvdp1 reviewed Mar 12, 2024

View reviewed changes

perazz added 3 commits March 14, 2024 08:57

OpenMP: replace cpp macros with !$ conditional compilation

edfeb59

force example.dat back into repo

da5d90c

free-form !$omp continuation style

d379e9c

perazz mentioned this pull request Mar 16, 2024

Provide state/error handling for linear algebra #774

Merged

5 tasks

gnikit reviewed Mar 22, 2024

View reviewed changes

indent OpenMP sentinels

2be3340

Revert "indent OpenMP sentinels" This reverts commit 35b721b. indent OpenMP, do not strip EOL

perazz force-pushed the blas_lapack_backend branch from 35b721b to 2be3340 Compare March 23, 2024 14:43

jvdp1 reviewed Mar 24, 2024

View reviewed changes

ci/fpm-deployment.sh Outdated Show resolved Hide resolved

test/stats/array3.dat Outdated Show resolved Hide resolved

src/stdlib_linalg_constants.fypp Outdated Show resolved Hide resolved

remove FPM_DEPLOYMENT

592e085

perazz added 2 commits March 27, 2024 08:44

dnrm2/snrm2: fix missing interface type

e30b2ec

shorten doc comment

745bcf8

jalvesz approved these changes Mar 27, 2024

View reviewed changes

lapack interface: import procedure interfaces

c12b3c3

perazz merged commit e88daa3 into fortran-lang:master Apr 2, 2024
17 checks passed

ZedThree mentioned this pull request Apr 2, 2024

Massive slow down in docs generation #783

Open

perazz mentioned this pull request Apr 5, 2024

fpm branch: .f90 -> .F90 if preprocessed #787

Merged

wpbonelli mentioned this pull request Apr 12, 2024

LAPACK and BLAS Support fortran-lang/setup-fortran#55

Open

zoziha mentioned this pull request Jul 3, 2024

Add findBLAS support to CMakeLists.txt #844

Open

Provide BLAS, LAPACK backends and interfaces #772

Provide BLAS, LAPACK backends and interfaces #772

Conversation

perazz commented Mar 11, 2024 • edited Loading

gnikit commented Mar 11, 2024

perazz commented Mar 11, 2024 • edited Loading

jalvesz commented Mar 11, 2024 • edited Loading

perazz commented Mar 12, 2024

jalvesz commented Mar 12, 2024 • edited Loading

perazz commented Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

perazz Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvdp1 Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvdp1 commented Mar 12, 2024

gnikit left a comment

Choose a reason for hiding this comment

Comments

jvdp1 left a comment

Choose a reason for hiding this comment

gnikit commented Mar 26, 2024

jvdp1 commented Mar 27, 2024

perazz commented Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

perazz commented Mar 28, 2024

jvdp1 commented Apr 2, 2024

perazz commented Apr 2, 2024

jvdp1 commented Apr 2, 2024

perazz commented Apr 2, 2024

jvdp1 commented Apr 2, 2024 via email

cyrilgandon commented Apr 5, 2024

perazz commented Apr 5, 2024 • edited Loading

cyrilgandon commented Apr 5, 2024

jalvesz commented Apr 5, 2024

gnikit commented Apr 5, 2024

perazz commented Apr 5, 2024

zoziha commented May 11, 2024

perazz commented May 12, 2024

zoziha commented May 12, 2024

perazz commented May 12, 2024

jvdp1 commented May 12, 2024 • edited Loading

zoziha commented May 12, 2024 • edited Loading

perazz commented Mar 11, 2024 •

edited

Loading

perazz commented Mar 11, 2024 •

edited

Loading

jalvesz commented Mar 11, 2024 •

edited

Loading

jalvesz commented Mar 12, 2024 •

edited

Loading

perazz commented Mar 12, 2024 •

edited

Loading

perazz Mar 13, 2024 •

edited

Loading

jvdp1 Mar 13, 2024 •

edited

Loading

perazz commented Mar 27, 2024 •

edited

Loading

perazz commented Apr 5, 2024 •

edited

Loading

jvdp1 commented May 12, 2024 •

edited

Loading

zoziha commented May 12, 2024 •

edited

Loading