stdlib group with separate but related repositories?

[leonfoks]

There is quite a wide variety of proposals.

Does it make sense ahead of time to split the stdlib into defined parts like numpy and scipy have done?
I can see maths, sorting, random numbers, linalg etc. going into NumFortran?
KdTrees, interpolation, splines, etc. going into SciFortran? (Theres already a SciFortran but you get what I mean)
Meshing? Python has a package called Discretize, but something similar could be handled in Fortran.
They could all be under the umbrella of the stdlib.

If i'm missing the point of anything stdlib related, please let me know!

[scivision]

That structure does seem to work well for a lot of projects in other languages. It could help ensure each part is kept to high quality while keeping agility of development. It could also facilitate the continuum of such efforts as many projects will probably be affiliated and API compatible, like the Numpy stack, h5py, xarray, pandas, etc.

For example w.r.t. file I/O, some libraries will simply have an iso_c_binding interface, some will be shims to polymorphic Fortran interfaces, some will be pure Fortran modules, etc.
This Fortran stdlib doesn't map exactly to the C++ concept of stdlib or other languages. It considers the math/analysis nature of Fortran.

[certik]

So having a rich ecosystem of Fortran packages is something we need. But we need #44 to make that approach work well. Languages like Julia first started with richer standard library, but lately it seems their approach is to use separate packages for functionality and a smaller standard library. This part of a blog post argues this point also:

http://cliffle.com/blog/m4vga-in-rust/#rust-has-a-package-manager

But: even if we succeed and in few years we have a rich ecosystem of packages and a large community and easy to create your own package, I think there is large value in having stdlib with the scope as defined in the README (https://github.com/fortran-lang/stdlib#scope), and the reason is that it will make Fortran very much with batteries included just like Matlab or Python+SciPy. The reason NumPy is a separate package is that Python itself is not numerically oriented, unlike Fortran, which already has a large subset of NumPy already built-in. Then stdlib is roughly in the range of SciPy (plus utilities). SciPy could be split into 10 packages, but there is high value in having just one package, as the "base", that gets you started to prototype your code and has everything you need to do numerical computing. Then, as you need more specialized packages, you will install such packages using fpm (Fortran Package Manager), as discussed in #44.

As an example of this how it can work, see the issue #101. I think h5fortran should stay as a separate package. But we can think if it makes sense to provide some simple high level functionality in stdlib itself.

[marshallward]

I would personally prefer a leaner standard library focused on generic "computer science" operations such as sorting and I/O, and rely on custom libraries for numerical work, perhaps provided by a robust distribution (if not packaging) system. There's only a few variations in output when sorting a list (stable and unstable, for example) but many ways to do an interpolation.

[leonfoks]

Good points. I never really thought about the importance of the package manager and its influence on project development.

This probably goes without saying but I might suggest that repo organization follows an approach where we are splitting the code base into well defined sections such that. If in the future a package manager makes it feasible, the code base could be split at a later date if it makes sense.

[milancurcic]

I think the target audience is a factor. Python started out as a general-purpose language and its stdlib reflects that.

This exercise is interesting to do with any language that has a stdlib. Try Python, C, C++, Haskell, Go, Rust, Nim. The result: Every stdlib has different scope.

If we look at Fortran's target audience (read: who uses Fortran?), then the functionality covered by numpy and scipy is largely in the center, not on the outskirts. Based on this, I'd argue that numpy+scipy stuff is more in scope of Fortran's stdlib than collections and sorting (though I think they are very much in scope too).

[milancurcic]

Sorry, I didn't answer the question.

I think yes (for more specialized stuff), but only eventually down the road. Splitting by functionality would be counterproductive this early. We're still learning who uses Fortran, how they use it, and what do they want.

[certik]

To add to @milancurcic's answer --- everything in stdlib currently is in the "experimental" namespace. That means we are reserving the freedom for ourselves to change things, to move things around, or to (down the road) split them into separate packages. Logistically, because we are just starting, and our community is still small, let's collaborate around just one repository. As our community becomes big and stdlib becomes big and hopefully we have a working fpm at that point, let's revisit this discussion.

Update: to be even more specific, let's do steps 1. - 4. from https://github.com/fortran-lang/stdlib/blob/master/WORKFLOW.md in this stdlib repository. Once we get to the point that we are ready for step 5. for some features, then it's time to think hard if they should go into stdlib, or perhaps if it makes sense to create more libraries. Until then, let's work together to build a community, and to work on standardizing the API for the features that we all want, and put them into "experimental" into stdlib.

[certik]

Here is a nice blog post from somebody learning Rust:

https://hackernoon.com/programming-in-rust-the-good-the-bad-the-ugly-d06f8d8b7738

and he has an excellent point near the end:

While not as bad as the Node.js ecosystem, the list of dependencies for every library has become quite long. I wrote a small GitHub bot called LabHub, and I’m still surprised by how many dependencies get pulled in (181 crates, if you’re wondering). To me, this suggests fragmentation and duplication, which could perhaps be improved by slowly graduating some widely needed features into the standard library (something that C++ has done very slowly).

So I suggest we keep trying to agree on common functionality that we all want, and agree an an API and put into experimental. At the same time, let's work on fpm (#44) and try to create a healthy Fortran package ecosystem like in Rust. But even if we are successful, as Rust shows, there is still a need for a high quality stdlib. And for the scope of stdlib, we already have about 10 Fortran packages that people use, see the list at #1. So we are at the point when it's time to consolidate all these libraries into just one library and all of us to collaborate on that.

[leonfoks]

Cool yeah these are all great reasons and ideas to get moving!

[milancurcic]

Stdlib is currently a single repository with no plans to split into sub-packages. If a need comes up, let's revisit it.