[RFC] Shrink the standard library

[asterite]

This issues is very controversial, but it was never really discussed in the past.

I've been thinking more and more about moving out pieces of the standard library to shards. Examples of these are the YAML and XML modules, though we could include more modules (CSV, Zip, Big*, maybe even HTTP).

These shards will most probably live under the crystal-lang organization, so they will be kind of "blessed".

The pros of doing this are:

• These shards can now evolve independently from the compiler. If YAML would have been a shard, we could already have the new functionality without having to wait for a next compiler release.
• These shards can have more "core" collaborators than the crystal repository. Because they are less critical, we could accept more contributors.
• These shards don't need to freeze their API in a backwards-compatible way after 1.0, unlike the standard library
• It simplifies the maintenance work for the crystal core developers
• It simplifies the release process. For example we lost a lot of time in the past release (0.24.0) because the version of libxml2 that comes with Debian 7 doesn't work in Crystal. Specs weren't (and aren't) passing there. But nobody complains, so maybe nobody uses Debian 7 anymore, or maybe nobody uses Crystal+XML in Debian 7. If XML were a shard, we wouldn't have to worry about all of this (well, yes, but not in each release, we could solve this faster and better), and we would have 0.24.0 some weeks ago.
• These shards could provide more features than what they provide now. We tend to be very conservative with the standard library because every public method is something that needs to be maintained and kept for backwards compatibility. We will never add CSS3 style selectors because it's too much work and we wouldn't want to have to maintain that, and maybe it's not appropriate for the standard library. If XML is a shard then all of these reasons disappear, and it can evolve faster and better
• It's already known that some standard libraries didn't age well. Ruby had Tk in the standard library. Maybe it was popular before, but now it's not. Eventually they moved it out of the standard library. With Java, Python and C# the same happened, either because they became obsolete, or because an alternative, better solution could be developed, one that didn't have to live by the backwards-compatible rule.
• Should the Crystal distribution depend on libyaml and libxml2? Well, they come in the standard library, but they are only needed if you use them. But the compiler doesn't use them (this is why they were removed at one point from homebrew, then added back because they are needed just in case someone wants to use those modules). Well, if they are outside the standard library and the compiler distibution that problem goes away. You install the XML shard and there it says you need to also install libxml2, or maybe it comes with a postinstall that will install it for you.

Of course there are cons too. These are:

1. It becomes less easy to use one of these modules. Instead of just require "xml" we now have to write a shard.yml, add the xml github repo, run shards, etc.
2. It becomes a bit harder to find out these modules. There's no XML in the standard library? Where's a shard I can use?
3. Coupled with the above, since, say, XML is no longer in the standard library, chances of someone doing an alternative XML shard become more likely, so there's more segmentation and redundant work.
4. Because of 3, and I've seen it in Ruby and Elixir, one project that consists of multiple dependencies might end up with two or more libraries for basically doing the same thing (like and HTTP::Client or a JSON parsing library).

I have replies for these 3 points.

1. One of nice things about Crystal is that it is "batteries-included". The reasons for this, at least in the past (for example for Java and Python) is that internet wasn't as ubiquitous as now, so it was better to just download Java/Python and have most of what you needed right there. Another reason is lack of a good package manager, so installing new packages was a burden. But with Crystal, as with all new programming languages, this is not a problem anymore. You can read this Rust reddit thread and this Rust blog post to see these reasons applied in Rust because of Cargo (their dependency manager). So if these shards are in the crystal-lang organization, they are easy to find (if we put them somewhere in the docs saying these are "blessed"/recommended shards). The only inconvenient part is writing the dependencies in shards.yml, but I wouldn't use this as the only reason not to go with this.
2. If we document well that the crystal-lang organization hosts "blessed", well-maintained, up-to-date shards, then hopefully people will look there before someone else (stars in a GitHub repo might do the trick as well for internet searches).
3. This is actually happening now! Even though there's a good HTTP::Client in the standard library, people go and create others, like cossack and crest. Of course HTTP::Client is missing fundamental stuff like redirections but there's not enough man-power to tackle this. Moving it to an external shard would make collaboration much easier as more people could join, and it could evolve faster.
4. Because of 3 (it's already happening) I guess it's kind of inevitable, whether the module is in the standard library or in a shard.

Basically, I'd like to move almost everything that the compiler doesn't use to a shard.

If we agree on this, we could discuss doing so for individual modules.

I believe Crystal should come with a standard library that mostly doesn't depend on external C libraries (it's a bit impossible, but the less, the better), and one that provides a foundation for other libraries to build on top of it. Examples of that are IO, which is evented by default; the runtime, with its concurrency model, spawn, Channel, etc.; Array, Hash and Regex, which are a core part of the language, mainly because they are tied to literals; and probably other types for similar reasons.

[matiasgarciaisaia]

I still don't have a strong opinion about this issue. Just wanted to add something we (Ary & me) have discussed last week - we could help the adoption of this blessed shards by making crystal init generate a shards.yml that already includes those dependencies listed.

So they are almost part of the stdlib (ie, most projects will probably "just include them") but they aren't coupled to the compiler's release process.

[asterite]

Oh, yes, that's a great idea. In fact it could ask you what you want/need:

- Do you need YAML? (yes/no)
- Do you need XML? (yes/no)

[RX14]

I think the right boundary between usefulness is somewhere between JSON/HTTP (which should be in the stdlib) and YAML/XML (which should not). JSON is simple, and it's fairly common. HTTP (at least version 1) is simple, and very common. YAML is a lot more complex of a standard, and is much less common than JSON. XML is very common (although not as much as json) yet is extremely complex to the point that we really need a prebuilt library to parse it.

Here's a list of top-level modules I (personally) don't think should be in the stdlib:

• Adler32 - never heard of it, seems to be there to satisfy the compiler only
• Big* - not so sure on this one but they're really quite slow and not really high quality and not well used
• BitArray - i'm 50/50 since it's so simple (but not well used). This is a very borderline case to me
• Complex - should we include matrices in the stdlib? is complex truly special among mathematical constructs to want to be in the stdlib?
• CRC32 - see Adler32. Perhaps this one is common enough to live in the stdlib under Digest
• Debug - do we want a fully featured dwarf parser in the stdlib or just enough to get us debug info on stacktraces? I'd say this should really be private.
• DL - well this one should just be removed because it's fairly useless (what's it's usecase?)
• INI - seems much less commonly used
• Levenshtein - only here for the compiler (should be a shard which the compiler depends on)
• LLVM - only here for the compiler (should be a shard which the compiler depends on)
• Markdown - ditto
• OAuth - complex and specifc, not sure why it's here
• OpenSSL - the only crypto primitives we should provide imo is a few Digests, HMAC and TLS.
• OptionParser - somewhat opinionated and only here because of the compiler
• Readline - named after a library, instead of a concept (like OpenSSL but even less popular/useful)
• Termios - no docs, idek what it does lol
• XML
• YAML

Perhaps the compression stuff (Zip, etc.) should be rethought a bit too. I think we should provide a few decomrpessors, but move them into their own module, and perhaps think about removing the file formats, and only keeping the raw compression formats.

[ghost]

I am not a crystal user, only a ruby user, so please feel free to ignore me completely. I am sure that whatever way you pick will be the one that will be right.

I am a bit confused as to why json is considered superior to yaml but since it also does not affect me,
I'll not comment on this further.

But one thing about distribution packaging, e. g. debian:

For example we lost a lot of time in the past release (0.24.0) because the version of libxml2
that comes with Debian 7 doesn't work in Crystal. Specs weren't (and aren't) passing there.
But nobody complains, so maybe nobody uses Debian 7 anymore, or maybe nobody
uses Crystal+XML in Debian 7.

I would not worry about this. Distributions such as debian will already package things up
anyway, and it should be THEIR responsibility to do so. If you depend on them and then
wait on this or that distribution, then in my opinion this leads to frustration only.

Ruby has the xmas release and it does not matter what any distributions do in regards to
xmas - xmas gifts are coming that day.

If you were to depend on a distribution, then that distribution has a crippling effect on
upstream (in this case, you, as programming language developers), and I feel that
this is a very ... awkward constraint to be had.

[drhuffman12]

I would love to see a stdlib shard, which would in turn depends on shards of these 'blessed' "pieces of the standard library", and likewise for other high-level groups of libraries (ui?, web?, gl?, etc). Smaller chunks [shards]; easier for community involvement and more optimizable code. [Kinda makes me think of MRuby's granularly configurable dependencies.]

[calebuharrison]

@drhuffman12 this just feels right to me. Having a shard (or group of shards) like this allows for the standard library to be even more conservative about what is considered a de-facto component of the language. If this would (as @asterite suggests) make version updates more nimble, then I'm all for it!

[unreadable]

I really do agree for shrinking the stdlib. Keeping the core smaller will lead to a better maintenance. Rust has a small stdlib and noone seems to complain, so I don't think it'll be a problem for Crystal too..would be nice though to keep at least HTTP/1.0 as part of the core lib.

[lbguilherme]

Basically, I'd like to move almost everything that the compiler doesn't use to a shard.

Is it a problem if the compiler itself ends up depending on (blessed) shards?

[RX14]

I disagree on the principle of moving everything the compiler doesn't use to a shard. I believe there are many things that the compiler does depend on that should be in a shard, and many things the compiler does not depend on which should be in core.

For example I think that a common batteries-included HTTP abstraction is essential for HTTP libraries and frameworks to play together. And the correct way to ensure this is to put HTTP in the stdlib. The compiler doesn't (apart from play) use this. But I still think it should be in the stdlib.

[asterite]

@RX14 I agree. I think LLVM could perfectly be a shard on which the compiler depends. It makes very little sense to have it in the standard library unless you are going to develop a compiler or interpreter, or contribute with the compiler itself.

I think in the Markdown discussion we concluded that it would make distributing Crystal a bit harder, but I'm not so sure about that.

I also kind of agree regarding a common HTTP server base. I don't know about HTTP::Client, maybe that applies too because OAuth can easily integrate with it thanks to before_request. I guess eventually HTTP::Client will have all of the current missing features.

For now, we can discuss the YAML and XML modules.

Regarding XML, I think it's always required by spec to provide the JUnit formatter output, which is based on XML. That also is a bit strange: if you don't have libxml2 in your system, you can't run specs, even if you don't want to use XML. That formatter should be a shard which enhances the spec library. Then we could extract XML to a shard.

Regarding YAML, it seems the specs for crystal tool init use it. But this could easily be tested in a different way without requiring YAML.

We could then open separate issues to discuss extracting other stuff, like Big*, to separate shards. Maybe if this idea is liked I could close this issue and open two separate issues, one for YAML and other for XML so we can weights the cons and pros of each.

[faultyserver]

I think that if common libraries like YAML are going to get moved out of the standard library, there should be some first-party way of maintaining local copies of shards, much like how Bundler installs gems locally.

A common use case for me is working on a plane or a train, where I do not have access to the internet, or it is spotty/slow at best. If I wanted to start a project while in transit, I'd be sad to find out that a common library I want/need to use isn't available because I can't download it in the moment, and won't be able to for another 5-12 hours.

I can't say I'm necessarily a big fan of Bundler because it ends up keeping various versions of everything around and you end up being forced to bundle exec everything, but being able to at least "cache" a more fleshed out (potentially customized?) "stdlib" would be really nice.

[faultyserver]

As a bit of a side note: seeing how shards.yml is (obviously) a YAML file, doesn't that mean that it needs YAML to run? I'm not entirely sure of how it all works, but it seems a little odd that the system for installing dependencies would need to install dependencies to run. But I guess that's essentially what bootstrapping is....

[RX14]

@faultyserver shards is compiled, it doesn't need the yaml shard, or any sourcecode to run.

It does depend on libyaml though.

[RX14]

I also think that if we include a http server we should have a http client. I don't think the current HTTP module is well designed at all though. HTTP::Server is usable but HTTP::Client is a mess and I don't like how client and server share Request but not Response classes. HTTP::Client needs middleware to allow shards to extend it or it'll replaced by something which actually does what people want (redirects, proxy, etc.)

[akzhan]

I just think that shardening shall be later. Please don't touch it, it's simple before 1.0.

Multiprocessing is feature I want :)

[drhuffman12]

Can the parser/compiler be used to generate a dependency tree? If so, maybe such a tree could be used to help note the 'from', to help clarify what might be easiest/harder to move to shards, and to help clarify what might need to be adjusted/de-coupled/etc in order to be moved to a shard?

[drhuffman12]

@akzhan , sounds good to me! Saving the sharding for 2.0 [or 1.x] would be fine with me. [Maybe sharding would help Crystal get to 1.0 sooner and I would love the pro's; but, I'd rather err on the side of less scope creap.]

[asterite]

@akzhan It's actually simpler if we move stuff out of the standard library to shards. Right now we have to tackle YAML, XML, etc., plus parallelism. If YAML, XML, etc., were shards, maintained by others (with the core team keeping an eye on them, but not necessarily having to worry too much because it's not in the standard library) it will give the core team more time to fiddle with parallelism and other more important stuff.

I actually started implementing tar some time ago, but never pushed it because it wasn't feature-complete. Now I'm thinking that I could have created a shard and others could have probably helped me fill the gaps. It doesn't matter if it's not feature-complete or if it's buggy because it's not in the standard library, and can evolve easier and separate from the language/compiler.

Another way to say it: a module that's in the standard library has to be well thought, documented and complete, and will be frozen once we reach 1.0. That means the core team has to spend more time on this.

Just look at how fast Elixir evolved. Their standard library is pretty small, but it has the core stuff in there. In fact, the tool for generating docs is not even included in Elixir, it's a separate tool. That lets the core developers focus on the language and a small core subset. Same goes with Rust, which already reached 1.0. Go is a different story because they had a lot more time and resources than those projects.

[faustinoaq]

I agree, an smaller stdlib will be easier to maintain and focus on parallelism or even windows support.

I also think documentation generator should be moved to a separated shard.

Maybe we can move some crystal tool too, like formatter, implementation or expand benefiting the creation of tools like Scry and facilitating crafting new tools like rename or refactoring, without waiting for a new version of crystal.

[konovod]

I agree with @faultyserver, impossibility to create anything useful without internet access seems sad. So it would be great to have offline way to install shards, so e.g. all crystal-lang shards will be installed by default when installing crystal itself. On the other hand, right now it is possible to just create app once with all needed shards and then copy them from lib folder, so it is not critical inconvenience, just minor issue.

[kostya]

remove many libs from stdlib decrease possibility to use crystal for scripting. when you need to write 10 lines script, and you don't want to create project and install shards.

[lbguilherme]

impossibility to create anything useful without internet access seems sad.

yarn for node (and I think npm does the same as well) creates a cache folder where it downloads every installed package there. When add a package it first checks the cache instead of downloading directly. This way you can install both faster and without Internet.

The omnibus release could prefill this "cache" with some of the blessed shards so that no internet is needed after crystal itself is installed.

remove many libs from stdlib decrease possibility to use crystal for scripting

We could have a shard macro to automatically install the shard into some standard location and include that location into the import path. Something like this:

shard "github/mamantoha/crest", ">0.9" # optional version restriction
require "crest"

pp Crest.get("http://example.com/resource")

This can even be used for bigger project (actually, I think this is better than having a yml file, but that's another story).

Maybe we can move some crystal tool too

The compiler can import some tool shards and expose then on the command line. This way they are both included in the compiler (transparent for the user), and with easier maintenance for the project.

[faustinoaq]

@lbguilherme Yeah, caching shards in some common place like node would be very useful, see crystal-lang/shards#180

[faustinoaq]

remove many libs from stdlib decrease possibility to use crystal for scripting

@kostya Maybe we should implement shards install foo allowing us to require common installed shards without using a macro see crystal-lang/shards#144

[RX14]

I'm not sure I like the idea of removing doc and the formatter from the compiler. At least these tools should always be installed. They're also tied into the compiler internals, which means that refractors to the compiler will affect these tools. Which makes putting them in a seperate repo rather hard with a changing master. Also more tools means more binaries we have to distribute.

[asterite]

Regarding offline usage of Crystal, I believe the problem will always be there. For example if I want to use MessagePack, well, it's not in the standard library, and I won't get it on a plane.

One can always download the shards in a local directory and then reference them with path.

I do agree about Crystal becoming less suitable for scripts. But I think scripts mostly use Array, Hash, File, IO, etc. Well, maybe JSON and CSV. We could keep those in the standard library because they have no dependencies, plus they are relatively simple.

But if scripts become a bit harder to write, I wouldn't mind. In any case I don't see the point in writing a small file, saving it in /usr/bin and compiling and executing it every time you want to run it. It's probably better to create a project, compile it with --release and put it in /usr/bin. I hope Crystal is more suitable for medium/large projects than for small scripts :-)

(we never said Crystal is a scripting language)

@faustinoaq Small side note, but doing a refactoring/renaming tool in Crystal is plain impossible, so I wouldn't spend time on that.

[faustinoaq]

I hope Crystal is more suitable for medium/large projects than for small scripts :-)

Well said! 👍 :)

doing a refactoring/renaming tool in Crystal is plain impossible

@asterite Maybe because crystal has the same issue that C++ about refactoring tools ?

[RX14]

Should probably remove ARGF too. Does anyone here use it?

[j8r]

I think JSON::Serializable and YAML::Serializable would better leave in an organization/project outside the stdlib (like serde). Rust and Go have only basic encoder/decoders too, this complexity can be delegated to external shards.

• this will avoid having various Serializable implementations for formats not implemented in the stdlib like TOML, XML, CON.
• they could be grouped in an organization that can ensure a consistent API for all supported data formats, group the docs and maintain them
• projects like @Blacksmoke16 's CrSerializer, that extends Serializable features, may even be part of this organization
• this free time for Crystal's maintainers - the concept is powerful and flexible, but no so simple underground (there are numbers of issues/PR)
• the stdlib will become more agnostic - people are free to chose whatever PullParser frontends they want. Serializable moved out, there will be still one way to serialize with the stdlib: JSON/YAML.mapping

[chocolateboy]

Should probably remove ARGF too. Does anyone here use it?

Yes, pretty much always.

[didactic-drunk]

Proposal to decouple stdlib with batteries included.

Requirements:

• Common libraries are available without extra steps (batteries included)
• Official documentation includes stdlib (batteries included)
• crystal eval works without requiring explicit shards (batteries included).
• short scripts can still use stdlib (batteries included)
• stdlib is split in to a different repo possibly maintained by a wider group of people addressing @asterite's concerns
• Each stdlib library goes in it's own repo.
• Can upgrade the compiler (maybe for better performance) without changing stdlib.
• Can upgrade stdlib (maybe to get newer features) without changing the compiler.
• Can upgrade individual stdlib libs keeping stdlib semi locked to a specific version.
• Can require individual stdlib libs and forgo stdlib.
• Airplane/disconnected mode still works @asterite.
• No new configuration necessary (older apps will need to add 1 shard dependency. shards command could do this automatically or implicitly)
• No new commands to run.

This proposal requires the following minor changes:

1. The crystal compiler includes a default shard file that contains 1 dependency for "stdlib".
2. Repos are created for each lib. You could start by moving 1 lesser used library for testing (Termios?).
3. crystal-community/stdlib is created that only has dependencies on other shards.
4. crystal init app adds a dependency for crystal-community/stdlib with an advisory version of same version of stdlib used with the compiler.
5. crystal eval and other crystal commands uses the compiler default stdlib if it can't find a project shard file.

That's the basic idea. To most developers all the commands stay the same and 2 lines are added to new shard.yaml files by default. Documentation is identical. Library availability online or off is identical. crystal and shard commands are identical. Project management is improved. Upgradability is improved. Long term support is improved. Documentation generation is improved (see below).

The remainder of changes are for the shards command, documentation generator, or other plumbing that doesn't concern the average user or usability of crystal.

Addressing documentation and lack of batteries.

Minor changes are made to the documentation generator to include stdlib dependencies in the official documentation and mark them as such.

Changes to the documentation generator to include shards and annotate which shard the API is from (if it doesn't already) would have other benefits. Now documentation of each project can be generated with all available API's for the specific versions used in your projects. Going to work on a ~4yr old project would be much better with it's own documentation rather than relying on google. If you tried to search for crystal 0.x.y, it wouldn't give you documentation for libfoo 0.y.z all in one place.

Addressing monolithic stdlib

Within this proposal no project needs to require stdlib. It can be removed by deleting the dependency in shard.yml and adding dependencies to the individual libraries used.

In addition more focused stdlibs could be published by anyone such as stdlib-http which includes dependencies for at least [http-client, openssl, json] that are validated and tested together against various compiler versions.

Addressing what to put in stdlib

It almost doesn't matter. Everything not required by the compiler (@asterite's preference) can be put in it's own library and still be available for use and documented as it is right now.

Addressing compiler/stdlib incompatibilities over time.

Automated testing:

• Each new compiler release goes in a pipeline to test each minor stdlib shard version against the new compiler release and publishes it's results.
• Each new stdlib release gets tested against every minor compiler release and publishes it's results.
• The results of both are aggregated in a format parsable by the shards command whatever the format or method ends up being.

Addressing individuals concerns

• @asterite (1-2 in first post) are completely addressed. 3-4 about competing libraries is already he already addressed himself. Turning it over to faster community development as in this proposal will probably reduce the need for extra libraries.
• @Wulfklaue maintenance and LTS is easier under this proposal as stdlib is still tested against all compiler releases. If anything support is improved with the ability to use mismatched stdlib/compiler versions that are tested or let the shard command tell you that it's nonfunctional. Almost all of your concerns are addressed.
• @kostya, @straight-shoota all stdlib shards would be available for use within this proposal. Scripting wouldn't be harmed.
• @MrSorcus stdlib isn't required with this proposal. Use individual libraries or none. Pick the versions or let them auto resolve.
• @Svenskunganka stdlib would still be maintained and official, possible more maintained than it is now within this proposal.

Why advisory versions?

App "foo" is using stdlib v1 but wants to use xml v2. If stdlib is locked to v1 then it's not possible to upgrade shards a piece at a time which is often necessary for old applications.

Perhaps a better name would be "default locked version". Locked to a specific version unless specified elsewhere.

Shards specific (behind the scenes) changes

• Needs to track shard "foo" tested "ok" or "fail" against various compilers. This seems like a general purpose change not limited to stdlib.

• • Wouldn't it be nice to have a standard docker build that publishes the results of all published crystal versions tested against your own libraries?
• • Wouldn't it nice if you put "kemel" v99 in your shard file and it told you that v99 only works on crystal v4.4.4-8.8.8 and you need to upgrade crystal or only upgrade to "kemal" v69 for your current crystal v2.2.2?
    -- How the information is tracked and stored is a minor implementation detail mostly concerning @ysbaddaden.

• Default shard dependency.

• • Possibly have a special case for the shards dependency where if a version isn't specified it defaults to the current compiler defaults. This is to generate the lockfile only. After it's locked the versions are set.
• • shard upgrade on a new compiler uses the new compiler defaults.
• • shard upgrade with a version specified uses the version string as normal.
• • Or maybe it's the missing github/url, or extra flag in shard.yml to use the compiler default. Minor detail.

Best of all

The code required to implement (most) of this proposal is minimal. A handful of defaults and additional search paths. Additional glue is necessary in shards to check compiler versions but that's probably necessary over the long term anyway for all libraries not just stdlib.

Should I have put this here or in a new PR?

[didactic-drunk]

@RX14 Was there something I didn't address? Or some part that can be improved?

[bcardiff]

Something that would be actionable in an iterative way and beneficial whether the std-lib is split or not is to separate the prelude from core.

Letting core be a minimalistic prelude alternative will:

• improve build time in the compiler specs that currently require the whole prelude.
• allow porting the std-lib and runtime to new targets easily

Currently, low-level stuff like Pointer ends up using use modules, macros, and classes that are convenient for a full batteries experience but are not required in a minimalistic environment. Nice to_s, conversions to other types, non-essential operations, etc.

Another example is Exception, to extract the backtrace depends on lots of things. For minimal behaviour, we could have no backtraces, but still, have the begin/rescue semantic.

Due to refactors is easy to end up with circular dependencies across types in the prelude. Having a core will help to put some strict boundaries there. Right now the prelude has some first modules included in a specific order, but leaving only those does not compile.

Core should have the minimal runtime, most primitives (but not necessarily all), and some classes/operations but not things like map/filter/etc.

I'm not sure the std should/can be splitted in the near future, but even so, I think that what I am proposing is still a step needed.

[girng]

I think this would introduce massive breaking-changes for current Crystal projects. Which will cause Crystal devs a lot of pain that is not really necessary. Especially considering the state Crystal is in now.

Shards exist for this very reason. Instead of all the shards being added to the core, they are.. shards. This is already sufficient. Crystal already has great modularity. and the current codebase is in no way bloated. I'm sorry, I just don't see any justification in doing this. In fact, I beg asterite, bcarddiff, and the other lead devs to never even consider doing something like this.

The lead devs decide on what is added to the core language. They've already merged what we have, as they thought it would be good for the core at the time. There is no reason to backtrack.

Also, if someone thinks just because they personally don't find something useful, you can't use that as an argument to get it removed; because you don't know if another Crystal dev is already utilizing it.

[ysbaddaden]

@didactic-drunk an actionable plan is the easy part: create repositories, move files there, use submodules or a shard.yml with loose patch level constraints (~> x.y.z) to make it easy to install, ..

There is more to the problem: confidence loss in merging changes and/or a complexified crystal test suite, interdependences, ...

It would be great to see some libs like LLVM or OpenSSL extracted and each tested against different versions, to be able to install patches quicker for them, but it's not that easy to put up.

[didactic-drunk]

I think this would introduce massive breaking-changes for current Crystal projects.

None. No breaking changes. That's part of "Requirements 1." implemented as "minor changes 1.".

[didactic-drunk]

@bcardiff What you propose sounds entirely different from splitting stdlib and core. I think you're talking about minimizing dependencies within core. I'm not up on compiler internals but stdlib is right up my alley.

[didactic-drunk]

@girng Rust does something similar. The compiler and stdlib can run at different versions. The proposal I listed allows the same mixing and matching with different version for long term support in a way that's seamless. You won't know about it unless you try to change the stdlib version manually.

Here's an example of how this benefits long term support:

• Develop app on crystal v1.
• 2 years go by and crystal v3 is released.
• You want to upgrade the http module ONLY which is currently part of core.

With core + stdlib as a single package you can:

• Upgrade crystal.
  • Your entire application potentially breaks from all the failing dependencies of what you specifically didn't want to upgrade.
  • Multiple people above said they didn't want to do this.
• Try backporting http and it's dependencies yourself.
  • Which files are required by http? url/* probably. OpenSSL? depends. Anything else? What do those depend on? How many layers does this go?
  • Testing with those specific versions? Good luck. You're completely on your own.
• Wait for someone else to maybe do it.
  • This always works out well.

Or with stdlib split from core:

• Specify stdlib-http ~>= 3.0
  • Let shards work out the dependencies for you.
    • With cross dependency testing it will figure out what works and what doesn't for you.
  • Only upgrade the parts of the application related to http and it's dependencies instead of everything at the same time.

It's way less work and way less breakage.

[didactic-drunk]

@ysbaddaden If crystal provided nightly ci builds to test against would this address the confidence issue? Could tests for stdlib-a run both with and without the current stdlib to check for integration issues? Without stdlib would only add it's listed dependencies to the require path. With stdlib (the default) all stdlib is available in the require path.

Would a travis image with all supported crystal versions preinstalled help? Tests could run against all available or a listed set.

Should there be tooling to aggregate the test results for all packages in a single place or will the existing ci infrastructure work?

[oprypin]

If YAML would have been a shard, we could already have the new functionality without having to wait for a next compiler release.

True.

Because they are less critical, we could accept more contributors.

This can be done without splitting the repo and the corresponding downsides. Can also be done more granularly.

These shards don't need to freeze their API in a backwards-compatible way after 1.0, unlike the standard library

And is that a good thing? If there is no HTTP client that maintains its API, what are people supposed to use?

It simplifies the maintenance work for the crystal core developers

If you mean that ignoring problems is a simplification, sure. In fact, with how PRs don't get merged, it's already pretty simple :)

It simplifies the release process.

Maybe the particular reason you provided is true, but I'm pretty sure that overall it will complicate things. Even starting with things like backwards-incompatible changes, though thankfully those are handled decently now.
But surely 1 repo is easier than N repos?

These shards could provide more features than what they provide now. We tend to be very conservative with the standard library because every public method is something that needs to be maintained and kept for backwards compatibility.

True, but moving code to a different git repository shouldn't completely correlate with dropping all conservatism and is also not necessary for it.

[didactic-drunk]

Do you have a complete solution for long term maintenance and incremental upgrades? How do you upgrade only YAML in a 1 year old application? Or upgrade YAML as far as it will go including it's dependencies? If stdlib is part of crystal you can't except by copying files yourself and tracking dependencies manually. If it's split cross version testing will tell shards what works and what doesn't saving everyone that has to upgrade an application lots of time.

There's also the question of whether Manas will give more people write access to crystal-lang/crystal. I suspect they don't want to which may require a split to get more contributors.

[didactic-drunk]

These shards don't need to freeze their API in a backwards-compatible way after 1.0, unlike the standard library

And is that a good thing? If there is no HTTP client that maintains its API, what are people supposed to use?

With split stdlib if you really need old HTTP functionality you can have it by setting the version. There's additional flexibility for forward of backward versions. With that flexibility API's can evolve without pissing off as many people.

Most of the complexity you worry about can be automated for crystal core developers. It doesn't concern end user developers. (Is that a term?) They still install crystal the same way they did before mostly via package managers.

[didactic-drunk]

It simplifies the release process.

Maybe the particular reason you provided is true, but I'm pretty sure that overall it will complicate things. Even starting with things like backwards-incompatible changes, though thankfully those are handled decently now.
But surely 1 repo is easier than N repos?

I assume the current release process is:

• Choose a version number
• Run a shell script.

The shell script will get more complex to handle the split but the release process will probably stay the same.

[RX14]

This is an issue about removing modules from the standard library, not splitting it up into multiple git repositories.

Please create another issue about that as they are different problems.

But please be aware that it'll never happen, as it creates a bunch of migratory work for the core team for almost no benefit over the code maintainers model oprypin mentioned. I don't even think crystal is large enough for that model yet.

The problem is a social organisational problem, with crystal, it's community, and mostly the core team. Treating it as a technical problem won't give the correct solution.

[straight-shoota]

I think this issue can be closed. The stdlib has been consolidated since this started, and we have reached a good compromise between batteries-included and unnecessary baggage.
Some of the features discussed here have been removed. Others are still around, and it seems to stay that way. With the recent 1.0 release we're committed to API stability, so there's not going to be a major cleanup anytime soon.
That doesn't mean that stdlib features can't be removed, but it's less likely and requires more detailed consideration about the specific case.