explore adding support / design implications of "totality at all costs" api

[cartazio]

its pretty tricky to actually have a humane api for "total array programming" for dynamically sized arrays (even in something like ATS/Idris/Agda, with static sized arrays, having a good UX for this can be tricky)

I think it turns into a related subproblem: what are total combinators that have nice definitions in terms of the Stream and Bundle datatypes used within the vector fusion framework

[cartazio]

feature request from @kindaro

[cartazio]

example being: "safeHead" perhaps (though i think there maybe more compelling examples .. head isn't really something you should do for nontrivial vector codes i think)

[Shimuuar]

For safeHead we have !? 0 and rest is awfully vague. Closing

[kindaro]

First you @cartazio delete my issue and write something else instead (only partially transferring what I had to say), then you @Shimuuar close it saying «awfully vague». It is hard for me to say something constructive in this situation.

I have withdrawn from the open source community because of situations like this.

I think everyone will benefit if we all try to be a little bit more constructive and work together. I understand that maintenance of a package is work that is not always rewarding. But what do you gain by dismissing your users? You only transfer your frustration to others.

There were certain issues I experienced working with this library. Likely other people also experience the same issues — not everyone will take time to open an issue. If you try to pretend these issues are not here, you will have less happy users and even fewer open issues. Is this what you want to achieve?

[kindaro]

So far as I can remember, my offer was in the essence the following:

1. «Safe» as a quality of a function means for the Haskell community overall that calling said function with defined arguments will evaluate to a defined value. So, it is the same as «total» — whether this is good or bad, this is how the words «safe» and «unsafe» are used.  For the maintainers of vector however «safe» means «the underlying code is memory safe». This was causing confusion in some conversations I was seeing back then in 2019, where someone would say «the API of vector is unsafe» and a maintainer would answer «no, it is perfectly safe».
2. I offered that the maintainers of vector accept complaints to functions being unsafe in the sense of sometimes taking defined arguments to undefined values, and offer an API in which all functions are safe in this sense. For example, the function drop is safe, and the function unsafeDrop is not safe, but also the function head is not safe. The principle «all pure functions are safe» is important for many people writing production code, to the extent that whole new preludes are published that promise to export unsafe functions only from modules called Something.Unsafe. For example, instead of head they would export vectorToMaybe ∷ ∀ α. Vector α → Maybe α.

This is a well justified and reasonable offer. It was 4 years ago and it is still. I do interpret your brushing it off like you did as disrespect. But I am going to make an effort to calm down right now and ask you again very politely: please entertain the thought of meeting this offer for a moment and let me know what you think.

Who am I to ask this?

• I love Haskell.
• I use Haskell at work and as a hobby.
• I have contributed a number of patches to various libraries, including core libraries, in the past.

If you do not value me as a representative of your user base, then who is your user base?

[Shimuuar]

Sorry, but after single round of broken telephone issue did become quite vague.

So problem is two possible definition of unsafe: one is function may throw exception and another function may violate some important runtime invariant: memory safety, alias buffer unexpectedly, etc. Nasal demons are OK. It's important to distinguish two. unsafe usually means latter: unsafePerformIO, unsafeCoerce and abomination like accursedUnutterablePerformIO. Former is usually discussed in terms total/partial.

Request is to make sure that all functions are total. That simply won't happen. When it comes to working with indexes directly invariants become too complicated to be expressed in haskell's type system. Returning Maybe everywhere is a non-started too. This simply forces user to unwrap maybes and that gets old very quick.

Better documentation on which functions may throw exception and when will be quite useful

[kindaro]

Yes, some parametrically polymorphous partial functions cannot be extended to total functions, head is an example. And proving correctness of working with indices directly would be out of scope for vector, whether feasible or not. I am also fine with calling partial functions «partial» and functions that never corrupt memory on some inputs «safe» — the only problem is that this is not how «safe» was used in actual discourse, last time I checked.

So, partial functions can be made total by completing the target type by wrapping it in Maybe. Providing to every partial function f: a ⇸ b a total counterpart f': a → Maybe b seems to always be routinely possible. Are there any partial functions in vector for which it is hard to say on which inputs they will be undefined? Surely not head. Indeed vector already provides (!?): Vector α → Int → Μaybe α as a total counterpart to (!): Vector α → Int → α. Same can be done for other partial functions.

Whether to use the total interface or the partial interface is a non-trivial choice specific to a particular use case. Sometimes it is clear enough that a partial function will never receive an input for which it is undefined. Other times I want this function to return Maybe so that I can pattern match on it and write an error in the Nothing case with a message explaining why this code path cannot ever be reached. In many cases I actually know that a partial function like head could be called with arguments for which it is undefined, and I want to handle this case — this is when I need a total counterpart like listToMaybe.

So, what are possible ways forward?

• A radical way forward is to do like rio — put partial functions into a module of their own called Partial and deprecate and eventually remove their exports from other modules.
• A moderate way would be to instead offer a module called Total that exports only safe and total functions. Who would object to that?

Something that can be done in either case is to make sure all partial functions have a total counterpart and their documentation refers to it. If I remember PVP right, this can be done with merely a minor version bump.

Let me explain the motivation. What matters here is not whether some specific case can be made to work or not. As you say,(!? 0) can be used instead of head, and so on. What matters, however, is confidence. A working software engineer is often a tired software engineer, and when a partial function like head is always at hand it will one day get used, and used wrongly. What I want is to import a module with confidence that it will not let me break my program. I want a module that says: «all functions exported here are total». If I then want to get clever and juggle some indices, an import statement like import Data.Vector.Partial will clearly tell the future programmer that they should be attentive to index arithmetic when modifying the code.

[Bodigrim]

@kindaro have you tried vector-sized? It offers total head and index.

A moderate way would be to instead offer a module called Total that exports only safe and total functions. Who would object to that?

Are there downsides to fleshing out such module in a separate package? This would provide even better UX for a "tired software engineer" - if you depend on, say, vector-total instead of vector, you do not need to read any code to have confidence that nothing is partial.

vector is already quite a nightmare to maintain, document and keep consistent between different flavours. I'm not sure that extending its API even further is a feasible load for current maintainers.

[Shimuuar]

I think that adding variants of functions that report error as Maybe instead of throwing exception is quite reasonable. Which ones do we miss by the way?

But adding several module of reexports puts too much burden on maintainers with marginal benefits

[kindaro]

• @kindaro have you tried vector-sized? It offers total head and index.

  Length indexed arrays are a good thing. But use cases for length indexed and dynamically sized arrays are different. I use that which is appropriate for the task — sometimes both in the same function definition.

• Are there downsides to fleshing out such module in a separate package? This would provide even better UX for a "tired software engineer" - if you depend on, say, vector-total instead of vector, you do not need to read any code to have confidence that nothing is partial.

  Either I have to look at import statements in Haskell or build-depends sections in Cabal language. Since Haskell code can be split into modules but Cabal language code cannot, for a sufficiently big program it will be more tedious to read the latter. For example, I have at hand a package which description is 510 lines long,

  At the same time, adding or removing dependencies is harder than adding or removing imports, because constraint solving will be affected, version conflicts can surface all of a sudden and Cabal freeze file may need to be regenerated.

---

Open source projects do not have official management, but this does not mean that all issues can be solved technically. This issue is not about a single case of a function with a certain type signature and behaviour (vectorToMaybe) being missing. It is not even about a whole class of functions (total completions of partial functions) being missing or incomplete. This issue is first about management and only then about engineering. There are three aspects of this issue that are managerial.

1. maintainer no listen

   This issue is evidence to the case that the maintainers of core libraries do not always listen to their customer — the Haskell user base. The maintainers removed my issue the same day I opened it, and instead made their own — this here issue. Why? Only to silence my voice. I see no other reason. If my issue was not clear, not respectfully written, had other flaws — they could have worked with me to improve it. I always come back and edit my messages when I see that they are not good enough. But no, they simply removed whatever I wrote. No warning, I just woke up to it being gone.

2. strong contributor be going elsewhere

   There is a whole industry of maintaining packages which purpose is separating total functions from partial. Some of our best people spent countless hours on this: Michael Snoyman and allies with rio, Dmitrii Kovanikov and allies with relude. They could have worked on core libraries instead, if the maintainers of core libraries were open to the idea of separating total functions from partial (and to other improvements that were being suggested over time).

3. new contributor be no coming

   The message to prospective contributors is incongruent — on the one hand we like to think that anyone is welcome to contribute, but on the other hand we see that a person needs to understand all the unspoken assumptions and expectations that the small group of maintainers impose and enforce. Why was my issue removed? Must have been some expectation I did not anticipate. What expectation? I have no way of ever finding out. I am not some kind of a casual passerby — I have dedicated years to learning Haskell and understanding the Haskell community. If I do not have the expertise, the empathy, the toughness to be contributing, who do you think will?

Anyway. Let us look at the specifics of this issue again. @Bodigrim Would you say adding vectorToMaybe is technically challenging? This is fine. We are engineers, solving technical challenges is what we do. What is the specification? We can start by writing type signatures, then test cases that show how Just ∘ head — the trivial solution — is not everywhere defined, then see. Thing is, we do not get to this point of supposed technical challenge, because we cannot agree on type signatures to be exported. Type signatures are not technically challenging! The issue as it stands is politics. The challenge is to persuade the maintainers. If I wanted to do politics, I should have become a politician, not a software engineer.

Here is one example of a statement from a maintainer that is simply arbitrary. No research, no justification.

But adding several module of reexports puts too much burden on maintainers with marginal benefits

@Shimuuar, how do you know what the benefit will be for the users of the library? Have you researched? I have brought evidence: rio, relude. Expert Haskell software engineers put hundreds of hours into these libraries. Why is it worthy for them and not for you? That you ignore the obvious contrary evidence and feel no need of justifying your executive statements shows that you simply speak from the height of formal authority. This is not respectful.

A maintainer of an open source library is at once an engineer and a manager. Reducing problems to technical challenges is a worthy skill, but one cannot solve all problems by reduction to a technical challenge. Here are some problems that are not solved and that make vector open to issues like this one:

• What is the vision? For example:

  • Is vector supposed to offer a minimal set of highly efficient primitives for working with vectors, or popular and helpful composite operations as well?
  • Is vector supposed to promote best practices, or is backwards compatibility a priority, even if that makes writing modern code adhering to best practices harder?

• What are the priorities? There is a set of stuff that must be done on time — for example, supporting that many last major versions of GHC, fixing security issues. There is also other stuff that would be good to do eventually. Where is this written down? For example, why is breakR (recently added in 9bc08ba) more important than vectorToMaybe?

  If you do not have the time to write total counterparts of partial functions right now, this is fine — you can add it to the list of low priority tasks. This is no reason to dismiss the issue.

• On what basis can people come and work on vector?

  For example, I can easily contribute to fourmolu (and I have) because the way is clear and the maintainers are respectful.

  • With vector, you create this atmosphere of wizardry. How about you add a little bit of documentation instead? I look at the code and I see that head is implemented in terms of (!). Surely vectorToMaybe can be implemented in terms of (!?) in the same way. What is nightmare to maintain, document and keep consistent between different flavours here? Why is writing module X (y) where {import Y (y)} too much burden for you and not to people whom you force to work on alternative preludes?

    This is not a rhetorical question. If vector is so poorly architected that even trivial changes are infeasible, then we have a bigger problem than just some missing functions. Haskell's value proposition includes ease of refactoring. How come a core library, maintained by the brightest minds of our community, cannot demonstrate this value?

  • I should contribute to fourmolu because I know the maintainers are respectful, so my mental health will go uphill. I should not contribute to Cabal or to vector because I had one or more experiences where my mental health was damaged. It is as simple as that.

I have seen some improvements to core libraries. For example, Data.List will advise you to use the total function uncons instead of head, and singleton has been added. Recall one argument from that lengthy discussion:

I would strongly encourage anyone inclined to say things like “there's no benefit in having the function” to consider reframing the sentiment as “I wouldn’t benefit from having the function, and don’t understand the benefit others will gain.” Once you’ve done that reframe, you can notice that there’s a lack of understanding toward which the appropriate emotional stance is curiosity, not dismissiveness. A lot of people have clearly expressed in this thread that they would benefit from this function, and when you assert as fact that such benefit does not exist, you’re implicitly dismissing and devaluing their experience.

This was 4 years ago. Persuading the core libraries committee to add singleton was a massive effort spanning several media. We cannot do like so for every small improvement. Have we not learned any better?

Technical expertise is important. But also management effort is important. Why not write a clear vision, priorities and guidelines, why not stay curious and respectful? Then, issues like this one will not be needed. Why not solve the problem at the root?

[lehins]

1. maintainer no listen

Carter has not been maintainer of vector for a long time. So I believe this issue is resolved. Please don't put current maintainers into the same bucket.

2. strong contributor be going elsewhere

I've personally contributed to rio and now also maintain vector. I really like how rio structured, but it is opinionated, which does not suite everyone in the Haskell community. I am also a big proponent of total functions. Problem is you and I are not the only customers of vector. And huge chunk of people believe that vector should have all the functions partial by default and that safe Haskell is overrated. So, don't take it up with us, the maintainers, take it up with the community.

3. new contributor be no coming

Why was my issue removed? Must have been some expectation I did not anticipate. What expectation? I have no way of ever finding out. 

I have no idea. I tried to find the issue you are speaking about, but I can't find any record of that. Take it up with @cartazio He's been removed from being vector maintainer over three years ago.

On what basis can people come and work on vector?
... Surely vectorToMaybe can be implemented in terms of (!?) in the same way.

Please, by all means. PRs are welcome. No-one will stop you from contributing to vector. If something is unclear we'll give you guidance. If you have questions, we'll try to answer them.
Remember though, we do this for free! So please don't jump with expectation that we should accommodate request of every vector user, just because they think it is important.

Why not write a clear vision, priorities and guidelines, why not stay curious and respectful? Then, issues like this one will not be needed.

This is a good idea. I'd totally support that. However, personally I am in a situation right now where I can't dedicate too much time to open source work, since I currently have very tight deadlines at work.

[Shimuuar]

@kindaro please, telling people what should they do and doing so with sense of absolute certainty is quite impolite.

You also seem to forget that in open source people do what they want to do. There's no contractual obligations, no manager to tell them what to do.

For example, why is breakR (recently added in 9bc08ba) more important than vectorToMaybe?

Because I felt like implementing it. Why there's no vectorToMaybe? No one wrote it. Why documentation is lacking? No one wrote it. And you can't change what others do.

"Talk is cheap. Show me the code"

[kindaro]

Hello Alexey @lehins!

1. maintainer no listen

Carter has not been maintainer of vector for a long time. So I believe this issue is resolved. Please don't put current maintainers into the same bucket.

3. new contributor be no coming

Why was my issue removed? Must have been some expectation I did not anticipate. What expectation? I have no way of ever finding out.

I have no idea. I tried to find the issue you are speaking about, but I can't find any record of that. Take it up with @cartazio He's been removed from being vector maintainer over three years ago.

But Aleksey @Shimuuar does not listen to me either. I said (and brought evidence) that API made out of total functions is important, but to him «adding several module of reexports puts too much burden on maintainers with marginal benefits». His latest comment shows that he misunderstood me big way, and I do not see any effort of constructive communication from him. This is evidence that «maintainer no listen» is still a correct summary.

Anyway — this is not how trust works. GitHub does not show me the list of maintainers, so I cannot tell who is or is not a maintainer at any given time. Instead, there is collective responsibility. For example, I know that Mikolaj Konarski is respectful and I should always work with him. Last time I checked he was a maintainer of Cabal, and this is good for me. But I also know other people who were maintainers of Cabal at some point (or were behaving as such de facto — I cannot know) and who were not respectful. I cannot take the risk. I shall think three times before even looking at their issue tracker. Likewise with vector: I shall do whatever Andrew @Bodigrim says because I know he is respectful and caring, but nevertheless even simply talking here is stressful enough.

If a maintainer breaks the code and disappears, other maintainers are accountable. You do not get to say «it was not me». Likewise, if trust is broken, maintainers are accountable. You are not line workers — you are the top management. You are ultimately accountable for the success of the library — not me and not someone who was a maintainer 3 years ago.

By the way, let us not put responsibility specifically on Carter @cartazio. Someone with moderator level access removed my issue 4 years ago. We have not found out who or why. I remain a friend and supporter of anyone who contributes to Haskell. What matters now is repairing the trust that was damaged and picking up the responsibility that was dropped.

I've personally contributed to rio and now also maintain vector. I really like how rio structured, but it is opinionated, which does not suite everyone in the Haskell community. I am also a big proponent of total functions. Problem is you and I are not the only customers of vector. And huge chunk of people believe that vector should have all the functions partial by default and that safe Haskell is overrated. So, don't take it up with us, the maintainers, take it up with the community.

What matters is not what people believe in itself. You would not believe what some people I met believe if I told you. Belief can be manipulated — what matters is solving the customers' problems and sustainability of business.

• Exporting a full set of total functions from Data.Vector.Safe is a design choice.

  • Does it solve customers' problems? Well, we should talk to customers. We have a few here: me, you, Andrew, Aleksey. We can also send a message to Michael and to Dmitrii and ask them if such a change would help rio and relude. We can also think, read a book or two and come up with other ideas to validate the market fit. If you stand by my side, together we can do it.
  • Does it keep the business sustainable? Well, we should talk to maintainers. We should ask Andrew and Aleksey how many more human hours per release of GHC will be needed to maintain the patched code and at what level of expertise these human hours will need to be.

• Deprecating partial and otherwise unsafe functions from Data.Vector and exporting them from Data.Vector.Unsafe is a design choice.

  • Does it solve customers' problems? Some will be happy, some will be hurt. Once again, we can ask. We can also measure the damage directly. How much human hours would it be to update all of Hackage by hand? Can a specially written script, or even a modern code editor, repair the imports automatically?
  • Does it keep the business sustainable? The slogan of Haskell is «if it compiles, it works». Everything we can do to support this statement helps Haskell. This is a strong reason to go with this plan if we care about the long term. But also breaking changes strengthen an image of instability. I think the maintainers of core libraries should come together and develop a policy of what grounds are sufficient for a breaking change.

  You suggest that there are people that specifically want some partial functions among the API of core libraries, presumably all mixed together. We should find a few such persons and do some in-depth interviews. This is the case where qualitative research needs to be done.

• We could also build a whole standalone package vector-safe. This is another design choice.

  • Does it solve customers' problems? Kind of yes but also kind of no. It is less flexible. You cannot always edit build-depends because there is also Cabal freeze file and who knows what other consequences. It is more convenient to be able to import partial or unsafe functions when you need them without touching Cabal's stuff.
  • Does it keep the business sustainable? At first glance, nope. I should help maintain a core library because it is common good, but I shall not maintain some random library with 3 users. I cannot even find time to maintain my private custom prelude… However, maybe we can find a way to automatically generate a copy of vector with unsafe and partial functions hidden. Maybe we can even make it import stuff from vector instead of being a literal copy. Then we can make the release script for vector also release that copy. (You do have a release script, right?)

• Lastly, keeping things as they are is also a design choice.

  • Does it solve customers' problems? No.
  • Does it keep the business sustainable? Kind of yes but also kind of no. Haskell's value proposition includes type safety. If it compiles it better not throw me an error from pure code. If Haskell cannot deliver on its promise, why not use a more popular language like Python or Rust? Plausibly core libraries that deliver on the promise of a programming language are important for the long term success of said language, but I do not have any research at hand to quantify this effect.

The big problem right now is that objections like «nightmare to maintain», «you and I are not the only customers» and «in open source people do what they want to do» crowd out deep and constructive thinking. This is why management matters. Someone needs to think deeply and constructively about stuff more abstract than code.

On what basis can people come and work on vector?
... Surely vectorToMaybe can be implemented in terms of (!?) in the same way.

Please, by all means. PRs are welcome. No-one will stop you from contributing to vector. If something is unclear we'll give you guidance. If you have questions, we'll try to answer them. Remember though, we do this for free! So please don't jump with expectation that we should accommodate request of every vector user, just because they think it is important.

You are simplifying to the point that truth is compromised.

• It is a real risk that I write a patch and it will be declined — hours and days of work to waste. I have a record of merging patches to open source projects that you can easily see. I know how it works. Worse, it could happen that a patch is merged but has bad long time consequences. Maintainers exist for a reason!
• Accommodating requests of every user is also obviously impossible (though this is the ideal) — I never suggest anything like that. What I say is rather that a systematic way of managing requests of users must be in place. I never say «why you no do what I say» — I say «why you no listen». I am always respectful. If I was not respectful in any of my messages, point it out to me and I shall fix it.

Why not write a clear vision, priorities and guidelines, why not stay curious and respectful? Then, issues like this one will not be needed.

This is a good idea. I'd totally support that. However, personally I am in a situation right now where I can't dedicate too much time to open source work, since I currently have very tight deadlines at work.

I understand you and be sure I deeply appreciate the work you put in vector. The problem here is that I can write patches but only the maintainers, in consultation with other experts, can set the vision, the priorities and the guidelines. I am a line worker — you are the top management. If I can help in some specific way, ask!

---

Hello Aleksey @Shimuuar!

@kindaro please, telling people what should they do and doing so with sense of absolute certainty is quite impolite.

Would you say the essence or the topic of any of my messages above is telling people what they should do with the sense of absolute certainty? I accept that this is a possible reading, but I cannot see specifically how you managed it. Understanding one another on the Internet is also a kind of work — it is not always easy. I make an effort to write clear and respectful messages, but I accept that I may have failed at some point. What I intended to deliver is that there is evidence to certain problems that make the experience of working with vector worse than it could be, both as a user and a contributor.

You also seem to forget that in open source people do what they want to do. There's no contractual obligations, no manager to tell them what to do.

For example, why is breakR (recently added in 9bc08ba) more important than vectorToMaybe?

Because I felt like implementing it. Why there's no vectorToMaybe? No one wrote it. Why documentation is lacking? No one wrote it. And you can't change what others do.

You are the manager, Aleksey. You are the top management.

There is no such thing as «no manager». Choosing to do nothing is also a choice. Choosing to do whatever you feel like is also management. Moreover, as a spokesperson and maintainer of a core library, you are privileged to be a top manager of Haskell. With great power comes great responsibility.

"Talk is cheap. Show me the code"

Your wish is my command.

-- | /O(1)/ Just the first element — or nothing when there are none.
vectorToMaybe :: Vector v a => v a -> Maybe a
{-# INLINE_FUSED vectorToMaybe #-}
vectorToMaybe v = v !? 0

See, you're the boss.

[Bodigrim]

By the way, let us not put responsibility specifically on Carter @cartazio. Someone with moderator level access removed my issue 4 years ago. We have not found out who or why. 

It is theoretically possible (though implausible) that your issue was removed by one of organisation admins. However, what I can tell you very certainly is that none of the current team (me + Alexey + Aleksey) is responsible for it simply because we did not have access rights at the time. Unfortunately, we did not know about the (now deleted) history of the issue and Aleksey did not mean to insult or impinge you when he closed the issue two days ago.

I feel that we got off on the wrong foot here, shall we try and start anew?

[kindaro]

It is theoretically possible (though implausible) that your issue was removed by one of organisation admins. However, what I can tell you very certainly is that none of the current team (me + Alexey + Aleksey) is responsible for it simply because we did not have access rights at the time. Unfortunately, we did not know about the (now deleted) history of the issue and Aleksey did not mean to insult or impinge you when he closed the issue two days ago.

The only specific consequence of that event is that I generally trust the moderators less. I am not here to pin blame on someone. If you can promise that issues will not be disappearing on your watch, then good. If you allow for the possibility that this issue disappears tomorrow, then I should take extraordinary measures, such as saving some screen shots. This is not technically difficult — the trouble for me is that I do not know what to expect, what the policy is. Likewise, I cannot tell what kind of patches will be merged and which will be declined. Is vectorToMaybe getting into the code base? I have no idea! Open source collaboration can only be built upon the ground of common knowledge about shared values, and right now I cannot tell what the values shared by the moderators of vector are.

So far my main contribution is the summary of the 4 different ways to resolve this issue:

• Exporting a full set of total functions from Data.Vector.Safe.
• Deprecating partial and otherwise unsafe functions from Data.Vector and exporting them from Data.Vector.Unsafe.
• Building a whole standalone package vector-safe.
• Keeping things as they are.

In the ideal, I want to have such clear understanding of values shared by the maintainers of vector that I could tell even before opening an issue which way they will choose. But in reality even after talking for 2 days I have no idea. This is a problem for me because I need to understand where Haskell is going. The question «should I keep working with Haskell or should I switch to another language» is very important for me and the answer hinges on my ability to tell where Haskell will be next year or 5 years in the future.

For example, opaleye makes breaking changes all the time. But I am confident depending on it because I know what values we share with Tom Ellis and I trust him to take the package to the better future. I do not have the same feeling about the core libraries. Some functions you folks add after a single request, other functions only after a month long electronic mail debate with pressure from other social media, yet other functions never. Honestly, my strongest feeling about core libraries overall is helplessness.

If I understand that Haskell's core libraries will be full of partial functions forever, at least I could put a big cross on them and stop worrying. But sometimes you make small steps to improvement. Where are you going? Where will Haskell be next year or 5 years in the future?

The simplest way to build common knowledge about shared values is to enforce them from the position of authority. For example, if you say «our value is eternal backwards compatibility, anyone who disagrees should fork», everything will become clear. Or you can say «our value is best practices», «our value is minimal interface», «our value is high performance». But you cannot say «our value is doing whatever we feel like» — this does not allow your customer to make any decision because it does not describe the product.

Are there other ways to build common knowledge about shared values? There are, but I cannot offer anything specific. This is why I say that maintainers are top managers — you are in the best position to build this common knowledge. And without this common knowledge it is very hard to achieve open source collaboration.

I feel that we got off on the wrong foot here, shall we try and start anew?

Whatever you say I shall do. If you can give me clear instructions I shall follow these instructions.

[Bodigrim]

If you can promise that issues will not be disappearing on your watch, then good.

The general policy is not to delete issues, unless their content is inappropriate (hate language, profane language, etc.). We are not in position to give any hard promises - after all we are at mercy of organisation admins and even well intentioned people can fat finger by mistake, but a removal of an issue is an exceptional, almost unheard of event. I'm really sorry for your experience here 4 years ago, I don't know what happened (and I'm not implying that it was your fault).

[lehins]

Exporting a full set of total functions from Data.Vector.Safe.

@kindaro I recommend reading our discussion in this ticket #361 in full. You will see that we not only already considered this exact approach, but this has actually been done in the past in vector-0.9 and has been reverted since.

We, as maintainers decided that even splitting Unsafe functions into their own modules is not worth of pursuing, therefore we are certainly not going to induce a whole lot of breakage on downstream users with shifting around safe and partial functions.

Therefore, I suggest going the safest and the most lenient route possible: "Building a whole standalone package vector-safe." That package can re-export all of the vector's functionality from relevant modules, similar to how rio does this. It seems like you have tremendous interest in this issue, so I would encourage you on starting such package. If it receives enough traction over time it could be transferred under haskell's organization wing.

Stability of core packages is very important, often more important than API improvements. We appreciate your interest in Haskell ecosystem, and I hope you can respect our decision as maintainers to not pursue these changes. That being said, if you still feel very strongly about making these improvements directly in the vector package at the cost of breaking a lot of code, then I would recommend bringing this discussion to a wider audience through channels like Haskell discord, mailing list, Haskell reddit, etc. See if you can get support there and maybe even find some volunteers to do the work.

That is my, hopefully clear instruction on how to go forward about this issue.

As far as vector package is concerned I recommend closing this ticket as "wontfix". @Bodigrim and @Shimuuar, any objections?

[Shimuuar]

None.

Although I want to add that additions of total counterparts of partial function and documentation improvements are welcome

[lehins]

Just a quick follow up. Addition of total counterparts to all partial functions from the current API that don't currently exist deserve is definitely issue worthwhile on pursuing. Although a separate issue and a proper discussion is necessary. At least that would have a much narrower scope that this ticket.

For example, I'd rather have a name headMaybe, rather than vectorToMaybe, which means such additions require some discussions. But I totally agree that such functions would be useful.

@kindaro So, a separate suggestion I have is creating an issue that lists all of total functions that are missing from current API. And collectively we could work on something like that.

[lehins]

Although I want to add that additions of total counterparts of partial function and documentation improvements are welcome

😄 This is too funny, we said the same thing at the same time. I literally pressed the button at the same time as your comment appeared

[Bodigrim]

I do not have the same feeling about the core libraries. Some functions you folks add after a single request, other functions only after a month long electronic mail debate with pressure from other social media, yet other functions never. Honestly, my strongest feeling about core libraries overall is helplessness.

The fundamental answer is that core libraries are understaffed (and, well, unpaid). We simply do not have resources to maintain a consistent service level for all requests. Sometimes a maintainer has more spare time than usual and gets nerd snipped into doing something, sometimes it indeed takes months even for myself to make things happen.

It's also a sign of maturing ecosystem, and I do not see "a month long debate" too high of threshold to change core libraries, affecting thousands of users. We do routinely have much longer discussions at https://github.com/haskell/core-libraries-committee.

---

@lehins let's carry on here for a little bit longer before closing.

[Bodigrim]

Speaking of values, I do prefer total functions to partial ones. However, as I learned in course of haskell/core-libraries-committee#87, this is not a value shared by all Haskell community.

I'm not sure that separating total functions into their own module makes sense:

• For Mutable vectors the majority of operations involve raw indexing. Technically you can separate, say, new and replicate to Data.Vector.Mutable.Total, but it does not make sense to me: surely you want to read and write, otherwise you'd stick to immutable interface.
• For immutable vectors it's vice versa: the only partial functions in the API are indexing-related (head / tail / ! / etc.). So Data.Vector.Total would be almost a full clone of Data.Vector. Maintaining its consistency with Data.Vector would be a significant cost, I'm not yet (but I can be) convinced that it's worth it.

I'm happy to merge total counterparts for head / tail and friends, please raise a separate issue to discuss the hardest of problems - naming. Once there are total counterparts available, we can also consider our next steps.

[kindaro]

Unbelievable to see such great headway after going out for a short walk!

Speaking of naming, let me add a small note. In Mathematics, there is this concept called «mathematical red herring principle». In this case:

• Partial functions are not functions — they are relations that reflect distinctions.
• Total functions are no more total than natural numbers are non-negative. Does one say «non-negative natural numbers»? Relations that respect elements and reflect distinctions are called «functions», simply «functions».
• Unsafe functions are not functions — they are not even relations with the specified source and target.

I think reframing the conversation about supposed functions in this context will help clarify the grand scheme into which things should eventually fit. There is already a disastrous article on the Internet that conflates functions and procedures, as if functional programming did not exist. We can do better.

on disappearing issues

If you can promise that issues will not be disappearing on your watch, then good.

The general policy is not to delete issues, unless their content is inappropriate (hate language, profane language, etc.). We are not in position to give any hard promises - after all we are at mercy of organisation admins and even well intentioned people can fat finger by mistake, but a removal of an issue is an exceptional, almost unheard of event. I'm really sorry for your experience here 4 years ago, I don't know what happened (and I'm not implying that it was your fault).

Also cosmic rays can flip bits and GitHub can be razed by ransomware. So can you write this general policy down? And who are the organization admins, by the way? Is there a reason not to put the list of people with moderator level access to the documentation? Sorrow for the past does not matter — positive changes for the sake of better future are what matters.

on long discussions

I do not have the same feeling about the core libraries. Some functions you folks add after a single request, other functions only after a month long electronic mail debate with pressure from other social media, yet other functions never. Honestly, my strongest feeling about core libraries overall is helplessness.

The fundamental answer is that core libraries are understaffed (and, well, unpaid). We simply do not have resources to maintain a consistent service level for all requests. Sometimes a maintainer has more spare time than usual and gets nerd snipped into doing something, sometimes it indeed takes months even for myself to make things happen.

It's also a sign of maturing ecosystem, and I do not see "a month long debate" too high of threshold to change core libraries, affecting thousands of users. We do routinely have much longer discussions at https://github.com/haskell/core-libraries-committee.

So, how about we instead have a long discussion once that sets the vision for the core libraries with a timeline of 1 year, 5 years and eternity, and then summarize this discussion into a vision document? Seems like this could save some time for the understaffed maintainers. Has this been tried?

on shared values

Speaking of values, I do prefer total functions to partial ones. However, as I learned in course of haskell/core-libraries-committee#87, this is not a value shared by all Haskell community.

No value will ever be shared by all. It is a situation best described by a probabilistic model. What matters is that all changes add up to a great product that solves a specific problem in a way that is sustainable. Right now I see that at least three of us agree that in the eternity functions should be sundered from more general relations, and all the more so from things that are not even relations.

It will take me some time to read that conversation. I see you have great fun over there.

on adding total functions

So, say we have total counterparts to partial functions. How can they be marked? How would the user tell which supposed function is a function and which is merely a partial function? If functions are marked as such in a uniform way, then it is easy to write a safe package generator, a linting tool or any other kind of tool. In the ideal also partial functions should point to their total counterparts. Is there a clear way of how this can be done? A feature that users do not get to know about is not complete.

I am on the way to raising a separate issue to discuss naming.

[Bodigrim]

@kindaro you are asking great questions, but unfortunately way above our pay grade. Yes, it would be nice to have a clear governance (see haskell/meta#1) and it would be nice if someone has time to design a grand vision of Haskell future for 5 years ahead.

[cartazio]

Please stop @ ing me. I edited your comments at the time because I have and had clinical anxiety / generalized panic issues, and I wanted to focus on the technical parts of your ideas And a person in my family was in the hospital at the time on life support so I didn’t have the time [omitted - Bodigrim].

[Bodigrim]

Carter, I'm sorry for your mental health issues, but let's maintain respectful attitude and avoid personal insults. I edited your message above.

I think it's really time we continue the discussion of total functions elsewhere, in a separate thread.

[kindaro]

Hey Carter. I know from the inside what anxiety and panic attacks feel like. I love you as a brother, stay strong!