Advanced Extents & Segments Proposal re C# 8.0 Ranges

[verelpode]

Advanced Extents & Segments Proposal

Rough sketch -- Version 1 (2019/05/07)

The potential for advanced Extents and Segments in C# is exciting and under-estimated. I am writing this proposal because I feel that the current version of the C# 8.0 Range proposal is not yet ready for release. It would be sad if a half-baked version is released prematurely and set in concrete forever. Although future extensions are possible in C# 8.5, this becomes difficult when it breaks compatibility with the syntax and design in C# 8.0, therefore it would be best to design the advanced scenarios NOW not later.

This is a rough sketch and I've been forced to rush it, thus I'm sure it contains some mistakes and omissions, but it is nevertheless worthwhile to present. Example:

int myOffset = ...;
int myLimitOffset = ...;
Segment<char> mySegment = ...;
Segment<char> resultSegment = mySegment[myOffset from start until element is CharMatchers.VerticalWhiteSpace 
     && element != '\u2029' with fence myLimitOffset with maxLength 5000*2];

The above example does this: Beginning with the Segment<char> (or Span<char> or Array or List<char>) named "mySegment", make a sub-segment of mySegment (alternatively an Extent alone) that begins at offset "myOffset" from the start, and continues until an element (char) is a vertical whitespace character and does not equal "paragraph separator" (U+2029), but do not extend the Extent or Segment any further than "myLimitOffset", and limit Extent.Length to 5000*2.

If you want to get the Extent (offset and length) alone without making a new Segment<T> corresponding to that Extent, then you can write:

Extent x = [mySegment at myOffset from start until element is CharMatchers.VerticalWhiteSpace 
     && element != '\u2029' with fence myLimitOffset with maxLength 5000*2];

Relative offsets: The "from" keyword makes an offset relative to something else. The default is "from start" thus writing "myOffset from start" is the same as writing simply "myOffset" without any "from" keyword. Supported "from" keywords include:

	100 from start
	100 from end
	100 from myPositionX forwards
	100 from myPositionX backwards
	100 from myPositionX backwards for 300+10
	100 from last '%' backwards
	100 from first CharMatchers.UppercaseLetter forwards
	100 from first CharMatchers.UppercaseLetter || CharMatchers.WhiteSpace forwards

The "for" keyword makes an Extent. For example, 100 for 20 means Extent.Offset = 100 and Extent.Length = 20.
100 from myPositionX backwards for 300+10 means the Extent.Length is 300+10, and the Extent.Offset is calculated like this: Begin at offset myPositionX then slide the offset backwards by 100. The calculation is simply:

	Extent.Offset = myPositionX - 100;
	Extent.Length = 300+10;

Supported "with" keywords include:

	with fence myLimitOffset
	with maxLength myMaxLength
	with minLength myMinLength
	with length myLength    // maybe, see also "for".
	with direction backwards
	with direction forwards

CharMatchers.VerticalWhiteSpace is an immutable singleton class that implements IElementMatcher<char>. Anyone can write their own classes that implement IElementMatcher<T> and match elements using any criteria desired.

Naturally if you want the opposite of a particular matcher, you can use the logical-not operator with the "is" operator:

element is !CharMatchers.Alphabetic && element is !CharMatchers.WhiteSpace

Without using a matcher, the following example begins at offset "myStartOffset" and continues until an element (char) is the carriage-return character or the newline character:

Segment<char> resultSegment = mySegment[myStartOffset until element == '\r' || element == '\n'];
// Or if you only want the Extent:
Extent x = [mySegment at myStartOffset until element == '\r' || element == '\n'];

"while" keyword

Note the difference between the "until" and "while" keywords. The following example begins at "myStartOffset" and continues while the elements are decimal digits or underscore:

Segment<char> resultSegment = mySegment[myStartOffset while element is CharMatchers.DecimalDigit || element == '_'];
// Or if you only want the Extent:
Extent x = [mySegment at myStartOffset while element is CharMatchers.DecimalDigit || element == '_'];

"where" keyword

The following loop iterates through each segment/run of consecutive whitespace characters:

foreach (Segment<char> whitespaceSeg in mySegment[myStartOffset where element is CharMatchers.WhiteSpace])
{ ... }

The foreach loops functions with the "where" keyword because "where" produces IEnumerable<Segment<T>>:

IEnumerable<Segment<char>> results = mySegment[myStartOffset where element is CharMatchers.WhiteSpace];

Note that the returned IEnumerable instance is not a pre-generated array, rather it produces its enumerable elements on-demand like how System.Linq.Enumerable does. This can be implemented using the preexisting yield return feature of C#. Also on the topic of efficiency, note that Segment<T> is a struct, thus the where keyword does not create thousands of objects in the heap.

TO DO: The "where" keyword gives you the matching segments. Decide whether to make an option or variation of "where" that gives you both the matching and non-matching segments.

Moving Average

A "moving average" example was provided on the following webpage, but the version there as of 2019/05/07 looks defective unfortunately.
https://docs.microsoft.com/en-us/dotnet/csharp/tutorials/ranges-indexes
Many elements are missed/ignored in that moving average because it does start += 100" yet "Range r = start..start+10;
Furthermore, it has a function named "MovingAverage" that doesn't return a MovingAverage at all, surprisingly. It is named MovingAverage but it returns the same as System.Linq.Enumerable.Average.

The following design of MovingAverage attempts to eliminate the defects. In addition, it is better because it smooths-out the average by overlapping the segments.

using System.Linq.Enumerable;
static IEnumerable<double> MovingAverage(Segment<int> inData)
{
	foreach (Segment<int> seg in inData.CutAndSplice(inPieceLength: 20, inOverlapLength: 10))
	{
		yield return seg.Average();
	}
}

See also: https://en.wikipedia.org/wiki/Moving_average

Factionalize

The "factionalize" operation splits a segment into a sequence of "factionalized segments". A "factionalized segment" is a Segment<T> with a faction value assigned. For example, factionalization could be used as a stage in syntax parsing of text that is formatted to obey particular syntax rules. Example:

enum MyFaction { ... }
Segment<char> mySegment = ...;
foreach (FactionalizedSegment<char,MyFaction> in mySegment.Factionalize<MyFaction>(MyFactionalizer))
{ ... }

The foreach loops functions with Segment<T>.Factionalize because Factionalize returns IEnumerable<FactionalizedSegment<TElement,TFaction>>.

If Factionalize is used as a stage in syntax parsing of a programming, scripting, or data language, then TFaction could be defined as an enumeration like this:

enum TokenKind
{
	None = 0,
	WhiteSpace,
	Number,
	Word,
	Keyword,
	PlusSymbol,
	MinusSymbol,
	MultiplySymbol,
	...
}

An instance of the delegate DlgFactionalizer is given to the Segment<T>.Factionalize method. See definition of this delegate within the .cs file.

Other slicing operations

See the .cs file for other slicing operations in Segment<T> such as:

public IEnumerable<Segment<T>> CutUp(int inPieceLength)
public static IEnumerable<Segment<T>> operator / (Segment<T> inSegment, int inPieceLength)
public IEnumerable<Segment<T>> CutAndSplice(int inPieceLength, int inOverlapLength)
public IEnumerable<Segment<T>> Dice(int inPieceLength, int inSkipLength, bool inInvert = false)
public IEnumerable<Segment<T>> Split(DlgSegmentSplitter<T> inSplitter)
public static IEnumerable<Segment<T>> Interleave(int inLength, params Segment<T>[] inSegments)
public Segment<T> ChopOffStart(int inPieceLength)
public Segment<T> ChopOffEnd(int inPieceLength)
public Segment<T> CleaveStart(int inPieceLength)
public Segment<T> CleaveEnd(int inPieceLength)
public void SeverStart(int inPieceLength, out Segment<T> outStartPiece, out Segment<T> outRemainder)
public void SeverEnd(int inPieceLength, out Segment<T> outEndPiece, out Segment<T> outRemainder)
public void Sever(Extent inExtent, out Segment<T> outStart, out Segment<T> outMiddle, out Segment<T> outEnd)
public (Segment<T> partA, Segment<T> partB) Bisect(int inOffset)
public IEnumerable<FactionalizedSegment<T,TFaction>> Factionalize<TFaction>(DlgFactionalizer<T,TFaction> inFactionalizer)
public Segment<T> Truncate(int inMaxLength)
public Segment<T> GetExtent(Extent inExtent)
public Segment<T> GetExtent(int inOffset, int inLength)
public Segment<T> GetExtentToEnd(int inStartOffset)
public Segment<T> GetExtentToStart(int inOffset)
public Segment<T> GetExtentAtStart(int inLength)  // might be renamed to GetPrefix
public Segment<T> GetExtentAtEnd(int inLength)    // might be renamed to GetSuffix
public Segment<T> GetRange(int inStartOffset, int inEndOffset)
public Segment<T> GetRangeInclusive(int inStartOffset, int inEndOffset)

Syntax for basic extents and ranges

Basic examples following (basic meaning not using the until, while keywords etc).

Extent x = [20 for 30];   // makes Extent.Offset=20, Extent.Length=30
Extent rangeA = [20..50]; // makes Extent.Offset=20, Extent.EndOffset=50
Extent rangeToEnd = [20..];
Extent rangeToStart = [..20];
Extent singleElement = [5];  // makes Extent.Offset=5, Extent.Length=1 or maybe Length=0
Extent negativeOrdinal = -[5]; // Extent.Offset == DataLength - 5
Extent reverseOrdinal = ~[5];  // makes 0-based ordinal in reverse, as if System.Array.Reverse was executed.
Extent rangeB = [20 .. -[5]];  // Extent.EndOffset == DataLength - 5

The following are the same, which means that -[5] actually invokes the operator - (Extent) overload method that is defined in struct Extent.

Extent negativeOrdinal = -[5];
Extent negativeOrdinal = -(new Extent(5, 1));

Likewise ~[5] invokes the operator ~ (Extent) overload method that is defined in struct Extent.

Iteration in reverse order

Example:

Segment<char> seg = ...;
int len = seg.Length;
for (int i = 0; i < len; i++)
{
	char ch = seg.InReverse[i];
	// Also supported:
	char ch = seg.GetItemReverse(i);
}

Builder

See Segment<T>.Builder in the .cs file.

https://github.com/verelpode/Advanced-Extents-Segments

[HaloFour]

What makes a Segment<T> or Extent<T> different from a Span<T>? Most of this seems like an API that could be furnished by a library. The proposed language changes seem ... odd. A reinvention of LINQ but specific for finding an offset? That also feels like it could be easily furnished by an API using fluent methods rather than so many permutations of combining keywords.

[verelpode]

@HaloFour

What makes a Segment<T> or Extent<T> different from a Span<T>?

Extent (non-generic) is instead of System.Range, thus Extent and System.Range don't contain any array elements whereas Span<T> does point to elements. Segment<T> is the one that is comparable with Span<T>. The difference is that Segment<T> includes much more functionality than Span<T> but I'd also be happy if the final design is to add the new functionality to Span<T> and/or System.Memory<T> rather than creating Segment<T> separately. However for the purposes of compiling and testing the .cs file, it is easier to test it as struct Segment<T> independent of Span<T>.

Most of this seems like an API that could be furnished by a library. ... That also feels like it could be easily furnished by an API using fluent methods rather than so many permutations of combining keywords.

Yes as I said in a previous message: "I'm unconvinced that it should be a language feature"

However language/syntax ideas were specifically requested of me, so that's what I did. For example, @CyrusNajmabadi requested "explanation of what that means at a language level". Also we're discussing this in dotnet/csharplang therefore the proposal should include language/syntax ideas at least for brainstorming and inspiration purposes, regardless of whether some of those language features are deleted or modified before the final release.

The proposed language changes seem ... odd.

As you said in your previous message:
"They'll learn it once and move on. Any new feature requires getting used to."

A reinvention of LINQ but specific for finding an offset?

Linq is part of the C# language. Linq is not a separate programming language. In fact, Linq is an acronym for "Language Integrated Query", and what language is it integrated into? C# ofcourse. Linq is C#. Thus it makes perfect sense for my proposal to remain consistent with C# by reusing preexisting keywords in C#. Why should I invent new different keywords when the preexisting keywords in C# are already good? The similarity with Linq is an advantage of my proposal, not at all something strange.

[CyrusNajmabadi]

This seems incredibly esoteric and niche to be at the language level. I second the question about why this simply isn't an API. The examples shown don't seem to be compelling enough to warrant a language change here...

[CyrusNajmabadi]

Linq is part of the C# language. Linq is not a separate programming language.

Linq introduces patterns and practices that are felt to be wildly applicable to large swaths of programming domains. I'm not seeing that here. You're introducing a complex and esoteric syntax... for a use case that i can't recall a single person before you asking for.

Is this modeled after some existing successful system? Why would this particular API warrant such a large language change, given the limited use and applicability to the ecosystem?

[CyrusNajmabadi]

Also... i see nothing here that would have any impact on the existing Range/Slicing feature. You're proposing an altogether different sort of API with different language integration that seems entirely unrelated. So i don't see anything that would warrant not doing the existing range feature just as it's being planned today.

i.e. ranges are well served by syntax like a..b. Even if i thought that something to help with your 'extent' domain was warranted in the language (which i don't), it would be simple to just pick a different syntax here instead of using the simple and easy to grok range syntax.

[CyrusNajmabadi]

I'm not seeing how this really helps at a language level. You could just write:

// Your approach:
Segment<char> resultSegment = mySegment[myOffset from start until element is CharMatchers.VerticalWhiteSpace  && element != '\u2029' with fence myLimitOffset with maxLength 5000*2];

// Simple flexible API approach:
Segment<char> resultSegment = GetSubSpan(mySegment, from: start, until: e => CharMatchers.VerticalWhiteSpace(e) && e != '\u2029', fence: myLimitOffset, maxLength: 5000*2)

The API approach is actually simpler and clearer (imo). You have your domain specific API and your domain specific concepts. You can expand on it as much as you want with whatever operators are appropriate (sounds like you have a lot).

[HaloFour]

@verelpode

The difference is that Segment<T> includes much more functionality than Span<T> but I'd also be happy if the final design is to add the new functionality to Span<T> and/or System.Memory<T> rather than creating Segment<T> separately.

Seems like this could all easily be extension methods for Span<T> and for IEnumerable<T> in general. You could write them and publish them to NuGet.

I'm unconvinced that it should be a language feature

The LDM's default position is to not add features. Ranges had to meet a fairly high bar to get as far as it did, as it seems as far as the LDM is concerned it is worthy of being a language feature.

[verelpode]

@CyrusNajmabadi

I second the question about why this simply isn't an API.

And I'll third the question. Yes, some features are better when they are simply methods in a class in a library. I mentioned this previously. On the other hand, some features are good to integrate into the language, thus it's a judgement call as always.

As for the connection with Linq, I'm not the person who started that. Linq is already used in the current version of the C# 8.0 Range feature. For example, the C# 8 "Indices and ranges" tutorial uses System.Linq.Enumerable.Range and System.Linq.Enumerable.Average.

You're introducing a complex and esoteric syntax...

Well which is it? On one hand, I'm hearing that its Linq-like, and on the other hand, I'm hearing that it's "esoteric", but it can't be both of those, so therefore I ask: Well which is it?
If you'd like to increase the consistency with C# 7.x by modifying my proposal to bring the syntax even closer to Linq, then I'd be happy to go along with that.

for a use case that i can't recall a single person before you asking for.

I'm surprised to read that, because I thought about which example to place as the very first example in my proposal, and I chose that one specifically for the reason that it's a common operation. So I'm quite surprised to read your message describing it as the exact opposite.

To be precise, ofcourse it's not common if you use it exactly as per my example without changing any of the parameters. But if you do change the parameters and change the match criteria to a different expression, then the pattern is a common pattern in programming. The one specific example is uncommon but the general pattern is common.

i see nothing here that would have any impact on the existing Range/Slicing feature.

The syntax in my proposal is different and incompatible with the syntax in the current C# 8 Range proposal. One example of the multiple differences/incompatibilities is that the C# 8 Range proposal says...

Range r = 20 .. ^5:

whereas my proposal says the same thing would use this syntax:

Extent x = [20 .. -[5]];

For multiple reasons, my proposal impacts in breaking ways on the current version of the C# 8 Range proposal.

[CyrusNajmabadi]

However language/syntax ideas were specifically requested of me, so that's what I did. For example, @CyrusNajmabadi requested "explanation of what that means at a language level".

It was requested of you not to say that it was good to have syntax changes. But because it sounded like you wanted syntax changes, and it was really impossible to tell what you were asking for. I was thinking you wanted some small tweak. I didn't realize you wanted to make a new large subdialect in the language for this small API domain.

Note that the burden here to get this cahnge is enormous, as per our previous conversation:

1. you'd have to demonstrate that this would be quite useful for a wide segment of users (i.e. >50%).
2. you could say this was beneficial for a smaller group. But then you'd have to show that group is SOL with existing solutions and this approach at hte language is fundamentally necessary for them to be effective.

I think showing either is going to be very difficult to do. For one, i'm not sure i can see even 5% of users using this new feature. Second, even if there was a group that would find 'extent-based programming' useful, they'd be served completely well by just doing this as a library/API as i showed above (and not requiring an actual language change).

Note that the slice/range feature was felt to hit '1'. This was based on actually examining codebases out there and dealing with the consistent and steady feedback from many groups across many domains that this would be useful.

[CyrusNajmabadi]

Well which is it? On one hand, I'm hearing that its Linq-like,

Linq itself was, and remains to be, highly esoteric. But we accepted that because of the enormous value it presented to the ecosystem.

and on the other hand, I'm hearing that it's "esoteric", but it can't be both of those,

Yes it can. Note that the language has never since taken on anything 'linq-like'. That's very intentional. Nothing has been able to show it warrants the type of approach we took there.

If you could somehow demonstrate value on the level of, or higher than, Linq, then we could potentially justify another esoteric embedded API domain language like this. But it would be a very tall order.

[CyrusNajmabadi]

To be precise, ofcourse it's not common if you use it exactly as per my example without changing any of the parameters. But if you do change the parameters and change the match criteria to a different expression, then the pattern is a common pattern in programming. The one specific example is uncommon but the general pattern is common.

Can you summarize what you think "the general pattern" is here that you're trying to design an API/langauge-feature around? It sounds like a general "allow me to to compute start/end points of my range using combinators", but i that's not right, please let me know. Thanks!

[CyrusNajmabadi]

The syntax in my proposal is different and incompatible with the syntax in the current C# 8 Range proposal. One example of the multiple differences/incompatibilities is that the C# 8 Range proposal says...

So don't use .. Now you're no longer conflicting with Ranges. Why is it so imperative for your feature that it must use ..? Why can't it use some other syntactic form that hasn't been used? Thanks!

[CyrusNajmabadi]

Also, @verelpode Have you considered hashing some of this out on gitter? it might be easier to converse on it, rather than the async nature of all these github messages.

[verelpode]

@HaloFour

Ranges had to meet a fairly high bar to get as far as it did,

Normally the C# team does have high bars and high standards, but this Range feature is an exception because it is unusually sloppy, unusually poorly described, and unusually poorly justified, and unusually poorly accompanied by practical examples. For example, the provided "moving average" example is terribly sloppy, so bad that it should actually be deleted or rewritten. In fact, I don't think I've ever seen such a sloppy example coming from the C# team. Normally they have high standards and do impressive work. I don't know how this mistake happened, but everyone makes mistakes from time to time, so it's no big deal. Mistakes are only bad if they remain uncorrected.
I know I'm also guilty of sloppy work on a few past occasions, but I went back and fixed my mistakes and ended up proud of my work.

[CyrusNajmabadi]

because it is unusually sloppy

What is sloppy about ranges/slices?

unusually poorly described

What is poorly described?

and unusually poorly justified

I have no idea waht this means. What are you looking for in a justification?

and unusually poorly accompanied by practical examples

it literally hasn't shipped yet.

For example, the provided "moving average" example is terribly sloppy, so bad that it should actually be deleted or rewritten. In fact, I don't think I've ever seen such a sloppy example coming from the C# team.

What on earth does this sample have to do with the language feature? Why do you think it's something the C# team wrote?

Mistakes are only bad if they remain uncorrected.

Even if tehre is a problem with this sample, that has nothing to do with any sort of claim that there's a problem with the actual feature.

[verelpode]

@CyrusNajmabadi

you'd have to demonstrate that this would be quite useful for a wide segment of users (i.e. >50%).

My proposal is relevant for 90% of software engineers. On second thought, it's probably fair to say that my proposal is relevant to 99% of software engineers because a software engineer who doesn't use extents just doesn't exist. OK maybe in Antarctica there exists a software engineer who doesn't use extents, but 99% of software engineers in the world use extents, although with different names for this pattern. "Extent" isn't a common name but the pattern is extremely common. The .NET Framework contains hundreds of usages of extents. Here's just a few of the many examples:

System.String.Substring(int startIndex, int length)
System.Array.Copy(Array sourceArray, int sourceIndex, Array destinationArray, int destinationIndex, int length)
System.Array.Clear(Array array, int index, int length)
System.Array.IndexOf<T>(T[ ] array, T value, int startIndex, int count)
System.Array.BinarySearch<T>(T[ ] array, int index, int length, T value)
.....and many more.....

Can you summarize what you think "the general pattern" is here

Another general pattern is the element matcher. For example, open the Object Browser window in Visual Studio, and set it to browse .NET Framework 4.7.2, and then search for "TryParse". Many examples appear for "TryParse", and that's not the only thing. Many other examples of this general pattern exist in real-world software such as the .NET Framework / Core.

[CyrusNajmabadi]

@verelpode You realize that this feature hasn't even been completed, let alone shipped. By what basis are you judging how the feature is coming along versus other features in the past? I can't recall any sort of approach with features in the last 5 years that did things differently here.

Looking at teh C# proposals, this seems in line with the rest: https://github.com/dotnet/csharplang/tree/master/proposals/csharp-8.0

[CyrusNajmabadi]

Here's just a few of the many examples:

All of your examples are served fine by ranges... so i'm not sure what the problem is... :)

You're proposing a vast API here and a large set of language changes to integrate them into the language. The onus is on you to demonstrate that this will provide clear and unambiguous value to the greater C# ecosystem.

[CyrusNajmabadi]

Another general pattern is the element matcher. For example, open the Object Browser window in Visual Studio, and set it to browse .NET Framework 4.7.2, and then search for "TryParse". Many examples appear for "TryParse", and that's not the only thing. Many other examples of this general pattern exist in real-world software such as the .NET Framework / Core.

I don't see how your 'extent' language feature helps out all for any of the TryXXX apis. Can you clarify how those would benefit?

[verelpode]

Why do you think it's something the C# team wrote?

OK, well, I'm sorry, I didn't know that you wrote it. But it's no big deal. I've read multiple messages from you in various other threads and I found those comments good and helpful. Certainly I also produce sloppy work on occasions, but I go back and fix it.

[CyrusNajmabadi]

My proposal is relevant for 90% of software engineers. On second thought, it's probably fair to say that my proposal is relevant to 99% of software engineers because a software engineer who doesn't use extents just doesn't exist. OK maybe in Antarctica there exists a software engineer who doesn't use extents, but 99% of software engineers in the world use extents, although with different names for this pattern.

I honestly don't know how you're going from the idea that programmers use some APIs that specify a 'start+length' pair into thus, there should be this extremely large API and assorted helper functions provided here *and* that API should be mapped into C# with a large embedded DSL.

Do you see the leap you're making, and how you haven't justified it? It's like saying developers use integers, therefor it's necessary for C# to have a symbolic mathematical system built into it a-la Mathematica.

You're making a logical leap that you're not supporting.

[CyrusNajmabadi]

OK, well, I'm sorry, I didn't know that you wrote it. But it's no big deal. I've read multiple messages from you in various other threads and I found those comments good and helpful. Certainly I also produce sloppy work on occasions, but I go back and fix it.

Again, i don't have any idea who wrote this. Most likely it was the MSDN docs team. You're welcome to give them feedback that the example shoudl be improved (the page itself allows you to give feedback). I don't believe the C# team writes these docs.

However, regardless, i don't see what that has to do with the actual language feature itself. Even if this one example page is bad, that has no bearing on the language feature itself and whether or not it has been properly designed and is the right feature to ship.

[CyrusNajmabadi]

@verelpode As before, gitter is likely a better medium for this. However, i strongly recommend you read: https://blogs.msdn.microsoft.com/ericgu/2004/01/12/minus-100-points/

The onus on someone proposing a language change (especially one as large and esoteric as this) is to provide an enormous amount of evidence to justify the value of the cahnge. Language changes start in a position of being assumed to be a negative. So, in order to be done, they not only have to outweigh their substantial initial negativeness, but they also have to demonstrate a high enough value to warrant doing ahead of everything else that can be done.

In your case, you've stated you want this very large feature, but you haven't demonstrated at all why it's appropriate to exist in the language, or why it's even something that valuable to the C# ecosystem as a whole.

I'm a compiler writer (among many other things), and i'm commonly working with streams of text. And i can tell you that i can only think of a tiny handful of cases this would be helpful. And that means adding a massive feature for niche applicability. On the other hand, if you wanted to put this in a Nuget package, i think you might get some adoption. THat would likely help drive a more focused API that might eventually have some impact back on the language itself.

[CyrusNajmabadi]

///  The .NET Framework contains hundreds of usages of extents. Here's just a few of the many examples:

System.String.Substring(int startIndex, int length)
System.Array.Copy(Array sourceArray, int sourceIndex, Array destinationArray, int destinationIndex, int length)
System.Array.Clear(Array array, int index, int length)
System.Array.IndexOf<T>(T[ ] array, T value, int startIndex, int count)
System.Array.BinarySearch<T>(T[ ] array, int index, int length, T value)
.....and many more.....

All of these can be suitably handled well with Ranges. Indeed, they were many of the motivating examples for Range by both the C# LDM and CoreFx. Both sides felt that Range was a suitable abstraction here. i.e. String.Substring(Range) would work (along with just having a slicing indexer). The same would hold for the rest of those.

So the above justification really does not help your case. Because it's already believed to be the case that the Range/Slice feature is a very good solution here, and your proposal hasn't indicated why it is not, or why your proposal is substantively more desirable.

--

Note: as before, i recommend proceeding on gitter.

[verelpode]

I've made a significant effort to accommodate both of you (even though you're not MS staff members), so it would be nice if the favor is returned. Originally I didn't want the Pascal ordinals in the proposal at all, but to accommodate you two highly vocal participants, I've gone and done it anyway -- for you. The Pascal ordinals are included in my proposal document against my wish, in an attempt to satisfy you guys. I searched for a win-win solution and the best I could find is this syntax with the arity-1 negation operator with square brackets:

10 .. -[5]

I'm so accommodating and open-minded and team-friendly that I've included it in the proposal despite the fact that I still think it's complete nonsense to make C# 8 use both C and Pascal ordinals.

Originally I said that I'm unconvinced that the C# 8 Ranges should be a language feature as opposed to API. But you guys wanted language features super strongly, so I went and put it in the proposal to the best of my ability, in an attempt to accommodate you guys.

I'm searching for win-win solutions. Are you?

[verelpode]

The onus on someone proposing a language change (especially one as large and esoteric as this) is to provide an enormous amount of evidence to justify the value of the cahnge.

Exactly, and that's the huge problem here, because that principle has been followed for other C# features but not for the Range feature. Somehow the Range feature has become an emotional religious-style issue. Emotional decisions are inappropriate in the design of C# 8.

[svick]

@verelpode

But you guys wanted language features super strongly

I believe people asked about language features, because discussing language features is the purpose of this repository and so they assumed that if you want to discuss something, it's a language feature.

I understand that you don't like Ranges. That's unfortunate, but no feature can satisfy everyone. You also think that Advanced Extents & Segments would be valuable, but it seems you don't think it has to be a language feature. But doesn't that mean you can create what you want as an API and ignore Ranges in your own code? Isn't that win-win, or at least as close to win-win we can get when people have different opinions?

[HaloFour]

@verelpode

You also explicitly stated that this proposal would be incompatible with ranges, so I assumed that you had an alternate syntax in mind. But I don't see that as being the case here. In fact, I think that these APIs would benefit from ranges, especially where it comes to calculating extents.

[verelpode]

@svick

it seems you don't think it has to be a language feature. But doesn't that mean you can create what you want as an API and ignore Ranges in your own code?

Maybe but it's too early to determine that. Some parts of my proposal may well be justifiable as language features, either in their current form or in a tweaked form.

System.Range can be used as an API instead of a language feature. It may well function better as an API. For example, Range.All is better than writing 0..^0 and Index.End is better than ^0 in my opinion. It is important for every new language feature to begin at minus 100 points and then work its way up. Somehow this minus 100 points procedure has been circumvented or bypassed for the Range feature and instead it has become an emotionally-charged debate. I also feel concerned about the possibility of fanaticism/fanboyism creeping into the decision-making process. I don't know how this happened, but it would be sensible to wait until the emotions subside. Defer the feature to C# 8.1 and then try to make a cold logical non-emotional analysis of the pros and cons of the proposal and the alternative proposals.

A large amount of evidence to justify the value of the language change is needed. In particular, multiple real-world practical non-academic examples that demonstrate that the new syntax is significantly superior than the previous way of writing the same thing. Credible justifications of why the Range feature should be a syntax and not an API. Multiple real examples demonstrating practical advantages in real-life scenarios, not emotions, and not sloppy examples like the broken "MovingAverage" example that was provided.

[CyrusNajmabadi]

I think you might not have realized that Microsoft and the C# team do not owe you anything in return for your unpaid participation in this project.

Here's the thing: I never have expected otherwise. I even said that explicitly. After I explained what you could expect from a proposal, I also said: "The same applies for me."

Feel free to take it to gitter if you want to converse more.

[CyrusNajmabadi]

I want to read critique of my ideas

Please read my posts. There were several critiques posted

[CyrusNajmabadi]

A lot of your comments are fundamentally on the topic of yourself (the comments where you describe your own behavior),

I was one of the people who designed the range/slice feature. I'm giving you insight into the thinking and process that went through it from my direct experience there.

I'm explaining what it takes to make a feature a reality to help you understand what is necessary on your side to do the same.

[CyrusNajmabadi]

FYI: Roslyn was named after https://en.m.wikipedia.org/wiki/Roslyn,_Washington

A town outside seattle (where MS is based)

[CyrusNajmabadi]

It would be helpful for every participant if you could please make an effort to keep your emotional attachment out of the design process and remain analytical, non-emotional, logical, and practical.

Sure. Feel free to respond to the direct critiques I've made of the proposal. There are several points that haven't been responded to.

That said, I would highly highly highly recommend taking this to gitter. It's just a better medium for hashing out ideas like this.

[verelpode]

V1 of my proposal attempted to support the reverse-1-based ordinals. This feature is against my own opinion, but I'm doing it anyway out of respect for the opinions of some other people who claim it is a justifiable feature. The question is, how can the following problem be eliminated: Apparently the current design of ^0 will trigger never-ending complaints over years or decades because of the inconsistent introduction of 1-based ordinals into a language that was forever 0-based.

How can this problem be eliminated? One possibility is to design it with the principle that subtraction means subtraction. It's hard to complain about a design where subtraction means subtraction. People cannot complain that it's confusing when the principle is so simple that subtraction means subtraction.

If the syntax [5] produces an instance of System.Extent or System.Range, and if the Extent or Range struct includes a normal C# operator-overload-method for the arity-1 minus operator (meaning static Extent operator - (Extent) { ... }), then the syntax -[5] invokes the operator-overload-method in the standard manner, which returns an Extent or Range that represents the equivalent of Array.Length - 5.

The upcoming decades of never-ending complaints are prevented because it's not confusing anymore when the - operator means -.

So that's one problem hopefully eliminated. Next, the second problem needs to be eliminated. The second problem is what @lostmsu mentioned: The majority of the reason for this feature to exist is academic scenarios. The added complexity and performance penalty of supporting negative ordinals may well outweigh the benefit in real-world scenarios, if any meaningful amount of benefit exists at all. A method of addressing this problem is to collect a good quantity of real examples from real software and then perform an analysis of the cost versus benefit in each example when it is converted to use negative ordinals.

[CyrusNajmabadi]

The question is, how can the following problem be eliminated: Apparently the current design of ^0 will trigger never-ending complaints over years or decades because of the inconsistent introduction of 1-based ordinals into a language that was forever 0-based.

It's your opinion that this will be a problem. This is not an opinion shared by the LDM.

In my personal opinion i don't see an issue here. People will learn how indexing will work, and will move on. And that's that. So there's no issue to solve, and no need to 'eliminate' anything.

[CyrusNajmabadi]

Next, the second problem needs to be eliminated. The second problem is what @lostmsu mentioned: The majority of the reason for this feature to exist is academic scenarios. The added complexity and performance penalty of supporting negative ordinals may well outweigh the benefit in real-world scenarios,

It doesn't appear as if that's the case. So the "may well" part is not an issue.

if any meaningful amount of benefit exists at all. A method of addressing this problem is to collect a good quantity of real examples from real software and then perform an analysis of the cost versus benefit in each example when it is converted to use negative ordinals.

This has been done. It was used to help drive and refine the feature. CoreFx also helped here which led to some later tweaks on the impl.

[CyrusNajmabadi]

Apparently the current design of ^0 will trigger never-ending complaints over years or decades because of the inconsistent introduction of 1-based ordinals into a language that was forever 0-based.

Note: we examined the broad ecosystem of languages that support features like this. We couldn't find any sort of similar sentiment to the one you're espousing for them. Their communities were able to pick up and understand how this sort of indexing works, and there's no indication from preliminary usage that the C# ecosystem is different here. Your concern, while well-meaning, seems speculative and doesn't match the reality of the greater programming community.

[jnm2]

@verelpode

As jmn2 said, this is tiring and absorbs a lot of time and energy,

Please don't take my name in vain. That is not what I was talking about. I was referring to your desire to redesign a language feature that was mostly out the door without taking into account the years of study that went into it.

[jnm2]

Maybe the name "Roslyn" was a mistake because it's a female name and thus makes it more difficult for males to remain non-emotional

This is horrifying. Where are you coming up with this stuff?

[jnm2]

@verelpode That was a really low-effort and sucky thing for me to say. I'm leaving it here just to take responsibility for it, but I was only thinking about myself. I'm sorry.

My impression is that you could get a lot of mileage in these conversations by framing your points in a way that makes it easy for folks to want to agree with you, but it totally makes sense if I've lost the right for you to listen to me with my last comment.

[verelpode]

In connection with the academic nature problem, here's one way of eliminating the problem of the dubious at-runtime complexity and performance penalty: Support this usage of negative ordinals:

char[] dataArray = ...;
Span<char> slice = dataArray[20 .. -[5]];

...but don't support negative ordinals in this case:

Range r  = [20 .. -[5]];
Extent x = [20 .. -[5]];

...because the first example can be implemented with zero penalty at runtime whereas the second example cannot. In terms of a cost-vs-benefit analysis, the second example is much more difficult to justify than the first example where the total length is known. This is a matter of finding a reasonable balance or compromise without springing to either extreme. This isn't a black and white issue.

I'm searching for ways to eliminate the above-mentioned problem because, as a general policy, I always try to eliminate a problem instead of burying my head in the sand and pretending that the problem doesn't exist. On a few occasions in the past, I've used self-delusion to make myself feel better, but it only worked short-term and in the long run I felt overall worse, so I try not to do that anymore. (This is a 100% truthful description of my experience in general and it is not directed at any particular participant here. Also note my comments can benefit "invisible" readers here who read this thread but haven't posted any messages.)

[verelpode]

@jnm2

My impression is that you could get a lot of mileage in these conversations by framing your points in a way that makes it easy for folks to want to agree with you

Should I desire to have that mileage in this case? Your situation is different than mine. When clicking on your name, your profile says ".NET Foundation member" whereas my profile deliberately does not name my employer. Furthermore, I'm not a Microsoft staff member paid to work on C# 8. I'm paid to work on a different project. I don't have the onus nor responsibility to solve problems in C# 8 nor to prove any of my ideas for C# 8.

@CyrusNajmabadi said in reply to me "The onus is on you", but that's incorrect. The applicable Microsoft staff members have the onus and responsibility. The staff who are paid to work on a project have the onus and responsibility to ensure that the problems in that project are addressed and eliminated, and that customer feedback is analyzed for valid and useful points. My onus and responsibility is to eliminate the problems in the project that I'm paid to complete, and that project is not C# 8. I also have a responsibility to make an extra effort for customers of my employer but nobody here is a customer of my employer.

@CyrusNajmabadi said "you've stated you want this very large feature", but I never stated that. My message clearly states: "This is a rough sketch and I've been forced to rush it, thus I'm sure it contains some mistakes and omissions"

It doesn't make sense to say that I want something that contains mistakes and omissions. It's also not my responsibility to fix all of the mistakes and omissions.

[svick]

@verelpode

You can become a member of the .NET Foundation too.

I don't have the onus nor responsibility to solve problems in C# 8 nor to prove any of my ideas for C# 8.

Nobody says that it's your job to fix C#. But it seems it's your opinion that the proposed version of C# 8.0 is problematic. And if you want that to change, then it's up to you to make your case.

[verelpode]

@svick Thanks for the link. By the way, when I said "academic", I meant the general sense, not specifically people who work in academic institutions. Anyone may produce an academic-only design. I don't work in an academic institution but nevertheless I've produced academic-only designs on few occasions and I had to go back and produce a new version/revision of the design in order to make it suitable for use in a real situation, but that wasn't bad news because I wasn't forced to throw the whole design in the garbage bin, rather I just revised it and fixed the problems.

I've also personally known a few people who DO work in academic institutions and who have impressive two-way connections between academic and practical applications, and impressive bidirectional knowledge transfer between teacher and students -- not only the teacher-to-student direction, but also student-to-teacher whenever possible.

if you want that to change, then it's up to you to make your case.

It's not as simple as that. Projects can be terribly bogged down by the people-politics and human psychology. Some projects are not technically difficult but are delayed for many years because of human psychology. It would be much simpler if criticism was only criticism, but sometimes humans state a criticism even when they don't believe it themselves! It can be difficult to distinguish between a genuine critique versus an ulterior motive or knee-jerk reaction triggered by an emotion. Fear is also a common problem -- sometimes humans imagine that they or their position is being threatened even when it isn't, and this can trigger harassment/mobbing or an attack against the imagined enemy who is no enemy in reality.

[verelpode]

@svick That webpage is unfortunately vague and short on details. Please don't be offended by the following question because I don't believe it's true. In fact, the following sounds like a conspiracy theory, if you ask me. But anyway, I'm curious to hear your response so here it is despite my skepticism. Here's what I heard: Large corporations in the stockmarket are legally required/obligated by the current laws to maximize profits for shareholders. Software engineers complete more work in less time when they work overtime, thus higher profits, but it's not legal to ask any employee to work more than 40 hours per week. To work around this law, software engineers can be encouraged to become a member of a non-profit organization, and then they usually work additional hours during evenings and weekends. In total they work much more than 40 hours per week for the same salary as 40 hours. I'm reluctant to repeat everything that I've heard but now my curiosity has gotten the better of me, so what are your thoughts about the inappropriate and disagreeable claim that some open-source organizations are "for suckers" and "naive young software engineers" who don't realize they're being manipulated into working much harder and longer for much less salary than they're entitled to, in order to maximize profits for the shareholders of the large stockmarket-based corporations.

I know, I know, crackpot conspiracy theory, but you work in an academic institution therefore I guess you're the same as me in viewing it as intellectually stimulating to discuss the purely-theoretical merits and flaws of a theory even when it's a wild crackpot theory.

I've also attended debating clubs at universities where sometimes a controversial crackpot theory is deliberately chosen as the topic of the debate, and it does produce an entertaining intellectual debate for the club members to enjoy.

[jnm2]

Should I desire to have that mileage in this case? [...] I don't have the onus nor responsibility to solve problems in C# 8 nor to prove any of my ideas for C# 8.

Why should you not want to benefit yourself or others by this conversation? That's all "mileage" meant. There's not much purpose in having a conversation if you don't want to be heard, and if you do want to be heard, it just makes sense to choose to say things in such a way that people will want to converse with you and consider what you have to say. I don't think this is controversial.

Your situation is different than mine. When clicking on your name, your profile says ".NET Foundation member" whereas my profile deliberately does not name my employer. Furthermore, I'm not a Microsoft staff member paid to work on C# 8. I'm paid to work on a different project.

My profile does not name my company either. I'm a part owner of a local business. The link @svick supplied is a good explanation of what a .NET Foundation membership is.

Just to be frank, it's super weird that you're doing things like looking up our profiles, coming to conclusions, and expounding on those conclusions at length in this conversation. For example, the whole "academic" section above. This is not normal or benefiting the topic.

[svick]

@verelpode

Large corporations in the stockmarket are legally required/obligated by the current laws to maximize profits for shareholders.

They're not.

Software engineers complete more work in less time when they work overtime

They don't.

To work around this law, software engineers can be encouraged to become a member of a non-profit organization, and then they usually work additional hours during evenings and weekends.

I don't see how that makes sense. You do need to contribute to a .NET Foundation project to become a member, but that can be just a one-off thing and contributions during work hours count too. And once you are a member, your only obligation is to pay membership dues, and even that is optional.

People who contribute to open source projects in their free time do it because they want to, not because they joined a foundation and now feel they have to.

what are your thoughts about the inappropriate and disagreeable claim that some open-source organizations are "for suckers" and "naive young software engineers" who don't realize they're being manipulated into working much harder and longer for much less salary than they're entitled to, in order to maximize profits for the shareholders of the large stockmarket-based corporations

Personally, I do think that compensation for open source work is a problem in the industry. But that applies to all open source projects with volunteer contributors, I don't see how manipulation or profits enter the picture.

Though all of this off-topic here. If you want to discuss these kinds of theories, I suggest you choose another venue.

[verelpode]

Example of a technique in a debating club. The following example isn't true -- please don't flip out -- this is nothing more than an theoretical example of the intellectual fun at a club.

@HaloFour wrote:

as a previous member and as continuing contributor they do have a great deal of respect for his input. Infinitely moreso than yours.

So in a debating club, here's how we respond if the opposing team says something like the above comment. We reply by saying that actually they don't respect him, rather they're using or exploiting him in order to advance their own career. He believes they respect him but actually they merely view him as a useful utility.

THAT'S NOT REAL. That's just a debating club example.

[msedi]

I'm just listening to this, and we also had a lot of discussions in our team about the range feature - to be honest, no one liked it, although everyone liked that there is finally something that can describe ranges and our team has a lot of different experiences from other languages (MATLAB, Maple, Mathematica, Python, R, etc.).

We had already implemented this features years ago, without the help of C# for our matrix classes and other classes that needed range support. The solution was quite simple, we created an Indexer struct (in your words Range) and each class that should be capable using this extended property indexer needed to implement this[Indexer idx] {get; set;}. This worked just fine except that we didn't have language support during parsing the code. I guess this is the main feature (not really the Range itself) that the compiler is able to resolve the Range notation and create a Range instance for you (a code rewrite at compile that I would like to see more often, since we have more of these things, e.g. physical units).

Without the help of C# we had to write the indexing elements in string representation, so for example

Indexer i = "10:20"

translated to an Indexer that took all elements between 10 and 20 (including both), as this is the case with Range (Start=10; End=20).

Since we are coming from the MATLAB world, we took start and end as additional parameters to be able to describe the extents of the range without having to know the real start and end index. So using

Indexer i = "start:end"

simply takes the whole range of the object using the Indexer.

We also made it possible to have offsets from the start and end, including some math operations on it.

Indexer i = "start+10:end-10"

We could also produce Indexer ranges with a userdefined increment:

Indexer i = "10:2:20" // take every second element from 10 to 20
Additionally we could produce Indexer arrays:

Indexer i = "10,11,12-14,18-end"

Also it worked for multidimensional indexing;

Indexer i = "10,11,12-14,18-end;1-10"

Everything works really fine, except, as mentioned above that we don't have compiler support to resolve it.

In the end, I think the Range is useful for very primitive scenarios, as for example String.Substring() or some other methods that require simple ranges.

For more complex cases I assume it's not so useful.

I know this is a bit off this topic now and a separate request: If roslyn would support code rewrites people could create their own Ranges, Indexer or whatever.

[verelpode]

@svick

once you are a member, your only obligation is to pay membership dues

So, I give my work to the foundation and I have to pay the foundation (it's a small $ amount to pay but nevertheless I pay the foundation something). Shouldn't it be the reverse? Shouldn't the foundation pay me for my long hard work in developing stuff for the foundation? Stuff for corporations to use and profit from?

[verelpode]

@svick
Aha, that info was interesting! So a couple of points in the theory are inaccurate. The theory said that the profit maximization is forced by law, but Cornell Uni says it's not law, rather it's the directors. The corporate directors decide to maximize profits because of pressure from shareholders etc.

Usually maximizing shareholder value is not a legal obligation, but the product of the pressure that activist shareholders, stock-based compensation schemes and financial markets impose on corporate directors.

The second inaccuracy is that overtime increases profits. It probably doesn't, but the corporate directors may want overtime anyway because they mistakenly believe it increases profits, and refuse to believe otherwise.

[verelpode]

@jnm2

There's not much purpose in having a conversation if you don't want to be heard

True.... I thought hard about what you said and in conclusion I think you're right. There's not much purpose in me continuing to participate here, especially because zero Microsoft employees are participating in the discussion. So it seems like the sensible and logical thing to do here is to end my participation. It also appears that there's no logical reason for me to join the foundation.

[jnm2]

@verelpode None of those are things I said. What I actually said (a few times now) is very much more difficult than leaving: it's only worthwhile to present your ideas in an appealing way.

[CyrusNajmabadi]

True.... I thought hard about what you said and in conclusion I think you're right. There's not much purpose in me continuing to participate here, especially because zero Microsoft employees are participating in the discussion.

This has not been a problem for many others in the past. Perhaps you are jumping to conclusions on how things with based on minimal participation while disregarding the advice and experience of many here we have been involved a lot in Roslyn over the last decade. I would strongly suggest trying to approach things more from a perspective that there could bring positive outcomes here and that people here are trying to help you get to that point.

You're welcome to leave if you want, but you'll only be going against your item professed still interests.

[CyrusNajmabadi]

@CyrusNajmabadi said in reply to me "The onus is on you", but that's incorrect.

No, it is correct. You are proposing the feature. The onus is on you entirely to get it to a state that it could be accepted by teh LDM. You can hope that others will help you take up the mantle, but no one is obligated at all to do so. It's entirely on you how you put it out and how you engage in the discussion around it.

Many people (including several conversing with you) have been successful in the past doing this sort of thing. And many here are giving you strong and sound advice on what you can do to be more successful. You can either listen and attempt to adopt that advice, or you can choose not to. But the outcomes of that choice are basically all on you.

The applicable Microsoft staff members have the onus and responsibility.

No. They don't. This is simply false.

The staff who are paid to work on a project have the onus and responsibility to ensure that the problems in that project are addressed and eliminated,

The staff believes the existing feature has been satisfactorily designed and implemented to their standards.

and that customer feedback is analyzed for valid and useful points.

As all your points were raised and addressed during the design portion of this feature, the staff believes that that has already been done.

@CyrusNajmabadi said "you've stated you want this very large feature", but I never stated that.

I presumed that by stating you wanted a different approach to ranges in the previous thread, that you wanted this. And i presumed that by opening this issue and outlining this large feature that that was tacitly stating you wanted it. If you don't want it, i apologze for the assumption. Nearly every other proposal ever made has been by someone who actually wants it done, so it seemed reasonable to assume the same of yours.

It's also not my responsibility to fix all of the mistakes and omissions.

It is if you want this proposal to go anywhere. Unless you can convince someone else to help you, then the onus is entirely on you to get the issue to the state that it could be meaningful.

And, if you're not going to work positively with others to get them to help you, and you're not going to put in the effort yourself, the expected outcome woudl be that nothing would come of this.

[CyrusNajmabadi]

So it seems like the sensible and logical thing to do here is to end my participation

As mentioned many times over, there are better avenues for you to discuss this sort of thing. For example: gitter

There are long and fruitful discussions there about both C# and the Roslyn project. And many team members participate there. So far, like with much of the rest of the advice provided, you have ignored approaching this problem space in any sort of different way than the one way you prefer. You are under no obligation to change or try something different. But you're very likely to run into much more limited success here if you continue behaving in this same manner for all issues that bother you.