Parallel for in the calculation of MaxWidth

[pyscripter]

@MShark67
I have introduced a parallel for loop (from System.Threading) in the calculation of MaxWidth. Could you please test it with you 250K file and check whether it makes a difference in performance?
Thanks!

[MShark67]

Very cool! I'll do some benchmarking tomorrow. Great job on those other fixes.

[DGH2112]

As an observation. I updated yesterday to include the fix for wide files however my very wide and long files still took minutes to load and the interface was blocked during this time. I saw the parallel MaxWidth work in the tree so assume this should be working. If I set a highlighter after opening the file all the time is spent in MaxWidth.

[DGH2112]

I tried a few things yesterday in turns of loading with and without highlighters - it seems to be the highlighters causing the slowness. In the app at the moment, the file is opened without a highlighter and the file is loaded within a second however when a general highlighter is assigned it then takes this time calcing the MaxWidth. Yes, I'm using the latest as I pulled the 65536 character wide fixed you made yesterday. It is very slow to scroll vertically through that "wide" area with each page taking 30 seconds at a time (PgDn). I used SynEdit\SynEdit before which had a fixed width so the file loaded relatively quickly.

[pyscripter]

I am a bit confused. Once the MaxWidth is calculated it should not affect speed any further. And as it is now it is calculated irrespective of whether there is a highlighter or not.

Assigning a highlighter should affect things initially as the text is parsed from start to end. It also may affect painting time if you have such long lines. After you assign the highlighter is editing and moving around much slower that with the old SynEdit?

Compared to the SynEdit/SynEdit, once you give up the notion that every character has the same fixed width (Chinese, emojis etc.) even character positioning requires to calculate a DirectWrite layout for the current line. It seems that if the line is very long that may be expensive.

[pyscripter]

Yes, I'm using the latest as I pulled the 65536 character wide fixed you made yesterday.

I have made more changes after that fix. Please make sure you have the latest.

[DGH2112]

I'll test again later in the week (the day job is not coding).

[pyscripter]

It is worth also checking the same with WordWrap.

[MShark67]

Testing done; results spectacular. Testing on an AMD mini pc with a mobile 4 core (Ryzen 7 3750H), Windows 11, Delphi 11, 32bit and 64bit. I see the new routine being roughly 4 times faster in every test I tried. Here are the logs. New and Old are the new routine vs the pre-parallel routine. Res is just the maxwidth calculated. The first pair is a 36 meg on disk script file with 3 hundred thousand lines of the same string (I wanted to do a test where all the lines were the same length to see what affect that might have). The second pair is the 250K bracketed SQL file I've mentioned before. The third par is a largish .pas file. I haven't tested "long line" files mentioned above. I'll give that a try and see what the difference is. [Update- These tests are without any syntax highlighting. Single line testing shows almost the exact same timings between the parallel and non-parallel version, which makes sense.] So far, I'd say this change is a win!

32bit Log: vvvvvvvvvvvvvvvvvvvvvv
New: 00:00:01.0337885 Res: 1089
Old: 00:00:03.9810809 Res: 1089
New: 00:00:00.4428788 Res: 531
Old: 00:00:01.6764738 Res: 531
New: 00:00:00.0509227 Res: 2493
Old: 00:00:00.1728124 Res: 2493

64bit Log: vvvvvvvvvvvvvvvvvvvvvv
New: 00:00:00.8369260 Res: 1089
Old: 00:00:03.1794440 Res: 1089
New: 00:00:00.3732312 Res: 531
Old: 00:00:01.3233780 Res: 531
New: 00:00:00.0435875 Res: 2493
Old: 00:00:00.1571905 Res: 2493

[MShark67]

I created a test file that looks like:
Line 1: 500,000 chars (content is just repeated sql statements)
Line 2: 250,000 chars
Line 3: 125,000 chars
Line 4: 75,000 chars
Line 5: 25,000 chars
Line 6 through 20: 10,000 chars each

File loads very quickly (less than a second.) Moving the caret through the characters is very interesting. Left and right arrowing at the top line shows about a second delay between each move. Careting down gets faster with every line, when on the last lines careting is back to normal. My use case for working with super wide files is usually opening a file, realizing it's just a single long line and doing a search and replace all of some delimiter with that delimiter and a CRLF to make the file normal. This is not in debug mode btw.

[MShark67]

It would be good to know what was the change that fixed the error, since I am not sure I did much related to line numbers.

I can't say exactly what helped the issue as I never fully diagnosed it. It would only happen while either loading a very large file (400 lines, 100K chars each) and at the point where the gutter line numbers would be painted I'd get an "invalid parameter" exception on the line:

  CheckOSError(TSynDWrite.ImagingFactory.CreateBitmap(Width, Height,
    @GUID_WICPixelFormat32bppPBGRA, WICBitmapCacheOnDemand, FWicBitmap));

However, it would only happen at run time (in both 32bit and 64bit) and not while debugging (though I tried a number of different settings to see if it changed anything, including temporarily removing the threading (just in case.)) I don't know what the invalid parameter was, maybe a negative width or height? I got the above info using a run time stack tracer.

Also testing performance on big files (and long lines) with WordWrap would be useful to understand the limits of SynEdit.

Agreed! I'll be looking at this as I test in general. I did have a (probably sophomoric) question about the HitTestTextPosition usage. It looks like in RowColumnToPixels it always does the layout with the entire line (which is needed for the widthIncludingTrailingWhitespace calculation, but could it use a leading portion of the line if it's doing the hittest part? This is really just a random thought as I'm not sure how much of that works.

[pyscripter]

It looks like in RowColumnToPixels it always does the layout with the entire line (which is needed for the widthIncludingTrailingWhitespace calculation, but could it use a leading portion of the line if it's doing the hittest part?

Yes, this is an optimization I had in mind for long lines. It is not that straightforward, because you cannot assume char width is fixed. But if we end up restricting the line length then it is not that important.

[MShark67]

In my large file testing I'm still seeing that error I mentioned above related to TSynDWrite.ImagingFactory.CreateBitmap, however I've managed to find out what the value is.. it's being called with a height of zero. It only ever happens in release mode, never in debug mode, and only when loading a very large file, pasting a block of long lines, or switching word wrap on the same large file. Any thoughts? It's really hard to track down unfortunately.

[pyscripter]

In TSynGutterBand.PaintLineNumbers put at the top

if ClipR.IsEmpty then Exit;
or
Assert(not ClipR.IsEmpty);

Is the assertion failing or do you still get an error with the first statement?

[MShark67]

It looks like I was adding a new issue at the same time you pretty much already diagnosed it. When the problem happens the clipping region is empty. It's hard to replicate as it has something to do with timing. I do think that your suggestion will fix it. I'll report back.

[pyscripter]

@DGH2112 @MShark67
I am open to suggestions as to how to handle files which are very large and/or have very long lines.

[pyscripter]

Created #161 and provided a VS code like solution. Please test

[MShark67]

This works well. I notice that in cases like this (a large wide file) that due to the time to perform the wrap, you can't really resize the window. You also get a double wrap sometimes since showing scrollbars causes a window resize which re-wraps. I'm not sure what the solution is (if any), maybe delay updating wrapping until the resize is complete? I think these are fairly low priority issues, I never work with files like this in real life, and when I do, I quickly reformat them to be something more manageable.

[MShark67]

I wonder if a better idea than automatically turning on wordwrap would be to use the eoWrapWithRightEdge feature instead. This would have the benefit of wrapping just once and still allow the control to be resized and fairly responsive on a large wide file. Another alternative would be to do nothing at all (I've gone back and forth on it lol.)

[pyscripter]

wonder if a better idea than automatically turning on wordwrap would be to use the eoWrapWithRightEdge feature instead.

It is a good idea. Implemented.

Have a look at https://en.delphipraxis.net/topic/6277-synedit-just-got-a-major-uplift/?tab=comments#comment-54651
Do you have any such experience? (I do not use sql)

[pyscripter]

@MShark67
I have committed a new version of DrawIndentGuides that draws lines directly to the Render target, but draws connected segments as a single line. So overall it draws much fewer but longer lines.

I wonder whether you can compare the performance of this vs the previous version. Many thanks!

[MShark67]

I'm testing the Indent Guides using VirtualTrees.pas (it's long and has a fairly high indent level) and without a highlighter. When I set the line style to igsSolid, I don't see any lag or stuttering at all. I can grab that scroll thumb and zip up and down and it keeps right up. If I use igsDotted it lags and becomes choppy; it's a very noticeable difference. I prefer igsSolid btw. The lines themselves look really good!

[pyscripter]

Major commit.

• Now all painting is with Direct2d/DirectWrite
• Reverted DrawIndentGuides "optimization".
• Remove "fix" of Super long lines slow down SynEdit #161 (Fix Long lines fix #166) by popular demand.
• Rationalized default options (removed eoScrollPastEOL).