Syntax highlighting breaks after save

[nsommer]

Since using this plugin with TextMate version 2.0-rc.4 the syntax highlighting frequently breaks after saving a file. By breaking I mean the whole file or a part of the file is only displayed in black/white text. An example is the screenshot of a ruby file below after pressing CMD+S. Syntax highlighting is missing in lines 31-49. Before saving it (and therefore before applying the editorconfig rules), the syntax highlighting was just fine. When I insert some key and save again, it is displayed correctly again.

This is my .editorconfig file:

[*]
charset = utf-8
end_of_line = lf

[*.{rb,yml,html,js,css,scss,erb}]
indent_style = space
indent_size = 2
insert_final_newline = true
trim_trailing_whitespace = true

Is there any information I could provide to help finding the cause of this?

[Mr0grog]

Ugh, that looks terrible. I’ll take a look at this as soon as I have time and am able, but I’m not sure if that will be any sooner than later this week.

It’s a good bet that this is due to updates to the document just prior to saving (https://github.com/Mr0grog/editorconfig-textmate/blob/master/source/additions/NSView%2BEditorConfig.m#L145-L147). That’s caused by the insert_final_newline and trim_trailing_whitespace settings. If it’s the problem, removing those or setting them to false should resolve the issue in the short term. I know that’s not a great solution, though :\

[Mr0grog]

Need to see if I can create my own std::multimap<std::pair<size_t, size_t>, std::string> here and pass it to OakDocument -performReplacements:replacements checksum:crc32 (where the checksum isn’t checked for a loaded doc) or maybe directly to OakDocumentEditor -performReplacements:replacements.

[Mr0grog]

Oh, actually, I might be able to just leverage OakDocument -matchesForString:options:, since its find results include a CRC32. (See values of find::options_t at https://github.com/textmate/textmate/blob/fb929aee10c271c1a5b87ddf1ff5e172bc7a8396/Frameworks/regexp/src/find.h#L8-L22)

See how TM does this internally: https://github.com/textmate/textmate/blob/52fda481dc9ee52fcf71a1134b900b61bc8f839d/Frameworks/Find/src/Find.mm#L242-L277

[Mr0grog]

Existing code that needs to utilize the above stuff is at

editorconfig-textmate/source/additions/NSView+EditorConfig.m

Lines 76 to 143 in b305927

	if (settings.trimTrailingWhitespace) {
	NSMutableString *newContent = [NSMutableString string];
	// Selections are an NSArray of NSValues boxing NSRanges
	// FIXME: this approach collapses column selections into a single contiguous range. Not sure on a better approach for now.
	// http://lists.macromates.com/textmate/2017-January/040185.html
	selections = [self accessibilityAttributeValue:NSAccessibilitySelectedTextRangesAttribute];
	NSCharacterSet *whitespace = [NSCharacterSet whitespaceCharacterSet];
	NSCharacterSet *newline = [NSCharacterSet newlineCharacterSet];
	NSScanner *scanner = [NSScanner scannerWithString:content];
	// By default scanners skip newlines and whitespace -- exactly what we're scaning for :P
	scanner.charactersToBeSkipped = nil;
	NSString *line;
	NSUInteger currentIndex = 0;
	while (!scanner.isAtEnd) {
	if ([scanner scanUpToCharactersFromSet:newline intoString:&line]) {
	NSUInteger i = 0;
	for (i = line.length; i > 0; i--) {
	if (![whitespace characterIsMember:[line characterAtIndex:i - 1]]) {
	break;
	}
	}
	[newContent appendString:[line substringToIndex:i]];
	NSUInteger removedOnThisLine = line.length - i;
	currentIndex += i;
	if (removedOnThisLine > 0) {
	didChangeContent = YES;
	for (NSUInteger selectionIndex = 0; selectionIndex < selections.count; selectionIndex++) {
	NSRange selection = [[selections objectAtIndex:selectionIndex] rangeValue];
	NSUInteger selectionEnd = NSMaxRange(selection);
	if (selectionEnd > currentIndex) {
	NSUInteger newLocation = selection.location;
	if (selection.location >= currentIndex + removedOnThisLine) {
	newLocation -= removedOnThisLine;
	}
	else if (selection.location > currentIndex) {
	newLocation = currentIndex;
	}
	if (selectionEnd >= currentIndex + removedOnThisLine) {
	selectionEnd -= removedOnThisLine;
	}
	else {
	selectionEnd = currentIndex;
	}
	selection = NSMakeRange(newLocation, selectionEnd - newLocation);
	[selections replaceObjectAtIndex:selectionIndex withObject:[NSValue valueWithRange:selection]];
	didUpdateSelection = YES;
	}
	}
	}
	}
	if ([scanner scanCharactersFromSet:newline intoString:&line]) {
	[newContent appendString:line];
	currentIndex += line.length;
	}
	}
	content = newContent;
	}

[hovsater]

@Mr0grog did you manage to progress on this? Anything I can do to help?

[Mr0grog]

Hey @KevinSjoberg, I’ve been super busy lately and have not had time to get to it :(

If you are interested in trying out the approach outlined in the comments above, that would be wonderful! I’d certainly be happy to take a PR.

[Mr0grog]

From an old e-mail list exchange with Alan Odgaard:

The most abstract way to change the document content (from the outside) is via search and replace, although this is easier done from a macro than a plug-in, so you might have to basically execute a macro to get this done.

So possibly the ideal approach is to see if there's a way to register and trigger a macro from ObjC code.

[Mr0grog]

Tripped over this issue again today after a long while. I still need to get around to #45 first, and I don’t think I’ll be able to address this in the next couple weeks, but I did go down a small rabbit-hole looking into possible fixes here.

I think there are 4 main options:

1. OakTextView (where we are catching the documentWillSave notification and running the code noted above) has a (void) insertText:replacementRange: method that is ObjC-friendly and looks like it should let us replace text ranges pretty easily. Using that instead of straight-up replacing the document content should hopefully mean we don’t need to manage selections and should also keep the syntax highlighting intact. This is probably the most straightforward fix, assuming it works.

2. OakTextView also has a (void) performFindOperation: method that can do a “replace all” operation, which would also do the trick. It takes an object implementing OakFindServerProtocol, which unfortunately involves two C++ types (find_operation_t and find::options_t), but they are both simple enums, so that might not be too big a deal. It’s messier, though.

3. I couldn’t find an obvious way to call an arbitrary macro, BUT OakTextView has an (IBAction) playScratchMacro: method that we could abuse. It’s designed to replay a macro you just recorded (either because you just did something you want to repeat a bunch, or you want to test it before saving). We could save a copy of the current scratch macro, replace it, call the method, then put the old scratch macro back:

   NSArray* scratchMacro = [NSUserDefaults.standardUserDefaults arrayForKey:@"OakMacroManagerScratchMacro"];
   
   // Replace the scratch macro and run it.
   
NSArray* macroToRun = (Make an NSArray with the macro contents);
   
[NSUserDefaults.standardUserDefaults setObject:[macroToRun copy] forKey:@"OakMacroManagerScratchMacro”];
   
macroToRun = nil;

   [textView playScratchMacro: self];

   
   // Reset everything.
   [NSUserDefaults.standardUserDefaults setObject:[scratchMacro copy] forKey:@"OakMacroManagerScratchMacro”];

   scratchMacro = nil;

4. Put a bundle with the relevant macros inside the plugin, then install it on startup if it isn’t already installed using the [BundlesManager sharedInstance]. Then you can run the OakTextView (void) performBundleItem: method. This is pretty complicated! It’s also not entirely clear how to get a pointer to the actual bundle item after installing (maybe by digging around in the bundles menu? Ugh. This also raises weird race conditions, like what happens if someone uninstalls the bundle before saving their document? Not great.

I think (1) is the most straightforward if it works, otherwise (3) is probably the next best. ((2) is pretty slick and feels better, perf-wise, than running all the macro machinery, but relies on more complicated internal enums that are probably also more prone to change, and therefore hard make compatible across TM versions.)

As far as the various macro solutions go, here are the two macros we’d want in text plist format (macros — at this level — are just plists in NSArray/NSDictionary form):

Insert final newline:

{
	commands = (
		{
			argument = {
				action = replaceAll;
				findString = "([^\\r\\n\\s])\\z";
				regularExpression = YES;
				replaceString = "$1\\n";
			};
			command = "findWithOptions:";
		},
	);
	name = "Insert Final Newline";
	uuid = "<UUID GOES HERE>";
}

Trim trailing spaces:

{
	commands = (
		{
			argument = {
				action = replaceAll;
				findString = "[\\t ]+$";
				regularExpression = YES;
				replaceString = "";
				wrapAround = YES;
			};
			command = "findWithOptions:";
		},
	);
	name = "Trim Trailing Spaces";
	uuid = "<UUID GOES HERE>";
}