Support "regex" filters

[ansis]

We've talked about this before but it was never implemented. How would it be specified?

Carto uses =~. We could also use regex.

[divya1c]

Hi! is the regex feature added to mapbox gl yet?

[tmcw]

If the issue is open, the task isn't done yet.

[1ec5]

regex (or like) would be more discoverable/memorable, since ~ has wildly different meanings in every language.

[tmcw]

To unpack what's necessary to implement this feature:

1. Naming - regex or ~
2. Compatibility across GL JS and GL Native

1 will likely be 10% of the work or less. Problem number 2 is more complex.

If the JavaScript port (GL JS) uses JavaScript's built-in RegExp object, then GL Native will need to include a compatible implementation of regular expressions in order to ensure that maps render exactly the same in a native environment. There are many particular flavors of regular expressions, so picking a feature-filled Native regular expression engine would mean that styles break in less feature-filled GL JS RegExp implementations.

There's also the question of whether GL Native should use platform-provided regular expression libraries, or bring its own on the C++ level. From diving into my handy V8 source checkout (get your own today! they're great), V8's implementation is Irregexp).

[1ec5]

@jfirebaugh points out that we can use std::regex’s ECMAScript regex support in gl-native to ensure compatibility between GL JS and the native SDKs. However, note that std::regex supports some features that browsers don’t, like [:alnum:].

In any case, I would be very much prefer that we use the platform-provided regex facilities in gl-native, similar to how we use the platform-provided facilities for uppercasing and lowercasing strings. (This does lead to minor discrepancies among the platforms: mapbox/DEPRECATED-mapbox-gl#21 (comment).) Using the platform-provided regex facilities means we don’t incur an increase in the SDK’s size, and it ensures that the runtime styling API is compatible with any other regex the developer uses in their application.

To illustrate my point, style specification filters are represented on iOS and macOS as NSPredicate objects. This is as natural as representing strings as NSString objects. NSPredicate format strings accept a SQL-like syntax, where the MATCHES operator is documented to support ICU regex syntax. However, if core code only supports ECMAScript syntax, then the iOS and macOS SDKs need to transform ICU regex to EMCAScript regex, rejecting any regex that doesn’t translate, and transform in the other direction as well when getting the predicate of a style layer. Otherwise, without this SDK-level transformation, the SDK’s behavior would be perceived as a bug.

I recognize that bringing ICU regex to GL JS would be a challenging task, and that some Studio users could expect ECMAScript regex since the live preview is implemented using GL JS. Fortunately, there’s enough overlap between the two syntaxes that I think we should declare a common subset of ICU and ECMAScript to be the syntax we want to support for filters; anything else (like lookbehind or Unicode properties in ICU, or matching Turkish İ with [A-Z] in ECMAScript) comes with a caveat that it isn’t guaranteed to work across platforms. I think most Studio users would come to Studio without knowing that there are different regex syntaxes, and they’re just as likely to try an ICU or PCRE regex as they are to try an ECMAScript regex.

[1ec5]

In chat, @tmcw brought up a valid concern that a user might input a regex for a filter that appears to do the right thing in GL JS, but it happens to prevent the layer from showing up at all on iOS. There’s no substitute for testing, but I agree that we should aim to make the live preview in Studio as faithful to the rendered output as possible.

The thing is, I think the subset of features in ICU but not ECMAScript is pretty small, and the subset in ICU but not ECMAScript even smaller. If Studio could detect the use of these features and display a warning icon, that would essentially enforce the subset of regex that we do officially support.

[lucaswoj]

This issue was moved to mapbox/mapbox-gl-js#4089