Cannot build with PCRE2_CODE_UNIT_WIDTH=0

[tobil4sk]

Hi, I'm porting a code base which used to use pcre1. It used to compile in pcre_* files along with pcre16_* files as it can use both string encodings. The files are unified in prce2, so pcre2 says Use 8, 16, or 32; or 0 for a multi-width application.. However, compiling with PCRE2_CODE_UNIT_WIDTH=0 causes compilation errors:

 - /pcre2-10.42/src/pcre2_match_data.c 
Error: In file included from /pcre2-10.42/src/pcre2_internal.h:143,
                 from /pcre2-10.42/src/pcre2_maketables.c:52:
/pcre2-10.42/src/pcre2.h:826:53: error: unknown type name ‘pcre2_general_context_’; did you mean ‘pcre2_general_context_8’?
  826 | #define pcre2_general_context          PCRE2_SUFFIX(pcre2_general_context_)
      |                                                     ^~~~~~~~~~~~~~~~~~~~~~
/pcre2-10.42/src/pcre2.h:981:25: note: in definition of macro ‘PCRE2_SUFFIX’
  981 | #define PCRE2_SUFFIX(a) a
      |                         ^
/pcre2-10.42/src/pcre2_maketables.c:82:18: note: in expansion of macro ‘pcre2_general_context’
   82 | pcre2_maketables(pcre2_general_context *gcontext)
      |                  ^~~~~~~~~~~~~~~~~~~~~
/pcre2-10.42/src/pcre2.h:826:53: error: unknown type name ‘pcre2_general_context_’; did you mean ‘pcre2_general_context_8’?
  826 | #define pcre2_general_context          PCRE2_SUFFIX(pcre2_general_context_)
      |                                                     ^~~~~~~~~~~~~~~~~~~~~~
/pcre2-10.42/src/pcre2.h:981:25: note: in definition of macro ‘PCRE2_SUFFIX’
  981 | #define PCRE2_SUFFIX(a) a
      |                         ^
/pcre2-10.42/src/pcre2_maketables.c:154:23: note: in expansion of macro ‘pcre2_general_context’
  154 | pcre2_maketables_free(pcre2_general_context *gcontext, const uint8_t *tables)
      |

I've followed the NON-AUTOTOOLS-BUILD guide, using the generic config.h along with:
-DHAVE_CONFIG_H -DPCRE2_STATIC -DPCRE2_CODE_UNIT_WIDTH=0 -DSUPPORT_PCRE2_8 -DSUPPORT_PCRE2_16 -DSUPPORT_UNICODE.
Not sure if I've missed something.

[PhilipHazel]

I'm afraid you have misunderstood - perhaps the documentation isn't clear (I have made a note to look at this point in due course). When building PCRE2 you must set PCRE2_CODE_UNIT_WIDTH to 8, 16, or 32. Each setting builds a different library. If you have an application that needs to use more than one PCRE2 library (e.g. 8-bit and 16-bit), then you can set PCRE2_CODE_UNIT_WIDTH to zero in your application, before including pcre2.h. The application must then refer to functions by their full names, e.g. pcre2_compile_8() instead of the generic pcre2_compile() and must be linked with all relevant PCRE2 libraries. The bottom line is that there is no PCRE2 library that handles more than one code unit width. (One day there might be; I have had some discussions with a user about this.)

[tobil4sk]

I see, thanks. So for now we'll need to build both pcre2-8 and pcre2-16 separately as libraries and link both.

In all fairness, NON-AUTOTOOLS-BUILD does only mention 8, 16, or 32, but I thought perhaps it is an undocumented setting.

Maybe the .c source files could give an error as well if PCRE2_CODE_UNIT_WIDTH is set to 0, mentioning that the value is not permitted when building the library itself. pcre2_internal.h looks like it should be the place for this, but pcre2test.c includes it as well and it uses PCRE2_CODE_UNIT_WIDTH=0 so that won't work.

[Uzume]

I wasn't confused about how PCRE2_CODE_UNIT_WIDTH=0 works (or lack thereof), however I wonder about this:

[...]The bottom line is that there is no PCRE2 library that handles more than one code unit width. (One day there might be; I have had some discussions with a user about this.)

@PhilipHazel: What sort of discussion have you had and is there someplace they can be found online?

Thanks.

[PhilipHazel]

It was a series of private emails, I'm afraid. The other party is otherwise occupied at the moment, so the "project" (if that's not too grand a word) is currently on hold for a couple of months. The discussions were very preliminary, but the rough idea was to have another library where the compile and match functions have another parameter, specifying the code unit width. The compiled code would always look like the current 32-bit code. Obviously, checking the width every time the code needs to load a character will slow things down. Converting the strings before starting - which of course can be done externally now - is not right because some subject strings are very, very long. PCRE2 already loads characters using macros (e.g. GETCHAR, see pcre2_intmodedep.h). The first step will be to have different versions for compile and match. A major decision will be whether to add an argument to the compile/match functions, or to retain compatibility and have an "extra options" setting.

[Uzume]

That seems like a very specialized set of use cases. Thanks for the quick feedback.

[tobil4sk]

Managed to solve this in the end by compiling the sources twice, once for 8 bit support and 16 bit, and putting them into separate object files, before linking everything together. So it wasn't too complicated to compile in a similar way to how we did it with pcre1, and now we can easily use both 8 bit and 16 bit functions.

I can close this now, unless you'd like to keep this around as a reminder for looking at the documentation as you mentioned.

For the library with multiple encoding support, I agree it sounds quite specialised. In our case, we have separate code paths for 8 bit and 16 bit so we use our own logic for deciding which one to use. I think it would be a shame to break the api again for a feature like this which seems like it would be used in fairly rare cases, but I'm not sure how common it is for other applications to want to use pcre with multiple string encodings.

[ZwergNaseXXL]

Not sure if you updated the docu yet, but I just fell into the same trap. Which means that it's still ambiguous, even though I read in several places, after a search for PCRE2_CODE_UNIT_WIDTH in all file types.

The docu led me to believe that PCRE2 will always contain 8, 16 and 32 bit versions of all functions and that PCRE2_CODE_UNIT_WIDTH just adds convenience macros for your "favorite" bit width. And that PCRE2_CODE_UNIT_WIDTH=0 just omits those convenience macros. And even though I now compile with PCRE2_CODE_UNIT_WIDTH=16, it still has the pcre2_..._8 types.

[PhilipHazel]

I think I did some work on the documentation, which is why I closed the issue. I've just had another check, and it seems to be quite clear. For example, in the "pcre2" man page it says "The source code for PCRE2 can be compiled to support strings of 8-bit, 16-bit, or 32-bit code units, which means that up to three separate libraries may be installed, one for each code unit size." And in "pcre2build" it says "By default, a library called libpcre2-8 is built, containing functions that take string arguments contained in arrays of bytes, interpreted either as single-byte characters, or UTF-8 strings. You can also build two other libraries, called libpcre2-16 and libpcre2-32, which process strings that are contained in arrays of 16-bit and 32-bit code units, respectively."

If you still think there is a lack of clarity about the current state of the documentation in HEAD, please say so, and I will try to improve it.

[Uzume]

@PhilipHazel: That documentation seems quite clear however, I believe the issue is with respect to PCRE2_CODE_UNIT_WIDTH itself. The documentation you quoted does not mention that macro at all or how to use it (and pcre2 always requires it).

Maybe have a summary somewhere of what acceptable values for PCRE2_CODE_UNIT_WIDTH mean:

• 8: for 8-bit code unit library, either building or access via code unit generic client API
• 16: for 16-bit code unit library, either building or access via code unit generic client API
• 32: for 32-bit code unit library, either building or access via code unit generic client API
• 0: for multiple (8, 16, and 32) code unit libraries access via code unit specific client APIs (since there is no way to specify the code unit width within the code unit generic client API); cannot be used for building code unit specific libraries

and of course state it defaults to 8.

[ZwergNaseXXL]

Yes, it was the scattered documentation of PCRE2_CODE_UNIT_WIDTH, which confused me.

Also, in regard to the last sentence in my last comment: If building with PCRE2_CODE_UNIT_WIDTH=16 only generates libpcre2-16, then why do I still have the the pcre2_..._8 types? Meanwhile I noticed that only the declarations are there, not the function definitions. Still a bit weird and adding to the confusion.

[PhilipHazel]

Anybody building using Autotools or Cmake doesn't need to know about PCRE2_CODE_UNIT_WIDTH, because those tools handle it for you. For those that build "by hand", the NON-AUTOTOOLS-BUILD file has, I hope, reasonable instructions.

Users of PCRE2 do, of course, need to know about it. I would hope that users would look at the "pcre2api" man page, where it says "Many applications use only one code unit width. For their convenience, macros are defined whose names are the generic forms such as pcre2_compile() and PCRE2_SPTR. These macros use the value of the macro PCRE2_CODE_UNIT_WIDTH to generate the appropriate width-specific function and macro names. PCRE2_CODE_UNIT_WIDTH is not defined by default. An application must define it to be 8, 16, or 32 before including pcre2.h in order to make use of the generic names."

Note that, in contradiction to @Uzume 's comment above, there is no default value for PCRE2_CODE_UNIT_WIDTH.

Interesting that it's taken 8 years for this issue to be raised. The fact that there is only one pcre2.h (rather than pcre2-8.h, etc) makes is easier to maintain and at the time of creating PCRE2 from PCRE1 it was the straightforward thing to do.

I am, incidentally, in the process of arranging for a compiler error if an attempt is made to compile any of the library modules with an invalid value for PCRE2_CODE_UNIT_WIDTH.

[Uzume]

Note that, in contradiction to @Uzume 's comment above, there is no default value for PCRE2_CODE_UNIT_WIDTH.

@PhilipHazel: The default of 8 was only in reference to your earlier quote of "By default, a library called libpcre2-8 is built..." and thus I assumed the build scripts defaulted to setting it to 8 (when not provided directions otherwise).