Preprocessor: # operator does not follow standards

[sheumann]

The # operator for stringizing tokens behaves non-standardly in at least the following respects:

• It does not insert spaces corresponding to white space between its argument tokens.
• When applied to a string literal, it should generate a string corresponding to the original representation of the string in the source file, including " delimiters and escape sequences using \ (and those characters are at least conceptually escaped with \ to permit this). Currently, it gives a string literal equivalent to the one passed to it, but does not include the delimiters and escape sequences from its actual representation in the source file.

[sheumann]

Commit 76eb476 addresses several issues related to the # operator, but still leaves some others. These include that escape sequences in strings are not handled properly and that comments between tokens can cause problems.

Here is a program that illustrates some remaining issues:

#include <stdio.h>
#include <string.h>

#define stringize(x) #x
#define stringize2(x, y) stringize(x+y)
#define strstr(x) (stringize(#x))

int main (void)
{
    puts(stringize(    a   +/**/b "C\t" 'g' <:p   ));
    printf("%lu\n", strlen(stringize(    a   +/**/b "C\t" 'g' <:p   )));
    puts(strstr(a   +b "C"));
    printf("%lu\n", strlen(strstr(a   +b "C")));
    puts(stringize2(z,a   +b "C\tD\x75\n"));
    printf("%lu\n", strlen(stringize2(z,a   +b "C\tD\x75\n")));
}