gh-52551: Fix encoding issues in strftime() #125193

serhiy-storchaka · 2024-10-09T14:25:06Z

Fix time.strftime(), the strftime() method and formatting of the datetime classes datetime, date and time.

Characters not encodable in the current locale are now acceptable in the format string.
Surrogate pairs and sequence of surrogatescape-encoded bytes are no longer recombinated.
Embedded null character no longer terminates the format string.

Issue: time.strftime() and Unicode characters on Windows #52551

Issue: Different behavior of C and Python impls of datetime.strftime with non-UTF-8-encodable strings #78662

Issue: datetime.strftime strings can be terminated by "\x00" literals #124531

Fix time.strftime(), the strftime() method and formatting of the datetime classes datetime, date and time. * Characters not encodable in the current locale are now acceptable in the format string. * Surrogate pairs and sequence of surrogatescape-encoded bytes are no longer recombinated. * Embedded null character no longer terminates the format string. This fixes also pythongh-78662 and pythongh-124531.

serhiy-storchaka · 2024-10-10T16:25:42Z

@vstinner, could you please review this PR?

vstinner · 2024-10-16T12:12:59Z

Modules/_datetimemodule.c

+                                                newformat, timetuple, NULL);
+        Py_DECREF(newformat);
+    }
+    if (0) {


This if (0) is weird. Why not adding a label after _PyUnicodeWriter_Dealloc() and using goto?

This would require changing more code. Renaming label Done to Error (there are many goto Done in not changed code) and adding new Done label. The if (0) idiom is weird, but used in many places.

vstinner · 2024-10-16T12:14:23Z

Modules/_datetimemodule.c

    }
+    if (newformat != NULL) {


I would prefer to exit early: if (newformat == NULL) goto ....

By the way, only the _PyUnicodeWriter_Finish() code path can fail, no?

This would require adding more labels. The flow would look more complicated.

Modules/_datetimemodule.c

vstinner · 2024-10-16T12:37:49Z

Modules/timemodule.c

-    format = PyUnicode_EncodeLocale(format_arg, "surrogateescape");
-    if (format == NULL)
+    format_size = PyUnicode_GET_LENGTH(format_arg);
+    format = PyMem_Malloc((format_size + 1)*sizeof(time_char));


I suggest to add an explicit integer overflow check: fail with PyErr_NoMemory() on overflow.

vstinner · 2024-10-16T12:39:59Z

Modules/timemodule.c

+             * will be ahead of time...
+             */
+            while (1) {
+                outbuf = (time_char *)PyMem_Realloc(outbuf, bufsize*sizeof(time_char));


You may also add an explicit integer overflow check here.

vstinner · 2024-10-16T12:50:17Z

Modules/timemodule.c

+            Py_DECREF(unicode);
+        }
+
+        j = i;


You should declare the j variable here, and I suggest to rename it to start.

vstinner · 2024-10-16T12:53:22Z

Modules/timemodule.c

+            }
+            format[fmtlen++] = (char)c;
+        }
+        if (fmtlen) {


This function is quite big. You might move the code handling format in a sub-function (the whole if block).

This may be inconvenient, because this code changes several outer variables.: outbuf and bufsize.

serhiy-storchaka · 2024-10-17T14:01:25Z

I addressed all comments. Perhaps it is now more difficult to review.

vstinner

LGTM. I just left minor coding style suggestions that you can ignore.

Thanks for the exhaustive added tests, it's very helpful.

vstinner · 2024-10-17T14:10:49Z

Modules/timemodule.c

+        *outbuf = (time_char *)PyMem_Realloc(*outbuf,
+                                                *bufsize*sizeof(time_char));


Suggested change

*outbuf = (time_char *)PyMem_Realloc(*outbuf,

*bufsize*sizeof(time_char));

*outbuf = (time_char *)PyMem_Realloc(*outbuf,

*bufsize*sizeof(time_char));

vstinner · 2024-10-17T14:12:38Z

Modules/timemodule.c

@@ -776,27 +776,100 @@ the C library strftime function.\n"
 #endif

 static PyObject *
-time_strftime(PyObject *module, PyObject *args)
+strftime1(time_char **outbuf, size_t *bufsize,


nitpick: you might rename the function to time_strftime1() to avoid any conflict with a potential strftime1() name from the C library.

serhiy-storchaka · 2024-10-17T15:18:31Z

Thank you for your review @vstinner.

miss-islington-app · 2024-10-17T15:47:02Z

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

miss-islington-app · 2024-10-17T15:47:05Z

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker ad3eac1963a5f195ef9b2c1dbb5e44fa3cce4c72 3.13

miss-islington-app · 2024-10-17T15:47:21Z

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker ad3eac1963a5f195ef9b2c1dbb5e44fa3cce4c72 3.12

…5193) Fix time.strftime(), the strftime() method and formatting of the datetime classes datetime, date and time. * Characters not encodable in the current locale are now acceptable in the format string. * Surrogate pairs and sequence of surrogatescape-encoded bytes are no longer recombinated. * Embedded null character no longer terminates the format string. This fixes also pythongh-78662 and pythongh-124531. (cherry picked from commit ad3eac1) Co-authored-by: Serhiy Storchaka <[email protected]>

Fix time.strftime(), the strftime() method and formatting of the datetime classes datetime, date and time. * Characters not encodable in the current locale are now acceptable in the format string. * Surrogate pairs and sequence of surrogatescape-encoded bytes are no longer recombinated. * Embedded null character no longer terminates the format string. This fixes also pythongh-78662 and pythongh-124531. (cherry picked from commit ad3eac1)

bedevere-app · 2024-10-17T17:21:10Z

GH-125657 is a backport of this pull request to the 3.13 branch.

…5657) Fix time.strftime(), the strftime() method and formatting of the datetime classes datetime, date and time. * Characters not encodable in the current locale are now acceptable in the format string. * Surrogate pairs and sequence of surrogatescape-encoded bytes are no longer recombinated. * Embedded null character no longer terminates the format string. This fixes also gh-78662 and gh-124531. (cherry picked from commit ad3eac1)

…5193) (pythonGH-125657) Fix time.strftime(), the strftime() method and formatting of the datetime classes datetime, date and time. * Characters not encodable in the current locale are now acceptable in the format string. * Surrogate pairs and sequence of surrogatescape-encoded bytes are no longer recombinated. * Embedded null character no longer terminates the format string. This fixes also pythongh-78662 and pythongh-124531. (cherry picked from commit 08ccbb9) Co-authored-by: Serhiy Storchaka <[email protected]> (cherry picked from commit ad3eac1)

…5657) (GH-125661) Fix time.strftime(), the strftime() method and formatting of the datetime classes datetime, date and time. * Characters not encodable in the current locale are now acceptable in the format string. * Surrogate pairs and sequence of surrogatescape-encoded bytes are no longer recombinated. * Embedded null character no longer terminates the format string. This fixes also gh-78662 and gh-124531. (cherry picked from commit 08ccbb9) (cherry picked from commit ad3eac1) Co-authored-by: Serhiy Storchaka <[email protected]>

vstinner · 2024-10-18T09:03:01Z

Congrats! That's a nice change. Using Unicode instead of bytes for the internal buffer is a good move.

serhiy-storchaka added needs backport to 3.12 bug and security fixes needs backport to 3.13 bugs and security fixes labels Oct 9, 2024

serhiy-storchaka requested review from pganssle and abalkin as code owners October 9, 2024 14:25

bedevere-app bot mentioned this pull request Oct 9, 2024

time.strftime() and Unicode characters on Windows #52551

Closed

bedevere-app bot added the awaiting core review label Oct 9, 2024

This was linked to issues Oct 9, 2024

time.strftime() and Unicode characters on Windows #52551

Closed

datetime.strftime strings can be terminated by "\x00" literals #124531

Closed

serhiy-storchaka changed the title ~~gh-52551: Fix encoding issues in strftime()~~ gh-52551, gh-78662, gh-124531: Fix encoding issues in strftime() Oct 9, 2024

serhiy-storchaka changed the title ~~gh-52551, gh-78662, gh-124531: Fix encoding issues in strftime()~~ gh-124531: Fix encoding issues in strftime() Oct 9, 2024

bedevere-app bot mentioned this pull request Oct 9, 2024

datetime.strftime strings can be terminated by "\x00" literals #124531

Closed

serhiy-storchaka changed the title ~~gh-124531: Fix encoding issues in strftime()~~ gh-78662: Fix encoding issues in strftime() Oct 9, 2024

bedevere-app bot mentioned this pull request Oct 9, 2024

Different behavior of C and Python impls of datetime.strftime with non-UTF-8-encodable strings #78662

Closed

serhiy-storchaka changed the title ~~gh-78662: Fix encoding issues in strftime()~~ gh-52551: Fix encoding issues in strftime() Oct 9, 2024

serhiy-storchaka mentioned this pull request Oct 9, 2024

gh-124531: Fix strftime() with embedded null characters #125003

Closed

serhiy-storchaka requested a review from vstinner October 10, 2024 16:24

vstinner reviewed Oct 16, 2024

View reviewed changes

serhiy-storchaka added 3 commits October 17, 2024 14:43

Merge branch 'main' into strftime

690e64a

Address some review comments.

c75eb0c

Address other review comment.

ca215dd

vstinner approved these changes Oct 17, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Oct 17, 2024

Apply style suggestions.

d4fcbf3

serhiy-storchaka enabled auto-merge (squash) October 17, 2024 15:19

serhiy-storchaka merged commit ad3eac1 into python:main Oct 17, 2024
37 checks passed

bedevere-app bot removed the awaiting merge label Oct 17, 2024

miss-islington-app bot assigned serhiy-storchaka Oct 17, 2024

serhiy-storchaka deleted the strftime branch October 17, 2024 16:18

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Oct 17, 2024

serhiy-storchaka removed the needs backport to 3.12 bug and security fixes label Oct 17, 2024

serhiy-storchaka removed their assignment Oct 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-52551: Fix encoding issues in strftime() #125193

gh-52551: Fix encoding issues in strftime() #125193

serhiy-storchaka commented Oct 9, 2024 •

edited

Loading

serhiy-storchaka commented Oct 10, 2024

vstinner Oct 16, 2024

serhiy-storchaka Oct 17, 2024

vstinner Oct 16, 2024

serhiy-storchaka Oct 17, 2024

vstinner Oct 16, 2024

vstinner Oct 16, 2024

vstinner Oct 16, 2024

vstinner Oct 16, 2024

serhiy-storchaka Oct 17, 2024

serhiy-storchaka commented Oct 17, 2024

vstinner left a comment

vstinner Oct 17, 2024

vstinner Oct 17, 2024

serhiy-storchaka commented Oct 17, 2024

miss-islington-app bot commented Oct 17, 2024

miss-islington-app bot commented Oct 17, 2024

miss-islington-app bot commented Oct 17, 2024

bedevere-app bot commented Oct 17, 2024

vstinner commented Oct 18, 2024

		outbuf = (time_char )PyMem_Realloc(*outbuf,
		bufsizesizeof(time_char));

gh-52551: Fix encoding issues in strftime() #125193

gh-52551: Fix encoding issues in strftime() #125193

Conversation

serhiy-storchaka commented Oct 9, 2024 • edited Loading

serhiy-storchaka commented Oct 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serhiy-storchaka commented Oct 17, 2024

vstinner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serhiy-storchaka commented Oct 17, 2024

miss-islington-app bot commented Oct 17, 2024

miss-islington-app bot commented Oct 17, 2024

miss-islington-app bot commented Oct 17, 2024

bedevere-app bot commented Oct 17, 2024

vstinner commented Oct 18, 2024

serhiy-storchaka commented Oct 9, 2024 •

edited

Loading