Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ #108704

Merged
merged 6 commits into from
Aug 31, 2023

Conversation

ethanfurman
Copy link
Member

@ethanfurman ethanfurman commented Aug 31, 2023

@ethanfurman ethanfurman added type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir 3.11 only security fixes 3.12 bugs and security fixes needs backport to 3.11 only security fixes 3.13 bugs and security fixes needs backport to 3.12 bug and security fixes labels Aug 31, 2023
@ethanfurman ethanfurman self-assigned this Aug 31, 2023
@ethanfurman ethanfurman changed the title [Enum] raise TypeError if super().__new__ called in custom __new__ gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ Aug 31, 2023
Doc/howto/enum.rst Outdated Show resolved Hide resolved
@ethanfurman ethanfurman added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 31, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @ethanfurman for commit 561dac5 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 31, 2023
@ethanfurman ethanfurman added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 31, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @ethanfurman for commit 0435142 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 31, 2023
@ethanfurman ethanfurman merged commit d48760b into python:main Aug 31, 2023
17 checks passed
@miss-islington
Copy link
Contributor

Thanks @ethanfurman for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 31, 2023
…custom __new__ (pythonGH-108704)

When overriding the `__new__` method of an enum, the underlying data type should be created directly; i.e. .

    member = object.__new__(cls)
    member = int.__new__(cls, value)
    member = str.__new__(cls, value)

Calling `super().__new__()` finds the lookup version of `Enum.__new__`, and will now raise an exception when detected.
(cherry picked from commit d48760b)

Co-authored-by: Ethan Furman <[email protected]>
@miss-islington
Copy link
Contributor

Sorry, @ethanfurman, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker d48760b2f1e28dd3c1a35721939f400a8ab619b8 3.11

@bedevere-bot
Copy link

GH-108733 is a backport of this pull request to the 3.12 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.12 bug and security fixes label Aug 31, 2023
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM Raspbian 3.x has failed when building commit d48760b.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/424/builds/4786) and take a look at the build logs.
  4. Check if the failure is related to this commit (d48760b) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/424/builds/4786

Failed tests:

  • test.test_concurrent_futures.test_shutdown

Failed subtests:

  • test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

442 tests OK.

10 slowest tests:

  • test_cppext: 10 min 34 sec
  • test_venv: 10 min 20 sec
  • test_largefile: 9 min 28 sec
  • test_io: 4 min 36 sec
  • test_math: 4 min 30 sec
  • test_mailbox: 3 min 28 sec
  • test_regrtest: 3 min 10 sec
  • test_hashlib: 2 min 14 sec
  • test_zipfile: 1 min 57 sec
  • test.test_multiprocessing_spawn.test_processes: 1 min 56 sec

1 test failed:
test.test_concurrent_futures.test_shutdown

19 tests skipped:
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test_devpoll test_idle
test_ioctl test_kqueue test_launcher test_perf_profiler
test_startfile test_tcl test_tkinter test_ttk test_ttk_textonly
test_turtle test_winconsoleio test_winreg test_winsound test_wmi
test_zipfile64

1 re-run test:
test.test_concurrent_futures.test_shutdown

Total duration: 40 min 25 sec

Click to see traceback logs
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)
AssertionError: b'Exception in thread Thread-1:\nTraceback (most recent call last):\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 1059, in _bootstrap_inner\n    self.run()\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/concurrent/futures/process.py", line 339, in run\n    self.add_call_item_to_queue()\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/concurrent/futures/process.py", line 394, in add_call_item_to_queue\n    self.call_queue.put(_CallItem(work_id,\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/queues.py", line 94, in put\n    self._start_thread()\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/queues.py", line 177, in _start_thread\n    self._thread.start()\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 978, in start\n    _start_new_thread(self._bootstrap, ())\nRuntimeError: can\'t create new thread at interpreter shutdown\nTraceback (most recent call last):\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/forkserver.py", line 274, in main\n    code = _serve_one(child_r, fds,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/forkserver.py", line 313, in _serve_one\n    code = spawn._main(child_r, parent_sentinel)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/spawn.py", line 132, in _main\n    self = reduction.pickle.load(from_parent)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__\n    self._semlock = _multiprocessing.SemLock._rebuild(*state)\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n' is not false

Yhg1s pushed a commit that referenced this pull request Aug 31, 2023
… custom __new__ (GH-108704) (#108733)

gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ (GH-108704)

When overriding the `__new__` method of an enum, the underlying data type should be created directly; i.e. .

    member = object.__new__(cls)
    member = int.__new__(cls, value)
    member = str.__new__(cls, value)

Calling `super().__new__()` finds the lookup version of `Enum.__new__`, and will now raise an exception when detected.
(cherry picked from commit d48760b)

Co-authored-by: Ethan Furman <[email protected]>
ethanfurman added a commit to ethanfurman/cpython that referenced this pull request Aug 31, 2023
…custom __new__ (pythonGH-108704)

When overriding the `__new__` method of an enum, the underlying data type should be created directly; i.e. .

    member = object.__new__(cls)
    member = int.__new__(cls, value)
    member = str.__new__(cls, value)

Calling `super().__new__()` finds the lookup version of `Enum.__new__`, and will now raise an exception when detected.

(cherry picked from commit d48760b)
@bedevere-bot
Copy link

GH-108739 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Aug 31, 2023
ethanfurman added a commit that referenced this pull request Sep 8, 2023
… custom __new__ (GH-108704) (GH-108739)

When overriding the `__new__` method of an enum, the underlying data type should be created directly; i.e. .

    member = object.__new__(cls)
    member = int.__new__(cls, value)
    member = str.__new__(cls, value)

Calling `super().__new__()` finds the lookup version of `Enum.__new__`, and will now raise an exception when detected.

(cherry picked from commit d48760b)
Comment on lines +1117 to +1121
# still not found -- verify that members exist, in-case somebody got here mistakenly
# (such as via super when trying to override __new__)
if not cls._member_map_:
raise TypeError("%r has no members defined" % cls)
#
Copy link

@tyronexj tyronexj Oct 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it ok to move _member_map_ check after _missing_ hook?
I have a derived class has no members and hook _missing_, it raise exception here.
@ethanfurman

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, no. I tried to do that, but then the bug wasn't fixed.

What does your missing() hook do?

If you have no other alternative, you'll need to subclass EnumType and override the __call__ method.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick reply. here is an example:

from enum import Enum

class DynamicEnum(Enum):
    @classmethod
    def _missing_(cls, value):
        temp = object.__new__(cls)
        temp._name_ = f"{cls.__name__}_{value}"
        temp._value_ = value
        return temp

dynamic_enum = DynamicEnum(3)
print(dynamic_enum)
print(dynamic_enum.value)

which prints

DynamicEnum.DynamicEnum_3
3

before this change.

As my understanding, hook _missing_ handle the missing input, a dynamic value in the above case.
so I raise the question: hook _missing_ before empty check.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll chime in here as well. I have the same question and got hit by this change as well. We have a setup where we do like:

from enum import Enum

class Result(Enum):
    @classmethod
    def _missing_(cls, value):
        v = None
        for subclass in cls.__subclasses__():
            try:
                v = subclass(value)
            except:
                pass

        return v


class CommonResult(Result):
    OK = 0
    ERR_FOO = -0x1
    ERR_BAR = -0x2

Previously, Result(0) returned CommonResult.OK. Now throws. It was a nifty pattern to use, and it sort of breaks the combination of using the _missing_ and subclasses of Enum.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes 3.13 bugs and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants