segmentation fault for protos with optional fields

[arithmetic1728]

What version of protobuf and what language are you using?
Version: v3.12.4
Language: Python

What operating system (Linux, Windows, ...) and version?
gLinux

What runtime / compiler are you using (e.g., python version or gcc version)
python 3.6.9, gcc 9.3.0

What did you do?
Steps to reproduce the behavior:

1. Write the following to foo.proto:

syntax = "proto3";
 
message Nested {
  optional bool test = 1;
}
 
message Foo {
  optional string name = 1;
  Nested nested = 3;
}

2. Compile with protoc: protoc --experimental_allow_proto3_optional --python_out=. foo.proto

3. Write the following to repro.py:

from foo_pb2 import Foo
 
foo = Foo()
foo.name = 'name'
foo.nested.test = False
 
foo.Clear()

4. Run: python repro.py, segv happens at line foo.Clear()

python repro.py 
Fatal Python error: Segmentation fault

Current thread 0x00007f981a49c4c0 (most recent call first):
  File "repro.py", line 7 in <module>
Segmentation fault

[arithmetic1728]

Filed on behavior of ads team. This blocks their August release. @BenRKarl

[BenRKarl]

Additional detail -

This error occurs in protos generated with protoc versions 3.12.2, 3.12.3 and 3.12.4, with Python versions 3.8.2 and 3.7.0, and protobuf release candidates 4.0.0rc1 and 4.0.0rc2.

[anandolee]

Does 3.12.0 have same issue?

[BenRKarl]

@anandolee Yes it appears so.

[bobhancock]

In the protobuf package in message.py Message.Clear() is defined as:

def Clear(self):
"""Clears all data that was set in the message."""
raise NotImplementedError

which is the line being called where the SEGFAULT occurs. Since Message is the abstract base class, the derived class should implement this method, but I don't see the implementation in api-core.

[anandolee]

I believe it is using cpp extension, so it will call the Clear() in message.cc
https://github.com/protocolbuffers/protobuf/blob/master/python/google/protobuf/pyext/message.cc#L1653