Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Failure (go-kafka-serde returned non-zero exit status 1) in SchemaRegistryAutoAuthTest.test_serde_client #10001

Closed
ztlpn opened this issue Apr 12, 2023 · 8 comments · Fixed by #10184
Assignees
Labels
area/schema-registry Schema Registry service within Redpanda ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages

Comments

@ztlpn
Copy link
Contributor

ztlpn commented Apr 12, 2023

https://buildkite.com/redpanda/redpanda/builds/26840#01877093-ce14-4e23-8fd7-91498e58fb0e

This is v22.3.x and a PR build (but I think the error is unrelated)

Module: rptest.tests.schema_registry_test
Class:  SchemaRegistryAutoAuthTest
Method: test_serde_client
Arguments:
{
  "client_type": 3,
  "protocol": 2
}
test_id:    rptest.tests.schema_registry_test.SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Golang
status:     FAIL
run time:   14.731 seconds


    Exception('SerdeClient-0-140510478558704-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/serde_client.py", line 93, in _worker\n    for line in ssh_output:\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-9: Command \'/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-13:9092,docker-rp-21:9092,docker-rp-22:9092 --topic serde-topic-PROTOBUF-Golang --schema-registry http://docker-rp-21:8081 --consumer-group bf4ddacc-5d5c-4130-a0a6-99bbc9ab6ed7 --protocol PROTOBUF --count 2 --debug --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/schema_registry_test.py", line 1162, in test_serde_client
    client.wait()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 74, in wait
    self._propagate_exceptions()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 100, in _propagate_exceptions
    raise Exception(self.errors)
Exception: SerdeClient-0-140510478558704-worker-1: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
    self._worker(idx, node)
  File "/root/tests/rptest/services/serde_client.py", line 93, in _worker
    for line in ssh_output:
  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next
    return next(self.iter_obj)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-9: Command '/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-13:9092,docker-rp-21:9092,docker-rp-22:9092 --topic serde-topic-PROTOBUF-Golang --schema-registry http://docker-rp-21:8081 --consumer-group bf4ddacc-5d5c-4130-a0a6-99bbc9ab6ed7 --protocol PROTOBUF --count 2 --debug --security '{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}'' returned non-zero exit status 1.
@ztlpn ztlpn added kind/bug Something isn't working area/schema-registry Schema Registry service within Redpanda ci-failure labels Apr 12, 2023
@ztlpn
Copy link
Contributor Author

ztlpn commented Apr 12, 2023

@michael-redpanda can you take a look? Looks like the test is quite new.

@ztlpn
Copy link
Contributor Author

ztlpn commented Apr 12, 2023

looking at the test code, how do we guarantee that we've processed all producer events before checking numDelivered?

@michael-redpanda
Copy link
Contributor

looking at the test code, how do we guarantee that we've processed all producer events before checking numDelivered?

I was just looking at that. Yeah it's probably that. I'll fix it in dev and the backport it to.

@michael-redpanda michael-redpanda self-assigned this Apr 12, 2023
@dotnwat
Copy link
Member

dotnwat commented Apr 17, 2023

@dotnwat
Copy link
Member

dotnwat commented Apr 19, 2023

@dotnwat
Copy link
Member

dotnwat commented Apr 20, 2023

FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=AVRO.client_type=Golang (1/35 runs)
  failure at 2023-04-20T06:48:04.201Z: Exception('SerdeClient-0-140338125742112-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-23: Command \'/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-15:9092,docker-rp-18:9092,docker-rp-12:9092 --topic serde-topic-AVRO-Golang --schema-registry http://docker-rp-18:8081 --consumer-group bd59515e-c5cb-4f56-b581-5d03c4571a61 --protocol AVRO --count 2 --debug --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/27568#01879d32-69b6-4002-897e-8e5cf251a18f
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=AVRO.client_type=Python (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566306121296-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-12-222: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-12-52:9092,ip-172-31-15-8:9092,ip-172-31-8-225:9092 --topic serde-topic-AVRO-Python --schema-registry http://ip-172-31-8-225:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol AVRO --count 2 --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Golang (2/35 runs)
  failure at 2023-04-19T16:34:16.694Z: Exception('SerdeClient-0-139990264732912-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-12: Command \'/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-23:9092,docker-rp-11:9092,docker-rp-24:9092 --topic serde-topic-PROTOBUF-Golang --schema-registry http://docker-rp-23:8081 --consumer-group f15559ad-227c-487c-a112-a896cba33540 --protocol PROTOBUF --count 2 --debug --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/27503#01879a11-f89d-4bf0-bd3c-cc634598500c
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=False (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566064571888-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-2-84: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-12-52:9092,ip-172-31-15-8:9092,ip-172-31-12-222:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-12-52:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=True (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566065297200-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-12-222: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-15-8:9092,ip-172-31-12-52:9092,ip-172-31-8-225:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-8-225:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryTest.test_serde_client.protocol=AVRO.client_type=Python (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566064243824-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-2-84: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-15-8:9092,ip-172-31-12-52:9092,ip-172-31-12-222:9092 --topic serde-topic-AVRO-Python --schema-registry http://ip-172-31-12-52:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol AVRO --count 2\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=False (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566065676688-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-8-225: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-2-84:9092,ip-172-31-15-8:9092,ip-172-31-12-222:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-15-8:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=True (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566306121296-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-2-84: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-12-222:9092,ip-172-31-7-239:9092,ip-172-31-15-8:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-7-239:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247

@michael-redpanda
Copy link
Contributor

The failures in golang are known (and being addressed in #10184).

The python ones are due to an ImportError:

│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]: Traceback (most recent call last):                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:   File "/tmp/python_librdkafka_serde_client.py", line 27, in <module>                                                                                                                                                                                                           │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:     from payload_pb2 import Payload as ProtobufPayloadClass                                                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:   File "/tmp/payload_pb2.py", line 5, in <module>                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:     from google.protobuf.internal import builder as _builder                                                                                                                                                                                                                    │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]: ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/usr/local/lib/python3.10/dist-packages/google/protobuf/internal/__init__.py) 

@michael-redpanda
Copy link
Contributor

I kicked off a cdt arm on #10184 and did not see those python errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/schema-registry Schema Registry service within Redpanda ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants