Add cloud event to core #16800

rakshith91 · 2021-02-18T06:43:52Z

Creating a different PR for specifically core related changes - will rebase the other PR for EG related ones

xiangyan99 · 2021-02-18T18:41:24Z

sdk/core/azure-core/azure/core/messaging.py

+    from datetime import timezone
+    TZ_UTC = timezone.utc  # type: ignore
+except ImportError:
+    class UTC(tzinfo):


I am thinking if we can leverage/refactor the code https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/core/azure-core/azure/core/pipeline/policies/_utils.py#L30

sdk/core/azure-core/azure/core/messaging.py

xiangyan99 · 2021-02-18T21:55:28Z

Double check:

https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/core/azure-core/azure/core/pipeline/policies/_utils.py imports requests which is heavy.

Do you really want it? That's the reason I said maybe we want to refactor the code. :)

(this is not blocker. just want to make sure we take it into account)

sdk/core/azure-core/azure/core/messaging.py

johanste · 2021-02-18T22:24:06Z

sdk/core/azure-core/azure/core/messaging.py

+        if data and data_base64:
+            raise ValueError("Invalid input. Only one of data and data_base64 must be present.")
+        return cls(
+        id=event.pop("id", None),


This indentation looks odd?

lmazuel · 2021-02-18T23:09:23Z

sdk/core/azure-core/azure/core/messaging.py

+        type=event.pop("type", None),
+        specversion=event.pop("specversion", None),
+        data=data or b64decode(data_base64),
+        time=event.pop("time", None),


if time is datetime, need deserializaiton here

it'd be a string

sdk/core/azure-core/tests/test_messaging_cloud_event.py

iscai-msft · 2021-02-19T15:38:32Z

/azp run python - autorest - pr

azure-pipelines · 2021-02-19T15:38:44Z

Azure Pipelines successfully started running 1 pipeline(s).

check-enforcer · 2021-02-19T21:44:01Z

This pull request is protected by Check Enforcer.

What is Check Enforcer?

Check Enforcer helps ensure all pull requests are covered by at least one check-run (typically an Azure Pipeline). When all check-runs associated with this pull request pass then Check Enforcer itself will pass.

Why am I getting this message?

You are getting this message because Check Enforcer did not detect any check-runs being associated with this pull request within five minutes. This may indicate that your pull request is not covered by any pipelines and so Check Enforcer is correctly blocking the pull request being merged.

What should I do now?

If the check-enforcer check-run is not passing and all other check-runs associated with this PR are passing (excluding license-cla) then you could try telling Check Enforcer to evaluate your pull request again. You can do this by adding a comment to this pull request as follows:
/check-enforcer evaluate
Typically evaulation only takes a few seconds. If you know that your pull request is not covered by a pipeline and this is expected you can override Check Enforcer using the following command:
/check-enforcer override
Note that using the override command triggers alerts so that follow-up investigations can occur (PRs still need to be approved as normal).

What if I am onboarding a new service?

Often, new services do not have validation pipelines associated with them, in order to bootstrap pipelines for a new service, you can issue the following command as a pull request comment:
/azp run prepare-pipelines
This will run a pipeline that analyzes the source tree and creates the pipelines necessary to build and validate your pull request. Once the pipeline has been created you can trigger the pipeline using the following comment:
/azp run python - [service] - ci

johanste · 2021-02-20T00:28:52Z

sdk/core/azure-core/azure/core/messaging.py

+        :type event: dict
+        :rtype: CloudEvent
+        """
+        data = event.pop("data", None)


I don't think we want to mutate the argument passed in. I would be a bit surprised to notice that my dict had lost (all of its) keys when the method returned.

changed this

johanste · 2021-02-20T00:31:44Z

sdk/core/azure-core/azure/core/messaging.py

+            source=event.pop("source", None),
+            type=event.pop("type", None),
+            specversion=event.pop("specversion", None),
+            data=data or b64decode(data_base64),


What happens if data is ''?

changed this a lil bit - data '' is accepted

johanste · 2021-03-02T20:10:12Z

sdk/core/azure-core/azure/core/messaging.py

+                        "Extension attributes should be lower cased and alphanumeric."
+                    )
+        except KeyError:
+            self.extensions = cast(Dict, None)


This cast does not look correct - it looks like self.extensions should be Optional[Dict], not Dict?

johanste · 2021-03-02T20:15:51Z

sdk/core/azure-core/azure/core/messaging.py

+                "Invalid input. Only one of data and data_base64 must be present."
+            )
+        if 'data' not in event and 'data_base64' not in event:
+            kwargs.setdefault("data", None)


Why are you using setdefault here? It is generally used when you get a partially filled out dict and you don't want to overwrite existing values. But you are defining kwargs in the method (i.e. it is not a parameter passed in to the method).

Please rename, and use direct assignment (e.g. `eventargs['data'] = None)

...also, do you really need to set "data" to None here? The dict passed in didn't have it, so you should be able to create the CloudEvent instance without passing in the keyword argument, right?

johanste · 2021-03-02T20:30:45Z

sdk/core/azure-core/azure/core/messaging.py

+        self.id = kwargs.pop("id", str(uuid.uuid4()))  # type: str
+        self.time = kwargs.pop("time", datetime.now(TZ_UTC))  # type: datetime
+
+        _optional_attributes = ["datacontenttype", "dataschema", "subject", "data"]


By looping through the attribute names like this, I'm pretty sure that you will not get the right type annotations for the attributes (but I may be wrong). Since it is only 4 names, I'd add them explicitly rather than looping through the names:

self.datacontenttype = kwargs.pop('datacontenttype', None)

This is both easier to read and (less important here) more performant.

johanste

The type annotations should be fixed before we merge (don't fix things by doing incorrect casts). Also, not looping through attribute names will provide/allow for typing for those attributes as well.

johanste · 2021-03-03T03:28:15Z

sdk/core/azure-core/azure/core/messaging.py

+
+        return cls(
+            id=event.get("id"),
+            source=cast(str, event.get("source")),


If there is no "source" key in event, you will be (incorrectly) casting None to str. It is better to raise a ValueError here if the either of required keys ("type" and "source") is not in event or else the error will surface much later.

left it to event["source"]

johanste

Please address the cast. As a general principle, we need to be careful about casts since doing them incorrectly negates all the value one get from type checking. In which case it is probably better to have them typed as Any rather than making guarantees that one cannot live up to.

…into http_request_json * 'master' of https://github.com/Azure/azure-sdk-for-python: (147 commits) [text analytics] add perf tests (Azure#17060) Add cloud event to core (Azure#16800) [Perf] Small fixes to storage-blob (Azure#17055) [EG] Regenerate Code (Azure#17053) Scrub batch shared keys (Azure#17030) [Tables] Add SAS to tables (Azure#16717) T2 containerservice 2021 03 03 (Azure#17050) Addressing issues with CredScan (Azure#16944) Communication chat preview4 (Azure#16905) (Azure#17037) remove first query section (Azure#17033) [formrecognizer] temp disable sample tests until service bug fixed (Azure#17036) [device update] allow device update pylint failures (Azure#17034) fix build (Azure#17029) update artifact names for ALL packages to align with the actual package name Create azure-iot-nspkg (Azure#17026) [Communication]: SMS 1:N Messages, Custom Tags, and Idempotence (Azure#16836) Fixing credentials to use AAD (Azure#16885) T2 deviceupdate 2021 03 02 (Azure#17016) T2 cosmosdb 2021 02 23 (Azure#16875) T2 datadog 2021 03 02 (Azure#17004) ...

…into add_sample_check * 'master' of https://github.com/Azure/azure-sdk-for-python: (388 commits) [text analytics] add normalized_text (#17074) Renaming with_token identity function (#17066) Adapt to azure core's cloud event (#17063) align perf tests with js (#17069) [Perfstress][Storage] Added FileShare perf tests (#15834) [formrecognizer] Adding custom forms perf test (#16969) Fix LanguageShort typo (#17068) sas creds updates (#17065) [eventgrid] Fix Sample eh (#17064) [Perfstress][Storage] Added Datalake perf tests (#15861) [text analytics] Healthcare n-ary relations (#16997) ServiceBus dict-representation acceptance and kwarg-update functionality (#14807) [text analytics] add perf tests (#17060) Add cloud event to core (#16800) [Perf] Small fixes to storage-blob (#17055) [EG] Regenerate Code (#17053) Scrub batch shared keys (#17030) [Tables] Add SAS to tables (#16717) T2 containerservice 2021 03 03 (#17050) Addressing issues with CredScan (#16944) ...

…into add_query_kwarg * 'master' of https://github.com/Azure/azure-sdk-for-python: (69 commits) march release (Azure#16966) Release of Device Update for IoT Hub SDK for Python. (Azure#17005) Add Get-AllPackageInfoFromRepo (Azure#16947) Track1 package is incorrectly set as track2 (Azure#17075) [text analytics] add normalized_text (Azure#17074) Renaming with_token identity function (Azure#17066) Adapt to azure core's cloud event (Azure#17063) align perf tests with js (Azure#17069) [Perfstress][Storage] Added FileShare perf tests (Azure#15834) [formrecognizer] Adding custom forms perf test (Azure#16969) Fix LanguageShort typo (Azure#17068) sas creds updates (Azure#17065) [eventgrid] Fix Sample eh (Azure#17064) [Perfstress][Storage] Added Datalake perf tests (Azure#15861) [text analytics] Healthcare n-ary relations (Azure#16997) ServiceBus dict-representation acceptance and kwarg-update functionality (Azure#14807) [text analytics] add perf tests (Azure#17060) Add cloud event to core (Azure#16800) [Perf] Small fixes to storage-blob (Azure#17055) [EG] Regenerate Code (Azure#17053) ...

…into add_reprs * 'master' of https://github.com/Azure/azure-sdk-for-python: (71 commits) EG - more docs imrpovement (Azure#17079) [EventHubs] add logging.info to warn the usage of partition key of non-string type (Azure#17057) march release (Azure#16966) Release of Device Update for IoT Hub SDK for Python. (Azure#17005) Add Get-AllPackageInfoFromRepo (Azure#16947) Track1 package is incorrectly set as track2 (Azure#17075) [text analytics] add normalized_text (Azure#17074) Renaming with_token identity function (Azure#17066) Adapt to azure core's cloud event (Azure#17063) align perf tests with js (Azure#17069) [Perfstress][Storage] Added FileShare perf tests (Azure#15834) [formrecognizer] Adding custom forms perf test (Azure#16969) Fix LanguageShort typo (Azure#17068) sas creds updates (Azure#17065) [eventgrid] Fix Sample eh (Azure#17064) [Perfstress][Storage] Added Datalake perf tests (Azure#15861) [text analytics] Healthcare n-ary relations (Azure#16997) ServiceBus dict-representation acceptance and kwarg-update functionality (Azure#14807) [text analytics] add perf tests (Azure#17060) Add cloud event to core (Azure#16800) ...

…into http_response_json * 'master' of https://github.com/Azure/azure-sdk-for-python: (165 commits) EG - more docs imrpovement (Azure#17079) [EventHubs] add logging.info to warn the usage of partition key of non-string type (Azure#17057) march release (Azure#16966) Release of Device Update for IoT Hub SDK for Python. (Azure#17005) Add Get-AllPackageInfoFromRepo (Azure#16947) Track1 package is incorrectly set as track2 (Azure#17075) [text analytics] add normalized_text (Azure#17074) Renaming with_token identity function (Azure#17066) Adapt to azure core's cloud event (Azure#17063) align perf tests with js (Azure#17069) [Perfstress][Storage] Added FileShare perf tests (Azure#15834) [formrecognizer] Adding custom forms perf test (Azure#16969) Fix LanguageShort typo (Azure#17068) sas creds updates (Azure#17065) [eventgrid] Fix Sample eh (Azure#17064) [Perfstress][Storage] Added Datalake perf tests (Azure#15861) [text analytics] Healthcare n-ary relations (Azure#16997) ServiceBus dict-representation acceptance and kwarg-update functionality (Azure#14807) [text analytics] add perf tests (Azure#17060) Add cloud event to core (Azure#16800) ...

Add cloud event to core

abe2389

ghost added the Azure.Core label Feb 18, 2021

extensions

9ed935d

rakshith91 marked this pull request as ready for review February 18, 2021 07:31

rakshith91 requested review from lmazuel and xiangyan99 as code owners February 18, 2021 07:31

rakshith91 requested a review from johanste February 18, 2021 07:31

rakshith91 added 2 commits February 18, 2021 10:14

raise on both

d11e02f

minor

ec3474c

xiangyan99 reviewed Feb 18, 2021

View reviewed changes

rakshith91 mentioned this pull request Feb 18, 2021

move cloud events to azure core #16661

Closed

rakshith91 commented Feb 18, 2021

View reviewed changes

sdk/core/azure-core/azure/core/messaging.py Outdated Show resolved Hide resolved

more changes

e658ce7

rakshith91 commented Feb 18, 2021

View reviewed changes

sdk/core/azure-core/azure/core/messaging.py Outdated Show resolved Hide resolved

Update sdk/core/azure-core/azure/core/messaging.py

50de6e6

johanste reviewed Feb 18, 2021

View reviewed changes

sdk/core/azure-core/azure/core/messaging.py Outdated Show resolved Hide resolved

johanste reviewed Feb 18, 2021

View reviewed changes

sdk/core/azure-core/azure/core/messaging.py Outdated Show resolved Hide resolved

johanste reviewed Feb 18, 2021

View reviewed changes

lmazuel reviewed Feb 18, 2021

View reviewed changes

sdk/core/azure-core/tests/test_messaging_cloud_event.py Show resolved Hide resolved

comments

9f3624b

rakshith91 added 3 commits February 19, 2021 12:06

changes

04acd47

test fix

f18f35d

test

9400e0e

johanste reviewed Feb 20, 2021

View reviewed changes

rakshith91 added 6 commits March 1, 2021 13:49

doc

e8b53b0

doc

3a16ca9

unconditional

37e887c

test fix

8e196ef

mypy

b4c726e

wrong import

3f49138

johanste reviewed Mar 2, 2021

View reviewed changes

johanste requested changes Mar 2, 2021

View reviewed changes

rakshith91 added 8 commits March 2, 2021 12:47

type annotations

22db8b8

data

6e9ce14

coment

43a79c1

assets

ba654a0

lint

4f9d80e

unnecessary none

0a3aa87

format

5051034

cast to str

c7afd6e

johanste reviewed Mar 3, 2021

View reviewed changes

johanste approved these changes Mar 3, 2021

View reviewed changes

remove cast

9613fd4

lmazuel approved these changes Mar 3, 2021

View reviewed changes

rakshith91 merged commit 303ff1f into Azure:master Mar 3, 2021

rakshith91 deleted the core_cloud_eventy branch March 3, 2021 18:20

dependabot bot mentioned this pull request Mar 17, 2021

Bump azure-core from 1.11.0 to 1.12.0 splunk/security_content#1256

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cloud event to core #16800

Add cloud event to core #16800

rakshith91 commented Feb 18, 2021 •

edited

Loading

xiangyan99 Feb 18, 2021

xiangyan99 commented Feb 18, 2021

johanste Feb 18, 2021

lmazuel Feb 18, 2021

rakshith91 Feb 18, 2021

iscai-msft commented Feb 19, 2021

azure-pipelines bot commented Feb 19, 2021

check-enforcer bot commented Feb 19, 2021

johanste Feb 20, 2021

rakshith91 Feb 21, 2021

johanste Feb 20, 2021

rakshith91 Feb 21, 2021

johanste Mar 2, 2021

johanste Mar 2, 2021

johanste Mar 2, 2021

johanste Mar 2, 2021

johanste left a comment

johanste Mar 3, 2021

rakshith91 Mar 3, 2021

johanste left a comment

Add cloud event to core #16800

Add cloud event to core #16800

Conversation

rakshith91 commented Feb 18, 2021 • edited Loading

Choose a reason for hiding this comment

xiangyan99 commented Feb 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iscai-msft commented Feb 19, 2021

azure-pipelines bot commented Feb 19, 2021

check-enforcer bot commented Feb 19, 2021

What is Check Enforcer?

Why am I getting this message?

What should I do now?

What if I am onboarding a new service?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johanste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johanste left a comment

Choose a reason for hiding this comment

rakshith91 commented Feb 18, 2021 •

edited

Loading