Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add webhooks #4863

Merged
merged 159 commits into from
Sep 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
159 commits
Select commit Hold shift + click to select a range
74dd8f2
add webhooks urls
sizov-kirill Jul 28, 2022
6142cac
add webhooks app in settings
sizov-kirill Jul 28, 2022
9aa1463
apps/webhooks: post, get
sizov-kirill Jul 29, 2022
b6032af
remove secret and useless fields during serialization
sizov-kirill Aug 1, 2022
e06e261
add constraint: not allow to have both project and organization id
sizov-kirill Aug 1, 2022
859a362
add filtering and searching fields; add patch, put, delete methods
sizov-kirill Aug 1, 2022
dd55458
migrations
sizov-kirill Aug 1, 2022
6bbed81
rest api tests: add webhook in dump_objects
sizov-kirill Aug 2, 2022
5660395
webhook/apps.py: add signals in appconfig
sizov-kirill Aug 2, 2022
75ad895
extend event type choice
sizov-kirill Aug 2, 2022
1cce895
add WebhookDelivery model
sizov-kirill Aug 2, 2022
898ec6f
webhook: update migrations
sizov-kirill Aug 2, 2022
33c49f2
engine/mixins: override updatemodelmixin for sending signals
sizov-kirill Aug 3, 2022
5929a9d
webhooks/serializers: exclude unnecessary fields during serialization…
sizov-kirill Aug 3, 2022
4cd5e0d
jobs view: use overridden mixin for sending update webhooks
sizov-kirill Aug 3, 2022
13fbabe
event types: delete non-existent events
sizov-kirill Aug 3, 2022
2faa9df
add signals file
sizov-kirill Aug 3, 2022
84b3993
WebhookDelivery model: delete data, updated_data fields; add redelive…
sizov-kirill Aug 4, 2022
5ff353b
Update Webhook serializers
sizov-kirill Aug 4, 2022
db15e3a
signals: add information about sender
sizov-kirill Aug 4, 2022
eb83692
Webhook view: pass owner info during creation, add GET webhook/{wid}/…
sizov-kirill Aug 4, 2022
ce5b732
add request and response fields for WebhookDelivery model
sizov-kirill Aug 4, 2022
0847036
delete WebhookDeliveryWriteSerializer
sizov-kirill Aug 5, 2022
a9164fc
add redeliver method
sizov-kirill Aug 5, 2022
db157c3
webhook serializer: events as a list
sizov-kirill Aug 5, 2022
6e6dd58
add ping event
sizov-kirill Aug 5, 2022
f656055
add webhook redis queue
sizov-kirill Aug 8, 2022
b29e3a3
add base tests
sizov-kirill Aug 10, 2022
1b81cff
different event types for different webhook types
sizov-kirill Aug 10, 2022
90c4ddb
GET available events method
sizov-kirill Aug 11, 2022
5d40d58
added initial version of list/create/update pages
klakhov Aug 11, 2022
b9a4586
fix event types validator
sizov-kirill Aug 11, 2022
8edbc63
fix validator for events fields
sizov-kirill Aug 11, 2022
f7cced4
add webhook queue
sizov-kirill Aug 11, 2022
5caf00d
add base tests for webhooks
sizov-kirill Aug 11, 2022
ef02a5b
add last_delivery_date and description fields
sizov-kirill Aug 11, 2022
af21b8a
fix update signal reciever
sizov-kirill Aug 11, 2022
d62eacb
Fixed headers
bsekachev Aug 12, 2022
9893b86
Some cvat-core refactoring
bsekachev Aug 12, 2022
85ff520
add more tests
sizov-kirill Aug 12, 2022
e38f679
Some code on cvat-core side
bsekachev Aug 12, 2022
8f43d08
Merge branch 'develop' into sk/add-webhooks
sizov-kirill Aug 12, 2022
98a9ef5
Merge branch 'kl/webhooks-ui' into sk/add-webhooks
bsekachev Aug 12, 2022
275f378
fix pylint warnings
sizov-kirill Aug 12, 2022
e24eb53
Implemented cvat-core part
bsekachev Aug 12, 2022
185abe1
Merge branch 'sk/add-webhooks' of github.com:cvat-ai/cvat into sk/add…
bsekachev Aug 12, 2022
b586cc6
Implemented webhooks adding
bsekachev Aug 12, 2022
725355d
Updating webhooks
bsekachev Aug 12, 2022
6013825
Implemented deleting
bsekachev Aug 12, 2022
d8a8f68
Implemented ping
bsekachev Aug 12, 2022
2d42da9
Using last delivery and last status
bsekachev Aug 12, 2022
6126eee
is active is enabled by default
bsekachev Aug 12, 2022
bbd81d2
add webhooks queue in launch.json settings
sizov-kirill Aug 12, 2022
3f29811
Implemented pagination
bsekachev Aug 12, 2022
db7298f
Merge branch 'sk/add-webhooks' of github.com:cvat-ai/cvat into sk/add…
bsekachev Aug 12, 2022
79e2849
Disabled some not implemented fields
bsekachev Aug 15, 2022
9d61c34
Adjusted styles
bsekachev Aug 15, 2022
85f52b3
Fixed webhooks pages out of an organization
bsekachev Aug 15, 2022
f45d654
add task_created and task_updated events; handle response from target…
sizov-kirill Aug 15, 2022
15d9216
Merge branch 'sk/add-webhooks' of https://github.com/cvat-ai/cvat int…
sizov-kirill Aug 15, 2022
2192e02
fix pylint warns
sizov-kirill Aug 15, 2022
972421e
Merge branch 'sk/add-webhooks' of github.com:cvat-ai/cvat into sk/add…
bsekachev Aug 15, 2022
9bfea72
Fixed some issues on editing page
bsekachev Aug 15, 2022
ede05f8
Merge branch develop into the sk/add-webhooks
Aug 26, 2022
233edc1
resolve conflicts
Aug 26, 2022
6024f26
Refactored styles on webhook list page
klakhov Aug 30, 2022
caadba1
Enabled 'select all' option and changed styles
klakhov Aug 31, 2022
2178759
add project_id as filter parameter
Sep 2, 2022
c7c49cc
fix unique constraint
Sep 5, 2022
958ba9b
use iam context during creation; fix unique constraint
Sep 5, 2022
1d3e4c9
added project webhooks ui
klakhov Sep 6, 2022
b9be372
IAM webhooks: create and update scopes
Sep 6, 2022
95bbf74
remove organization_id from post data
Sep 6, 2022
e4eaaf5
add tests for POST and PATCH webhooks (iam point of view)
Sep 6, 2022
f5a135f
fix some projects rest api tests
Sep 6, 2022
fa10d38
Merge branch 'sk/add-webhooks' of https://github.com/opencv/cvat into…
Sep 6, 2022
addf8dc
add tests
Sep 9, 2022
18cfacf
IAM webhooks: delete, list, view scopes
Sep 9, 2022
cb3c256
update tests
Sep 9, 2022
55bcaee
Merge branch develop into the sk/add-webhooks
Sep 12, 2022
361ff6e
resolve conflicts
Sep 12, 2022
ac96885
Update WebhookDelivery model
Sep 12, 2022
7df0d68
Update WebhookDelivery serializer
Sep 12, 2022
673f169
Create WebhookDelivery object before sending webhook
Sep 12, 2022
ddfffcb
Return information about webhook delivery on `ping` request
Sep 12, 2022
2311df1
Fix Pylint warnings
Sep 12, 2022
e9486cf
Add new migrations
Sep 12, 2022
6c7d6f5
added ping delivery status polling
klakhov Sep 12, 2022
454cc45
updated status tooltips
klakhov Sep 13, 2022
1f59ae9
WebhookDelivery model: rename date fields
Sep 13, 2022
e676875
Extend list of filter fields
Sep 13, 2022
def9cf8
added search, sort, filtration
klakhov Sep 14, 2022
cbaf6ec
Add methods for getting project and organization ids
sizov-kirill Sep 15, 2022
a833999
Use additional serialization of instance instead of calling serialize…
sizov-kirill Sep 15, 2022
5ca0e33
fix typo
sizov-kirill Sep 15, 2022
de30443
REST API tests: refactor test projects
sizov-kirill Sep 15, 2022
3fcd1e8
Add limitation for number of webhooks
sizov-kirill Sep 15, 2022
cd67aad
Generate rego tests for webhooks
sizov-kirill Sep 15, 2022
9a51c68
Add cvat_worker_webhooks service
sizov-kirill Sep 16, 2022
97c0510
enabled secret and some event fields
klakhov Sep 16, 2022
c633f2f
Merge branch 'develop' into sk/add-webhooks
klakhov Sep 16, 2022
b2fee89
fixed cvat-core merge errors
klakhov Sep 16, 2022
9f947f0
Webhook events: task_updated, task_created, task_deleted
sizov-kirill Sep 16, 2022
bd36054
Fix event types
sizov-kirill Sep 16, 2022
caa3ec8
Merge branch 'sk/add-webhooks' of https://github.com/opencv/cvat into…
sizov-kirill Sep 16, 2022
ee2e6a2
fixed organization change and empty search
klakhov Sep 16, 2022
bd9b827
Add webhooks for all event types
sizov-kirill Sep 16, 2022
cdd55d3
Merge branch 'sk/add-webhooks' of https://github.com/opencv/cvat into…
sizov-kirill Sep 16, 2022
79f0705
Add secret signature
sizov-kirill Sep 16, 2022
b31b0fd
Add event_name validation
sizov-kirill Sep 16, 2022
4374665
Fix pylint errors
sizov-kirill Sep 16, 2022
661c5ea
apps/webhooks: clean migrations directory
sizov-kirill Sep 19, 2022
208865b
apps/webhooks: change event name pattern to '<action>:<resource>'
sizov-kirill Sep 19, 2022
c1059b4
tests: update after changing event name pattern for webhooks
sizov-kirill Sep 19, 2022
25529ac
updated ui event name pattern
klakhov Sep 19, 2022
fd46b69
Merge branch 'develop' into sk/add-webhooks
sizov-kirill Sep 19, 2022
1a13503
fixed rendering for new events, fixed jest tests
klakhov Sep 19, 2022
dcc62fd
apps/webhooks: add last_delivery_date field
sizov-kirill Sep 20, 2022
4841c0b
apps/webhooks: add webhook timeout
sizov-kirill Sep 20, 2022
2ac11a1
applied comments
klakhov Sep 20, 2022
72f41ab
apps/webhooks: fix order field for deliveries list
sizov-kirill Sep 22, 2022
7064f00
Add tests for webhooks functionality
sizov-kirill Sep 22, 2022
f1a12d2
Restore db before testing webhooks sender
sizov-kirill Sep 22, 2022
c9b3ad6
Fix pylint warnings
sizov-kirill Sep 23, 2022
b99a7f8
added jest tests
klakhov Sep 23, 2022
255d0a1
fixed reload of update webhook page
klakhov Sep 23, 2022
88ab3a7
Add and update license headers
sizov-kirill Sep 23, 2022
5eda102
iam/rules: fix webhooks csv rules
sizov-kirill Sep 23, 2022
ae0205c
iam/rules: fix webhook rego file
sizov-kirill Sep 23, 2022
3f021d5
iam/rules: update webhooks csv
sizov-kirill Sep 23, 2022
d23ba9e
iam/rules: update webhooks rego for view scope
sizov-kirill Sep 23, 2022
45b3120
iam/rules: add scipt that generate rego tests for webhooks
sizov-kirill Sep 23, 2022
879a4ad
iam/permissions: add project permissions for webhook
sizov-kirill Sep 23, 2022
470c2a4
Delete opa binary
sizov-kirill Sep 26, 2022
3b5cdda
iam/rules: fix webhook input description
sizov-kirill Sep 26, 2022
1489c00
apps/webhooks: update webhook event list
sizov-kirill Sep 26, 2022
3190474
apps/models: remove field type annotation
sizov-kirill Sep 26, 2022
137884b
Move webhook urls definition
sizov-kirill Sep 26, 2022
de6e2fa
added cypress tests
klakhov Sep 26, 2022
b04b884
applied comments to tests
klakhov Sep 26, 2022
9aa2962
Delete unused imports
sizov-kirill Sep 27, 2022
c3fc5b3
remove console.log
klakhov Sep 27, 2022
eb59ef5
apps/webhooks: limit size of response body from target_url
sizov-kirill Sep 27, 2022
e8b79cc
apps/webhooks: improve status field for delivery info
sizov-kirill Sep 27, 2022
b31c340
apps/webhooks: add default value for status_code field
sizov-kirill Sep 27, 2022
3039a78
apps/webhooks: add migration
sizov-kirill Sep 27, 2022
35420e3
apps/webhooks: fix code style
sizov-kirill Sep 27, 2022
29a5321
apps/iam/permissions: make error message more correct
sizov-kirill Sep 27, 2022
ffd79b6
Merge branch 'sk/add-webhooks' of https://github.com/opencv/cvat into…
sizov-kirill Sep 27, 2022
5a51f56
apps/webhooks: reduce response size limit
sizov-kirill Sep 27, 2022
37ab9f6
tests/rest_api: declare port for webhook container as an env variable
sizov-kirill Sep 27, 2022
f29ba2b
Merge branch develop into the sk/add-webhooks
sizov-kirill Sep 27, 2022
353c6fc
adjusted webhook last status
klakhov Sep 28, 2022
8183cb9
apps/webhooks: remove unnecessary `set` for events types
sizov-kirill Sep 28, 2022
bc0a6b8
apps/webhooks: fix arguments for `ping` receiver function
sizov-kirill Sep 28, 2022
f7a607b
Merge branch 'sk/add-webhooks' of https://github.com/opencv/cvat into…
sizov-kirill Sep 28, 2022
2f588e0
completely removed eslint-plugin-header
klakhov Sep 28, 2022
d745d8d
Merge branch 'develop' into sk/add-webhooks
sizov-kirill Sep 28, 2022
05c2cd7
tests/rest_api: resolve conflicts
sizov-kirill Sep 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ module.exports = {
'.eslintrc.js',
'lint-staged.config.js',
],
plugins: ['@typescript-eslint', 'security', 'no-unsanitized', 'eslint-plugin-header', 'import'],
plugins: ['@typescript-eslint', 'security', 'no-unsanitized', 'import'],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@klakhov , why have we deleted 'eslint-plugin-header'?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'eslint-plugin-header' is not working with our current 'double-header' (doesnt support optional headers), it seems we need another solution

extends: [
'eslint:recommended', 'plugin:security/recommended', 'plugin:no-unsanitized/DOM',
'airbnb-base', 'plugin:import/errors', 'plugin:import/warnings',
Expand Down
20 changes: 20 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,25 @@
"env": {},
"console": "internalConsole"
},
{
"name": "server: RQ - webhooks",
"type": "python",
"request": "launch",
"justMyCode": false,
"stopOnEntry": false,
"python": "${command:python.interpreterPath}",
"program": "${workspaceRoot}/manage.py",
"args": [
"rqworker",
"webhooks",
"--worker-class",
"cvat.simpleworker.SimpleWorker",
],
"django": true,
"cwd": "${workspaceFolder}",
"env": {},
"console": "internalConsole"
},
{
"name": "server: git",
"type": "python",
Expand Down Expand Up @@ -285,6 +304,7 @@
"server: django",
"server: RQ - default",
"server: RQ - low",
"server: RQ - webhooks",
"server: RQ - scheduler",
"server: git",
]
Expand Down
37 changes: 36 additions & 1 deletion cvat-core/src/api-implementation.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
// Copyright (C) 2019-2022 Intel Corporation
// Copyright (C) 2022 CVAT.ai Corporation
//
// SPDX-License-Identifier: MIT

Expand All @@ -17,13 +18,14 @@ const config = require('./config');
checkObjectType,
} = require('./common');

const User = require('./user');
const User = require('./user').default;
const { AnnotationFormats } = require('./annotation-formats');
const { ArgumentError } = require('./exceptions');
const { Task, Job } = require('./session');
const Project = require('./project').default;
const { CloudStorage } = require('./cloud-storage');
const Organization = require('./organization');
const Webhook = require('./webhook').default;

function implementAPI(cvat) {
cvat.plugins.list.implementation = PluginRegistry.list;
Expand Down Expand Up @@ -286,6 +288,39 @@ const config = require('./config');
config.organizationID = null;
};

cvat.webhooks.get.implementation = async (filter) => {
checkFilter(filter, {
page: isInteger,
id: isInteger,
bsekachev marked this conversation as resolved.
Show resolved Hide resolved
projectId: isInteger,
filter: isString,
search: isString,
sort: isString,
});

checkExclusiveFields(filter, ['id', 'projectId'], ['page']);
const searchParams = {};
for (const key of Object.keys(filter)) {
if (['page', 'id', 'filter', 'search', 'sort'].includes(key)) {
searchParams[key] = filter[key];
}
}

if (filter.projectId) {
if (searchParams.filter) {
const parsed = JSON.parse(searchParams.filter);
searchParams.filter = JSON.stringify({ and: [parsed, { '==': [{ var: 'project_id' }, filter.projectId] }] });
} else {
searchParams.filter = JSON.stringify({ and: [{ '==': [{ var: 'project_id' }, filter.projectId] }] });
}
}

const webhooksData = await serverProxy.webhooks.get(searchParams);
const webhooks = webhooksData.map((webhookData) => new Webhook(webhookData));
webhooks.count = webhooksData.count;
return webhooks;
};

return cvat;
}

Expand Down
25 changes: 24 additions & 1 deletion cvat-core/src/api.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
// Copyright (C) 2019-2022 Intel Corporation
// Copyright (C) 2022 CVAT.ai Corporation
//
// SPDX-License-Identifier: MIT

Expand All @@ -23,14 +24,15 @@ function build() {
const { FrameData } = require('./frames');
const { CloudStorage } = require('./cloud-storage');
const Organization = require('./organization');
const Webhook = require('./webhook').default;

const enums = require('./enums');

const {
Exception, ArgumentError, DataError, ScriptingError, PluginError, ServerError,
} = require('./exceptions');

const User = require('./user');
const User = require('./user').default;
const pjson = require('../package.json');
const config = require('./config');

Expand Down Expand Up @@ -843,6 +845,26 @@ function build() {
return result;
},
},
/**
* This namespace could be used to get webhooks list from the server
* @namespace webhooks
* @memberof module:API.cvat
*/
webhooks: {
/**
* Method returns a list of organizations
* @method get
* @async
* @memberof module:API.cvat.webhooks
* @returns {module:API.cvat.classes.Webhook[]}
* @throws {module:API.cvat.exceptions.PluginError}
* @throws {module:API.cvat.exceptions.ServerError}
*/
async get(filter: any) {
const result = await PluginRegistry.apiWrapper(cvat.webhooks.get, filter);
return result;
},
},
/**
* Namespace is used for access to classes
* @namespace classes
Expand All @@ -864,6 +886,7 @@ function build() {
FrameData,
CloudStorage,
Organization,
Webhook,
},
};

Expand Down
3 changes: 2 additions & 1 deletion cvat-core/src/comment.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
// Copyright (C) 2020-2022 Intel Corporation
// Copyright (C) 2022 CVAT.ai Corporation
//
// SPDX-License-Identifier: MIT

const User = require('./user');
const User = require('./user').default;
const { ArgumentError } = require('./exceptions');

/**
Expand Down
26 changes: 26 additions & 0 deletions cvat-core/src/enums.ts
Original file line number Diff line number Diff line change
Expand Up @@ -438,3 +438,29 @@ export enum StorageLocation {
LOCAL = 'local',
CLOUD_STORAGE = 'cloud_storage',
}

/**
* Webhook source types
* @enum {string}
* @name WebhookSourceType
* @memberof module:API.cvat.enums
* @property {string} ORGANIZATION 'organization'
* @property {string} PROJECT 'project'
* @readonly
*/
export enum WebhookSourceType {
ORGANIZATION = 'organization',
PROJECT = 'project',
}

/**
* Webhook content types
* @enum {string}
* @name WebhookContentType
* @memberof module:API.cvat.enums
* @property {string} JSON 'json'
* @readonly
*/
export enum WebhookContentType {
JSON = 'application/json',
}
3 changes: 2 additions & 1 deletion cvat-core/src/issue.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
// Copyright (C) 2020-2022 Intel Corporation
// Copyright (C) 2022 CVAT.ai Corporation
//
// SPDX-License-Identifier: MIT

const quickhull = require('quickhull');

const PluginRegistry = require('./plugins').default;
const Comment = require('./comment');
const User = require('./user');
const User = require('./user').default;
const { ArgumentError } = require('./exceptions');
const serverProxy = require('./server-proxy').default;

Expand Down
3 changes: 2 additions & 1 deletion cvat-core/src/organization.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
// Copyright (C) 2021-2022 Intel Corporation
// Copyright (C) 2022 CVAT.ai Corporation
//
// SPDX-License-Identifier: MIT

Expand All @@ -8,7 +9,7 @@ const { MembershipRole } = require('./enums');
const { ArgumentError, ServerError } = require('./exceptions');
const PluginRegistry = require('./plugins').default;
const serverProxy = require('./server-proxy').default;
const User = require('./user');
const User = require('./user').default;

/**
* Class representing an organization
Expand Down
2 changes: 1 addition & 1 deletion cvat-core/src/project.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import { Storage } from './storage';
const PluginRegistry = require('./plugins').default;
const { ArgumentError } = require('./exceptions');
const { Label } = require('./labels');
const User = require('./user');
const User = require('./user').default;
const { FieldUpdateTrigger } = require('./common');

/**
Expand Down
Loading