Shotgrid: Add production beta of shotgrid integration #2921

victorbartel · 2022-03-18T16:38:54Z

Brief description

Add shotgrid module

Description

The purpose of this PR is to add a brand-new module which allows synchronizing data from Shotgrid to Openpype.

Additional info

This module is still under development and is not in its final shape. Main features to be covered are:

Batch synchronization(initialise and ingest Shotgrid data into Avalon format)
Publishing from Openpype to Shotgrid
~~Incremental event synchronisation (Shotgrid to Avalon)~~

Documentation (add "type: documentation" label)

module documentation
server documentation

Testing notes:

start with this tools/run_tray.sh
configure project settings under shotgrid section
install and run leecher server part

openpype/modules/deadline/plugins/publish/submit_publish_job.py

BigRoy

Haven't tested the module but did a quick glance over the code changes. Commented some things that stood out and also had these notes:

I feel like the addition of the following files don't belong to this PR:

This file seems to be doing nothing of use:

openpype/modules/shotgrid/hooks/post_shotgrid_changes.py

The file openpype/settings/entities/enum_entity.py has a lot of code style changes unrelated to the "logic change". I'm fine with the changes since a lot of them help readability. Just wanted to note that it tweaks quite a few lines of code that aren't absolutely required for this PR. Likely it's an automated thing from your code editor.

openpype/modules/shotgrid/plugins/publish/collect_shotgrid_session.py

BigRoy · 2022-03-24T10:21:50Z

openpype/modules/shotgrid/lib/settings.py

+def get_shotgrid_servers() -> Dict[str, Any]:
+    return get_shotgrid_settings().get("shotgrid_settings", {})
+
+
+def get_leecher_backend_url() -> str:
+    return get_shotgrid_settings().get("leecher_backend_url")
+
+
+def filter_projects_by_login() -> bool:
+    return bool(get_shotgrid_settings().get("filter_projects_by_login", False))


These functions won't be very optimized - it might make sense more to instead add the result of get_shotgrid_settings as an argument. That way you don't need constantly requery the database for the settings (which can be quite a lot of data to query).

It's not too many calls currently that'd get reduced - but it'd remove a call here since the full settings just got queried a few lines above.

Thoughts?

These functions won't be very optimized

Premature optimization is the root of all evil (or at least most of it) in programming - Donald Knuth

We are talking about usage of MongoDB, which is reputed of being highly performant and has strongly optimized throughput.
Despite the fact that it's a premature optimization, I've added LRU cache for those functions.

openpype/modules/deadline/plugins/publish/submit_publish_job.py

victorbartel · 2022-03-25T09:45:29Z

The file openpype/settings/entities/enum_entity.py has a lot of code style changes unrelated to the "logic change".

I would like to avoid those stylish changes, but the hound didn't leave us any choice with its "lines too long" checks. So, black formatted.

BigRoy

The code updates look good to me - unfortunately I can't test it in production myself.

I did have a question about the Py3 type hints in the code base. I like it, however I'm not sure we can adopt that already. The collectors would run within the hosts - e.g. Maya 2020, with Py 2.7 - and would import code of the module and thus fails with SyntaxErrors. Or would that actually work currently? It might be nice to test the logic/functionality in a Py2 host to be sure. The same for f-string formatting.

It would be a bit of a code style loss if we'd need to preserve Py2 in that area - but we might still need to currently? Thoughts?

victorbartel · 2022-03-25T15:20:45Z

It would be a bit of a code style loss if we'd need to preserve Py2 in that area - but we might still need to currently? Thoughts?

Let me first accomplish few tests under Maya 2018 and come back here with more elements.

victorbartel · 2022-04-11T08:50:48Z

Hey @BigRoy, @mkolar,

We have made a lot of extra tests under Maya 2018/2020 in order to be sure that everything works fine. As a result, all non py27 compliant features were removed. The publishing works like a charm.

BigRoy · 2022-04-14T09:03:07Z

@pberto were you able to give this a go?

pberto · 2022-04-15T00:52:58Z

This is something we will test soon.

mkolar · 2022-04-19T08:20:28Z

@victorbartel thank you for the work on this. We're unable to test this internally at the moment, however as soon as we get a second opinion from @pberto or someone else, we're happy to move forward and merge this in, considering how well self contained it is!

pberto · 2022-04-19T08:48:26Z

We plan to test it the second week of MAY as we need to be on cutonmpr site.

iLLiCiTiT · 2022-06-14T16:18:49Z

openpype/modules/shotgrid/lib/server.py

+_LOG = Logger().get_logger("ShotgridModule.server")
+
+
+def find_linked_projects(email):


Unused function

iLLiCiTiT · 2022-06-14T16:36:57Z

openpype/modules/shotgrid/plugins/publish/collect_shotgrid_entities.py

+    return {}
+
+
+def _get_shotgrid_task(avalon_project, avalon_asset, avalon_task):


I have few questions regarding this function.

Why is shotgrid creating separated mongo database for data that should be in avalon database?
At least it looks it's done that way. I maybe missed that part of shotgrid implementation?

Why _id contains strings?

Few comments:
Please try avoid using avalon in variables and also be more specific about the content (e.g. asset_doc or task_name tells more about it's content), also pass project document just to use it's name can be simplified to pass just the name?

In my memory, it seemed simpler ans more maintainable for @victorbartel to store shotgrid data in another database and store in avalon on the asset that corresponds to the id of this database

That would make sense in SQL database, not in mongo.
Now you have to query twice from 2 different databases to get data related to the same entity. To get shotgun entity based on asset name you have to query it from avalon database to get it's id, then based on the id query it's shotgrid id from shotgrid database and then query it from shotgrid server? Performance of that with cloud mongo wouldn't be great.
And _id key is not ObjectId but string, which is not requirement, but the ObjectId is used for indexing in mongo collections.

openpype/modules/shotgrid/plugins/publish/collect_shotgrid_entities.py

poetry.lock

Update and revert poetry lock changes

Added shotgun-api3 source files to poetry lock

iLLiCiTiT · 2022-06-20T13:25:31Z

openpype/settings/defaults/system_settings/modules.json

@@ -131,6 +131,13 @@
            }
        }
    },
+    "shotgrid": {
+        "enabled": true,


The module should be disabled by default.

Suggested change

"enabled": true,

"enabled": false,

iLLiCiTiT · 2022-06-20T13:26:01Z

openpype/settings/defaults/system_settings/modules.json

@@ -131,6 +131,13 @@
            }
        }
    },
+    "shotgrid": {
+        "enabled": true,
+        "filter_projects_by_login": true,


This is unused setting right now

mkolar · 2022-06-20T17:08:21Z

@ClementHector I believe the last are the two small things regarding settings that @iLLiCiTiT commented about

…Pype into shotgrid_module

ClementHector · 2022-06-21T06:32:02Z

I made the changes for the settings, for the base in the mongo I will try to do this in the next two weeks.

iLLiCiTiT

There are few thing that require some attention, but that's for future PRs. I would say this is ready to merge.

mkolar · 2022-06-22T08:24:58Z

Brilliant. Thank yo guys for this. It took a long while, but is a brilliant addition to OpenPype. I'm happy to merge this.

BigRoy · 2022-06-22T08:40:28Z

Thanks for the major contribution. Would love to see some demo videos of this in action! :)

mkolar added Bump Minor Pull requests that update a dependency file type: feature Larger, user affecting changes and completely new things labels Mar 18, 2022

victorbartel force-pushed the shotgrid_module branch from cf47c91 to f258d1a Compare March 21, 2022 08:57

BigRoy reviewed Mar 24, 2022

View reviewed changes

openpype/modules/deadline/plugins/publish/submit_publish_job.py Outdated Show resolved Hide resolved

BigRoy reviewed Mar 24, 2022

View reviewed changes

victorbartel force-pushed the shotgrid_module branch from f258d1a to 1156b6f Compare March 25, 2022 08:28

BigRoy reviewed Mar 25, 2022

View reviewed changes

openpype/modules/deadline/plugins/publish/submit_publish_job.py Outdated Show resolved Hide resolved

BigRoy added the module: Shotgrid label Mar 25, 2022

victorbartel requested a review from BigRoy March 25, 2022 10:14

BigRoy reviewed Mar 25, 2022

View reviewed changes

victorbartel added 7 commits April 8, 2022 14:01

Add shotgrid module

0e96071

Fix hound complaints

00285ca

Fix hound Exception complaints

feae04f

Add fixes after first PR

521b562

Add fixes for hound

68941da

Remove an empty line

1214cb9

Reformat code after hound barking

d106ba4

victorbartel force-pushed the shotgrid_module branch from 7a14643 to d106ba4 Compare April 8, 2022 12:06

victorbartel and others added 5 commits April 8, 2022 15:12

Drawn back to the obsolete version of python

3c1ecfb

Make the publish work under maya 2018

3a3e552

Add memoize implementation for settings purposes

0c85093

Fix after hound notes

16dc433

Fix after hound notes, part 2

57386db

iLLiCiTiT reviewed Jun 14, 2022

View reviewed changes

openpype/modules/shotgrid/plugins/publish/collect_shotgrid_entities.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed Jun 14, 2022

View reviewed changes

poetry.lock Outdated Show resolved Hide resolved

iLLiCiTiT and others added 7 commits June 20, 2022 13:42

reverted poetry.lock changes

d49c2ba

Merge branch 'develop' into bugfix/rever_poetry_lock_changes

3485529

remove unused import

b4c6cb4

Remove unused code and add sugestions

151bf29

remove unused method

ad0aea0

use context to get shotgrid project id

50ec61f

update poetry lock

47d2a61

ClementHector force-pushed the shotgrid_module branch from 0163eae to 47d2a61 Compare June 20, 2022 12:14

ClementHector and others added 5 commits June 20, 2022 14:16

Merge pull request #6 from pypeclub/bugfix/rever_poetry_lock_changes

1fd8723

Update and revert poetry lock changes

remove unused import

ad3380b

poetry lock rebase

0417fb1

added shotgun-api3 source files to poetry lock

e5fbdd1

Merge pull request #7 from pypeclub/bugfix/last_poetry_fix

f85865a

Added shotgun-api3 source files to poetry lock

iLLiCiTiT reviewed Jun 20, 2022

View reviewed changes

ClementHector added 2 commits June 21, 2022 08:29

change default settings

144ef08

Merge branch 'shotgrid_module' of https://github.com/Ellipsanime/Open…

83db3de

…Pype into shotgrid_module

iLLiCiTiT approved these changes Jun 21, 2022

View reviewed changes

mkolar assigned ClementHector Jun 22, 2022

mkolar merged commit e2cf641 into ynput:develop Jun 22, 2022

mkolar changed the title ~~Add shotgrid module~~ Shotgrid: Add production beta of shotgrid integration Jun 22, 2022

mkolar mentioned this pull request Jun 22, 2022

Revert "Shotgrid: Add production beta of shotgrid integration" #3390

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shotgrid: Add production beta of shotgrid integration #2921

Shotgrid: Add production beta of shotgrid integration #2921

victorbartel commented Mar 18, 2022 •

edited by BigRoy

Loading

BigRoy left a comment

BigRoy Mar 24, 2022 •

edited

Loading

victorbartel Mar 25, 2022 •

edited

Loading

victorbartel commented Mar 25, 2022

BigRoy left a comment

victorbartel commented Mar 25, 2022

victorbartel commented Apr 11, 2022

BigRoy commented Apr 14, 2022

pberto commented Apr 15, 2022

mkolar commented Apr 19, 2022

pberto commented Apr 19, 2022 •

edited

Loading

iLLiCiTiT Jun 14, 2022

iLLiCiTiT Jun 14, 2022

ClementHector Jun 20, 2022

iLLiCiTiT Jun 20, 2022

iLLiCiTiT Jun 20, 2022

iLLiCiTiT Jun 20, 2022

mkolar commented Jun 20, 2022

ClementHector commented Jun 21, 2022

iLLiCiTiT left a comment •

edited

Loading

mkolar commented Jun 22, 2022

BigRoy commented Jun 22, 2022

		_LOG = Logger().get_logger("ShotgridModule.server")


		def find_linked_projects(email):

		return {}


		def _get_shotgrid_task(avalon_project, avalon_asset, avalon_task):

Shotgrid: Add production beta of shotgrid integration #2921

Shotgrid: Add production beta of shotgrid integration #2921

Conversation

victorbartel commented Mar 18, 2022 • edited by BigRoy Loading

Brief description

Description

Additional info

Documentation (add "type: documentation" label)

Testing notes:

BigRoy left a comment

Choose a reason for hiding this comment

BigRoy Mar 24, 2022 • edited Loading

Choose a reason for hiding this comment

victorbartel Mar 25, 2022 • edited Loading

Choose a reason for hiding this comment

victorbartel commented Mar 25, 2022

BigRoy left a comment

Choose a reason for hiding this comment

victorbartel commented Mar 25, 2022

victorbartel commented Apr 11, 2022

BigRoy commented Apr 14, 2022

pberto commented Apr 15, 2022

mkolar commented Apr 19, 2022

pberto commented Apr 19, 2022 • edited Loading

iLLiCiTiT Jun 14, 2022

Choose a reason for hiding this comment

iLLiCiTiT Jun 14, 2022

Choose a reason for hiding this comment

ClementHector Jun 20, 2022

Choose a reason for hiding this comment

iLLiCiTiT Jun 20, 2022

Choose a reason for hiding this comment

iLLiCiTiT Jun 20, 2022

Choose a reason for hiding this comment

iLLiCiTiT Jun 20, 2022

Choose a reason for hiding this comment

mkolar commented Jun 20, 2022

ClementHector commented Jun 21, 2022

iLLiCiTiT left a comment • edited Loading

Choose a reason for hiding this comment

mkolar commented Jun 22, 2022

BigRoy commented Jun 22, 2022

victorbartel commented Mar 18, 2022 •

edited by BigRoy

Loading

BigRoy Mar 24, 2022 •

edited

Loading

victorbartel Mar 25, 2022 •

edited

Loading

pberto commented Apr 19, 2022 •

edited

Loading

iLLiCiTiT left a comment •

edited

Loading