Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Domain Tools Iris - First Release #4445

Merged
merged 32 commits into from
Nov 6, 2019
Merged

Domain Tools Iris - First Release #4445

merged 32 commits into from
Nov 6, 2019

Conversation

ChuckWoodraska
Copy link
Contributor

Status

Ready

Required version of Demisto

4.1.0

Does it break backward compatibility?

  • No

Must have

  • Tests
  • Documentation (with link to it) Not sure where this goes in the repo.
  • Code Review

Additional changes

Add List named tags for the IrisTags playbook

Technical writer review

Mention and link to the files that require a technical writer review.

@Itay4 Itay4 requested review from yuvalbenshalom and removed request for Itay4 September 22, 2019 06:02
@yuvalbenshalom yuvalbenshalom requested review from Itay4 and removed request for yuvalbenshalom September 26, 2019 11:12
Copy link
Contributor

@Itay4 Itay4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👍
A few general comments:

  1. Please generate integration documentation
  2. Please add tests for all content entities (playbooks and scripts) and not only for the integration.
  3. Please outputs types to all commands outputs.
  4. Add Python 3 docker image.
  5. File names should be same the directory name, so if the directory name is DomainTools_Iris, so should be the Python, YAML, etc.
  6. Please use ' instead of " in the code.
  7. Please change the command prefix to something like domaintools- or domaintoolsiris-
  8. Please don't access dictionary keys directly, use .get() instead.
  9. Please don't access list indices without checking length first.
  10. Please add Google style docstrings.
  11. Split each function to two, command and request. For more details please have a look here
  12. No need for the format functions, please move the parsing logic code to be inside the command functions instead.
  13. Please add DBot Score outputs.
  14. Make sure all entities id and name are the same, and not GUIDs
  15. Did not review playbooks and scripts yet

Integrations/DomainTools_Iris/CHANGELOG.md Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/iris_description.md Outdated Show resolved Hide resolved
TestPlaybooks/playbook-DomainTools_Iris_Test.yml Outdated Show resolved Hide resolved
TestPlaybooks/playbook-DomainTools_Iris_Test.yml Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/iris.yml Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/iris.py Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/iris.py Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/iris.py Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/iris.py Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/iris.py Outdated Show resolved Hide resolved
@ChuckWoodraska
Copy link
Contributor Author

I believe all the suggested changes for the integration are done other than what I have left unresolved. Still working on updating playbooks and scripts due to changes in the integration.

@ChuckWoodraska
Copy link
Contributor Author

Also can you point to examples of testing automation scripts and playbooks? I'm not seeing much in the docs on the best way to do this.

@Itay4
Copy link
Contributor

Itay4 commented Oct 12, 2019

Also can you point to examples of testing automation scripts and playbooks? I'm not seeing much in the docs on the best way to do this.

Example for automation script test: ParseEmailFiles

Example for playbook test: Phishing playbook

Copy link
Contributor

@Itay4 Itay4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, some general comments:

  1. Please use Python 3 docker image.
  2. Some of the docstrings missing details about the arguments, please improve these.
  3. Some of the context outputs have Unknown type, please change to the relevant type, if possible.
  4. Please add the DBotScore object to the context outputs in the YAML.
  5. Please update the command names in the test playbook.

Integrations/DomainTools_Iris/DomainTools_Iris.py Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/DomainTools_Iris.py Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/DomainTools_Iris.py Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/DomainTools_Iris.py Outdated Show resolved Hide resolved
TestPlaybooks/playbook-DomainTools_Iris_Test.yml Outdated Show resolved Hide resolved
TestPlaybooks/playbook-DomainTools_Iris_Test.yml Outdated Show resolved Hide resolved
TestPlaybooks/playbook-DomainTools_Iris_Test.yml Outdated Show resolved Hide resolved
Integrations/DomainTools_Iris/DomainTools_Iris.py Outdated Show resolved Hide resolved
@ChuckWoodraska
Copy link
Contributor Author

Is the repo for docker images an entirely different repo? Or where does that image/dockerfile have to be?

@Itay4
Copy link
Contributor

Itay4 commented Oct 16, 2019

Is the repo for docker images an entirely different repo? Or where does that image/dockerfile have to be?

you can see the python 3 docker image here
in order to use it in your integration, you need to specify in the yml dockerimage: demisto/python3:3.7.4.2718

Copy link
Contributor

@idovandijk idovandijk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for providing this contribution. I could find use to some of your playbooks, but I have some comments and there are things that need to be fixed to meet our standards and enter Demisto as out-of-the-box playbooks. I commented on the first lane of each relevant playbook, as commenting in exact areas in the YML can confuse. I added screenshots and task numbers for your convenience. There are also some general points for the playbooks:

  • Please provide meaningful descriptions for all playbooks and all tasks. If you write them for your integration commands, then upon creating the tasks, the description will be used automatically and save you the time of re-writing it.

  • Please set the "Auto extract indicators" setting of each task according to what fits that task best. You can read more about this setting in our documentation. Generally, this determines whether indicators should be extracted and have their reputation calculated, from the data that you print to the war-room in that specific task (its "results", not "outputs"). Sometimes data printed to the war-room does not need this functionality and it unnecessarily extends playbook run-time and provides excessive context. A rule of thumb is to set it to "none" unless that task prints to the war-room some interesting indicators on which we want to run automatically !ip, !domain and so on, in the background.
    image

  • Please change the task names to go hand in hand with our conventions. For example, condition tasks should be questions.
    You can refer to these playbooks, which meet our standards and are good examples, to see how we name different tasks: IP Enrichment - Internal - Generic v2, Extract Indicators From File - Generic v2. Note capitalization of letters and the meaningfulness and conciseness of the names.

  • Please replace all "DT" (things like ${something}) to our normal selectors, and add the "Unique" operator where possible and applicable. This prevents mistakes and data duplication. Do this for inputs of tasks and for inputs of the playbooks.
    image

image

image

image

DT should only be used in cases where complex logic unsupported by our filters and transformers is needed.

  • For all playbooks that work on input data, always check for the existence of that data before working with it. Never assume inputs are present, or the tasks will fail whenever an input is empty. The example playbooks I provided above demonstrate that as well (where we check if a file exists, or if an IP exists).

Thanks again for your efforts.

Playbooks/playbook-DomainTools_Domain_Auto_Enrichment.yml Outdated Show resolved Hide resolved
Playbooks/playbook-DomainTools_Guided_Pivot_Enrichment.yml Outdated Show resolved Hide resolved
Playbooks/playbook-DomainTools_Iris_Tags.yml Outdated Show resolved Hide resolved
@Itay4
Copy link
Contributor

Itay4 commented Oct 22, 2019

@ChuckWoodraska please pull from the base branch

@CLAassistant
Copy link

CLAassistant commented Oct 22, 2019

CLA assistant check
All committers have signed the CLA.

@Itay4
Copy link
Contributor

Itay4 commented Oct 29, 2019

@ChuckWoodraska looks good, just note 1 comment i've unresolved
pretty much done on my side, waiting for @idovandijk approval and @kirbles19 review

@idovandijk
Copy link
Contributor

Playbook review complete. Final review by @kirbles19 now

@demisto demisto deleted a comment Nov 4, 2019
@ChuckWoodraska
Copy link
Contributor Author

Is there an ETA for the final review being done?

@Itay4
Copy link
Contributor

Itay4 commented Nov 5, 2019

Is there an ETA for the final review being done?

mid next week

@richardbluestone
Copy link
Contributor

All docs have been reviewed.

@Itay4 Itay4 merged commit 1a86001 into demisto:ChuckWoodraska_master_base Nov 6, 2019
Itay4 pushed a commit that referenced this pull request Nov 6, 2019
* Domain Tools Iris - First Release (#4445)

* DomainTools Iris Integration - Initial Release

* Updated after failing Circle CI validation steps mostly dealing with comments and descriptions.

* Updated after failing Circle CI validation steps mostly dealing with comments, descriptions, and mypy errors.

* mypy and subtype update for integration.

* Integration updates to take into consideration the reviewers suggestions.

* Apply suggestions from code review

Co-Authored-By: Itay Keren <[email protected]>

* Integration, script, and playbook updates to take into consideration the reviewers suggestions.

* Trying to appease circle ci

* Trying to appease circle ci

* Take out unused playbooks and automation scripts. Update other playbooks.

* Fix circle ci errors

* Fix circle ci errors

* Fix circle ci errors dealing with type hints

* improve imports

* add type ignore

* remove typing import

* import dict

* import any

* update docker image

* Apply suggestions from code review

Co-Authored-By: Itay Keren <[email protected]>

* Requested updates from reviewer of the integration.

* Take out unused variable.

* Updated playbooks.

* Remove return error. Only raise exception.

* Update default inputs for pivot playbook.

* Update DomainTools_Iris.yml

Reviewed the yml file

* Update DomainTools_Iris.yml

* Update DomainTools_Iris_description.md

* Update playbook-Indicator_Pivoting-DomainTools_Iris.yml

Updated the file

* change integration id in conf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Contribution Thank you! Contributions are always welcome! docs-approved
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants