Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117

Abhishek-Dev09-zz · 2020-07-16T18:10:56Z

It will upload all .whl,.ABOUT, .NOTICE,.LICENCE files of thirdparty repo to new repo as an asset of specific OS and python and also generate archive for each Oses and all python

Release them as asset on Github for #2136

Signed-off-by: Abhishek Kumar [email protected]

Fixes #2068 aboutcode-org/skeleton#50 #2072

It will upload all .whl,.ABOUT, .NOTICE,.LICENCE files of thirdparty repo to github new repo as an asset of specific OS and python 3 version and also generate archive for each Oses and all python and release them an asset on github. Signed-off-by: Abhishek Kumar <[email protected]>

github_release.py

steven-esser

Can we put these in etc/scripts/ like in #2118?

Abhishek-Dev09-zz · 2020-07-20T15:23:10Z

Can we put these in etc/scripts/ like in #2118?

Yes, you can run from anywhere no problem

pombredanne

Looking good!
I have a few nitpickings for your consideration

etc/scripts/deps_archive.py

etc/scripts/github_release.py

pombredanne · 2020-07-21T17:19:35Z

etc/scripts/deps_archive.py

+    """
+    Generate an archive for dependencies for specific OS and
+    given version of python by taking directory as an input.
+    """


Can you remind me why and how this archive would be used?

It will generate an archive for each os and each python. For example - macOS_py36.tar.gz, win3_py36.zip, win64_py.36zip, linux_py36.tar.gz . For more info see PR description

I still do not know what these archives will be used for.... Are you saying that you plan to upload one such archive per OS/Python/scancode version to a GH release?
That archive will not be usable as a pip find-link then. And if there are dupes between OSes... that is going to be problematic. The archives we want to upload would be something built with etc/release/ scripts instead... e.g. full sdist with a streamlined thirparty

But you said earlier One archive for each OS/Python combo should be created and each should have its exact set of pinned requirements included in the built archive, no more and no less.

Yes, but I would prefer to have the sdist being use for these archives that's using a proper manifest and does not include things we do not want there. For this you should expand on the release script instead

FYI , this will give you 1 directory along with tar.gz ,if you are on linux /macOS.
if you are on window, you get 1 directory along with zip.

BTW i got directory of wheels of 30.1 MB and tar.gz of 28 MB on my macOS.

etc/scripts/deps_archive.py

Signed-off-by: Abhishek Kumar <[email protected]>

pombredanne · 2020-07-25T12:10:59Z

etc/scripts/deps_archive.py

+from __future__ import print_function
+
+import argparse
+from commoncode.system import on_windows


Sort your imports: commoncode is not from the stdlib, but our own code and should go in a separate block.
See for instance https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/cargo.py#L36

pombredanne · 2020-07-25T12:13:06Z

etc/scripts/deps_archive.py

+deps_archive.py \\
+  --input thirdparty \\
+  --req requirements.txt
+  --output_file macOS_py36 \\


Please remove that as it does help readability.

Signed-off-by: Abhishek Kumar <[email protected]>

pombredanne · 2020-07-26T15:07:13Z

etc/scripts/deps_archive.py

+    root_dir = os.path.abspath(archive_name)
+    output_dir = os.path.abspath(archive_name)
+    if on_windows:
+        make_archive(output_dir, 'zip', root_dir)


I am stil not clear about what these archives wrt. to what we need as sdist

here you get all wheel/sdist deps packed like sdist file that required for ur python/os.

Signed-off-by: Philippe Ombredanne <[email protected]>

Do not spawn a process but call a function instead. Clarify variable and options names and their help text. Use access token from a cli option or an environment variable. Signed-off-by: Philippe Ombredanne <[email protected]>

Abhishek-Dev09-zz · 2020-07-26T23:07:06Z

etc/scripts/github_release.py

@@ -27,127 +27,138 @@
 from __future__ import print_function

 import argparse
-from fnmatch import fnmatchcase


why you remove this?

Abhishek-Dev09-zz · 2020-07-26T23:09:06Z

etc/scripts/github_release.py

 from commoncode.fileutils import resource_iter

-python_version = str(sys.version_info[0]) + str(sys.version_info[1])


Why remove this section ,it will help you filter the python 3 files if your directory contains multiple python deps.

Abhishek-Dev09-zz · 2020-07-26T23:11:07Z

etc/scripts/github_release.py

-        or fnmatchcase(files, py_abi)
-        or (
-            fnmatchcase(files, '*tar.gz*')
-            and not fnmatchcase(files, '*py2-ipaddress-3.4.1.tar.gz*')


It will upload unwanted file also

Abhishek-Dev09-zz · 2020-07-26T23:13:08Z

etc/scripts/github_release.py

-        '--body-string',
-        help='Required : Text describing the release. Ignored if the release already exists.',
+        '--token',
+        help=TOKEN_HELP,


great detail here.

Abhishek-Dev09-zz · 2020-07-26T23:15:03Z

etc/scripts/github_release.py

-
-
-def main_with_args(args: str) -> None:
+    api = grr.GithubApi(


Either you call API or cmd . result will be same.

Abhishek-Dev09-zz · 2020-07-26T23:16:31Z

etc/scripts/github_release.py

        type=str,
-        default='10',
+        required=False,


no need to mention that

Abhishek-Dev09-zz · 2020-07-26T23:17:07Z

etc/scripts/github_release.py

        type=str,
-        required=True,
+        required=False,


no need to mention, by default it is false

Abhishek-Dev09-zz · 2020-07-26T23:18:18Z

etc/scripts/github_release.py



-def main() -> None:
+def main():


Why you remove anotations?

pombredanne

I am still not 100% clear about the deps archive and these will be used. But I am merging now so we can get a clearer picture and improve from that base.

Abhishek-Dev09-zz force-pushed the #295-github_release branch 2 times, most recently from ee13ae6 to a1c0500 Compare July 16, 2020 18:16

Abhishek-Dev09-zz force-pushed the #295-github_release branch from a1c0500 to f08eacf Compare July 16, 2020 18:18

Abhishek-Dev09-zz commented Jul 18, 2020

View reviewed changes

github_release.py Outdated Show resolved Hide resolved

steven-esser reviewed Jul 20, 2020

View reviewed changes

pombredanne requested changes Jul 21, 2020

View reviewed changes

Move all script to etc/scripts

89e957e

Signed-off-by: Abhishek Kumar <[email protected]>

Abhishek-Dev09-zz force-pushed the #295-github_release branch from e0ee198 to 89e957e Compare July 21, 2020 18:49

Use abspath instead of expanduser

90bff2b

Signed-off-by: Abhishek Kumar <[email protected]>

Abhishek-Dev09-zz requested a review from pombredanne July 23, 2020 05:52

Format the script properly

2e80d15

Signed-off-by: Abhishek Kumar <[email protected]>

pombredanne reviewed Jul 25, 2020

View reviewed changes

Sort imports,remove examples,use proper variable name

393fad4

Signed-off-by: Abhishek Kumar <[email protected]>

Abhishek-Dev09-zz requested review from pombredanne and steven-esser July 26, 2020 05:10

pombredanne mentioned this pull request Jul 26, 2020

Automate the creation of requirement.txt and fetch deps #2118

Merged

Improve the description and documentation of function

d0e1386

Signed-off-by: Abhishek Kumar <[email protected]>

pombredanne reviewed Jul 26, 2020

View reviewed changes

pombredanne added 2 commits July 26, 2020 17:16

Improve help texts

27c4e86

Signed-off-by: Philippe Ombredanne <[email protected]>

Improve release creation and upload aboutcode-org#2072

c062d21

Do not spawn a process but call a function instead. Clarify variable and options names and their help text. Use access token from a cli option or an environment variable. Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne force-pushed the #295-github_release branch from 2e6a01f to c062d21 Compare July 26, 2020 21:29

Abhishek-Dev09-zz commented Jul 26, 2020

View reviewed changes

etc/scripts/github_release.py

type=str,

default='10',

required=False,

Copy link

Author

Abhishek-Dev09-zz Jul 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to mention that

Abhishek-Dev09-zz commented Jul 26, 2020

View reviewed changes

etc/scripts/github_release.py

type=str,

required=True,

required=False,

Copy link

Author

Abhishek-Dev09-zz Jul 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to mention, by default it is false

Abhishek-Dev09-zz commented Jul 26, 2020

View reviewed changes

etc/scripts/github_release.py

def main() -> None:

def main():

Copy link

Author

Abhishek-Dev09-zz Jul 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you remove anotations?

This was referenced Jul 27, 2020

Dev utility: build and download the sdist and wheels of a single package and all its dependencies #2087

Closed

Dev utility: Create release scripts to build archives #2072

Closed

End to end approach for third-party packages handlings in ScanCode #2135

Closed

pombredanne approved these changes Jul 27, 2020

View reviewed changes

pombredanne merged commit 07685a8 into aboutcode-org:develop Jul 27, 2020

Abhishek-Dev09-zz deleted the #295-github_release branch July 27, 2020 17:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117

Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117

Abhishek-Dev09-zz commented Jul 16, 2020 •

edited by pombredanne

Loading

steven-esser left a comment

Abhishek-Dev09-zz commented Jul 20, 2020

pombredanne left a comment

pombredanne Jul 21, 2020

Abhishek-Dev09-zz Jul 21, 2020

pombredanne Jul 22, 2020

Abhishek-Dev09-zz Jul 22, 2020

pombredanne Jul 23, 2020

Abhishek-Dev09-zz Jul 23, 2020

pombredanne Jul 25, 2020

pombredanne Jul 25, 2020

pombredanne Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

Abhishek-Dev09-zz Jul 26, 2020

pombredanne left a comment

		from commoncode.fileutils import resource_iter

		python_version = str(sys.version_info[0]) + str(sys.version_info[1])

Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117

Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117

Conversation

Abhishek-Dev09-zz commented Jul 16, 2020 • edited by pombredanne Loading

steven-esser left a comment

Choose a reason for hiding this comment

Abhishek-Dev09-zz commented Jul 20, 2020

pombredanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pombredanne left a comment

Choose a reason for hiding this comment

Abhishek-Dev09-zz commented Jul 16, 2020 •

edited by pombredanne

Loading