Automate requirements.txt creation #2086

Abhishek-Dev09-zz · 2020-06-23T01:43:21Z

This script is use to build deps for python 3.6 #295 and release the deps as assets on github

Fixes aboutcode-org/skeleton#50 #2068

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁

Signed-off-by: Abhishek Kumar [email protected]

This script is use to build deps for python 3.6 aboutcode-org#295 and release the deps as assets on github Signed-off-by: Abhishek Kumar <[email protected]>

Abhishek-Dev09-zz · 2020-07-07T07:14:04Z

@pombredanne : Output of github_release.py - https://github.com/Abhishek-Dev09/thirdparty/releases/tag/v1.0 . It is automated and generated by this script only . Now i focussing for py3.6 for now. So don't panic for py 3.7 & 3.8 . Once we done with 3.6 , we can easily move to other py version.

pombredanne · 2020-07-07T07:54:33Z

@Abhishek-Dev09 can you check your ABOUT files?
bin/about check thirdparty/

Abhishek-Dev09-zz · 2020-07-07T07:59:16Z

@Abhishek-Dev09 can you check your ABOUT files?
bin/about check thirdparty/

mymac@ABHISHEKs-MacBook-Air scancode-toolkit % bin/about check thirdparty/ zsh: no such file or directory: bin/about

pombredanne

Thanks!
I posted some comments for a start.

I think you need to design some Requirements object (see dparse or packaging or pyup for libraries that may be worth adopting such as https://github.com/pyupio/pyup/blob/master/pyup/requirements.py ) and not read/write continuously to files with hardcoded names.

pombredanne · 2020-07-07T07:55:42Z

etc/scripts/freeze_and_update_reqs.py

+import subprocess
+
+
+os.chdir(os.pardir) #go to etc


Why not keeping this in a function?

pombredanne · 2020-07-07T07:56:10Z

etc/scripts/freeze_and_update_reqs.py

+os.chdir(os.pardir)  # go to scancode-toolkit
+cwd = os.getcwd()
+print("Current working directory:", cwd) 
+fp = open("requirements.txt", "wb")


Why do you open/close here? It does nothing and is possibly a source of failures

It create empty file . We need that because without this this scripts not work. This script assume that we already have requirements.txt file.

pombredanne · 2020-07-07T07:56:52Z

etc/scripts/freeze_and_update_reqs.py

+def is_package_exist_with_hashes(package_name,hash_of_file):
+    """ Check if any line in the file contains package with same hashes """
+    # Open the file in read only mode
+    line_with_hash = package_name + " \\" + "\n" + "    --hash=sha256:"+ hash_of_file


Can you use .format() with a mapping? It makes things much easier to read

pombredanne · 2020-07-07T07:57:24Z

etc/scripts/freeze_and_update_reqs.py

+    """ Check if any line in the file contains package with same hashes """
+    # Open the file in read only mode
+    line_with_hash = package_name + " \\" + "\n" + "    --hash=sha256:"+ hash_of_file
+    with open("requirements.txt", 'r') as read_obj:


You need to use single vs. double quotes consistently. We use single quotes unless this is a doc string.

Please do it everywhere.

pombredanne · 2020-07-07T07:58:07Z

etc/scripts/freeze_and_update_reqs.py

+        for line in read_obj:
+            # For each line, check if line contains the package with same hashes
+            if line_with_hash in line:
+                return True
+    return False
+
+def is_package_exist_without_hashes(package_name):
+    """ Check if any line in the file contains package without same hash """
+    # Open the file in read only mode
+    line_with_hash = package_name + " \\" + "\n" 
+    with open("requirements.txt", 'r') as read_obj:
+        # Read all lines in the file one by one
+        for line in read_obj:


Can you find a better name than read_obj

i use name as file

Yes, but find a better name: read_obj means nothing. What is this file about? what does it contain?

pombredanne · 2020-07-07T08:00:16Z

etc/scripts/freeze_and_update_reqs.py

+    """ Check if any line in the file contains package without same hash """
+    # Open the file in read only mode
+    line_with_hash = package_name + " \\" + "\n" 
+    with open("requirements.txt", 'r') as read_obj:


Could you craft some kind of object for the requirements like some Requirement class and not open the requirements.txt file over and over?
Also the file may very likely be named some else than requirements.txt?

In open("requirements.txt", 'r') use instead open('requirements.txt') as reading text is the default alright.

…ode-org#295 This script copies all thirdparty files(required) and place it in repo as release assets. Signed-off-by: Abhishek Kumar <[email protected]>

pombredanne

Thank you! here are some comment for your consideration:

Can you review and fix the formatting for your code?
Having an a Requirement object of sorts would really make more sense that always opening and writing a file.
Please do not mix adding or removing wheels in this PR, focus instead on the scripts only
you need to add also tests

pombredanne · 2020-07-08T13:31:18Z

etc/scripts/freeze_and_update_reqs.py

+    return hash.hexdigest()
+
+def main():
+    os.chdir(os.pardir) #go to etc


You should accept an argument with a path to a requirements files instead of trying to change the working dir.

pombredanne · 2020-07-08T13:31:23Z

etc/scripts/freeze_and_update_reqs.py

+    os.chdir(os.pardir)  # go to scancode-toolkit
+    cwd = os.getcwd()
+    print("Current working directory:", cwd) 
+    fp= open('requirements.txt','w') #Create empty file because it must exits before opening the file in read mode.


Use a with context manager.

pombredanne · 2020-07-08T13:33:14Z

etc/scripts/freeze_and_update_reqs.py

+    cwd = os.getcwd()
+    print("Current working directory:", cwd) 
+    fp= open('requirements.txt','w') #Create empty file because it must exits before opening the file in read mode.
+    for subdir, dirs, files in os.walk(cwd+"/thirdparty"):


You should accept an argument for the thirdparty directory. Also it would be likely easier to understand if you load all file paths at once and iterate after. You should also reuse the commonode.fileutils functions

You talking about which function in fileutils?

resource_iter()

pombredanne · 2020-07-08T13:35:01Z

etc/scripts/freeze_and_update_reqs.py

+        for filename in files:
+            filepath = subdir + os.sep + filename
+            if filepath.endswith(".whl") and (fnmatch.fnmatchcase(filename, "*py3*") or fnmatch.fnmatchcase(filename, "*cp36*")):
+                name = filename.split('-')[0]


I would really prefer that you use that https://github.com/jwodder/wheel-filename as a library.

pombredanne · 2020-07-08T13:35:49Z

github_release.py

+def main():
+    cwd = os.getcwd()
+    print("Current working directory:", cwd) 
+    subprocess.run(["pip3","install","github-release-retry"])


Do not do that. Have a requirements file instead and a README.rst that explains what to do

pombredanne · 2020-07-08T13:36:29Z

github_release.py

+
+    os.environ ["GITHUB_TOKEN"] = token
+
+    tag =  input ("Enter tag_name :")


do not use input, instead use arguments for main() and accept command line arguments

pombredanne · 2020-07-08T13:38:40Z

etc/scripts/freeze_and_update_reqs.py

+                    append_requirement_file(package_name,hs)
+                else: add_package(package_name,hs)
+
+            elif filepath.endswith(".gz") and not filename.startswith("py2-"):


which filename would ever start with "py2-"? and what is endswith(".gz")? about?

py2-ipaddress.tar.gz

Why would you single out a specific package name? What's so special about this?

because my function would not work for py2-ipaddress.tar.gz but don't worry this package will be removed after dropping python 2.

your function should not have any Python-package specific hack. The problem is in your code then

pombredanne · 2020-07-09T12:07:56Z

@Abhishek-Dev09 can you check your ABOUT files?
bin/about check thirdparty/

mymac@ABHISHEKs-MacBook-Air scancode-toolkit % bin/about check thirdparty/ zsh: no such file or directory: bin/about

You need to be properly configured with ./configure ... if you ran it once about would be there

pombredanne · 2020-07-26T08:26:12Z

Hi: Is this PR obsolete and replaced by #2118 ? If yes, may you can close this one here?

Abhishek-Dev09-zz · 2020-07-26T12:08:54Z

Hi: Is this PR obsolete and replaced by #2118 ? If yes, may you can close this one here?

Yes, i m closing this

Uxn is a virtual machine which represents a personal computing playground, and is described here: https://100r.co/site/uxn.html Uxntal assembly language is described here: https://wiki.xxiivv.com/site/uxntal.html The demo code piano.tal is used with permission.

Script to atomate requirements.txt

4d44bd3

This script is use to build deps for python 3.6 aboutcode-org#295 and release the deps as assets on github Signed-off-by: Abhishek Kumar <[email protected]>

Abhishek-Dev09-zz force-pushed the #295-requirement.txt branch 2 times, most recently from 519d18a to fad3166 Compare July 7, 2020 07:11

pombredanne requested changes Jul 7, 2020

View reviewed changes

Abhishek-Dev09-zz force-pushed the #295-requirement.txt branch 4 times, most recently from cf10c87 to d20fede Compare July 7, 2020 22:17

Expose epre-built wheels, tarballs and ABOUT files as releases aboutc…

3aca419

…ode-org#295 This script copies all thirdparty files(required) and place it in repo as release assets. Signed-off-by: Abhishek Kumar <[email protected]>

Abhishek-Dev09-zz force-pushed the #295-requirement.txt branch from d20fede to 3aca419 Compare July 7, 2020 22:23

pombredanne requested changes Jul 8, 2020

View reviewed changes

pombredanne changed the title ~~Script to atomate requirements.txt~~ Automate requirements.txt creation Jul 8, 2020

Abhishek-Dev09-zz closed this Jul 26, 2020


		os.environ ["GITHUB_TOKEN"] = token

		tag = input ("Enter tag_name :")

Automate requirements.txt creation #2086

Automate requirements.txt creation #2086

Conversation

Abhishek-Dev09-zz commented Jun 23, 2020 • edited Loading

Tasks

Abhishek-Dev09-zz commented Jul 7, 2020 • edited Loading

pombredanne commented Jul 7, 2020

Abhishek-Dev09-zz commented Jul 7, 2020

pombredanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abhishek-Dev09-zz Jul 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pombredanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abhishek-Dev09-zz Jul 8, 2020 • edited Loading

Choose a reason for hiding this comment

pombredanne Jul 9, 2020 • edited Loading

Choose a reason for hiding this comment

pombredanne commented Jul 9, 2020

pombredanne commented Jul 26, 2020

Abhishek-Dev09-zz commented Jul 26, 2020

Abhishek-Dev09-zz commented Jun 23, 2020 •

edited

Loading

Abhishek-Dev09-zz commented Jul 7, 2020 •

edited

Loading

Abhishek-Dev09-zz Jul 7, 2020 •

edited

Loading

Abhishek-Dev09-zz Jul 8, 2020 •

edited

Loading

pombredanne Jul 9, 2020 •

edited

Loading