Add poetry and fix problem from #23 #25

breno-jesus-fernandes · 2022-06-08T16:49:32Z

Based on feedback from #23 I remove "black" and "isort" style applied. Add links to documentation in README.md. Fix version compatibility from project, python, and pandas. I merge the last main in the current branch, so should be ok this time.

…kg structure now follows pep 561

# Conflicts: # .github/workflows/test.yml # setup.py

Dr-Irv

This all looks good, but we need to keep the tests that build and install the wheel.

.github/workflows/test.yml

Dr-Irv · 2022-06-08T19:11:36Z

I'm approving so the CI will run here. I still need to pull down this branch and test locally to get a sense of how this works.

setup.cfg

Dr-Irv · 2022-06-08T22:23:34Z

I'm comparing what we have in the pandas wheel with what poetry produces.

In the wheel file produced by poetry, a file called entry_points.txt in pandas_stubs-1.4.2.20220608.dist-info is included and it has the contents:

[console_scripts]
all_tests=scripts.tests:run_all
install_wheel=scripts.tests:install_wheel
rm_src_code=scripts.tests:remove_src_code

This shouldn't appear in our wheel file as it refers to the poetry scripts. Can you figure out how to not include it?

Also, for the version number, the convention I'd like to use is YYMMDD, so can you change that as well?

This is really close to me merging it in!

…rom setuptools to "poe" a poetry plugin to run util scripts.

breno-jesus-fernandes · 2022-06-08T23:10:42Z

I did not know that poetry.scripts it was equivalent to console-scripts from setup tools,looks like there is not way to remove from entry points, so now i'm using a simple poetry plugin to run all the util scripts called poe.

Dr-Irv · 2022-06-09T00:46:44Z

I pulled your latest version, and did a poetry install -vvv to update the installation, and got this message:

 Because pandas-stubs depends on poethepoet (^0.13.1) which doesn't match any versions, version solving failed.

Seems like pypi only has version 0.0.3 .

…update".

breno-jesus-fernandes · 2022-06-09T01:04:04Z

I think you already had a pre builded poetry.lock so came a conflicted, just type poetry update, to see if it works, all tests passed here.

Dr-Irv · 2022-06-09T12:22:39Z

OK, now that works, but there are still some issues:

After I do poetry run poe all_tests, the pandas-stubs package is still installed in my environment.
There are now 5 tests that are run:
- local mypy
- local pytest
- local pyright
- pyright against installed wheel
- mypy against installed wheel
  It would be useful to have some blank lines between the output of each of these tests, and markers between the outputs of the tests that say things like "Beginning mypy test on repo", "End mypy test on repo", with blank lines after the "End" messages.
Can we make the order of the tests mypy, then pyright, then pytest
Can we set things up to run only a subset of the tests (via a command line parameter)? i.e,. run only the local, or run only the wheel based test. And an option to run only mypy and pyright locally, skipping the pytest test. The reason I say this is that when you're working on the stubs, you often just need to run the typing tests to check for validity, as opposed to running everything.
When running the tests, this message appears multiple times:

<my-path-to-environment>\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(

Can that be avoided, or is this a known issue with poetry?

breno-jesus-fernandes · 2022-06-10T17:55:56Z

When we install the project dependencies with poetry install -vvv, actually is installing the project in "editable mode" but not building the wheel and installing, here is the log:

- Building package pandas-stubs in editable mode
- Adding pandas_stubs.pth to C:\Users\Ghost\Documents\pandas-stubs\.venv\Lib\site-packages for C:\Users\Ghost\Documents\pandas-stubs
- Adding the pandas_stubs-1.4.2.220608.dist-info directory to C:\Users\Ghost\Documents\pandas-stubs\.venv\Lib\site-packages

I think is a way that poetry defines a fixed path to the project in "pandas_stubs.pth" file to know where is the source code. We could easily avoid just typing poetry install -vvv --no-root or poetry update -vvv but seems mypy and pyright stop working properly. So for now, I think we do not need to worry about this because does not seems to cause any damage to the project.

Dr-Irv

Aside from the comment about reinstalling poetry, this is looking good.

When the install dist step is done, what's puzzling is the output where it seems to be uninstalling lots of things, then does the install of the dependent packages. I'm seeing this in the output:

===========================================
Beginning to run: 'Install Dist'
===========================================

Processing c:\code\pr25\dist\pandas_stubs-1.4.2.220608-py3-none-any.whl
Collecting matplotlib>=3.3.2
  Using cached matplotlib-3.5.2-cp39-cp39-win_amd64.whl (7.2 MB)
Collecting pandas<1.5.0,>=1.4.0
  Using cached pandas-1.4.2-cp39-cp39-win_amd64.whl (10.5 MB)
Collecting typing-extensions>=4.2.0
  Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting numpy>=1.17
  Using cached numpy-1.22.4-cp39-cp39-win_amd64.whl (14.7 MB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.33.3-py3-none-any.whl (930 kB)
Collecting python-dateutil>=2.7
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pillow>=6.2.0
  Using cached Pillow-9.1.1-cp39-cp39-win_amd64.whl (3.3 MB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting pyparsing>=2.2.1
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.2-cp39-cp39-win_amd64.whl (55 kB)
Collecting packaging>=20.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pytz>=2020.1
  Using cached pytz-2022.1-py2.py3-none-any.whl (503 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pyparsing, pytz, python-dateutil, pillow, packaging, numpy, kiwisolver, fonttools, cycler, typing-extensions, pandas, matplotlib, pandas-stubs
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
  Attempting uninstall: pyparsing
    Found existing installation: pyparsing 3.0.9
    Uninstalling pyparsing-3.0.9:
      Successfully uninstalled pyparsing-3.0.9
  Attempting uninstall: pytz
    Found existing installation: pytz 2022.1
    Uninstalling pytz-2022.1:
      Successfully uninstalled pytz-2022.1
  Attempting uninstall: python-dateutil
    Found existing installation: python-dateutil 2.8.2
    Uninstalling python-dateutil-2.8.2:
      Successfully uninstalled python-dateutil-2.8.2
  Attempting uninstall: pillow
    Found existing installation: Pillow 9.1.1
    Uninstalling Pillow-9.1.1:
      Successfully uninstalled Pillow-9.1.1
  Attempting uninstall: packaging
    Found existing installation: packaging 21.3
    Uninstalling packaging-21.3:
      Successfully uninstalled packaging-21.3
  Attempting uninstall: numpy
    Found existing installation: numpy 1.22.4
    Uninstalling numpy-1.22.4:
      Successfully uninstalled numpy-1.22.4
  Attempting uninstall: kiwisolver
    Found existing installation: kiwisolver 1.4.2
    Uninstalling kiwisolver-1.4.2:
      Successfully uninstalled kiwisolver-1.4.2
  Attempting uninstall: fonttools
    Found existing installation: fonttools 4.33.3
    Uninstalling fonttools-4.33.3:
      Successfully uninstalled fonttools-4.33.3
  Attempting uninstall: cycler
    Found existing installation: cycler 0.11.0
    Uninstalling cycler-0.11.0:
      Successfully uninstalled cycler-0.11.0
  Attempting uninstall: typing-extensions
    Found existing installation: typing-extensions 4.2.0
    Uninstalling typing-extensions-4.2.0:
      Successfully uninstalled typing-extensions-4.2.0
  Attempting uninstall: pandas
    Found existing installation: pandas 1.4.2
    Uninstalling pandas-1.4.2:
      Successfully uninstalled pandas-1.4.2
  Attempting uninstall: matplotlib
    Found existing installation: matplotlib 3.5.2
    Uninstalling matplotlib-3.5.2:
      Successfully uninstalled matplotlib-3.5.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
poetry 1.1.7 requires keyring<22.0.0,>=21.2.0; python_version >= "3.6" and python_version < "4.0", but you have keyring 23.4.0 which is incompatible.
poetry 1.1.7 requires packaging<21.0,>=20.4, but you have packaging 21.3 which is incompatible.
Successfully installed cycler-0.11.0 fonttools-4.33.3 kiwisolver-1.4.2 matplotlib-3.5.2 numpy-1.22.4 packaging-21.3 pandas-1.4.2 pandas-stubs-1.4.2.220608 pillow-9.1.1 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.1 six-1.16.0 typing-extensions-4.2.0

===========================================
End 'Install Dist', runtime: 28.683 seconds.
===========================================

Shouldn't installing the distribution just have to install the built wheel and not reinstall everything? This seems to be adding a lot of extra time.

Maybe it is the --force_reinstall that is doing that ?

Dr-Irv · 2022-06-11T21:03:09Z

scripts/test.py

+        Step(name="Run Pyright Against Dist", run=run_pyright_dist),
+        Step(name="Uninstall Dist", run=uninstall_dist),
+        Step(name="Restore Source Code", run=restore_src),
+        Step(name="Install Poetry", run=install_poetry),


why do you need to install poetry when it is already installed?

Dr-Irv · 2022-06-11T21:03:23Z

scripts/test.py

+        Step(name="Run Pyright Against Dist", run=run_pyright_dist),
+        Step(name="Uninstall Dist", run=uninstall_dist),
+        Step(name="Restore Source Code", run=restore_src),
+        Step(name="Install Poetry", run=install_poetry),


same comment as above about installing poetry

Dr-Irv · 2022-06-12T18:53:17Z

I think there is one more issue to check. Suppose that there is an error that occurs in mypy or pyright when testing the distribution via poetry run poe test_dist . Then the test will exit, and the source will not be restored.

To see this, add this line to tests/test_timefuncs.py:

 def test_types_init() -> None:
      ts: pd.Timedelta = pd.Timestamp("2021-03-01T12")

Then do poetry run poe test_dist .

So is it possible that if an error is discovered when testing the distribution that you automatically restore the source?

Dr-Irv · 2022-06-12T23:35:19Z

Also, if there is an error when testing the distribution, you should uninstall the distribution.

breno-jesus-fernandes · 2022-06-14T23:31:08Z

Not sure if I added in this PR, but I have finished the pipeline to publish in github and pypi the package through github actions.

Dr-Irv

This is good to go. Thanks @BrenoJesusFernandes .

Dr-Irv · 2022-06-15T01:34:48Z

Not sure if I added in this PR, but I have finished the pipeline to publish in github and pypi the package through github actions.

Please submit that as a separate PR.

Breno Fernandes added 13 commits June 5, 2022 15:40

Add poetry, all configs and dependences now are in pyproject.toml, pc…

c9c3efa

…kg structure now follows pep 561

Fix: Dependencies versions

cc394fb

Feature: Scratch files to document the project

3dabd25

Feature: Improve CI/CD with poetry and removing pyright workaround

697ade8

Fix: Add black and isort in dependencies

1a2c94f

Feature: Add a script to run all tests, just type "poetry run all_test"

7176fd0

Feature: Add something in documentations

13b1b19

Merge branch 'main' into feature/poetry

ea7e6c4

# Conflicts: # .github/workflows/test.yml # setup.py

Fix: Project version, Author, Python versions, Pandas version

2de548e

Fix: Remove old test file documentation

8bcb394

Fix: Remove old pyright configs

cf96463

Fix: Documentation in README.md with links

ec78fab

Fix: pyproject.toml python compatibility

9f44c9b

Dr-Irv requested changes Jun 8, 2022

View reviewed changes

.github/workflows/test.yml Show resolved Hide resolved

Breno Fernandes added 2 commits June 8, 2022 15:08

Fix: Add tests with wheel

34ec92a

Fix: Remove pytests from tests against dist

728e449

Dr-Irv requested changes Jun 8, 2022

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

Breno Fernandes added 3 commits June 8, 2022 15:41

Fix: Remove Source code distribution on tests after install wheel

7871b94

Fix: Using shutil to remove dir

e5040a4

Fix: Improve local tests and fix CI tests

9259b96

Dr-Irv reviewed Jun 8, 2022

View reviewed changes

setup.cfg Show resolved Hide resolved

Fix: Add metadata to pyproject.toml when build a wheel

8d05d17

Fix: Replace default "poetry.scripts" equivalent to console_scripts f…

34c8305

…rom setuptools to "poe" a poetry plugin to run util scripts.

Fix: Improving docs with instructions to update dependencies "poetry …

b9677bd

…update".

Breno Fernandes added 6 commits June 10, 2022 14:16

CI: All tests call poe to run

0e3ea0a

Docs: Improving setup and test documentation

fbff5dc

Refactor: Improving test_src (I still think it could be better)

ce3a4f5

Docs: Improving poe tests documentation

b398255

Style: Apply black and isort to new scripts code

40836f2

Fix: Bug when test_src code fails, test_dist continues to test

3cf7947

Fix: Error in test documentation

38a64d6

breno-jesus-fernandes mentioned this pull request Jun 10, 2022

error in 'egg_base' option: 'src.tmp' does not exist or is not a directory #26

Closed

Breno Fernandes added 4 commits June 10, 2022 18:44

Docs: Improving CLI help documentation to run tests

54a846a

Docs: Improving CLI help documentation to run tests

36fc564

CI: Runs CI only when change code or dependencies

0f73192

Docs: Fix Mc Donald's english errors

1d73e23

Dr-Irv reviewed Jun 11, 2022

View reviewed changes

Breno Fernandes added 3 commits June 12, 2022 01:18

Fix: Package test

fe21dd2

Fix: Package Test

bc550a6

CI: Fix package test

d4467e0

Fix: add project to sys path

0f4eb0b

Breno Fernandes (Linux OS) added 6 commits June 13, 2022 21:49

Refactor: Folder Hierarquie, Rollback job, refresh setup docs

5a17251

Fix: Add rollback to test_dist

5667178

Feature: Add code style check

4c6d838

Refactor: Improving rollback

ba94941

Feature: add clean cache option in local tests

3abae80

Fix: code style check

0aaeadb

Dr-Irv approved these changes Jun 15, 2022

View reviewed changes

Dr-Irv merged commit f2b783c into pandas-dev:main Jun 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add poetry and fix problem from #23 #25

Add poetry and fix problem from #23 #25

breno-jesus-fernandes commented Jun 8, 2022

Dr-Irv left a comment

Dr-Irv commented Jun 8, 2022

Dr-Irv commented Jun 8, 2022

breno-jesus-fernandes commented Jun 8, 2022

Dr-Irv commented Jun 9, 2022

breno-jesus-fernandes commented Jun 9, 2022

Dr-Irv commented Jun 9, 2022

breno-jesus-fernandes commented Jun 10, 2022

Dr-Irv left a comment

Dr-Irv Jun 11, 2022

Dr-Irv Jun 11, 2022

Dr-Irv commented Jun 12, 2022

Dr-Irv commented Jun 12, 2022

breno-jesus-fernandes commented Jun 14, 2022

Dr-Irv left a comment

Dr-Irv commented Jun 15, 2022

Add poetry and fix problem from #23 #25

Add poetry and fix problem from #23 #25

Conversation

breno-jesus-fernandes commented Jun 8, 2022

Dr-Irv left a comment

Choose a reason for hiding this comment

Dr-Irv commented Jun 8, 2022

Dr-Irv commented Jun 8, 2022

breno-jesus-fernandes commented Jun 8, 2022

Dr-Irv commented Jun 9, 2022

breno-jesus-fernandes commented Jun 9, 2022

Dr-Irv commented Jun 9, 2022

breno-jesus-fernandes commented Jun 10, 2022

Dr-Irv left a comment

Choose a reason for hiding this comment

Dr-Irv Jun 11, 2022

Choose a reason for hiding this comment

Dr-Irv Jun 11, 2022

Choose a reason for hiding this comment

Dr-Irv commented Jun 12, 2022

Dr-Irv commented Jun 12, 2022

breno-jesus-fernandes commented Jun 14, 2022

Dr-Irv left a comment

Choose a reason for hiding this comment

Dr-Irv commented Jun 15, 2022