Issue with conda-forge scrapy>1.6.0 on Windows

[b-per]

Issue: Already reported in the scrapy repo: scrapy/scrapy#4289

When installing scrapy from conda-forge, no error appears during the process but when trying to run scrapy the following error appears:

Fatal error in launcher: Unable to create process using '"d:\bld\scrapy_1584555945006\_h_env\python.exe" "C:\Users\benoi\anaconda3\envs\scrapy\Scripts\scrapy.exe" ': The system cannot find the file specified.

I have no D: drive on my machine.
As reported in the bug on the scrapy repo, installing withthe previous 1.6.0 version with conda install -c conda-forge scrapy=1.6.0 works.

Environment (conda list):

# packages in environment at C:\Users\benoi\anaconda3\envs\scrapy:
#
# Name                    Version                   Build  Channel
astroid                   2.3.3                    py38_0
atomicwrites              1.3.0                      py_0    conda-forge
attrs                     19.3.0                     py_0    conda-forge
automat                   20.2.0                     py_0    conda-forge
bcrypt                    3.1.7            py38h9de7a3e_1    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py38h32f6830_0    conda-forge
cffi                      1.14.0           py38ha419a9e_0    conda-forge
colorama                  0.4.3                      py_0    conda-forge
constantly                15.1.0                     py_0    conda-forge
cryptography              2.8              py38hb32ad35_1    conda-forge
cssselect                 1.1.0                      py_0    conda-forge
hyperlink                 17.3.1                     py_0    conda-forge
idna                      2.9                        py_1    conda-forge
importlib-metadata        1.6.0            py38h32f6830_0    conda-forge
importlib_metadata        1.6.0                         0    conda-forge
incremental               17.5.0                     py_0    conda-forge
isort                     4.3.21                   py38_0
lazy-object-proxy         1.4.3            py38he774522_0
libiconv                  1.15              hfa6e2cd_1006    conda-forge
libxml2                   2.9.10               h9ce36c8_0    conda-forge
libxslt                   1.1.33               heafd4d3_0    conda-forge
lxml                      4.5.0            py38h7b22a32_1    conda-forge
mccabe                    0.6.1                    py38_1
more-itertools            8.2.0                      py_0    conda-forge
openssl                   1.1.1f               hfa6e2cd_0    conda-forge
packaging                 20.1                       py_0    conda-forge
parsel                    1.5.2                    py38_0
pip                       20.0.2                     py_2    conda-forge
pluggy                    0.13.1                   py38_0    conda-forge
protego                   0.1.16                     py_0    conda-forge
py                        1.8.1                      py_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.20                       py_0    conda-forge
pydispatcher              2.0.5                      py_1    conda-forge
pyhamcrest                2.0.2                      py_0    conda-forge
pylint                    2.4.4                    py38_0
pyopenssl                 19.1.0                     py_1    conda-forge
pyparsing                 2.4.6                      py_0    conda-forge
pytest                    5.4.1            py38h32f6830_0    conda-forge
pytest-runner             5.2                        py_0    conda-forge
python                    3.8.2           h5fd99cc_5_cpython    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38hfa6e2cd_0    conda-forge
queuelib                  1.5.0                    py38_0
scrapy                    1.6.0                    py38_0
service_identity          18.1.0                     py_0    conda-forge
setuptools                46.1.3           py38h32f6830_0    conda-forge
six                       1.14.0                     py_1    conda-forge
sqlite                    3.30.1               hfa6e2cd_0    conda-forge
twisted                   20.3.0           py38h9de7a3e_0    conda-forge
vc                        14.1                 h869be7e_1    conda-forge
vs2015_runtime            14.16.27012          h30e32a0_1    conda-forge
w3lib                     1.20.0                     py_0    conda-forge
wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
wincertstore              0.2                   py38_1003    conda-forge
wrapt                     1.12.1           py38he774522_1
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h2fa13f4_1006    conda-forge
zope.interface            5.0.2            py38h9de7a3e_0    conda-forg

Details about conda and system ( conda info ):

     active environment : scrapy
    active env location : C:\Users\benoi\anaconda3\envs\scrapy
            shell level : 2
       user config file : C:\Users\benoi\.condarc
 populated config files : C:\Users\benoi\.condarc
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages :
       base environment : C:\Users\benoi\anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\benoi\anaconda3\pkgs
                          C:\Users\benoi\.conda\pkgs
                          C:\Users\benoi\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\benoi\anaconda3\envs
                          C:\Users\benoi\.conda\envs
                          C:\Users\benoi\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False

[GeorgeA92]

As I previously mentioned in this issue (scrapy/scrapy#4289 (comment))
there is a one difference between conda-forge scrapy packages with versions more than 1.6.0 (where this error occure) and the older versions.
/recipe/info/meta.yaml files from recent versions don't have these 2 lines:

    entry_points:
        - scrapy = scrapy.cmdline:execute

Also I noticed that this lines in corresponding file of this repository was changed as result of this commit

My assumption is - missing scrapy entry_point in /recipe/info/meta.yaml and rest of conda-forge package that built based on this /meta.yaml causing this error.
@Gallaecio , @dangra , @kmike , @redapple , @rmax , @wRAR

[yadalik]

Btw this line "d:\bld\scrapy_1584555997548" not only in meta.yaml files, but also hardcoded somewhere at \Scripts\scrapy.exe files at:

\user_env_name\Scripts\scrapy.exe
\miniconda3\pkgs\scrapy-2.0.1-py37hc8dfbb8_0\info\recipe\meta.yaml
\miniconda3\pkgs\scrapy-2.0.1-py37hc8dfbb8_0\Scripts\scrapy.exe

Looks like that's why it working if use python -m scrapy, cause in that case it not using that exe-file with those hardcoded lines, and that's why just editing yaml-files not helping.
Also, as a qiuck workaround - everything works fine with pip install scrapy instead of installing with conda install -c conda-forge scrapy.

[GeorgeA92]

@yadalik

Btw this line "d:\bld\scrapy_1584555997548" not only in meta.yaml files, but also hardcoded somewhere at \Scripts\scrapy.exe files at:

Yes. That's true. I noticed it from this comment.
I think that **\scrapy.exe binaries was generated using data from \recipe\meta.yaml.

Also, as a qiuck workaround - everything works fine with pip install scrapy

Not.
These can lead to this issues because of scrapy dependencies (as far as I know twisted and lxml packages require C++ compiler):
scrapy/scrapy#2881
scrapy/scrapy#4301
scrapy/scrapy#3865
scrapy/scrapy#3785
scrapy/scrapy#1819
scrapy/scrapy#2506

The only reliable and recommended way to avoid issues with pip intall scrapy - usage of scrapy package of conda-forge.

For conda-forge packages of scrapy>1.6.0 - we have current issue with d:\bld\scrapy_************.

[MarkusFraKS]

I stumbled over this issue, too. Thanks to @GeorgeA92 for the investigation and thanks to @camfrout for opening the issue. It seems to be an easy fix. Adding the missing entry-point as @GeorgeA92 mentioned, bumping the build number and do a new build should fix the problem. At least it worked building it on my local channel. No changes on scrapy itself are needed. Any chances that this is done soon or a reason it is not done yet?

@Gallaecio , @dangra , @kmike , @redapple , @rmax , @wRAR

[rmax]

Thanks for the research!

Indeed, looks like removing the entrypoint causes the missed exe. I would try:

• restore the entrypoint
• add scrapy.exe version # [win] as test

Edit: I remember @Gallaecio having some issues with the build in windows last time, not sure if those issue would persist in this case or there won't be additional issues.

[klauswarzecha]

I can confirm that the issue brought up by @camfrout still exists.

Here's what I did this morning:

• install Anaconda3-2020.02 (64 Bit) locally on a machine with Windows 10 (fresh install)
• add conda-forge as primary channel conda config --add channels conda-forge
• update packages conda update --all
• create environment conda create -n crawl python=3.7 ipython scrapy
• activate new environment conda activate crawl

Up to this point, everything seems to be fine. However, when I called scrapy startproject something,
the error message was

Fatal error in launcher: Unable to create process using '"d:\bld\scrapy_1587736721630_h_env\python.exe" "C:\Users\Klaus.Warzecha\Anaconda3\envs\crawl\Scripts\scrapy.exe" crawl': Das System kann die angegebene Datei nicht finden.

which is the localized German version of "The system cannot find the file specified."

Almost unnecessary to mention that the machine does not have a drive D:\ either.

This doesn't mean that the current version of scrapy from conda-forge is completely unusable. From within ipython, imports from scrapy seems to work fine. However, not being able to use scrapy shell is somewhat unsatisfying.

I can further confirm that from within the environment

• conda remove scrapy
• pip install scrapy

does indeed furnish a version of scrapy 2.1.0 from which the shell can be called. So, thanks to @GeorgeA92 for pointing out that.

Nevertheless, I'd still prefer being able to install as much as possible through conda ;-)

Cheers,
Klaus

[kmike]

As an alternative, is python -m scrapy shell working?

[MarkusFraKS]

As an alternative, is python -m scrapy shell working?

@kmike : Yes.

[rmax]

The latest build removed the scrapy.exe from the package. Here is the attempt to fix it: #40

[klauswarzecha]

As an alternative, is python -m scrapy shell working?

@kmike I was in planning and couldn't respond till now.
After pip uninstall -v -y scrapy and conda install scrapy on the machine described above, the shell opens with python -m scrapy shell and uses the ipython shell.
A quick fetch of url and the extraction of some text using xpath were successful too!

So, Yes, works like a charm! Thanks a lot!

[rmax]

Thank you all for your patience. New packages have been published for windows (v2.1.0_1).

This restores the scrapy command in windows. However, worth noting there are known limitations that are being worked upstream (related scrapy/scrapy#4458).