Cannot install synapser due to problems with MarkupSafe

[yocra3]

Operating system

Ubuntu 18.04.5 LTS

Description of the problem

There is an error when installing MarkupSafe:

> install.packages("synapser", repos=c("http://ran.synapse.org", "http://cran.fhcrc.org"))
Installing package into ‘/home/genetica/R/x86_64-pc-linux-gnu-library/3.6’
(as ‘lib’ is unspecified)
probando la URL 'http://ran.synapse.org/src/contrib/synapser_0.8.71.tar.gz'
Content type 'application/x-tar' length 132104 bytes (129 KB)
==================================================
downloaded 129 KB

* installing *source* package ‘synapser’ ...
** using staged installation
Loading required package: R6
Loading required package: rjson
Loading required package: R6
Loading required package: rjson
You are using pip version 18.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
You are using pip version 18.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
You are using pip version 18.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-c4ouwebu/MarkupSafe/
You are using pip version 18.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/tmp/RtmpSd9Qnv/R.INSTALL8fb572759ef/synapser/inst/python/installPythonClient.py", line 137, in main

    
], localSitePackages)
  File "/tmp/RtmpSd9Qnv/R.INSTALL8fb572759ef/synapser/inst/python/installPythonClient.py", line 172, in _install_pip

    
raise Exception("pip.main returned {} when installing {}".format(rc, package))
Exception
: 
pip.main returned 1 when installing MarkupSafe==1.0
Error in PythonEmbedInR::pyExec(command) : 
  An error has occurred while executing Python code. See traceback above.
Ejecución interrumpida
ERROR: configuration failed for package ‘synapser’
* removing ‘/home/genetica/R/x86_64-pc-linux-gnu-library/3.6/synapser’

The downloaded source packages are in
	‘/tmp/RtmpWEd6TS/downloaded_packages’
Warning message:
In install.packages("synapser", repos = c("http://ran.synapse.org",  :
  installation of package ‘synapser’ had non-zero exit status

The problem is due to that synapser is trying to install an old version of MarkupSafe. When trying to install it in command line, directly with pip, I can install MarkupSafe 1.1:

> pip3 install MarkupSafe
Collecting MarkupSafe
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: MarkupSafe
Successfully installed MarkupSafe-1.1.1

But not MarkupSafe 1.0:

> pip3 install MarkupSafe-1.0
Collecting MarkupSafe-1.0
Exception:
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
   status = self.run(options, args)
 File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 353, in run
   wb.build(autobuilding=True)
 File "/usr/lib/python3/dist-packages/pip/wheel.py", line 749, in build
   self.requirement_set.prepare_files(self.finder)
 File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
   ignore_dependencies=self.ignore_dependencies))
 File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
   require_hashes
 File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
   self.link = finder.find_requirement(self, upgrade)
 File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
   all_candidates = self.find_all_candidates(req.name)
 File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
   for page in self._get_pages(url_locations, project_name):
 File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
   page = self._get_page(location)
 File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
   return HTMLPage.get_page(link, session=self.session)
 File "/usr/lib/python3/dist-packages/pip/index.py", line 795, in get_page
   resp.raise_for_status()
 File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
   raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/markupsafe-1-0/

Expected behavior

Install synapser.

Actual behavior

Synapser fails to install.

Output of sessionInfo()

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=es_ES.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=es_ES.UTF-8        LC_COLLATE=es_ES.UTF-8    
 [5] LC_MONETARY=es_ES.UTF-8    LC_MESSAGES=es_ES.UTF-8   
 [7] LC_PAPER=es_ES.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=es_ES.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.6.3 tools_3.6.3   

[jkiang13]

Hi @yocra3

Can you try the following from the command line (double equal signs not the dash as above).

pip3 install MarkupSafe==1.0

The MarkupSafe installation as part of the synapser installation is into a Python interpreter that is embedded into the PythonEmbedInR library rather than your system Python interpreter, so installing it on the system Python interpreter will not affect the synapser installation, but it's possible your system Python will encounter the same underlying in which case we may get a more useful error message.

[yocra3]

This command is also failing:

> pip3 install MarkupSafe==1.0
Collecting MarkupSafe==1.0
 Cache entry deserialization failed, entry ignored
 Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
   Complete output from command python setup.py egg_info:
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/tmp/pip-build-i_7ufgop/MarkupSafe/setup.py", line 6, in <module>
       from setuptools import setup, Extension, Feature
   ImportError: cannot import name 'Feature'
   
   ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-i_7ufgop/MarkupSafe/

[jkiang13]

@yocra3

Could you try running the following commands in the R console where you tried to install synapser and paste the output here?

library(PythonEmbedInR)
pyExec('import setuptools; print(setuptools.__file__); print(setuptools.__version__);')

[yocra3]

@jkiang13 Here you have:

> library(PythonEmbedInR)
Loading required package: R6
Loading required package: rjson
> pyExec('import setuptools; print(setuptools.__file__); print(setuptools.__version__);')
/home/genetica/.local/lib/python3.6/site-packages/setuptools/__init__.py
50.3.0

[jkiang13]

@yocra3

Hmm, the PythonEmbedInR library that synapser uses installs an embedded version of python for use with this library. However that output indicates that in your case the library is loading some Python packages from your own python lib directory rather than the one with the embedded interpreter. This error seems to be a known issue with some versions of MarkupSafe and setuptools, but this ins't normally encountered because the setuptools in the embedded interpreter is compatible.

Your sessionInfo() doesn't show any reticulate usage, that can cause issues with the version of Python that is bound to. Are you a reticulate user?

Also can you try the following in your R console and show the output?

# before loading PythonEmbedInR
sysenv <- Sys.getenv()
sysenv[grepl("^PYTHON", names(sysenv))]

library(PythonEmbedInR)

# after PythonEmbedInR
sysenv <- Sys.getenv()
sysenv[grepl("^PYTHON", names(sysenv))]

On an Ubuntu installation I would expect after loading PythonEmbedInR that there would be entries like the below, but I think in your case it will show otherwise.

PYTHONHOME              /usr/local/lib/R/site-library/PythonEmbedInR
PYTHONPATH              /usr/local/lib/R/site-library/PythonEmbedInR/lib

[yocra3]

Hi,

Here you have the output of the commands:

> sysenv <- Sys.getenv()
> sysenv[grepl("^PYTHON", names(sysenv))]

> library(PythonEmbedInR)
Loading required package: R6
Loading required package: rjson
> sysenv <- Sys.getenv()
> sysenv[grepl("^PYTHON", names(sysenv))]
PYTHONHOME              /home/genetica/R/x86_64-pc-linux-gnu-library/3.6/PythonEmbedInR
PYTHONPATH              /home/genetica/R/x86_64-pc-linux-gnu-library/3.6/PythonEmbedInR/lib

Is that what you were expecting? On the other hand, I do not know what a reticulate user is. How can I check it?

[jkiang13]

@yocra3

Hmm, those are as expected. I do not think your issue is related to reticulate.

Rather it seems you have installed a version of setuptools in your user site packages directory (/home/genetica/.local/lib/python3.6/site-packages) that is taking priority over the typical setuptools version in the PYTHONPATH that includes the embedded python version that synapser normally uses. There is an incompatibility between that version of setuptools and the version of Markupsafe that is installed for use with synapser. I don't think we typically see this because users don't have a version of setuptools installed in their user site directory.

I have made a branch of synapser that upgrades the Markupsafe. Can you try installing this version as follows:

if (!require("PythonEmbedInR")) install.packages("PythonEmbedInR", repos=c("http://cran.fhcrc.org", "http://ran.synapse.org"))
if (!require("devtools")) install.packages("devtools")
devtools::install_github('jkiang13/synapser', ref='SYNR-1466-upgrade-markupsafe')

[yocra3]

HI @jkiang13,

That worked and I could install synapser. I close the issue.

Thank you very much for your support.

[jkiang13]

Okay great @yocra3

This change will be included in the next release of synapser. Note that if you reinstall the current production build in the meantime it will not yet include this change.