old html5lib makes pip dysfunctional #588

alanpurple · 2017-09-29T07:01:52Z

html5lib dependency should be updated to 1.0b10

windows 10 x64
Anaconda 5.0.0 with python 3.6.2
install tensorflow-gpu -> tensorboard -> html5lib downgraded -> pip dysfunctional

chihuahua · 2017-09-29T07:16:22Z

Interesting - in what way does pip become dysfunctional?

alanpurple · 2017-09-29T07:18:41Z

@chihuahua
TypeError: parse() got an unexpected keyword argument 'transport_encoding'

similar to Kozea/WeasyPrint#353

Resolved by copying newer version of html5lib folder and overwrite

chihuahua · 2017-09-29T07:29:34Z

Hmm, TensorBoard already requires 1.0b8, which comes after 1.0b1.

tensorboard/tensorboard/pip_package/setup.py

Line 34 in f6e82c4

'html5lib == 0.9999999', # identical to 1.0b8

alanpurple · 2017-09-29T07:31:04Z

@chihuahua my mistake, b10

Fixes #588.

jart · 2017-10-03T01:04:41Z

Upgrading to a newer html5lib broke our bleach dependency. We would need to upgrade Google to the latest Bleach in order to do this. What specific version of pip are you using? In what way does it become dysfunctional? What is the specific error, including traceback?

alanpurple · 2017-10-03T02:44:43Z

@jart

Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\Users\alanp> pip -V
pip 9.0.1 from C:\Anaconda3\lib\site-packages (python 3.6)
PS C:\Users\alanp> conda upgrade pip
Fetching package metadata ...............
Solving package specifications: .

All requested packages already installed.

packages in environment at C:\Anaconda3:

pip 9.0.1 py36hadba87b_3
PS C:\Users\alanp> conda upgrade setuptools
Fetching package metadata ...............
Solving package specifications: .

All requested packages already installed.

packages in environment at C:\Anaconda3:

setuptools 36.5.0 py36h65f9e6e_0
PS C:\Users\alanp> conda install bleach==1.5.0
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment C:\Anaconda3:

The following packages will be DOWNGRADED:

bleach:   2.0.0-py36h0a7e3d6_0       --> 1.5.0-py36_0
html5lib: 0.999999999-py36ha09b1f3_0 --> 0.9999999-py36_0

Proceed ([y]/n)?

html5lib-0.999 100% |###############################| Time: 0:00:00 11.67 MB/s
bleach-1.5.0-p 100% |###############################| Time: 0:00:00 0.00 B/s
PS C:\Users\alanp> pip -V
pip 9.0.1 from C:\Anaconda3\lib\site-packages (python 3.6)
PS C:\Users\alanp> pip install sqlalchemy==1.1.13
Collecting sqlalchemy==1.1.13
Exception:
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
status = self.run(options, args)
File "C:\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run
wb.build(autobuilding=True)
File "C:\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "C:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 554, in _prepare_file
require_hashes
File "C:\Anaconda3\lib\site-packages\pip\req\req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "C:\Anaconda3\lib\site-packages\pip\index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "C:\Anaconda3\lib\site-packages\pip\index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "C:\Anaconda3\lib\site-packages\pip\index.py", line 568, in _get_pages
page = self._get_page(location)
File "C:\Anaconda3\lib\site-packages\pip\index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "C:\Anaconda3\lib\site-packages\pip\index.py", line 811, in get_page
inst = cls(resp.content, resp.url, resp.headers)
File "C:\Anaconda3\lib\site-packages\pip\index.py", line 731, in init
namespaceHTMLElements=False,
TypeError: parse() got an unexpected keyword argument 'transport_encoding'
PS C:\Users\alanp> conda upgrade bleach
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment C:\Anaconda3:

The following packages will be UPDATED:

bleach:   1.5.0-py36_0     --> 2.0.0-py36h0a7e3d6_0
html5lib: 0.9999999-py36_0 --> 0.999999999-py36ha09b1f3_0

Proceed ([y]/n)?

PS C:\Users\alanp> pip install sqlalchemy==1.1.13
Collecting sqlalchemy==1.1.13
Installing collected packages: sqlalchemy
Found existing installation: SQLAlchemy 1.1.14
Uninstalling SQLAlchemy-1.1.14:
Successfully uninstalled SQLAlchemy-1.1.14
Successfully installed sqlalchemy-1.1.13
PS C:\Users\alanp> pip install sqlalchemy==1.1.14
Collecting sqlalchemy==1.1.14
Installing collected packages: sqlalchemy
Found existing installation: SQLAlchemy 1.1.13
Uninstalling SQLAlchemy-1.1.13:
Successfully uninstalled SQLAlchemy-1.1.13
Successfully installed sqlalchemy-1.1.14

jart · 2017-10-03T02:53:10Z

Maybe file an issue upstream with pip or probably conda? Having an old version of a package shouldn't break a package manager.

alanpurple · 2017-10-03T02:57:28Z

@jart

conda has nothing to deal with this
I just use conda for reproduce this situation safely(for safe and easy back)

pip install tensorflow-gpu->tensorboard->bleach1.5.0->html5lib b8 has same problem

jart · 2017-10-03T03:07:47Z

If what you're saying is the case, then all versions of pip released in past eleven months (since pypa/pip#4038) will break when TensorFlow is installed.

@dstufft would it be possible to make pip more bullet proof in cases where old versions of html5lib are installed?

dstufft · 2017-10-03T03:15:01Z

Uh, something is wrong because pip doesn't normally depend on anything, it bundles all of it's dependencies internally and uses them instead unless someone has patched your version to debundle them.

alanpurple · 2017-10-03T04:34:21Z

@dstufft well, apparently pip use htmllib's "parse"

@jart whatever the true cause is, you guys can't use old bleach and html5lib forever ( in this nature )

It will cause another problem after python 3.6.3, 3.7.0 or Anaconde 5.x.x, etc ( possibly )

dstufft · 2017-10-03T18:22:55Z

We do yes, but one one that has been vendored into pip. We don't use the installed version at all, see https://github.com/pypa/pip/blob/9.0.1/pip/index.py#L33.

henrysky · 2017-10-04T22:29:13Z

I am also suffering from this issue.

pip install tensorflow-gpu will break my pip by installing an outdated version of html5lib
You can resolve the issue temporarily by installing a newer version of html5lib wheel downloaded to your computer locally

But updating a newer version of html5lib will break tensorboard

(C:\Users\Henry\Anaconda3) C:\Users\Henry\Documents>tensorboard
Traceback (most recent call last):
File "c:\users\henry\anaconda3\lib\runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "c:\users\henry\anaconda3\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\Henry\Anaconda3\Scripts\tensorboard.exe_main.py", line 5, in
File "c:\users\henry\anaconda3\lib\site-packages\tensorboard\main.py", line 36, in
from tensorboard.plugins.audio import audio_plugin
File "c:\users\henry\anaconda3\lib\site-packages\tensorboard\plugins\audio\audio_plugin.py", line 27, in
from tensorboard import plugin_util
File "c:\users\henry\anaconda3\lib\site-packages\tensorboard\plugin_util.py", line 21, in
import bleach
File "c:\users\henry\anaconda3\lib\site-packages\bleach_init.py", line 14, in
from html5lib.sanitizer import HTMLSanitizer
ModuleNotFoundError: No module named 'html5lib.sanitizer'

jart · 2017-10-05T01:03:01Z

If pip shades its dependencies (e.g. html5lib becomes pip._vendor.html5lib) then the only likely explanation is the conda command is wrapping pip in such a way that the shading doesn't happen.

@kalefranz There have been multiple reports that installing TensorFlow with Conda will break the Conda installation, most likely due to pip's dependencies not being shaded. What can we do to help our friends?

straygar · 2017-10-19T05:44:30Z

@marcotama I'm confused. How are you supposed to pip install anything if the pip installation on the environment is broken, throwing an exception every time you try to do anything?

marcotama · 2017-10-19T06:15:18Z

@straygar pip install <file> installs a package from a local file, thereby working around the broken html5lib.

gwerbin · 2017-10-20T23:16:57Z

@jart this does sound like an upstream issue with Conda, I never had this problem until I switched from Pip to Conda.

kalefranz · 2017-10-21T01:44:15Z

This is only happening on windows, correct? Has anyone seen this on an OS other than windows?

msarahan · 2017-10-21T04:11:36Z

As an explanation of what's going on here, Anaconda needed to un-vendor pip due to concerns about Windows path length limit (260 characters). The vendored packages were exceeding that limit in our tests, causing conda installations of pip to fail. IIRC, it was the vendoring of vendored deps that was the most painful: https://github.com/pypa/pip/tree/9.0.1/pip/_vendor/requests/packages - this appears to be fixed in master, though, so we won't need to un-vendor future pip versions.

We're not sure exactly what the proper fix is yet for current pip, but we'll work on a fix ASAP. Any advice on how to un-vendor better is welcome.

jart · 2017-10-21T04:40:10Z

@msarahan I'm so happy you're here. If path length is an issue on Windows, then one thing you might be able to do is just stuff pip in a zip and add it to sys.path. As in cd lib/site-packages && zip -r pip.zip pip and then sys.path.append('pip.zip'); from pip import main; main().

I just tried it on my workstation and got an SSL error when I ran pip install, so there might be a couple lines in there that need to be monkey-patched, like possibly to load a pem file or something. But it should be doable.

The cherry on top, if you do this, is pip will have a better cold start time with Conda than anywhere else. Files within a zip tend to occupy contiguous memory, so fewer seeks and better readahead. Also way less system calls because stat() calls end up being userspace hash map lookups on the zip directory loaded from the end of the file into memory. We've actually been thinking about doing this exact same thing with TensorFlow, so it won't take a half a darn second to import.

elipeters · 2017-10-24T00:37:45Z

Windows 10 fix with anaconda
conda install -c anaconda html5lib

ghost · 2017-10-25T11:34:52Z

@elipeters solution has worked for me

msarahan · 2017-10-25T13:11:41Z

We also have a new build of pip available for testing. This new version restores vendoring in pip, but uses the master branch of pip for the vendorings, which are an updated, unvendored requests, which makes the path length issue go away. Unfortunately, it probably missed our bugfix release for the 5.0.0.1 version of Anaconda. You can test it with:

conda install -c c3i_test2 pip

back2yes · 2017-11-14T11:34:10Z

met same bug on windows 7

alanpurple · 2017-11-16T00:54:35Z

tensorflowers!!!
please stop making html5lib downgrade everytime

why is this necessary to force to downgrade?

msarahan · 2017-11-16T00:59:02Z

Our new pip version is available by default now. We hope this fixes the issue on the anaconda side.

conda update pip

alanpurple · 2017-11-16T01:00:22Z

@msarahan thanks, I'll try that one

BenjaminWegener · 2017-12-02T17:04:53Z

https://github.com/SerpentAI/SerpentAI/issues/96#issuecomment-336203943

halfhorst · 2018-01-09T19:52:52Z

The pip upgrade solved this for me. Thanks!

fatih-iver · 2018-01-11T11:28:12Z

pip install html5lib==1.0b10 solved my problem. Thanks!

MikeTheWatchGuy · 2018-01-12T15:51:54Z

marcotama ! THANK YOU !

You're the only post on the net so far that has said: DO THIS
I did and it fixed it... Thanks tons!

lukehsiao · 2018-03-07T23:17:39Z

When I include tensorboard as one of my dependiencies in my own python package, and try to python setup.py install, I'm getting:

error: html5lib 1.0b8 is installed but html5lib==0.9999999 is required by {'tensorboard'}

Note that 1.0b8 should be identical to 0.9999999 as noted in the comment. Is there an easy way for me to resolve this issue?

anentropic · 2018-03-13T17:36:08Z

I have problems with this outdated html5lib dependency when trying to use pipenv

pypa/pipenv#1716

TensorBoard uses bleach, which depends on html5lib, which made a big breaking change at some point in the past. Since upgrading these dependencies is not possible, we'll shade them under the `tensorboard._vendor` namespace, which is magically created by our build_pip_package script, which was rewritten, to support smoke testing and dash. The new process for building the TensorBoard pip package is: ```sh rm -rf /tmp/tensorboard bazel run //tensorboard/pip_package:build_pip_package pip install -U /tmp/tensorboard/*py2*.pip ``` Fixes tensorflow/tensorflow#16424 Fixes tensorflow#427 Fixes tensorflow#588 Closes tensorflow#607 Could help tensorflow#748

TensorBoard uses bleach, which depends on html5lib, which made a big breaking change at some point in the past. Since upgrading these dependencies is not possible, we'll shade them under the `tensorboard._vendor` namespace, which is magically created by our build_pip_package script, which was rewritten, to support smoke testing and dash. The new process for building the TensorBoard pip package is: ```sh rm -rf /tmp/tensorboard bazel run //tensorboard/pip_package:build_pip_package pip install -U /tmp/tensorboard/*py2*.pip ``` Fixes tensorflow/tensorflow#16424 Fixes #427 Fixes #588 Closes #607 Could help #748

alanpurple changed the title ~~old html5lib makes pip disfunctional~~ old html5lib makes pip dysfunctional Sep 29, 2017

chihuahua added type:bug type:build/install labels Sep 29, 2017

chihuahua added a commit that referenced this issue Sep 29, 2017

Update required version of html5lib to 1.0b10

2067422

Fixes #588.

chihuahua mentioned this issue Sep 29, 2017

Update required version of html5lib to 1.0b10 #589

Merged

chihuahua closed this as completed in #589 Sep 29, 2017

chihuahua added a commit that referenced this issue Sep 29, 2017

Update required version of html5lib to 1.0b10 (#589)

46c153f

Fixes #588.

chihuahua self-assigned this Sep 29, 2017

This was referenced Sep 30, 2017

Cherry picks for TensorBoard 0.1.8 #593

Merged

Revert "Update required version of html5lib to 1.0b10 (#589)" #604

Merged

jart reopened this Oct 3, 2017

jjallaire mentioned this issue Oct 3, 2017

Trouble with install_keras() on Windows 10 rstudio/keras3#146

Closed

chihuahua mentioned this issue Oct 3, 2017

Consider switching from bleach to html-sanitizer #607

Closed

jjallaire mentioned this issue Oct 3, 2017

Error installing tensorflow/keras with install_keras() rstudio/keras3#147

Closed

jart mentioned this issue Oct 5, 2017

Add conda environment and update README tensorflow/tensorboard-plugin-example#17

Closed

MikeTheWatchGuy mentioned this issue Jan 12, 2018

tensorflow-gpu install from pip breaks conda on windows tensorflow/tensorflow#14980

Closed

jart mentioned this issue Jan 21, 2018

Tensorflow should not depend on tensorboard tensorflow/tensorflow#12567

Closed

jart mentioned this issue Feb 3, 2018

Dependency on old version of bleach (1.5) tensorflow/tensorflow#16424

Closed

jart mentioned this issue Apr 19, 2018

Vendor bleach and html5lib in pip package #1142

Merged

jart closed this as completed in #1142 Apr 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

old html5lib makes pip dysfunctional #588

old html5lib makes pip dysfunctional #588

alanpurple commented Sep 29, 2017 •

edited

Loading

chihuahua commented Sep 29, 2017

alanpurple commented Sep 29, 2017

chihuahua commented Sep 29, 2017

alanpurple commented Sep 29, 2017

jart commented Oct 3, 2017

alanpurple commented Oct 3, 2017

jart commented Oct 3, 2017

alanpurple commented Oct 3, 2017 •

edited

Loading

jart commented Oct 3, 2017

dstufft commented Oct 3, 2017

alanpurple commented Oct 3, 2017 •

edited

Loading

dstufft commented Oct 3, 2017

henrysky commented Oct 4, 2017 •

edited

Loading

jart commented Oct 5, 2017

straygar commented Oct 19, 2017 •

edited

Loading

marcotama commented Oct 19, 2017

gwerbin commented Oct 20, 2017

kalefranz commented Oct 21, 2017

msarahan commented Oct 21, 2017

jart commented Oct 21, 2017

elipeters commented Oct 24, 2017

ghost commented Oct 25, 2017

msarahan commented Oct 25, 2017

back2yes commented Nov 14, 2017

alanpurple commented Nov 16, 2017

msarahan commented Nov 16, 2017

alanpurple commented Nov 16, 2017

BenjaminWegener commented Dec 2, 2017

halfhorst commented Jan 9, 2018 •

edited

Loading

fatih-iver commented Jan 11, 2018

MikeTheWatchGuy commented Jan 12, 2018

lukehsiao commented Mar 7, 2018 •

edited

Loading

anentropic commented Mar 13, 2018

old html5lib makes pip dysfunctional #588

old html5lib makes pip dysfunctional #588

Comments

alanpurple commented Sep 29, 2017 • edited Loading

chihuahua commented Sep 29, 2017

alanpurple commented Sep 29, 2017

chihuahua commented Sep 29, 2017

alanpurple commented Sep 29, 2017

jart commented Oct 3, 2017

alanpurple commented Oct 3, 2017

All requested packages already installed.

packages in environment at C:\Anaconda3:

All requested packages already installed.

packages in environment at C:\Anaconda3:

jart commented Oct 3, 2017

alanpurple commented Oct 3, 2017 • edited Loading

jart commented Oct 3, 2017

dstufft commented Oct 3, 2017

alanpurple commented Oct 3, 2017 • edited Loading

dstufft commented Oct 3, 2017

henrysky commented Oct 4, 2017 • edited Loading

jart commented Oct 5, 2017

straygar commented Oct 19, 2017 • edited Loading

marcotama commented Oct 19, 2017

gwerbin commented Oct 20, 2017

kalefranz commented Oct 21, 2017

msarahan commented Oct 21, 2017

jart commented Oct 21, 2017

elipeters commented Oct 24, 2017

ghost commented Oct 25, 2017

msarahan commented Oct 25, 2017

back2yes commented Nov 14, 2017

alanpurple commented Nov 16, 2017

msarahan commented Nov 16, 2017

alanpurple commented Nov 16, 2017

BenjaminWegener commented Dec 2, 2017

halfhorst commented Jan 9, 2018 • edited Loading

fatih-iver commented Jan 11, 2018

MikeTheWatchGuy commented Jan 12, 2018

lukehsiao commented Mar 7, 2018 • edited Loading

anentropic commented Mar 13, 2018

alanpurple commented Sep 29, 2017 •

edited

Loading

alanpurple commented Oct 3, 2017 •

edited

Loading

alanpurple commented Oct 3, 2017 •

edited

Loading

henrysky commented Oct 4, 2017 •

edited

Loading

straygar commented Oct 19, 2017 •

edited

Loading

halfhorst commented Jan 9, 2018 •

edited

Loading

lukehsiao commented Mar 7, 2018 •

edited

Loading