Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510444 - easy_install can't install anything because it uses HTTP but pypi.python.org requires HTTPS
Summary: easy_install can't install anything because it uses HTTP but pypi.python.org ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: python-setuptools
Version: 6.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Charalampos Stratakis
QA Contact: Mirosław Zalewski
URL:
Whiteboard:
Depends On:
Blocks: 1494507 1519849
TreeView+ depends on / blocked
 
Reported: 2017-11-07 12:24 UTC by Radek Bíba
Modified: 2018-08-21 16:17 UTC (History)
14 users (show)

Fixed In Version: python-setuptools-0.6.10-4.el6
Doc Type: If docs needed, set a value
Doc Text:
Previously, the easy_install utility failed to download packages from the Python Package Index (PyPI) repository. This happened because PyPI has recently switched to using the HTTPS protocol, which easy_install did not support. With this update, easy_install is now able to connect to PyPI using HTTPS, and, as a result, the described problem no longer occurs.
Clone Of:
: 1519849 (view as bug list)
Environment:
Last Closed: 2018-08-21 16:17:57 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Radek Bíba 2017-11-07 12:24:10 UTC
Description of problem:
The easy_install utility worked fine a few weeks ago but now it's broken. See, for example, here:

# easy_install stitches
Searching for stitches
Reading http://pypi.python.org/simple/stitches/
Couldn't find index page for 'stitches' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for stitches
error: Could not find suitable distribution for Requirement.parse('stitches')

Now, stitches *is* an existing project. Open the first URL -- http://pypi.python.org/simple/stitches/ -- and you'll see it. But, the URL has become https... and easy_install didn't switch to that one or didn't use the HTTPS URL automatically. FWIW, it does the latter in RHEL 7.

A potential workaround is using "-i https://pypi.python.org/simple". This works for some projects, but if the dependency chain contains something that wants to install stuff using easy_install again, the installation of that dependency fails. This is the case with stitches:

# easy_install -i https://pypi.python.org/simple stitches
Searching for stitches
Reading https://pypi.python.org/simple/stitches/
Best match: stitches 0.11
Downloading https://pypi.python.org/packages/9d/73/681005f0f0fb9483f0d94547a35059c776144fc998780b9ef00167a50f3d/stitches-0.11.tar.gz#md5=0ac0b6e418767e6ec5cf684460d1972f
Processing stitches-0.11.tar.gz
Running stitches-0.11/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ZMIU8w/stitches-0.11/egg-dist-tmp-82ctsZ
zip_safe flag not set; analyzing archive contents...
stitches.connection: module references __file__
Adding stitches 0.11 to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/stitches-0.11-py2.6.egg
Processing dependencies for stitches
...
...
Searching for pynacl>=1.0.1
Reading https://pypi.python.org/simple/pynacl/
Best match: PyNaCl 1.2.0
Downloading https://pypi.python.org/packages/fa/38/0f35ec4beb6562f1abfa07914db1cea978e93da409ba6293f810d9e677d6/PyNaCl-1.2.0.tar.gz#md5=7a31e95f5f457184b1660b71d6808b19
Processing PyNaCl-1.2.0.tar.gz
Running PyNaCl-1.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-a0vNJS/PyNaCl-1.2.0/egg-dist-tmp-QYvAk9
Checking .pth file support in .
/usr/bin/python -E -c pass
Searching for cffi>=1.4.1
Reading http://pypi.python.org/simple/cffi/
Couldn't find index page for 'cffi' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for cffi>=1.4.1
error: Could not find suitable distribution for Requirement.parse('cffi>=1.4.1')

So the only solution is hacking the source code:

# sed -i s,http://pypi.python.org,https://pypi.python.org, /usr/lib/python2.6/site-packages/setuptools/command/easy_install.py

Another (unrelated) issue appears later in the process (that's another story), but at least everything is downloaded using HTTPS. So please tweak the easy_install.py file.

Version-Release number of selected component (if applicable):
python-setuptools-0.6.10-3.el6.noarch

Comment 1 Charalampos Stratakis 2017-11-07 14:41:18 UTC
easy_install is not the recommended way of installing packages from pypi on RHEL 6 as unfortunately the version of setuptools that we ship is way too old. Combined with the fact that RHEL 6 is currently on production phase 3, this issue is not one that we can realistically fix.

The recommended way to install packages from pypi on RHEL 6 is through pip that is shipped with the python software collection.

Comment 2 Radek Bíba 2017-11-07 15:28:16 UTC
I prefer pip myself, but it doesn't always help -- in some cases easy_install is called while installing a dependency. Here's a reproducer:

# pip install pynacl
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting pynacl
/usr/lib/python2.6/site-packages/pip-9.0.1-py2.6.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.6/site-packages/pip-9.0.1-py2.6.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading PyNaCl-1.2.0.tar.gz (3.3MB)
    100% |████████████████████████████████| 3.3MB 369kB/s 
    Complete output from command python setup.py egg_info:
    Couldn't find index page for 'cffi' (maybe misspelled?)
    No local packages or download links found for cffi>=1.4.1
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-OCphNh/pynacl/setup.py", line 248, in <module>
        "Programming Language :: Python :: 3.6",
      File "/usr/lib64/python2.6/distutils/core.py", line 113, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python2.6/site-packages/setuptools/dist.py", line 221, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File "/usr/lib/python2.6/site-packages/setuptools/dist.py", line 245, in fetch_build_eggs
        parse_requirements(requires), installer=self.fetch_build_egg
      File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 538, in resolve
        dist = best[req.key] = env.best_match(req, self, installer)
      File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 780, in best_match
        return self.obtain(req, installer) # try and download/install
      File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 792, in obtain
        return installer(requirement)
      File "/usr/lib/python2.6/site-packages/setuptools/dist.py", line 293, in fetch_build_egg
        return cmd.easy_install(req)
      File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 466, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi>=1.4.1')
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-OCphNh/pynacl/

Comment 3 Charalampos Stratakis 2017-11-07 16:07:03 UTC
Reopening the bugzilla for the time being. Will request an opinion from PM.

Comment 4 Charalampos Stratakis 2017-11-07 17:53:32 UTC
High-level overview of the issue:

easy_install, a binary we ship with the python-setuptools package, no longer fulfills its primary purpose due to an API change in a third-party service.

easy_install is used to download packages from PyPI, a repository of third-party Python libraries. PyPI recently switched to supporting https requests only [0]. Due to that change, easy_install is not able to fetch packages from the default repository anymore, so use cases that required that functionality are currently broken on RHEL 6 systems.

We did not receive customer reports about this yet, but since this affects intallation, we worry that they'd only notice when they need to re-install a system in an emergency.

Thus requesting the opinion of PM in the matter. Please note that depending on the severity of the issue, this bugzilla can be considered a z-stream candidate.

[0] https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html

Comment 5 Honza Horak 2017-11-10 14:04:17 UTC
We talked about this with James yesterday, and it actually seems like something we should try to solve -- however, by solving we might either mean a fix, or maybe we can be fine with a work-wround/KCS article if there is something users can do to fix the issue.

So, the question is -- is there any way to mitigate the issue, other than fixing the package?

Comment 6 Petr Viktorin 2017-11-10 14:29:56 UTC
The tool itself has an option for alternate index URL, but transitive dependencies will call it unconfigured (see comment 2).

A possible workaround is to install transitive dependencies first, i.e. in the example above, install `cffi` with pip before installing `pynacl`.
However, there's no good way to list all transitive dependencies, so users would need to try installing what they need, on error install the first missing dependency, and repeat.

Comment 7 Charalampos Stratakis 2017-11-10 14:45:58 UTC
(In reply to Honza Horak from comment #5)
> We talked about this with James yesterday, and it actually seems like
> something we should try to solve -- however, by solving we might either mean
> a fix, or maybe we can be fine with a work-wround/KCS article if there is
> something users can do to fix the issue.
> 
> So, the question is -- is there any way to mitigate the issue, other than
> fixing the package?

Per comment 1 the fix to the package would be relatively simple:

--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -181,7 +181,7 @@ class easy_install(Command):
                 else:
                     self.all_site_dirs.append(normalize_path(d))
         if not self.editable: self.check_site_dir()
-        self.index_url = self.index_url or "http://pypi.python.org/simple"
+        self.index_url = self.index_url or "https://pypi.python.org/simple"
         self.shadow_path = self.all_site_dirs[:]
         for path_item in self.install_dir, normalize_path(self.script_dir):
             if path_item not in self.shadow_path:

A workaround could be that the same patch can be applied on a running system to the /usr/lib/python2.6/site-packages/setuptools/command/easy_install.py (and then the respective .pyc and .pyo files have to be deleted as well).

That will essentially have the same effect, however this could prove problematic due to meddling with protected directories here (e.g. SELinux will then always complain from that point when using easy_install, due to python trying to generate the .pyc and .pyo files).

All in all, a fix to the package seems less problematic than the available workarounds.

Comment 9 Marcel Kolaja 2017-12-01 08:14:56 UTC
(In reply to Radek Bíba from comment #0)
> Description of problem:
> The easy_install utility worked fine a few weeks ago but now it's broken.

Radek, does this mean that it is a Regression? Thanks!

Comment 10 Radek Bíba 2017-12-01 08:28:28 UTC
Kind of. There was no change on the setuptools side that would cause this breakage, but the change at pypi.python.org has a negative impact on easy_install. I guess we could call it an indirect regression.


Note You need to log in before you can comment on or make changes to this bug.