Twine check fails due to long-description sections

[bgklein]

Your Environment

1. Your operating system:
   Windows 10

2. Version of python you are running:
   Python 3.7.3

3. How did you install twine? Did you use your operating system's package manager or pip or something else?
   pip install twine

4. Version of twine you have installed (include complete output of):
   twine version 1.13.0 (pkginfo: 1.5.0.1, requests: 2.22.0, setuptools: 30.4.0,
   requests-toolbelt: 0.9.1, tqdm: 4.32.2)

5. Which package repository are you targeting?
   azure-batch-extensions

The Issue

twine check and also twine upload fails to upload long-description with section header. The error message received is

The project's long_description has invalid markup which will not be rendered on PyPI. The following syntax errors were detected:
line 1: Severe: Unexpected section title or transition.

=====

Current setup I am testing with is:

with open('README.rst', 'r', encoding='utf-8') as f:
    README = f.read()

setup(
    name='azure-batch-extensions',
    version=version,
    description='Microsoft Azure Batch Extended Features',
    long_description_content_type='text/x-rst',
    long_description=README,
    author='Microsoft Corporation',
    author_email='[email protected]',
    url='https://github.com/Azure/azure-batch-cli-extensions',
    classifiers=CLASSIFIERS,
    namespace_packages=[
        'azext'
    ],
    packages=[
        'azext.batch',
        'azext.batch.operations',
        'azext.batch.models'
    ],
    install_requires=DEPENDENCIES,
    extras_require=DEPENDENCIES_27,
)

and the README.rst is

title
=====

content

Interestingly the following README.rst passes

aaa
===

content

Steps to Reproduce

python setup.py sdist the above setup in a setup.py
twine check .\dist\*.tar.gz

[bgklein]

Looking at the renderer it seems to fail to render valid rst?

>>> from readme_renderer.rst import render
>>> README
'aaaaaaa\n=======\n\ncontent\n'
>>> render(README)
'<p>content</p>\n'

[di]

What is the exact value of the header? This may be a duplicate of pypa/readme_renderer#144

[bgklein]

An example of the README after read from file is

>>> README
'aaaaaa\n======\n\ncontent\n'

[sigmavirus24]

So there are several things that seem suspicious to me:

https://github.com/Azure/azure-batch-cli-extensions/blob/6455cfc3d2f906efde84b845930d2663d44d6c7a/setup.py#L48-L57

You're concatenating two files that may or may not flow well together.

Looking at https://raw.githubusercontent.com/Azure/azure-batch-cli-extensions/master/README.rst your Developer Installation section looks oddly formatted in plain-text:

Developer Installation
----------------------

Preparing your machine
++++++++++++++++++++++
1.	Install Python 3.5.x from http://python.org. Please note that the version of Python that comes preinstalled on OSX is 2.7. 
2.	Clone your repository and check out the master branch.
3.	Create a new virtual environment “env” for Python 3.5 in the root of your clone. You can do this by running:

    **Windows**

    .. code-block:: bash

        python -m venv <clone root>\env

    **OSX/Ubuntu (bash)**

    .. code-block:: bash

        python –m venv <clone root>/env

4.	Activate the env virtual environment by running:

    **Windows**

    .. code-block:: bash

      <clone root>\env\scripts\activate.bat

    **OSX/Ubuntu (bash)**

    .. code-block:: bash

      . <clone root>/env/bin/activate

5.	Install the dependencies and load the command module as a local package using pip.

  .. code-block:: bash
  
    python scripts/dev_setup.py

I'm not certain that those would cause the header to fail.

I also believe we use the long-description in your PKG-INFO file which you have not provided. Finally you seem to be using an older version of setuptools which could be contributing to the problem. I also don't know what the version of readme-renderer is that you're using. Perhaps that's not part of our version string reporting. We can't really help you without a full picture of your environment.

[bgklein]

I narrowed down the README section for debugging purposes. I have attached a simplified PKG-INFO from the above repro steps (simplified example of failure and converted to .txt for upload).
PKG-INFO.txt

[di]

Looking at the renderer it seems to fail to render valid rst?

>>> from readme_renderer.rst import render
>>> README
'aaaaaaa\n=======\n\ncontent\n'
>>> render(README)
'<p>content</p>\n'

This is not a failure to render, it would return None if it had failed. Instead readme_renderer is (by design) removing the first heading from the description.

[di]

The issue is that you have a mix of Unix and Windows style linebreaks in your PKG-INFO. Here's the same file with the correct line breaks, it passes twine check: PKG-INFO.txt

[di]

Which would make this a duplicate of pypa/setuptools#1390.