support python3

[hainm]

Hi,

(from recent issue) I know that mdanalysis does not support Python3 yet. But I think mdanalysis dev should consider update the code to compat with Python3. I've been playing with mdanalaysis for a while but it's a bit inconvenient to switch back to Python2 to use mdanalysis. (of course thing's easier with conda).

if mdanalysis does not heavily depend on external libraries that only work with python2, I think it's quick to use mdanalysis for both 2 and 3 version. I did this with my pytraj with the help from six.py module.

If you guys want to make compat and need suggestion, I could help.

cheers.
Hai

[richardjgowers]

Hi Hai,

I'm definitely on board to help port to python 3.

I'm not an expert, but I think six is really useful for 2.5+, and I think we only support 2.6 onwards? @orbeckst ? So I think we have less potential headaches anyway.

Truth be told, I'm not 100% sure on what the limiting factor is on the port... I know we still use some relative imports (ie. import core) in the package, but that's all I'm really aware of.

[orbeckst]

On 29 Apr, 2015, at 00:46, Richard Gowers wrote:

I'm not an expert, but I think six is really useful for 2.5+, and I think we only support 2.6 onwards? @orbeckst ? So I think we have less potential headaches anyway.

Yes, we only support 2.6+.

Truth be told, I'm not 100% sure on what the limiting factor is on the port... I know we still use some relative imports (ie. import core) in the package, but that's all I'm really aware of.

See the other issues that I tagged with Python3 � but @seb-buch had been looking into it recently.

Oliver Beckstein * [email protected]
skype: orbeckst * [email protected]

[tylerjereddy]

Apparently you can remove all __init__.py files in Python3 as they aren't needed for package imports anymore, although that may not be a good idea. Interestingly, I've seen a lot of argument for using from . import module syntax recently. David Beazley's recent PyCon talk on importing / modules goes over this stuff in excruciating detail.

[hainm]

this is what I did for pytraj when make it compat with 2 and 3.

1. always keep in mind writing a compat code for both of them.
2. I wrote code in python2.7 and then use 2to3 to convert "print ...." in python2 to "print ()" in python3.
   (of course needed to add from __future__ import print_function). I did not use 2to3 to convert anything else because I wanted my code to run in both 2 and 3 without adding converting script in setup.py file. (just use 2to3 -p)
3. Use "six.py" to use zip (zip, izip), string_types (bytes, str, unicode...), pickle, ...
4. Explicitly use list, list list(range(8)) if you mean to return a list
5. I think the most headache is to use string in python2 and 3.
   I wrote most of my code in Cython so it does great help in dealing with string. For example, I just use mystring.endcode() to convert to C++ string and mystring.decode() to convert to python string and cython will take care of using correct string type. (to check if an object is a string, we can use string_types in six.py rather using isinstance(obj, basetring)
6. I myself prefer to use Cython (vs swig or weave (in your distance.py))
7. per absolute/relative import, I also prefer to use (from . import blah_blah (with the help from from __future__ import absolute_import). But care should be taken since we can introduce loop import.
8. Use delay import to avoid code broken if one of library does not exist:

def _import(modname):
    """has_numpy, np = _import('numpy')"""
    has_module = False
    try:
        imported_mod = __import__(modname)
        has_module = True
        return (has_module, imported_mod)
    except ImportError:
        has_module = False
        return (has_module, None)

[seb-buch]

Hi,
porting our python code to py3 is pretty much straight forward but the real
deal is porting the extensions as some of them need a complete rewrite to
be compatible with py3.
Though, I would like to tackle this issue by this summer.
Séb
Le 29 avr. 2015 17:37, "Hai Nguyen" [email protected] a écrit :

this is what I did for pytraj when make it compat with 2 and 3.

1. always keep in mind writing a compat code for both of them.
2. I wrote code in python2.7 and then use 2to3 to convert "print ...."
   in python2 to "print ()" in python3. (of course needed to add from
   future import print_function). I did not use 2to3 to convert
   anything else because I wanted my code to run in both 2 and 3 without
   adding converting script in setup.py file. (just use 2to3 -p)
3. Use "six.py" to use zip (zip, izip), string_types (bytes, str,
   unicode...), pickle, ...
4. Explicitly use list, list list(range(8)) if you mean to return a
   list
5. I think the most headache is to use string in python2 and 3. I
   wrote most of my code in Cython so it does great help in dealing with
   string. For example, I just use mystring.endcode() to convert to C++ string
   and mystring.decode() to convert to python string and cython will take care
   of using correct string type. (to check if an object is a string, we can
   use string_types in six.py rather using isinstance(obj, basetring)
6. I myself prefer to use Cython (vs swig or weave (in your
   distance.py))
7. per absolute/relative import, I also prefer to use (from . import
   blah_blah (with the help from from future import absolute_import).
   But care should be taken since we can introduce loop import.
8. Use delay import to avoid code broken if one of library does not
   exist:

def _import(modname):
"""has_numpy, np = _import('numpy')"""
has_module = False
try:
imported_mod = import(modname)
has_module = True
return (has_module, imported_mod)
except ImportError:
has_module = False
return (has_module, None)

—
Reply to this email directly or view it on GitHub
#260 (comment)
.

[kain88-de]

Are there any other obvious blockers to support python 3?

[richardjgowers]

If I remember correctly, izip and xrange don't exist in py3, so any use of them will have to be fixed. I think six has some sort of fix for this? Otherwise it's fairly easy to wrap the imports in a try/except block

[richardjgowers]

Otherwise, you can try running the unit tests in a py3 environment and see what breaks and make a list here. Our test coverage is pretty good, so I think we can use that.

[kain88-de]

I've setup a Py3 virtual-env. There are two things needed to get the tests to run in py3

GridDataFormat

To install it I needed remove ez_setup.py and depend on the user installed setuptools.
This does not guarantee that everything here will work since the module uses regular
expressions. The standard module is not cross version but there is a new replacement module that supports 2 and 3 regex.

C-Extensions

Likely most of the C-Extensions have to be rewritten to support 2 and 3 and the same time. Cython does this automatically. They are using macros to determine with which python version the extension is compiled. I'm not sure about SWIG since I have never used it.

[richardjgowers]

So for GridData, we can accept PRs for that too to make it py3 compatible.

For the C extensions, I think it's best to work through them one by one (or make a table of Extension against Language). The Cython and Swig should be ok, and any CPython should be rewritten into Cython

[orbeckst]

Please open an issue for GridDataFormats to use six... or whatever is needed to make it work with Py3. Thanks!

[richardjgowers]

https://github.com/MDAnalysis/mdanalysis/wiki/List-of-extensions

So I think the main blocking thing currently is our extensions, I've made a list of them and their current state.

[kain88-de]

So some good news. After having a python3 version for GridDataFormats I can actually run the complete setup for MDAnalysis under python 3. I still can't import it successfully because of syntax errors and some other small things which I will slowly work through.

btw did I see it right that NamedStream is only used for testing the library? If so I would like to remove it since the class cannot be constructed in python3 (Which is due to either/both of the base classes now being implemented in C instead of python).

[richardjgowers]

Yeah it looks like you're making progress which is awesome. Once we've got 0.11.0 out the door we can add py3 into the travis build matrix (but allow it to fail) and gradually patch it up.

[hainm]

awesome.

[kain88-de]

You can follow my progress to get the library load and tests to run here.

https://github.com/kain88-de/mdanalysis/tree/python3

After this I should have a pretty good overview of MDAnalysis and a long wishlist for refactoring.

[orbeckst]

btw did I see it right that NamedStream is only used for testing the library? If so I would like to remove it since the class cannot be constructed in python3 (Which is due to either/both of the base classes now being implemented in C instead of python).

NamedStream is needed to seamlessly interface other code with the stream-readers; that was pretty much the whole point. If you can come up with a better solution I am all ears.

Btw, why is it not possible to implement NamedStream in Python 3?

[kain88-de]

@orbeckst It was not possible to have the two base classes since they are now implemented in C and different layout in the underlying python dict. But it seems to be enough to just depend on io.IOBase. Test are still passing for me then under python 2.

[kain88-de]

Looks like ubuntu is starting to move away from python 2.7 for good.

http://summit.ubuntu.com/uos-1511/meeting/22568/python3-only-on-the-images/

[orbeckst]

Am Nov 5, 2015 um 3:11 schrieb kain88-de [email protected]:

Looks like ubuntu is starting to move away from python 2.7 for good.

http://summit.ubuntu.com/uos-1511/meeting/22568/python3-only-on-the-images/

... Which will decrease the suitability of Ubuntu for scientific computing.

[arose]

Anyone, please feel free to try this. I likely have no time for it.

@kain88-de recently rewrote the xdr module, and it supports Py3 now, so once we've got DCD done you could probably just import the MDAnalysis versions directly if you wanted.

I might do that. However my initial motivation with simpletraj was to have a package with minimal dependencies so that it is easier to install.

[mnmelo]

While testing a python3 installation (python3.4 setup.py install), for #798, I get the following:

...
Extracting MDAnalysis-0.14.1.dev0-py3.4-linux-x86_64.egg to /scratch/manel/virtualenv/lib/python3.4/site-packages
  File "/scratch/manel/virtualenv/lib/python3.4/site-packages/MDAnalysis-0.14.1.dev0-py3.4-linux-x86_64.egg/MDAnalysis/analysis/contacts.py", line 951
    """
     ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2458-2459: truncated \uXXXX escape

Adding MDAnalysis 0.14.1.dev0 to easy-install.pth file
...

The install seems to continue just fine, and I can import MDAnalysis at the end of it. I'm guessing it'll be an easy fix for python3-experienced people.

[orbeckst]

This is going to happen this summer, also thanks to NumFOCUS: http://www.mdanalysis.org/2017/06/03/numfocus-grant/

[richardjgowers]

@MDAnalysis/coredevs so Py3 is probably finished today! Should we also make 3.6 our default python development version? Ie run coverage and minimal builds on 3.6 rather than 2.7 as we currently do. This would then relegate 2.7 to something we support, but not develop around

[kain88-de]

As long as we keep builds for 2.7 around I'm for that. I would also suggest to support python 3.5 and maybe 3.4. (Something along the lines of the last 2-3 stable releases)

[kain88-de]

Also @richardjgowers & @tylerjereddy you are making awesome progress!

[mnmelo]

@kain88-de, since our userbase can't use MDAnalysis with python 3 right now, there's probably little code to break if we directly start supporting only python 3.6. It'd save us yet a couple more worries with compatibility code... Or do you think skipping python 3.4 and 3.5 will leave out many users?

[kain88-de]

What does ubuntu LTS have right now? CentOS 7 (latest) has python 3.4. Someone who doesn't use conda but a system python and a long term support OS will likely not have 3.6 and we would exclude those from upgrading to use python 3.

[mnmelo]

Ah, of course, it makes sense not to force the users to upgrade python version just for running MDAnalysis.

[orbeckst]

Then I suggest we aim for

• 2.7.x (until 2020)
• ≥ 3.4

Should we add a Travis cron job for tests on 3.4 and 3.5, maybe they don't have to run every time?

[mnmelo]

Those seems like good targets.
And since we're starting to have a clear idea of version support timelines, would this be a good time to get our logo under the Python 3 Statement?

In the future, dropping 3.4 and 3.5 support can be done along Python's EOL timeline, although I realize some adjustment might be needed to allow for for lagging LTS distros.

[jbarnoud]

Wow! I turn my head for 24 hours and it is a frenzy here! Good works guys!

I am all in favour of making python 3 our primary target; especially to encourage us, developer, to use MDAnalysis on python 3.

For the Python 3 Statement, I would wait for 0.17.0 to be released as it would let us some time to actually experience python 3.

[kain88-de]

Btw did anyone check if our examples in the docs work on Python 3?

[richardjgowers]

@jbarnoud we've been having a productive time

[richardjgowers]

https://travis-ci.org/MDAnalysis/mdanalysis/builds/254989078

Ok py3.6 build is passing!

[orbeckst]

Woot!

[orbeckst]

I feel this momentous moment deserves an announcement... maybe at the end of the week when we see what else @richardjgowers and @tylerjereddy will have accomplished.

[richardjgowers]

We're busy porting the library to perl right now

…

On Wed, 19 Jul 2017, 3:41 p.m. Oliver Beckstein, ***@***.***> wrote: I feel this momentous moment deserves an announcement... maybe at the end of the week when we see what else @richardjgowers <https://github.com/richardjgowers> and @tylerjereddy <https://github.com/tylerjereddy> will have accomplished. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#260 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AI0jB6167jiD4QegVAD_e-dfq3W3lYlwks5sPoYEgaJpZM4ELMru> .