Parse settings from pyproject.toml

[miigotu]

This project is the last of my dependencies that can only read from setup.cfg. the domains that I think should be used are something like:

pyproject.toml:

[ tool.babel.extract_messages ]
message_extractors="""{
    'gui': [
        ('**/js/*.min.js', 'ignore', None),
        ('**/js/*.js', 'javascript', {'input_encoding': 'utf-8'})
    ]}"""
#options
[ tool.babel.compile_catalog ]
#options
[ tool.babel.init_catalog ]
#options
[ tool.babel.update_catalog ]
#options

[miigotu]

@akx @benselme @blagasz @kvesteri @mitsuhiko @sils
Sorry for the ping but I'd really like to see this, and I'm willing to add a PR to make this happen. Before I waste my time, is there a policy or stopper against accepting a change pr that accomplishes this? I just don'y want to waste my time if so.

[miigotu]

I'd also like to submit a patch for the f-string problem. There is not a nice way to do it, but I think we could add a hack for it (which includes a small speed expense) that depends on a config/pyproject setting or environment variable or something to enable. Possibly with a new/different import when you gettext_install. That way it would not be a breaking change, and would avoid slowing down every implementation unless they enabled the feature.

[akx]

@miigotu There's no policy or stopper preventing a PR for pyproject.toml.

However:

• care should be taken to keep Babel working even when no TOML capabilities are present,
• the format of the TOML configuration should be well-thought-out in advance. The draft in the original message here seems a bit icky with a Python literal being embedded in the TOML.

As for "the f-string problem"... I think that's quite unrelated here?

[miigotu]

Yeah I messed with the spec for it a bit last night, and pulled a global options out. Toml allows multiple definitions of the same key, so

[tool.babel.config.extractors]
type = "python" 
options =
paths =
match = "**/source/*.py"

[tool.babel.config.extractors]
type = "js"
match = "**/js/*.js

Works, but I need to think about it more.

[AbdealiLoKo]

Is my understanding right that if I am not using setuptools I cannot use any config file (even setup.cfg) ?
i.e. the pybabel CLI command does not read from any config file ? the only way to give it inputs is the command line arguments ?

I don't mind not using pyproject.toml - but if pybabel could read from a .babel or .babelrc or even just take configs from setup.cfg ... that would be fine

[miigotu]

Ideally nobody should have to add a secondary config file to their project when pyproject.toml is the preferred and python team recommended method for new projects, and handles all possible use cases in the spec for any tools that is willing to implement their config from it. It was sort of the purpose of the pep, to make it so we didn't need 10 different files for configuration cluttering up our source code, and searching all over the place for where settings came from in larger projects. Setuptools is being phased out in most projects. I understand keeping backwards compatibility, but forwards compatibility keeps tools relevant for people who are perfectionists and prefer some level of interoperability and conformity to standards.

[jpmckinney]

Setuptools is being phased out in most projects

? What are people using to build packages if not setuptools? (enscons?)

[AbdealiLoKo]

I am moving a package to use poetry - I pinged about the config file because of this

I also use basic build in some projects

[miigotu]

Setuptools is being phased out in most projects

? What are people using to build packages if not setuptools? (enscons?)

Poetry or flit are the leading tools to package new projects.

Peps 517, 518, 621, 631, 633, 660 (and more) provide for pyproject.toml to include project metadata, build system information, dependencies (including extras and optional named groups), configuration for third party tools and requirements in the expanded toml, elimination for the need of requirements.txt, requirements-dev.txt, requirements-win.txt, setup.py, setup.cfg, and numerous other files in a project. Ideally, a project should only need a readme, pyproject.toml, code, and possibly contributing.md and a LICENSE file. Everything else can be handled in the pyproject.toml provided the tools you use read configuration from it.

I use about a dozen tools in my project, python-babel is the only one that does not check the pyproject except for .editorconfig.

https://github.com/python-poetry/poetry
https://flit.pypa.io/en/stable/

These eliminate the need for multiple tools to build, package, and publish separately. Adding/removing depends, installing (editable or not), multiple virtualenv functionality for testing or building, building/packaging, and pushing to pypi are all handled with one tool.

Particularly with poetry, you dont need to use virtualenv, pipenv, tox, twine, pip, setuptools, pyenv, or anything. You can even define custom commands to run, inside the pyproject, using poethepoet. It's a complete tool, that is extensible with plugins.

Setuptools itself even works and reads config from pyproject.toml https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

[jpmckinney]

FWIW flake8 also doesn't support pyproject.toml PyCQA/flake8#234

Anyway, this project is open to a PR so why not just contribute one? #777 (comment)

Edit: Off-topic, but I use the pyenv-virtualenv Homebrew package, which automatically installs pip and setuptools, and pip-tools for my applications (not my packages). Then I use the https://github.com/pypa/gh-action-pypi-publish GitHub Action, which uses twine to publish the package. Don't need pipenv (alternative to pip-tools) or tox (just create a matrix in GitHub Actions). I guess one CLI with many packages as plugins might be preferred to multiple CLIs from many packages... Anyway, +1 to pyproject.toml support – I prefer to have fewer config files, too.

[akx]

Poetry or flit are the leading tools to package new projects.

I disagree - I've been starting to use PyPA's own hatch for my libraries as that's what the official tutorial primarily suggests, and build to do the building.

[miigotu]

FWIW flake8 also doesn't support pyproject.toml PyCQA/flake8#234

Anyway, this project is open to a PR so why not just contribute one? #777 (comment)

Edit: Off-topic, but I use the pyenv-virtualenv Homebrew package, which automatically installs pip and setuptools, and pip-tools for my applications (not my packages). Then I use the https://github.com/pypa/gh-action-pypi-publish GitHub Action, which uses twine to publish the package. Don't need pipenv (alternative to pip-tools) or tox (just create a matrix in GitHub Actions). I guess one CLI with many packages as plugins might be preferred to multiple CLIs from many packages... Anyway, +1 to pyproject.toml support – I prefer to have fewer config files, too.

Even though most projects are using something like black/blackd instead of flake8, flake8 does in fact support pyproject.toml
https://github.com/SickChill/sickchill/blob/master/pyproject.toml#L125

[miigotu]

Poetry or flit are the leading tools to package new projects.

I disagree - I've been starting to use PyPA's own hatch for my libraries as that's what the official tutorial primarily suggests, and build to do the building.

What are you disagreeing with? Flit and poetry are the two leading projects for pyproject based projects. I never said they are the ONLY options. Even hatch uses pyproject only.

[miigotu]

FWIW flake8 also doesn't support pyproject.toml PyCQA/flake8#234

Anyway, this project is open to a PR so why not just contribute one? #777 (comment)

Edit: Off-topic, but I use the pyenv-virtualenv Homebrew package, which automatically installs pip and setuptools, and pip-tools for my applications (not my packages). Then I use the https://github.com/pypa/gh-action-pypi-publish GitHub Action, which uses twine to publish the package. Don't need pipenv (alternative to pip-tools) or tox (just create a matrix in GitHub Actions). I guess one CLI with many packages as plugins might be preferred to multiple CLIs from many packages... Anyway, +1 to pyproject.toml support – I prefer to have fewer config files, too.

And as far as your edit, I was listing projects that you absolutely have no need for with flit or poetry, because the functionality of all of them is included. I also publish with GitHub actions (again, there's a thousand ways to do this)

The point is that setup.py and setup.cfg are antiquated ways of managing a project, especially if you only have one tool requiring a setup.cfg or .babelrc, when there is a file specifically created and extended for this purpose of configuration of 3rd party tools.

Open source software is always going to have many ways to do things. Our job is to make it easier to do them, and simplify the process, to foster future growth of developers and make their life easier while using our software (and the language we are using).

Just because something CAN be done a certain way, doesn't mean it is the best way. "Hacking" is literally the process of finding things that can be improved and then improving them. When I get around to this, I'll try to make the PR. Unless someone more familiar with this projects source and coding conventions can get to it before then.

[miigotu]

FWIW flake8 also doesn't support pyproject.toml PyCQA/flake8#234

Apologies, I use flake9 (fork of flake8) that keeps rebased on flake8 and accepts configuration from pyproject.toml, because of their similar reluctance to accept widely accepted PEPs.

Silly to me for flake8 to not accept PEPs when the entire project is a result of a PEP specifying code formatting 😂 Afaik, the initial refusal was due to a pip bug when pyproject was initially implemented, that was fixed 2 years ago. Flake8 itself might even work with the pyproject config now, but I have not personally tried it since flake9 has continued to stay up to date and all of the flake8 plugins usually have flake9 counterparts, such as flake9-isort.

I don't think that reluctance to adhere to a new configuration format should be a reason to cause a forking event, but some people have ideas in their head sometimes that are hard to change, myself included. It causes projects to become obsolete, which is why things like black are born to replace them. With the advent of f-strings, and not being able to use pyproject, something similar could happen to babel, but I think a few things could just be fixed here instead.

I just need to find the time away from work and all of my projects to attempt the PR here. It's my birthday today (40 ugh) so commenting is all I'm going to do for today lol.

[akx]

@miigotu Happy birthday! 🍰

Also, yeah, "I disagree" may have been a bit inaccurate - what I meant was that I don't think poetry and flit are necessarily the two most popular tools, and setuptools itself also does building based on TOML just fine...

[miigotu]

@miigotu Happy birthday! 🍰

Also, yeah, "I disagree" may have been a bit inaccurate - what I meant was that I don't think poetry and flit are necessarily the two most popular tools, and setuptools itself also does building based on TOML just fine...

Thanks. So yeah, setuptools does work with pyproject. Setuptools is not really recommended even by pyca or any of the other major groups anymore, but the true successor has not emerged just yet. The few build systems that have came about all have their pros and cons, but that's kind of what the purpose was, for options in how you want your project to be built and distributed.

Poetry might be the best option for rust and c extensions once the documentation is improved for that part, but it's definitely a fast changing environment for build and deployment ecosystems at the moment. Heck I might even release a project to work around f-strings in a nice way, and have it disabled unless someone absolutely needs translations because of the performance loss.

Point being there is no reason to continue being one of the last major projects that requires a setup.{cfg,py} or a separate configuration file. It's much nicer to have all of that configuration in one place.

[ikus060]

Sorry to bring that up again. It's a old issue but still un-resolved.

What is the roadmap to support the new pyproject.toml ? I'm using python setup.py extract_messages and other babel integration with setuptools in my project. I'm looking for a way to migrate it to pyproject.toml. If even possible.

[rossnomann]

@ikus060 You can use babel.cfg as a mapping file:

[python: **.py]
encoding = utf-8

[jinja2: **.jinja2.html]
encoding = utf-8

And run pybabel directly. I use a task runner plugin for poetry:

[tool.poe.tasks.xgettext]
cmd = "pybabel extract -F babel.cfg --input-dirs=path/to/project -o path/to/locale/messages.pot"

[blaise-io]

For those like @ikus060 who switched from setup.py to pyproject.toml and like to keep their setup.cfg:
You could use

python -c 'from setuptools import setup; setup()' extract_messages

[ThomasWaldmann]

Maybe there could be a quick hack until someone wants to do a bigger PR:

https://tox.wiki/en/latest/config.html#pyproject-toml

Something like tox has done there would already enable people to get rid of babel.cfg in their toplevel directory.

[jpmckinney]

This works fine for me in pyproject.toml, for defining new extractors:

[project.entry-points."babel.extractors"]
mystuff = "my.module:extract_mystuff"

Combined with a mapping in a babel.cfg file, that is.

[akx]

@jpmckinney Hi there! Since Babel 2.16 (brand new), you should be able to put the mapping configuration in pyproject.toml too; see the syntax over at #1108. Would be grand if you gave it a shot in a Real Life project :)

[jpmckinney]

@akx What does the configuration look like? There are many approaches discussed in the PR, and I can see the (supposedly working) TOML example in the PR's tests, but if I do: [[babel.mappings]] I get babel.messages.frontend.ConfigurationError: pyproject.toml: No 'tool.babel' section found in file, and if I do [[tool.babel.mappings]], then I get no errors but I also get no messages extracted.

Edit: Ah, I made a typo in my pyproject.toml config. Works like:

[[tool.babel.mappings]]
method = "jinja2"
pattern = "**.html"
encoding = "utf-8"
ignore_tags = "script,style"
include_attrs = "alt title summary"