Maximum Line length and other styleguides

[friederschueler]

Should we move away from the original 80 char limit per line?

Most of the industry already moved on, because there is no technical reason to force this char limit.
Linux Torvalds, Robert C. Martin (Uncle Bob) and Raymond Hettinger all support longer lines to improve readability.
Google, Facebook, etc. all moved away from 80 char limit.
Django as a big python project for example also uses 120 (119) char limit.

[acolomb]

Some reasoning for my choice of 100 (actually a little bit less, like 96 seems practical for me personally): I almost always have a split screen in my IDE (Emacs) with two panes side-by-side on a 16" laptop display. One is the file I work on, the other is usually Git status, search results, checker output list, or some other file used for API reference on the currently worked on code.

I wouldn't like a hard limit pre-configured, but rather decide on a case-by-case basis when we encounter lines longer than the 100 characters. The vast majority of code lines will not even reach the limit of 80, if the code is well structured. If you ask black, it mostly makes a difference between arguments within parentheses on the same line, indented one level on the next one, or split in several lines at each comma:

long_function_being_called_with_many_args(foo_argument, bar_argument, baz_argument)
slightly_longer_beyond_the_limit(
    foo_argument, bar_argument, baz_argument, frob_argument
)
really_long_line = long_function_being_called_with_many_args(
    foo_argument,
    bar_argument,
    baz_argument,
)

[acolomb]

If you need very long lines the solution in 99% of the cases is refactoring.

Since we're talking Python here, I'm always a little concerned about performance. E.g. does assigning to a local variable and passing that to a function instead of the expression iself actually generate more byte-code? Does it change anything w.r.t. garbage collection?

I think that's the most common reaction to line length issues, assigning smaller parts to local names instead of building monster expressions. In C, I'm used to such things getting optimized away, but (C)Python? Sure it's not much, but such things accumulate to slightly slow down a complete program or negatively affect the CPU cache miss rate for example. If it really means allocating (later, freeing) a structure for the local variable and then doing a namespace lookup for the same in various possible scopes, that's non-negligible code for the CPU to execute. Which many people probably don't see because it's just a "single line of Python" for "just refactoring".

[erlend-aasland]

In general, I follow the advice of Martin Fowler: write well-structured, idiomatic code, and don't try to trick the compiler; instead trust it to perform the needed optimisations. (This goes for any programming language, including C and Python.)

Regarding Python and performance: A lot of stuff has happened in the VM the recent years. Since 3.11, we've had a specialising adaptive interpreter (PEP-659¹), and since 3.13 (to be released in October), we've got an experimental JIT and free-threading. Moreover, many targeted optimisations like inlined comprehensions and zero-cost exceptions have been added.

Does it change anything w.r.t. garbage collection?

Probably not; see the CPython devguide for more details about the GC and reference counting (which is what's in play here, and not the cycle-breaking GC).

TL;DR: write Pythonic code²³ and trust the compiler :)

That being said, I've spotted some low-hanging fruit wrt. optimisations in canopen. But first: code coverage; I'd rather not go down that road unless I'm certain I'm not introducing regressions.

Footnotes

1. yes, it's draft, but that's because Mark is writing for documentation purposes only ↩

2. https://mas.to/@davidism/112288910847353545 ↩

3. https://fosstodon.org/@erlendaasland/112207377798931192 ↩

[acolomb]

Thanks for the insights. So what does happen with the added local variable? Is it as much extra CPU work as I feared?

[erlend-aasland]

It depends. I would not worry too much about it; IMO, readability and maintainability are more important aspects. Moreover, it's easier to spot an optimisation in well-structured code; refactor first, optimise later :)

See also:

• Speeding up Python-to-Python calls. faster-cpython/ideas#661

[acolomb]

Coming here with an embedded systems background, it would be reassuring to trust the compiler / interpreter will make such easy optimizations. As long as I don't know it does in principle, the concern cannot be ignored completely while writing code. I do like to trust in optimization, but knowing such a trivial one is not made (partly because of the flexibility in an interpreted language -- blessing and curse) makes it hard. I've worked with enough different terrible compilers to expect the worst and anything else by default.

The basic problem with "optimize later" is that hardly anyone ever does it. Therefore knowing where a perfomance penalty lingers does affect coding style. If the trade-off is between ~100 character lines (slightly longer) and refactoring with a real cost in CPU cycles, then I definitely favor the former to support maintainability.

We've reached the point where a whole generation (majority) of programmers doesn't know anything about how a computer actually works. Everything is hidden under proliferating layers of abstraction, which makes everything look extremely simple, while it actually degrades performance (though well hidden) with every added layer. No wonder the "Apple II" still leads the competition for minimum key-to-screen latency, which does get very noticeable in many modern applications on hardware that could have landed on a million moons simultaneously 50 years ago. I think it's important to point out such things from time to time, to raise awareness (from usually zero) about the real consequences of adding one line of code and the trade-offs. And to stress how important it is for compilers to try real hard for good optimizations -- that's what machines are good at and what's desperately needed when collaborating with human developers.

End of rant :-)

[sveinse]

Regarding Python and performance: A lot of stuff has happened in the VM the recent years. Since 3.11, we've had a specialising adaptive interpreter (PEP-6591), and since 3.13 (to be released in October), we've got an experimental JIT and free-threading. Moreover, many targeted optimisations like inlined comprehensions and zero-cost exceptions have been added.

With respect to recent development in Cpython: While working on the asyncio port of canopen I noticed that python-can use threads and runs all RX messaging as callbacks in a worker thread and thus this affects data reception methods in canopen too. I raised this in #284 . Now, I'm not here to beat a dead horse, but perhaps @erlend-aasland can shed some insights to what should be our best practices with regards to thread protection in canopen. Mabye moreso if the GIL is about to change.

[erlend-aasland]

Unfortunately, I think I need to get to know this library (and python-can!) better in order to answer your question, @sveinse. I only recently started getting to know these, not to mention getting to know CAN/CANopen. Moreover, free-threading is a fast-moving vehicle in CPython's main branch. Some patches are backported to the upcoming 3.13 branch and some will only find its way to 3.14 (which is main right now).

That being said, I've got virtual environments for 3.13 and 3.14 set up for my canopen development. I didn't try the free-threading executables yet, but I'll give them a go in a day or two :)

[sveinse]

FYI the thread is created here:

https://github.com/hardbyte/python-can/blob/main/can/notifier.py#L80

And the thread worker is:

https://github.com/hardbyte/python-can/blob/main/can/notifier.py#L111-L140

(For async use crossing threads is not a problem, as python-can is using .call_soon_threadsafe())