Python 3.11: What are the best new features?

One of - if not THE - most popular programming languages just received an update: Python 3.11 released on October 24th !

After new dictionary functions in Python 3.9 and structural pattern matching in Python 3.10, let’s explore this new version!

Key features of Python 3.11

Python 3.11 brings a number of quality-of-life improvements, and some of them can be really impactful on your projects! Here are the main features of this new version.

1. Increased performance

This is the major improvement of Python 3.11: depending on the task, it is announced to be between 10% to 60% faster!

To check this, let’s run the pyperformance benchmark suite.

The pyperformance benchmark suite is focused on real-world benchmarks and uses whole applications when possible. It is a great tool for detecting potential regressions and validating optimization changes!

Install it:

python3 -m pip install pyperformance

And then run it twice, once for each Python version you want to benchmark (be careful to change your python version between the tests!)

pyperformance run -o py310.json

# Change python version here (check the "How can I try Python 3.11?" section)

pyperformance run -o py311.json

The -o argument lets you choose where you want to save the results of the tests.

Once the benchmarks are done, you can compare them:

pyperformance compare py310.json py311.json

For each test, pypefomance will compare the results of each version of Python. Here are a few of them:

The results are pretty great: Python 3.11 is faster than 3.10 on 56 of the 61 tests!
On average, the tests are executed 30% faster than before!

Here is a breakdown of the performance improvements:

These speed improvements mainly come from:

• Faster startup: Python 3.11 caches code to speed up module loading
• Better handling of recursion: thanks to improved handling of the C implementation, recursive functions are now much faster and can go deeper than before
• Specialization of the interpreter: at runtime, Python tries to find patterns in the executing code, and applies specialized operations when it finds one

For more details about these improvements, you can read the release notes.

2. Improved error messages

Did you ever lose several hours trying to fix an error, just to realize that what you were trying to fix was already functional and that the mistake was somewhere else?

Usually, you then think “I should have paid attention to the error message, instead of trying to fix what I thought was the error!”.

Sure. But sometimes, the error message can be quite ambiguous!

Take this code for instance:

This code generates two objects, an array and a vector, which should look like this:

array: [[1, 5, 6], vector: [2, 6, 7, 1, 2, 6]
[1, 1, 2]

Now, suppose that we want to access and sum specific elements of these objects:

Until now, the python interpreter would have thrown the following error:

print(array[2][2] + vector[2])
IndexError: index 2 is out of bounds for axis 0 with size 2 

This is an accurate description of the error, but without pointing out the exact instruction causing it: is the issue about the array or the vector?

With Python 3.11, the same error becomes:

print(array[2][2] + vector[2])
      ~~~~~^^^
IndexError: index 2 is out of bounds for axis 0 with size 2

The new interpreter not only points out the incorrect instruction, but also the problematic argument!

3. Clearer exceptions

Python 3.11 comes with some improvements to exception handling.

Exception notes

First, exception notes: exceptions now possess a new .__notes__ attribute, which can be set using the .add_note method.

For example, let’s take a look at the following piece of code:

Since “foo” can’t be cast to a float, this code raises the following error:

ValueError: could not convert string to float: 'foo'

This message contains all the necessary information to understand what caused the error. However, you might want to give a bit more context while raising the error, to explain why it happened:

Although the print is a decent workaround, it is not a satisfying way to handle the issue:

• Conceptually: it disconnects the error message from the related object
• Practically: if the error is dealt with later, you risk having an orphan print, without the associated error message

Exception notes tackle these two issues:

The stack trace now displays:

ValueError: could not convert string to float: 'foo'
The user shouldn't be able to input string values at step XXX

These exception notes are particularly useful with another improvement of Python 3.11: Exception groups!

Exception groups

There are several cases where unrelated exceptions need to be propagated together:

• Some libraries (like asyncio or trio) handled async concurrency, running several tasks in parallel and then aggregating the results
• Python’s socket library may attempt to connect to different addresses and return multiple errors if all attempts fail
• Complex calculations can raise multiple errors, a good example of which is the hypothesis library. It tries variations of a similar calculation and thus can raise different exceptions

Until now, the interpreter could only propagate at most one exception at a time.

The new exception groups are a way to handle this issue.

raise ExceptionGroup("group", [KeyError("foo"), ValueError(13), KeyError("bar")])

Raises:

| ExceptionGroup: group (3 sub-exceptions)
  +-+---------------- 1 ----------------
    | KeyError: 'foo'
    +---------------- 2 ----------------
    | ValueError: 13
    +---------------- 3 ----------------
    | KeyError: 'bar'
    +------------------------------------

Exception groups can be nested, and provide a clear visualization of all the errors raised.

They handle exception notes:

keyerror1 = KeyError("foo")
keyerror1.add_note("foo is rarely an appropriate value")

raise ExceptionGroup("group", [keyerror1, ValueError(13), KeyError("bar")])

Raises:

| ExceptionGroup: group (3 sub-exceptions)
  +-+---------------- 1 ----------------
    | KeyError: 'foo'
    | foo is rarely an appropriate value
    +---------------- 2 ----------------
    | ValueError: 13
    +---------------- 3 ----------------
    | KeyError: 'bar'
    +------------------------------------

To conclude on exception groups, Python 3.11 adds a new clause specifically designed to handle them: except*

For example, by tweaking the previous snippet:

We can implement a different behavior for different exceptions.

Here, the result is:

| ExceptionGroup: group (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | KeyError: 'foo'
    | foo is rarely an appropriate value
    +---------------- 2 ----------------
    | KeyError: 'bar'
    +------------------------------------

The ValueError is simply ignored, while the KeyErrors are still raised!

Additional features

Besides the essential features already discussed earlier, Python 3.11 brings a ton of quality-of-life improvements!

Among them:

• A new lib in the standard library for parsing TOML files: tomllib
• New typing features: a new Self type, to annotate methods that return instances of the same class, a LiteralString type to prevent SQL injection, and the possibility to tag some items as optional in TypeDicts
• Some improvements to the re module: regular expressions now support atomic grouping and possessive quantifiers, which both improve performance when used appropriately

How can I try Python 3.11?

Python 3.11 released on October 24th, but you can download it here !

To learn more about Python version management, check this article which explains how to use pyenv and virtualenv!

Conclusion

This article covers the most impactful changes of Python 3.11, but there are many other technical upgrades: feel free to check the official documentation!

Finally, to strengthen your Python skills, I strongly recommend reading these articles about the zen of python  or python type testing

Contact-us for more informations