The Python language is designed to make complex tasks simple, so updates iterate relatively quickly and require us to keep up with them!

The release of a new version is always accompanied by new features and functions, and the first thing we need to understand and attribute these points before upgrading the version, so that we may use them flexibly in our programming later. Can’t wait, ready to go, so let’s start now!

pyton 3.10

1. PEP 604

New Type Union Operator

In previous versions when you wanted to declare a type to contain multiple types, you needed to use Union[] to contain multiple types, now you can just use the | symbol to do it.

  • Old Version

    1
    2
    3
    4
    
    from typing import Union
    
    def square(number: Union[int, float]) -> Union[int, float]:
        return number ** 2
    
  • New Version

    1
    2
    
    def square(number: int | float) -> int | float:
        return number ** 2
    
  • can be used in isinstance and issubclass.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    
    In : isinstance('s', (int, str))
    Out: True
    
    In : isinstance(1, (int, str))
    Out: True
    
    In : isinstance('s', int | str)
    Out: True
    
    In : isinstance(1, int | str)
    Out: True
    

2. PEP 613

TypeAlias

Sometimes we want to customize a type, so we can create an alias (Alias) to do so. However, for the type checker (Type Checker), it can’t tell if it’s a type alias or a normal assignment, and now we can easily tell by introducing TypeAlias.

  • The previous writing style

    1
    
    Board = List[Tuple[str, str]]
    
  • The new version of the writing style

    1
    2
    3
    4
    
    from typing import TypeAlias
    
    Board:TypeAlias = List[Tuple[str, str]]  # This is a type alias
    Board = 2                                # This is a module constant
    

3. PEP 647

User-Defined Type Guards

In contemporary static checking tools (such as typescript, mypy, etc.) there is a feature called Type narrowing. This is when a parameter type could have matched more than one type, but under certain conditions the type range can be narrowed down to a smaller range of type(s).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
In : def show_type(obj: int | str):  # The argument obj is either int or str
...:     if isinstance(obj, int):    # Implementing Type narrowing, mypy confirms that obj is int
...:         return 'int'
...:     return 'str'  # Since the previous qualification is int, here mypy will confirm that obj is str
...:

In : show_type(1)
Out: 'int'

In : show_type('1')
Out: 'str'
  • Problems identified - Deficiencies exist

    A more accurate knowledge of the object’s type is very friendly to mypy, and the conclusions of the check will be more accurate; type narrowing can be problematic in some scenarios.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    
    In : def is_str(obj: object) -> bool:
    ...:     return isinstance(obj, str)
    ...:
    ...:
    ...: def to_list(obj: object) -> list[str]:
    ...:     if is_str(obj):
    ...:         return list(obj)
    ...:     return []
    ...:
    
    In : def is_str(obj: object) -> bool:
    ...:     return isinstance(obj, str)
    ...:
    ...:
    ...: def to_list(obj: object) -> list[str]:
    ...:     if is_str(obj):
    ...:         return list(obj)
    ...:     return []
    ...:
    
    In : to_list('aaa')
    Out: ['a', 'a', 'a']
    
    In : to_list(111)
    Out: []
    

    This code is simpler than the one mentioned in PEP, they are both correct codes and the type comments are problematic. But running mypy prompts an error message. In the 2 functions obj is used because it is not sure of the object type, so object is used, and in fact to_list will only handle obj as type str. Originally if is_str(obj) would have narrowed the type, but because it was split into functions, isinstance didn’t succeed in narrowing it here.

    1
    2
    3
    4
    5
    6
    
     mypy wrong_to_list.py
    wrong_to_list.py:7: error: No overload variant of "list" matches argument type "object"
    wrong_to_list.py:7: note: Possible overload variants:
    wrong_to_list.py:7: note:     def [_T] list(self) -> List[_T]
    wrong_to_list.py:7: note:     def [_T] list(self, Iterable[_T]) -> List[_T]
    Found 1 error in 1 file (checked 1 source file)
    
  • User-defined Type Guards are provided in the new version to resolve

    Originally the type of the return value was bool, now we specify it as TypeGuard[str] so that mypy can understand its type. Actually, to put it another way, you can read TypeGuard[str] as an alias for bool with a type declaration, so please understand this carefully.

    1
    2
    3
    4
    
    from typing import TypeGuard
    
    def is_str(obj: object) -> TypeGuard[str]:
        return isinstance(obj, str)
    
    1
    2
    
     mypy right_to_list.py
    Success: no issues found in 1 source file
    

4. PEP 612

Parameter Specification Variables

The type system in Python has limited support for types of Callable (such as functions), it can only specify the type of the Callable, but it cannot be propagated for arguments to function calls.

  • This problem exists mainly in the usage of decorators

    The argument value received by join is supposed to be a list of strings, but mypy does not validate this last print(join([1, 2])) correctly. Because the type of args and kwargs in the inner function in the log decorator is Any, this causes the type of the arguments chosen for the call to be unverified, and frankly, it can be written any way.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    
    from collections.abc import Callable
    from typing import Any, TypeVar
    
    R = TypeVar('R')
    
    def log(func: Callable[..., R]) -> Callable[..., R]:
        def inner(*args: Any, **kwargs: Any) -> R:
            print('In')
            return func(*args, **kwargs)
        return inner
    
    @log
    def join(items: list[str]):
        return ','.join(items)
    
    print(join(['1', '2']))  # Correct usage
    print(join([1, 2]))      # Wrong usage, mypy should prompt type error
    
  • Newer versions can use ParamSpec to resolve

    By using typing.ParamSpec, the argument type of inner is passed directly through P.args and P.kwargs for validation purposes.

    typing.ParamSpec helps us to facilitate [referencing] positional and keyword arguments, and this PEP addition of `typing. is to provide an ability to add, remove or convert parameters of another callable object.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    
    from typing import TypeVar, ParamSpec
    
    R = TypeVar('R')
    P = ParamSpec('P')
    
    def log(func: Callable[P, R]) -> Callable[P, R]:
        def inner(*args: P.args, **kwargs: P.kwargs) -> R:
            print('In')
            return func(*args, **kwargs)
        return inner
    
    1
    2
    3
    4
    
     mypy right_join.py
    right_join.py:22: error: List item 0 has incompatible type "int"; expected "str"
    right_join.py:22: error: List item 1 has incompatible type "int"; expected "str"
    Found 2 errors in 1 file (checked 1 source file)
    

    The more common added arguments I can think of are the [injected] type of decorators. The join function has 2 arguments, but since the first argument logger is [injected] in the with_logger decorator, you only need to pass the value of the items argument when you use it.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    import logging
    from collections.abc import Callable
    from typing import TypeVar, ParamSpec, Concatenate
    
    logging.basicConfig(level=logging.NOTSET)
    R = TypeVar('R')
    P = ParamSpec('P')
    
    def with_logger(func: Callable[Concatenate[logging.Logger, P], R]) -> Callable[P, R]:
        def inner(*args: P.args, **kwargs: P.kwargs) -> R:
            logger = logging.getLogger(func.__name__)
            return func(logger, *args, **kwargs)
        return inner
    
    @with_logger
    def join(logger: logging.Logger, items: list[str]):
        logger.info('Info')
        return ','.join(items)
    
    print(join(['1', '2']))
    print(join([1, 2]))
    

    In addition to adding, removing and converting parameters can also be done with Concatenate, see another example of removing parameters. With the remove_first decorator, the first argument passed in is ignored, so add(1, 2, 3) is actually computing add(2, 3). Be careful to understand where this Concatenate is, if it is added, then Concatenate is added to the type declaration of the Callable of the decorator argument, if it is removed, it is added to the type declaration of the returned Callable.

5. PEP 618

Parameters of the zip function

I believe that every Python developer with much working experience has experienced this trap, when the elements within the argument are of different lengths, the part with the longer length is ignored, without any hint, when in fact it was originally mentioned in the documentation.

1
2
In : list(zip(['a', 'b', 'c'], [1, 2]))
Out: [('a', 1), ('b', 2)]

But for most developers no one really pays attention to this documentation, the problem is very implicit and requires developers to understand the zip problem. The newer versions of strict=True will report an error when the length of the elements within the argument is different.

1
2
3
4
5
6
7
In : list(zip(['a', 'b', 'c'], [1, 2], strict=True))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In , in <cell line: 1>()
----> 1 list(zip(['a', 'b', 'c'], [1, 2], strict=True))

ValueError: zip() argument 2 is shorter than argument 1

6. PEP 626

Better error messages

There are some examples of error messages listed in the official that were not obvious and friendly enough in the previous version, this version deals with them centrally. Here they are: Same SyntaxError Isn’t the hint in the new version easier to understand? Not only does it suggest the type of error, it also gives hints.

  • The old way of writing

    1
    2
    3
    4
    5
    
    In : print("Hello, World!)
        File "<ipython-input-7-bc0808c61f64>", line 1
            print("Hello, World!)
                                ^
    SyntaxError: EOL while scanning string literal
    
  • New writing style

    1
    2
    3
    4
    5
    
    In : print("Hello, World!)
        Input In []
            print("Hello, World!)
                ^
    SyntaxError: unterminated string literal (detected at line 1)
    
  • The old way of writing

    1
    2
    3
    4
    5
    
    In : if b = 1: pass
    File "<ipython-input-8-e4cfea4b3624>", line 1
        if b = 1: pass
            ^
    SyntaxError: invalid syntax
    
  • New writing style

    1
    2
    3
    4
    5
    
    In : if b = 1: pass
    Input In []
        if b = 1: pass
        ^
    SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?
    

7. PEP 634-636

Match-Case

Many Python core developers believe that Python does not need to add switch-case syntax, because the same effect can be achieved with if/elif/else. This new syntax is called Structural Pattern Matching in Chinese, and there are three PEPs to introduce it because there are a lot of new features

  • PEP 634: Introduces the match syntax and supported patterns.
  • PEP 635: Explains the reason for the syntax being designed this way
  • PEP 636: a tutorial introducing the concepts, syntax and semantics

match is followed by the variable to be matched, case is followed by a different condition, and then the statement to be executed if the condition is met. The last case is underlined to indicate a default match, so if the previous condition does not match, it will be executed in this case, which is equivalent to the previous else.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
def http_error(status):
    match status:
        case 400:
            return 'Bad request'
        case 401:
            return 'Unauthorized'
        case 403:
            return 'Forbidden'
        case 404:
            return 'Not found'
        case _:
            return 'Unknown status code'

But the match-case syntax can do much more than the switch-case syntax in C / Go languages, it is actually the match-case syntax in Scala / Erlang languages, which supports complex pattern matching, and I will demonstrate the flexibility of this new syntax and pythonic in detail with several pattern examples.

  • [1] Literal mode

    The above example is a literal pattern that uses the basic data structures that come with Python, such as strings, numbers, booleans, and None.

    1
    2
    3
    4
    5
    6
    
        case 0:
            print('zero')
        case 1:
            print('one')
        case 2:
            print('two')
    
  • [2] Capture mode

    Can match the assignment target of a single expression. If greeting is not null, it will be assigned to name, but note that if greeting is null a NameError or UnboundLocalError error will be thrown, since name has not been defined before.

    1
    2
    3
    4
    5
    6
    7
    8
    
    def capture(greeting):
        match greeting:
            case "":
                print("Hello!")
            case name:
                print(f"Hi {name}!")
        if name == "Santa":
            print('Match')
    
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    
    In : capture('Alex')
    Hi Alex!
    
    In : capture('Santa')
    Hi Santa!
    Match
    
    In : capture('')
    Hello!
    ---------------------------------------------------------------------------
    UnboundLocalError                         Traceback (most recent call last)
    Input In [4], in <cell line: 1>()
    ----> 1 capture('')
    
    Input In [1], in capture(greeting)
        1 def capture(greeting):
        2     match greeting:
        3         case "":
        4             print("Hello!")
        5         case name:
        6             print(f"Hi {name}!")
    ----> 7     if name == "Santa":
        8         print('Match')
    
    UnboundLocalError: local variable 'name' referenced before assignment
    
  • [3] Sequence mode

    The results can be used in match in either list or tuple format. You can also use the first, *rest = seq pattern to unpack as described in PEP 3132 - Extended Iterable Unpacking.

    The first element of this match condition needs to be 1, otherwise the match fails. The first case uses a list and unpacking, the second case uses a tuple, which is actually the same semantics as a list, and the third is still a list.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    
    In : def sequence(collection):
    ...:     match collection:
    ...:         case 1, [x, *others]:
    ...:             print(f"Got 1 and a nested sequence: {x=}, {others=}")
    ...:         case (1, x):
    ...:             print(f"Got 1 and {x}")
    ...:         case [x, y, z]:
    ...:             print(f"{x=}, {y=}, {z=}")
    ...:
    
    In : sequence([1])
    
    In : sequence([1, 2])
    Got 1 and 2
    
    In : sequence([1, 2, 3])
    x=1, y=2, z=3
    
    In : sequence([1, [2, 3]])
    Got 1 and a nested sequence: x=2, others=[3]
    
    In : sequence([1, [2, 3, 4]])
    Got 1 and a nested sequence: x=2, others=[3, 4]
    
    In : sequence([2, 3])
    
    In : sequence((1, 2))
    Got 1 and 2
    

    If the pattern after case is a single item, you can remove the parentheses and write it that way. However, note that case 1, [x, *others] cannot be written without the parentheses, and the logic of the decomposition will change if the parentheses are removed.

    1
    2
    3
    4
    5
    6
    7
    8
    
    def sequence2(collection):
        match collection:
            case 1, [x, *others]:
                print(f"Got 1 and a nested sequence: {x=}, {others=}")
            case 1, x:
                print(f"Got 1 and {x}")
            case x, y, z:
                print(f"{x=}, {y=}, {z=}")
    
  • [4] Wildcard pattern

    Use the single underscore _ to match any result, but not bound (not assigned to a variable or variables), the final case _ is the wildcard pattern, but of course there can be multiple matches, and the sequence pattern mentioned earlier also supports _. The use of wildcards requires attention to the logical order, putting the small range in front and the large range in the back to prevent it from not meeting expectations.

    1
    2
    3
    4
    5
    
    def http_error(status):
        match status:
            ...
            case _:
                return 'Unknown status code'
    
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    
    In : def wildcard(data):
    ...:     match data:
    ...:         case [_, _]:
    ...:             print('Some pair')
    ...:
    
    In : wildcard(None)
    
    In : wildcard([1])
    
    In : wildcard([1, 2])
    Some pair
    
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    In : def sequence2(collection):
    ...:     match collection:
    ...:         case ["a", *_, "z"]:
    ...:             print('matches any sequence of length two or more that starts with "a" and ends with "z".')
    ...:         case (_, _, *_):
    ...:             print('matches any sequence of length two or more.')
    ...:         case [*_]:
    ...:             print('matches a sequence of any length.')
    ...:
    
    In : sequence2(['a', 2, 3, 'z'])
    matches any sequence of length two or more that starts with "a" and ends with "z".
    
    In : sequence2(['a', 2, 3, 'b'])
    matches any sequence of length two or more.
    
    In : sequence2(['a', 'b'])
    matches any sequence of length two or more.
    
    In : sequence2(['a'])
    matches a sequence of any length.
    
  • [5] Constant value mode

    This pattern, mainly matches constants or enumeration values of the enum module.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    
    In : class Color(Enum):
    ...:     RED = 1
    ...:     GREEN = 2
    ...:     BLUE = 3
    ...:
    
    In : class NewColor:
    ...:     YELLOW = 4
    ...:
    
    In : def constant_value(color):
    ...:     match color:
    ...:         case Color.RED:
    ...:             print('Red')
    ...:         case NewColor.YELLOW:
    ...:             print('Yellow')
    ...:         case new_color:
    ...:             print(new_color)
    ...:
    
    In : constant_value(Color.RED)  # Match the first case
    Red
    
    In : constant_value(NewColor.YELLOW)  # Match the second case
    Yellow
    
    In : constant_value(Color.GREEN)  # Match the third case
    Color.GREEN
    
    In : constant_value(4)  # The constant values match the second case as well
    Yellow
    
    In : constant_value(10)  # Other constants
    10
    

    Note here that since case has a binding effect, you cannot use constants like YELLOW directly, which is wrong.

    1
    2
    3
    4
    5
    6
    
    YELLOW = 4
    
    def constant_value(color):
        match color:
            case YELLOW:
                print('Yellow')
    
  • [6] Mapping mode

    It’s actually case followed by support for using a dictionary to do the matching.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    
    In : def mapping(config):
    ...:     match config:
    ...:         case {'sub': sub_config, **rest}:
    ...:             print(f'Sub: {sub_config}')
    ...:             print(f'OTHERS: {rest}')
    ...:         case {'route': route}:
    ...:             print(f'ROUTE: {route}')
    ...:
    
    In : mapping({})
    
    In : mapping({'route': '/auth/login'})  # Match the first case
    ROUTE: /auth/login
    
    # Match dictionaries with sub keys, values are bound to sub_config and the rest of the dictionary is bound to rest
    In : mapping({'route': '/auth/login', 'sub': {'a': 1}})  # Match the second case
    Sub: {'a': 1}
    OTHERS: {'route': '/auth/login'}
    
  • [7] Class mode

    After case supports any object to do matching, this is because for matching the position needs to be determined , so the position parameter needs to be used to identify it.

    1
    2
    3
    4
    5
    6
    7
    8
    
    In : def class_pattern(obj):
    ...:     match obj:
    ...:         case Point(x=1, y=2):
    ...:             print(f'match')
    ...:
    
    In : class_pattern(Point(1, 2))
    match
    

    Another solution to this custom class that doesn’t use positional arguments for matching is to use __match_args__ to return an array of positional arguments, like this. Here Point2 uses the standard library’s dataclasses.dataclass decorator, which will provide the __match_args__ property, so it can be used directly.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    
    In : class Point:
    ...:     __match_args__ = ('x', 'y')
    ...:
    ...:     def __init__(self, x, y):
    ...:         self.x = x
    ...:         self.y = y
    ...:
    
    In : from dataclasses import dataclass
    
    In : @dataclass
    ...: class Point2:
    ...:     x: int
    ...:     y: int
    ...:
    
    In : def class_pattern(obj):
    ...:     match obj:
    ...:         case Point(x, y):
    ...:             print(f'Point({x=},{y=})')
    ...:         case Point2(x, y):
    ...:             print(f'Point2({x=},{y=})')
    ...:
    
    In : class_pattern(Point(1, 2))
    Point(x=1,y=2)
    
    In : class_pattern(Point2(1, 2))
    Point2(x=1,y=2)
    
  • [8] Combination (OR) mode

    Multiple literals can be combined to represent an or relation using |, and | can exist more than one within a case condition to represent multiple or relations.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    
    def or_pattern(obj):
        match obj:
            case 0 | 1 | 2: # 0,1,2 three numbers match
                print('small number')
            case list() | set():  # List or set matching
                print('list or set')
            case str() | bytes():  # String or bytes match
                print('str or bytes')
            case Point(x, y) | Point2(x, y):  # Borrowing from the previous 2 classes, one of them can match
                print(f'{x=},{y=}')
            case [x] | x:  # list and only one element or single value matches the
                print(f'{x=}')
    

    Note that the [x] in case [x] | x is not triggered because of the matching order, and x cannot be a set, string, byte, etc., because it will not be matched in the previous condition. Also, there is no case syntax in Python for the AND relationship.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    
    In : or_pattern(1)
    small number
    
    In : or_pattern(2)
    small number
    
    In : or_pattern([1])
    list or set
    
    In : or_pattern({1, 2})
    list or set
    
    In : or_pattern('sss')
    str or bytes
    
    In : or_pattern(b'sd')
    str or bytes
    
    In : or_pattern(Point(1, 2))
    x=1,y=2
    
    In : or_pattern(Point2(1, 2))
    x=1,y=2
    
    In : or_pattern(4)
    x=4
    
    In : or_pattern({})
    x={}
    
  • [9] AS mode

    The AS pattern was actually a walrus pattern in the early days, but it was later discussed that the use of the as keyword would give the syntax an edge. It should be noted that [0, int() as i] in this case is a subpattern, that is, it contains patterns within patterns: [0, int() as i] is the sequence pattern matched by case, and where int() as i is the subpattern, it is the AS pattern.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    
    In : def as_pattern(obj):
    ...:     match obj:
    ...:         case str() as s:
    ...:             print(f'Got str: {s=}')
    ...:         case [0, int() as i]:
    ...:             print(f'Got int: {i=}')
    ...:         case [tuple() as tu]:
    ...:             print(f'Got tuple: {tu=}')
    ...:         case list() | set() | dict() as iterable:
    ...:             print(f'Got iterable: {iterable=}')
    ...:
    ...:
    
    In : as_pattern('sss')
    Got str: s='sss'
    
    In : as_pattern([0, 1])
    Got int: i=1
    
    In : as_pattern([(1,)])
    Got tuple: tu=(1,)
    
    In : as_pattern([1, 2, 3])
    Got iterable: iterable=[1, 2, 3]
    
    In : as_pattern({'a': 1})
    Got iterable: iterable={'a': 1}
    
  • [10] Add conditions to the schema

    The pattern also supports adding an if judgment (called guard), which allows the match to be further judged, equivalent to achieving some degree of AND effect.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    In : def go(obj):
    ...:     match obj:
    ...:         case ['go', direction] if direction in ['east', 'north']:
    ...:             print('Right way')
    ...:         case direction if direction == 'west':
    ...:             print('Wrong way')
    ...:         case ['go', _] | _:
    ...:             print('Other way')
    ...:
    
    In : go(['go', 'east'])  # Matching condition 1
    Right way
    
    In : go('west')  # Matching condition 2
    Wrong way
    
    In : go('north')  # Match default conditions
    Other way
    
    In : go(['go', 'west'])  # Match default conditions
    Other way
    

8. BPO 12782

New Context Manager syntax

In previous versions, sometimes we wanted to put multiple contexts inside a with on a long line, and a common way to do this was to use a backslash, which would align open. But writing it this way caused code checking tools like pep8 and black to report an error: they all thought that backslashes should not be used to explicitly continue a line.

  • The previous writing style

    1
    2
    3
    4
    
    In : with open('input', mode='rb') as input, \
    ...:      open('output', mode='rb') as output:
    ...:     ...
    ...:
    
  • New writing style

    1
    2
    3
    4
    5
    6
    
    In : with (
    ...:     open('input', mode='rb') as input,
    ...:     open('output', mode='rb') as output
    ...: ):
    ...:     ...
    ...:
    

    Now you can add a bracket to these context managers and it’s perfect, in fact you can use with to enclose multiple context managers (Context Manager) in brackets.

9. Note

Introduce the new features of the module and analyze their use and usage.

  • [1] itertools.pairwise

Its a way to return iterators in the form of elements of an iterable object in order, with two elements placed next to each other in a group. The package more-itertools was available in previous versions and has now been added to the standard library.

1
2
3
4
5
In : list(pairwise([1, 2, 3, 4, 5]))
Out: [(1, 2), (2, 3), (3, 4), (4, 5)]

In : list(pairwise('ABCDE'))
Out: [('A', 'B'), ('B', 'C'), ('C', 'D'), ('D', 'E')]
  • [2] contextlib.aclosing

The standard contextlib.closing decorator has been introduced before, and contextlib.closing is actually just a version of async. As officially described, its role is to implement the following logic.

A good habit with Python is to close the file handle when the operation is done, and the built-in open with with is a good practice. It works by calling ff.__exit__ to automatically close the file handle when the block is finished.

1
2
3
In : with open('new.txt', 'w') as f:
...:     f.write('A\n')
...:
1
2
3
4
5
6
7
8
9
In : ff = open('new.txt', 'w')

In : ff.closed
Out: False

In : ff.__exit__()  # with will call it

In : ff.closed
Out: True

But note that not all open methods natively support with (or provide __exit__ and __enter__ methods), and of course there are other types of operations that need to be made sure to be closed (even if an exception is thrown). In fact, the with syntax is generally supported by standard libraries or well-known projects, but may not be available when writing your own or corporate projects, for example, I define a class like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
In : class Database:
...:     def close(self):
...:         print('Closed')
...:

In : with Database() as d:
...:     ...
...:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-23-f0ea085f7488> in <module>
----> 1 with Database() as d:
      2     ...
      3

AttributeError: __enter__

So, you can wrap it with contextlib.closing. And contextlib.closing is actually a version of asyncio.

1
2
3
4
5
6
In : import contextlib

In : with contextlib.closing(Database()) as d:
...:     ...
...:
Closed
1
2
3
4
5
6
from contextlib import aclosing

async with aclosing(my_generator()) as values:
    async for value in values:
        if value == 42:
            break
  • [3] contextlib.AsyncContextDecorator

    contextlibContext.Decorator was added in Python 3.2, which is the basis for contextlib.contextmanager, as expressed in the class name, and which serves to allow the context manager to be used as a decorator.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    
    # The most basic context manager without parameters
    class mycontext:
        def __enter__(self):
            print('Starting')
            return self
    
        def __exit__(self, *exc):
            print('Finishing')
            return False
    
    # You can use the with syntax like this
    In : with mycontext():
    ...:     print('Inner')
    ...:
    Starting
    Inner
    Finishing
    

    What if you want to change the management of this context to a decorator? Just make mycontext inherit from the base class contextlib.ContextDecorator, isn’t that convenient? So contextlib.AsyncContextDecorator is actually the version of asyncio.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    
    In : class mycontext(contextlib.ContextDecorator):  # Note the inheritance here
    ...:     def __enter__(self):
    ...:         print('Starting')
    ...:         return self
    ...:
    ...:     def __exit__(self, *exc):
    ...:         print('Finishing')
    ...:         return False
    ...:
    
    In : @mycontext()
    ...: def p():
    ...:     print('Inner')
    ...:
    
    In : p()
    Starting
    Inner
    Finishing
    
    1
    2
    3
    4
    5
    6
    7
    8
    
    class mycontext(AsyncContextDecorator):
        async def __aenter__(self):
            print('Starting')
            return self
    
        async def __aexit__(self, *exc):
            print('Finishing')
            return False
    

10. Reference