What is a with statement?

The with statement is a feature related to exception handling introduced in Python 2.6. The with statement is used when accessing resources to ensure that the necessary “cleanup” operations are performed to release resources, such as automatic closure of files after use, automatic acquisition and release of locks in threads, and so on, regardless of whether an exception occurs during use. automatically after a file is used, automatic acquisition and release of locks in threads, etc.

The with statement is a new control flow structure with the following basic structure.

1
2

with expression [as variable]:
    with-block

A good example is file handling, where you need to get a file handle, read data from the file, and then close the file handle. If you don’t use the with statement, the code would look like this.

1
2
3

file = open("foo.txt")
data = file.read()
file.close()

There are two problems here.

  • May forget to close the file handle
  • File read data exception occurred and no handling was done

Here is the code where the exception handling is added.

1
2
3
4
5

file = open("foo.txt")
try:
    data = file.read()
finally:
    file.close()

Although this code works well, it is too long. This is where with comes into its own. In addition to having a more elegant syntax, with can also handle exceptions generated by contextual environments very well. Here is the with version of the code.

1
2

with open("foo.txt") as file:
    data = file.read()

How does the with statement work?

Python is also smart about with. The basic idea is that the object whose value is being evaluated by with must have an enter() method and an exit() method. After the statement immediately following with is evaluated, the enter() method of the returned object is called, and the return value of this method is assigned to the variable following as. When the block of code following with has been executed, the exit() method of the preceding return object is called.

The following example specifies how with works.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

class Sample:
    def __enter__(self):
        print("In __enter__()")
        return "Foo"

    def __exit__(self, type, value, trace):
        print("In __exit__()")


def get_sample():
    return Sample()


with get_sample() as sample:
    print("sample:", sample)

The output after execution reads

1
2
3

In __enter__()
sample: Foo
In __exit__()

Specific flow.

  • enter() method is executed
  • The value returned by the enter() method - in this case “Foo” - is assigned to the variable ‘sample’.
  • The code block is executed, printing the value of the variable “sample” as “Foo”
  • The exit() method is called

The real power of with is that it can handle exceptions. You may have noticed that the exit method of the Sample class has three arguments: val, type and trace. These arguments are quite useful in exception handling. Let’s change the code to see exactly how it works.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

class Sample:
    def __enter__(self):
        return self

    def __exit__(self, type, value, trace):
        print("type:", type)
        print("value:", value)
        print("trace:", trace)

    def do_something(self):
        bar = 1 / 0
        return bar


with Sample() as sample:
    sample.do_something()

The output results are.

1
2
3
4
5
6
7
8
9

Traceback (most recent call last):
  File "test.py", line 16, in <module>
    sample.do_something()
  File "test.py", line 11, in do_something
    bar = 1 / 0
ZeroDivisionError: division by zero
type: <class 'ZeroDivisionError'>
value: division by zero
trace: <traceback object at 0x0000029739A7F508>

In this example, the with followed by get_sample() becomes Sample(). It doesn’t matter, as long as the object returned by the statement immediately following with has enter() and exit() methods. In this case, the enter() method of Sample() returns the newly created Sample object and assigns a value to the variable sample.

In fact, the exit() method is executed when any exception is thrown in the block following the with. As the example shows, when an exception is thrown, the type, value and stack trace associated with it are passed to the exit() method, so the ZeroDivisionError exception thrown is printed out. When developing libraries, clean up resources, close files, and so on, can be placed in the exit method.

Thus, Python’s with statement is an effective mechanism for making code more concise, and for making cleanup easier when exceptions are thrown.

Context Manager

In any programming language, input and output of files, disconnection of database, etc. are very common resource management operations. However, resources are limited, and when writing programs, we must ensure that these resources are released after use, or else they will easily cause resource leakage, which can make the system processing slow in mild cases or crash in severe cases.

To solve this problem, different programming languages have introduced different mechanisms. In Python, the corresponding solution is the context manager. Context managers help you automatically allocate and release resources, most typically in the form of with statements.

Another typical example is Python’s threading.lock class. For example, if I wanted to get a lock, perform the corresponding operation, and then release it when I was done, the code would look like this.

1
2
3
4
5
6

some_lock = threading.Lock()
some_lock.acquire()
try:
    ...
finally:
    some_lock.release()

The corresponding with statement, again, is very concise.

1
2
3

some_lock = threading.Lock()
with somelock:
    ...

We can see from these two examples that the use of the with statement simplifies the code and effectively prevents resource leaks from occurring.

Class-based Context Manager

Here, I customized a context manager class FileManager to simulate Python’s open and close file operations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

class FileManager:
    def __init__(self, name, mode):
        print('calling __init__ method')
        self.name = name
        self.mode = mode
        self.file = None

    def __enter__(self):
        print('calling __enter__ method')
        self.file = open(self.name, self.mode)
        return self.file

    def __exit__(self, exc_type, exc_val, exc_tb):
        print('calling __exit__ method')
        if self.file:
            self.file.close()


with FileManager('test.txt', 'w') as f:
    print('ready to write to file')
    f.write('hello world')

When we use a class to create a context manager, we must make sure that the class includes the method “__enter__()” and the method “__exit__()”. The method “__enter__()” returns the resources to be managed, and the method “__exit__()” usually contains operations to release or clean up the resources, such as closing the file in this example. And when we use the with statement to execute the context manager.

1
2

with FileManager('test.txt', 'w') as f:
    f.write('hello world')

The following four steps occur in sequence.

  • The method “__init__()” is called and the program initializes the object FileManager so that the file name (name) is “test.txt” and the file mode (mode) is ‘w’
  • The method “__enter__()” is called, the file “test.txt” is opened in write mode, and the FileManager object is returned to the variable f
  • The string “hello world” is written to the file “test.txt”.
  • The method “__exit__()” is called and is responsible for closing the previously opened file stream

Thus, the output of this program is.

1
2
3
4

calling __init__ method
calling __enter__ method
ready to write to file
calling __exit__ method

The parameters “exc_type, exc_val, exc_tb” in method “__exit__()” represent exception_type, exception_value and traceback respectively. When we execute a with statement with a context manager, if an exception is thrown, the information about the exception will be included in these three variables and passed to the method “__exit__()”.

Implementation of decorators and generators based on contextlib module

Python also provides a decorator for the contextmanager in the contextlib module, which further simplifies the way the context manager is implemented. The function is split into two parts by yield, with the statement before yield executed in the __enter__ method and the statement after yield executed in the __exit__ method. The value immediately after yield is the return value of the function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

from contextlib import contextmanager


@contextmanager
def file_manager(name, mode):
    try:
        f = open(name, mode)
        yield f
    finally:
        f.close()


with file_manager('test.txt', 'w') as f:
    f.write('hello world')

In this code, the function file_manager() is a generator that opens the file when we execute the with statement and returns the file object f. When the with statement is done, the close file operation in the finally block is executed. As you can see, when using a generator-based context manager, we no longer have to define the “__enter__()” and “__exit__()” methods, but be sure to add the decorator @contextmanager which is easy for newbies to overlook.