When programming in Python, you will often encounter operations that read and write files. The various modes of reading and writing files (such as read, write, append, etc.) can sometimes be really confusing, as well as confusing the use of methods such as open, read, readline, readlines, write, writelines, etc. can also throw you for a loop.

I hope this article will help you better understand how to read and write files and use the most appropriate methods in the most appropriate places.

What is a document?

Before we start looking at how to use files in Python, it’s important to understand exactly what files are and how modern operating systems handle certain aspects of them.

Essentially, a file is a contiguous set of bytes used to store data. This data is organized in a specific format and can be anything as simple as a text file, or as complex as a program executable. Finally, these byte files are translated into binary files1,0 for easier processing by the computer.

A file on most modern file systems consists of three main parts.

  • Header : metadata about the contents of the file (file name, size, type, etc.)
  • Data : the contents of the file as written by the creator or editor
  • End of File (EOF) : A special character indicating the end of the file

The content of the data representation depends on the format specification used and is usually indicated by the extension. For example, a file with a .gif extension is most likely to conform to the Graphics Interchange Format specification. There are hundreds, if not thousands, of file extensions.

File path

File paths are required to access files on the operating system. A file path is a string that represents the location of a file. It is divided into three main parts.

  • folder path: the location of the folder on the file system, with subsequent folders separated by a forward slash / (Unix) or a backslash \ (Windows)
  • filename: the actual name of the file
  • Extension: the file path is pre-set with a period (.) at the end , used to indicate the file type

Python file path-related operations.

  • Use getcwd() to get the current working directory, i.e., the path to the directory where the Python script is currently working, similar to the pwd command in Linux.
    • If you want to print a file name or path that contains Chinese in Windows, you need to use “GBK” to decode it.
  • Merge paths using path.join()
    • Use sep for backslashes in Windows and forward slashes in Linux (backslashes need to be escaped with \)
  • Relative paths
    • (.) Dot indicates the current folder
    • (. ) dot means parent folder
  • Relative path, absolute path conversion
    • path.abspath(path): return absolute path
    • path.isabs(path): determine if it is an absolute path
    • path.relpath(path,start): return relative path
  • path.split.
  • path.dirname(path): return the directory where the file is located, os.path.basename(path): return the file name
  • path.split(path) == (os.path.dirname(path), os.path.basename(path)).silit(os.sep) is splitting the path by separator
  • path.basename(): get the file name
  • path.splitext(): split the extension
  • View file size: path.getsize(filename), can only count files, not folders, if you need to count folders, you need to traverse by yourself.
  • Get file attributes: stat(file)
  • Check the validity of the path
    • path.exists: whether the path exists
    • path.isdir: if or not it is a directory
    • path.isfile: if or not it is a file
  • Return all files and directory names in the specified directory: listdir()

Other methods.

  • remove files: os.remove()
  • Remove directories: removedirs(r “c:\python”)
  • rename: rename(old, new) # files or directories are used with this command
  • Create multi-level directories: makedirs(r “c:\python\test”), which will create intermediate folders, similar to the Linux command mkdir -p
  • Create a single directory: mkdir(“test”)
  • Modify file permissions and timestamps: chmod(file)

Directory operations.

  • Copy a file.
    • copyfile(“oldfile”, “newfile”) # Both oldfile and newfile can only be files
    • copy(“oldfile”, “newfile”) # oldfile can only be a folder, newfile can be a file or a target directory
  • copy folder: copytree(“olddir”, “newdir”) # both olddir and newdir can only be directories, and newdir must not exist
  • Move a file (directory): move(“oldpos”, “newpos”)
  • Delete a directory
    • rmdir(“dir”) # Only empty directories can be deleted
    • rmtree(“dir”) # Empty directories and directories with contents can be deleted

End of line

A common problem encountered when working with file data is the representation of new lines or line endings. Line endings originated in the days of Morse code, when a specific symbol was used to indicate the end of a transmission or the end of a line.

Later, the International Organization for Standardization (ISO) and the American Standards Association (ASA) standardized teletypewriters. the ASA standard specifies that line endings should use carriage returns (sequence CR or \r) and line feed (LF or \n) characters (CR+LF or \r\n). However, the ISO standard allows CR+LF characters or LF characters only.

Windows uses the CR+LF character for newlines, while Unix and newer Mac versions use only the LF character. This can lead to some complications when you are working with files that originate on different operating systems. Here’s a simple example. Suppose we examine the file dog_breeds.txt created on a Windows system.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Pug\r\n
Jack Russel Terrier\r\n
English Springer Spaniel\r\n
German Shepherd\r\n
Staffordshire Bull Terrier\r\n
Cavalier King Charles Spaniel\r\n
Golden Retriever\r\n
West Highland White Terrier\r\n
Boxer\r\n
Border Terrier\r\n

The same output will be interpreted differently on Unix devices.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

Pug\r
\n
Jack Russel Terrier\r
\n
English Springer Spaniel\r
\n
German Shepherd\r
\n
Staffordshire Bull Terrier\r
\n
Cavalier King Charles Spaniel\r
\n
Golden Retriever\r
\n
West Highland White Terrier\r
\n
Boxer\r
\n
Border Terrier\r
\n

Solution.

1
2
3

with open('test.txt', 'r') as f:
    for line in f.readlines():
    line = line.strip('\n')

Python traversal method for folders

Use walk()

Output is always folder first and then file name

1
2
3
4
5
6
7
8
9

# -*- coding: utf-8 -*- 
import os 
def echo_path(root_dir): 
    list_dirs = os.walk(root_dir) 
    for root, dirs, files in list_dirs: 
        for d in dirs: 
            print os.path.join(root, d)
        for f in files: 
            print os.path.join(root, f)

Using listdir

The output follows the directory tree structure and is sorted alphabetically.

1
2
3
4
5
6
7
8
9

# -*- coding: utf-8 -*- 
import os 

def echo_path(root_dir): 
    for dir_list in os.listdir(root_dir): 
        path = os.path.join(root_dir, dir_list) 
        print(path)
        if os.path.isdir(path): 
            echo_path(path)

glob module

The glob module is one of the simplest modules, with very little content. Use it to find pathnames of files that match specific rules.

glob.glob

returns a list of all matching file paths. It has only one argument, pathname, which defines the file path matching rules, here it can be either absolute or relative paths. Example.

1
2
3
4
5
6
7

import glob  

#获取指定目录下的所有图片  
print(glob.glob(r"E:\Picture\*\*.jpg") )

#获取上级目录的所有.py文件  
print(glob.glob(r'../*.py') #相对路径)

Python reads and writes file contents

Open a file with the with statement

Reading and writing files in Python requires 3 steps.

  • Calling the open function, which returns a File object
  • Calling the read() or write() method of the File object
  • Calling the close() method of the File object to close the file

Common opening modes for files.

  • ‘r’: read-only (default. Throws an error if the file does not exist)
  • ‘w’: write-only (if the file does not exist, the file is automatically created)
  • ‘a’: append to the end of the file
  • ‘r+’: read and write

If you need to open the file in binary mode, you need to add the character “b” after mode, such as “rb”, “wb”, etc.

If you don’t use the with statement, the code is as follows.

1
2
3

file = open("/tmp/foo.txt")
data = file.read()
file.close()

There are two problems here :

  • Possible forgetting to close the file handle.
  • File read data exception occurred and nothing was done about it.

Enhanced code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

try:
    f = open('xxx')
except:
    print 'fail to open'
    exit(-1)
try:
    do something
except:
    do something
finally:
     f.close()

Although this code works well, it is too long. with, in addition to having a more elegant syntax, also does a good job of handling exceptions generated by the contextual environment.

The with version of the code.

1
2

with open("/tmp/foo.txt") as file:
    data = file.read()

The workflow of with.

  • After the statement immediately following with is evaluated, the __enter__() method of the returned object is called, and the return value of this method will be assigned to the variable following as.
  • When the block of code following the with has all been executed, the __exit__() method of the preceding return object is called.

Sometimes you may want to read a file and write to another file at the same time. If you use the example shown when learning how to write to a file, it can actually be merged into the following.

1
2
3
4
5

d_path = 'dog_breeds.txt'
d_r_path = 'dog_breeds_reversed.txt'
with open(d_path, 'r') as reader, open(d_r_path, 'w') as writer:
    dog_breeds = reader.readlines()
    writer.writelines(reversed(dog_breeds))

Text reading: the difference between read(), readline(), readlines()

read()

read() is the simplest way to read all the contents of a file at once in one big string, i.e., in memory.

1
2
3
4
5
6

f = open('test.txt') 
try:
    file_context = file_object.read() 
    file_context = open(file).read().splitlines()   // file_context是一个list，每行文本内容是list中的一个元素
finally:
    file_object.close()

Advantages of read().

  • Convenient and simple
  • Reads the file in one big string at a time, fastest

Disadvantages of read().

  • When the file is too large, it will take up too much memory

readline()

readline() reads text line by line, the result is a list

1
2
3
4
5

with open(file) as f:
    line = f.readline()
    while line:
        print(line)
        line = f.readline()

Advantages of readline().

  • Small memory footprint, read line by line

Disadvantages of readline().

  • Slower because it reads line by line

readlines()

readlines() reads all the contents of the text at once, the result is a list

1
2
3

with open(file) as f:
  for line in f.readlines():
     print(line)

This method reads text with a ‘\n’ line break at the end of each line (you can use L.rstrip(’\n’) to remove the line break)

Advantages of readlines().

  • Reads text content in one go, faster

Disadvantages of readlines().

  • As the text grows, it takes up more and more memory

Text writing: write() and writelines()

  • write() is passed a string
  • writelines() passes in an array
1
2
3
4
5
6
7

lines = ['line1', 'line2']
with open('filename.txt', 'w') as f:
    f.write('\n'.join(lines))

lines = ['line1', 'line2']
with open('filename.txt', 'w') as f:
    f.writelines("%s\n" % l for l in lines)