Python 中多进程模型 multiprocessing 在不同操作系统 api 不同，导致运行差异。本篇对其进行介绍。

介绍

导致差异的根本原因是多进程启动的方式

在 Linux 操作系统中，默认创建子进程的方式是 fork，而在 Mac/Windows 系统中，默认创建子进程的方式是 spawn。查看启动方式可以使用如下 Python 代码：

1
2
3
4
import multiprocessing

if __name__ == "__main__":
    print(multiprocessing.get_start_method())

在 Windows 上只能使用 spawn，但是，在 Linux 和 Mac（给予 Unix）上却可以设置采用哪种启动方式，方法如下：

1
2
3
4
5
import multiprocessing

if __name__ == "__main__":
    multiprocessing.set_start_method("fork")  # "fork" or "spawn" 都可以在 Linux or Mac 上; Windows only have spawn
    print(multiprocessing.get_start_method())

fork 和 spawn 的区别就是 spawn 启动多进程会默认将当前脚本或模块在主进程和子进程都执行一遍，如下面的代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import multiprocessing
import os


def print_hi(text: str):
    print(f"hello, {text}")


i = 0
i = i + 1
print(f"i={i}，当前进程id：{os.getpid()}，当前进程处理模块名：{__name__}")

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")
    print(f"多进程启动方式：{multiprocessing.get_start_method()}")
    print(f"父进程id：{os.getpid()}")
    # 创建2个子进程
    p = multiprocessing.Pool(processes=2)
    texts = ["xiaoming", "luban"]
    p.map(print_hi, texts)

输出结果是：

1
2
3
4
5
6
7
i=1，当前进程id：94695，当前进程处理模块名：__main__
多进程启动方式：spawn
父进程id：94695
i=1，当前进程id：94698，当前进程处理模块名：__mp_main__
i=1，当前进程id：94699，当前进程处理模块名：__mp_main__
hello, xiaoming
hello, luban

如果改成 fork 方式：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import multiprocessing
import os


def print_hi(text: str):
    print(f"hello, {text}")


i = 0
i = i + 1
print(f"i={i}，当前进程id：{os.getpid()}，当前进程处理模块名：{__name__}")

if __name__ == "__main__":
    multiprocessing.set_start_method("fork")
    print(f"多进程启动方式：{multiprocessing.get_start_method()}")
    print(f"父进程id：{os.getpid()}")
    # 创建2个子进程
    p = multiprocessing.Pool(processes=2)
    texts = ["xiaoming", "luban"]
    p.map(print_hi, texts)

那么输出结果是：

1
2
3
4
5
i=1，当前进程id：94714，当前进程处理模块名：__main__
多进程启动方式：fork
父进程id：94714
hello, xiaoming
hello, luban

可见，spawn 方式的会把当前脚本在子进程中原模原样执行，只是在子进程中遇到条件判断 if __name__ == "__main__" 时，因不符合条件而未执行里面的内容。

启动方法的区别

fork 和 spawn 的区别是什么呢？

  • fork：除了必要的启动资源外，其他变量、包、数据等都继承自父进程，并且是 copy-on-write 的。也就是说，共享了父进程的一些内存页，因此启动较快。但是由于大部分都用的父进程数据，所以是不安全的进程；
  • spawn：从头构建一个子进程，父进程的数据等拷贝到子进程空间内，拥有自己的Python解释器，所以需要重新加载一遍父进程的包，也因此导致启动较慢。但是由于数据都是自己的，安全性较高。

Windows 系统只支持 spawn，而 Linux/Mac 既支持 spawn 又支持 fork，可根据需要选择。

其实，在 Linux/Mac 上启动方法有三类：

1
2
3
4
In [1]: import multiprocessing

In [2]: multiprocessing.get_all_start_methods()
Out[2]: ['fork', 'spawn', 'forkserver']

各系统支持的方法：

  • Windows (win32): spawn
  • macOS (darwin): spawn, fork, forkserver
  • Linux (unix): spawn, fork, forkserver

各系统默认方法：

  • Windows (win32): spawn
  • macOS (darwin): spawn
  • Linux (unix): fork

三种区别：

  • spawn: start a new Python process.
  • fork: copy a Python process from an existing process.
  • forkserver: new process from which future forked processes will be copied.

Forking and spawning are two different start methods for new processes. Fork is the default on Linux (it isn’t available on Windows), while Windows and MacOS use spawn by default.

When a process is forked the child process inherits all the same variables in the same state as they were in the parent. Each child process then continues independently from the forking point. The pool divides the args between the children and they work though them sequentially.

On the other hand, when a process is spawned, it begins by starting a new Python interpreter. The current module is reimported and new versions of all the variables are created. The plot_function is then called on each of the the args allocated to that child process. As with forking, the child processes are independent of each other and the parent.

Neither method copies running threads into the child processes.

特点 fork spawn
Import module at start of each child process no yes
Variables have same id as in parent process yes no
Child process gets variables defined in name == main block yes no

设置启动方法

方法一

1
2
3
4
5
6
7
8
import multiprocessing

......
# protect the entry point
if __name__ == '__main__':
	# set the start method
	multiprocessing.set_start_method('spawn')
  ......

方法二

1
2
3
4
5
6
7
8
9
10
11
12
13
import multiprocessing

......
# protect the entry point
if __name__ == '__main__':
	# get a context configured with a start method
  context = multiprocessing.get_context('fork')
  # create a child process via a context
  process = context.Process(...)
  ...
  # set the start method
  context.set_start_method('spawn', force=True)
  ......

QA

如果在 Windows 上或者采用 spawn 启动方式的 Linux/Mac 上，没有使用 if __name__ == "__main__":，那么会报异常，测试代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import multiprocessing
import os


def print_hi(text: str):
    print(f"hello, {text}")


i = 0
i = i + 1
print(f"i={i}，当前进程id：{os.getpid()}，当前进程处理模块名：{__name__}")

# multiprocessing.freeze_support()
multiprocessing.set_start_method("spawn")
print(f"多进程启动方式：{multiprocessing.get_start_method()}")
print(f"父进程id：{os.getpid()}")
# 创建2个子进程
p = multiprocessing.Pool(processes=2)
texts = ["xiaoming", "luban"]
p.map(print_hi, texts)

报异常：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
i=1，当前进程id：94918，当前进程处理模块名：__main__
多进程启动方式：spawn
父进程id：94918
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 129, in _main
    prepare(preparation_data)
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 240, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 291, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jinzhongxu/Codes/torch-test/windows-spawn.py", line 14, in <module>
    multiprocessing.set_start_method("spawn")
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/context.py", line 247, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 129, in _main
    prepare(preparation_data)
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 240, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 291, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jinzhongxu/Codes/torch-test/windows-spawn.py", line 14, in <module>
    multiprocessing.set_start_method("spawn")
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/context.py", line 247, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set
i=1，当前进程id：94921，当前进程处理模块名：__mp_main__
i=1，当前进程id：94922，当前进程处理模块名：__mp_main__
i=1，当前进程id：94923，当前进程处理模块名：__mp_main__
i=1，当前进程id：94924，当前进程处理模块名：__mp_main__
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/miniconda/lib/python3.11/multiprocessing/spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^

或者在 Pytorch 多进程采样训练时，出现：freeze_support() 等报错。

这就是因为 spawn 方式会把当前脚本拷贝到多个子进程中，子进程有遇到启动多进程代码，会无限循环创建子进程执行。因此，最安全的方式是添加上: if __name__ == "__main__"。

参考文献

  1. python3 多进程代码，抛出 RuntimeError：省略.. The “freeze_support()” line can be omitted if the program …
  2. multiprocessing 中fork和spawn的区别
  3. multiprocessing — Process-based “threading” interface、
  4. Multiprocessing Start Methods
  5. Fork vs Spawn in Python Multiprocessing