python 导入numpy 导致多进程绑定同一个CPU问题解决方法-白红宇

python 如果有导入numpy模块的import语句，会导致默认将多进程程序的每个进程都绑定到同一个CPU core上，

失去了多进程在多核CPU上的性能优越性，这和CPU affinity(CPU亲和性)有关，解决办法：

Python refuses to use multiple cores – solution

I was trying to get to work and I noticed that if I run two Python scripts simultaneously – say, in two different terminals – they use the same core. Hence, I get no speedup from multiprocessing/parallel Python. After some searching around, I found out that in some circumstances importing causes Python to stick all computations in one core. This is an issue with , and apparently it only happens for some mixtures of Numpy and BLAS libraries – other packages may cause the CPU affinity issue as well.

There’s a package called (Linux only AFAIK) that lets you set and get CPU affinity. Download it, run python setup.py install, and run this in Python or :

                            
In [
1
]: 
import
affinity
In [
2
]: affinity.get_process_affinity_mask(
0
)
Out[
2
]: 
63
         
       

This is good: 63 is a bitmask corresponding to 111111 – meaning all 6 cores are available to Python. Now running this, I get:

                            
In [
4
]: 
import
numpy as np
In [
5
]: affinity.get_process_affinity_mask(
0
)
Out[
5
]: 
1
         
       

So now only one core is available to Python. The solution is simply to set the CPU affinity appropriately after import numpy, for instance:

                            
import
numpy as np
import
affinity
import
multiprocessing
affinity.set_process_affinity_mask(
0
,
2
*
*
multiprocessing.cpu_count()
-
1
)
         
       