1. How to Program, Part I
  2. How to Program, Part II
  3. How to Program, Part III
  4. How to Program, Part IV
  5. How to Program, Part V
  6. How to Program, Part VI
  7. exercises
  8. pyMPI tutorial
  9. Calculating PI, Part I
  10. Calculating PI, Part II
  11. Calculating PI, Part III
  12. Dividing Work
  13. More MPI
  14. Poogle - Web Search
  15. Mandelbrot Sets
  16. Mandelbrot, The Code
  17. Mandelbrot, The Images
  18. Mandelbrot In CUDA
  19. Conway's Life, Part I
  20. Life Code Listing
  21. Conway's Life, Part II
  22. MPI Life Code Listing

pyMPI tutorial

There are a lot of things left to explore in the base python programming language, but this tutorial was written with the intention of teaching people how to use MPI (Message Passing Interface) to program a cluster.

For this reason, we will need a version of python that understands MPI. You can get that here. If getting pyMPI to work on your favorite machine is proving to be tricky, you can download MEMPI from here.

To execute a parallel program you need to use the mpirun command, and specify the number of processes you want the program to run on. In essence, we are launching two nearly identical programs, the only thing that is different is the value of mpi.rank that each has. In the example below we will run in two processes. One process will have mpi.rank == 0, and the other will have mpi.rank == 1. Both will see mpi.size == 2 (the number of processes).

hello_mpi.py
1import mpi
2 
3print "rank=",mpi.rank,"/size=",mpi.size
$ mpiexec -np 2 python hello_mpi.py
rank= 1 /size= 2
rank= 0 /size= 2

Notice how confusing the output looks. This is because there is no way to coordinate output generated by the two processors. For this reason, we will normally use an if statement before each print:

hello_mpi2.py
1import mpi
2 
3if mpi.rank == 0:
4    print "size=",mpi.size
$ mpiexec -np 2 python hello_mpi2.py
size= 2

How can processors work together? In this case, each processor contributes a number to be added. In this case, everything is adding a "1."

mpisum.py
1import mpi 
2 
3procs = mpi.allreduce(1, mpi.SUM)
4if mpi.rank==0:
5  print 'total procs=',procs
$ mpiexec -np 2 python ./mpisum.py
total procs= 2

In this case, each processor contributes its rank. What do you get if you use 3 procs, 4 procs, or more?

mpisum2.py
1import mpi 
2 
3sum = mpi.allreduce(mpi.rank, mpi.SUM)
4if mpi.rank==0:
5  print 'total sum=',sum
$ mpiexec -np 4 python ./mpisum2.py
total sum= 6

You don't have to use SUM. You can also use other useful operations like MIN and MAX. Here's MIN:

mpimin.py
1import mpi 
2 
3min = mpi.allreduce(mpi.rank, mpi.MIN) 
4if mpi.rank==0:
5  print 'total min=',min
$ mpiexec -np 4 python ./mpimin.py
total min= 0

And here is MAX:

mpimax.py
1import mpi 
2 
3max = mpi.allreduce(mpi.rank, mpi.MAX) 
4if mpi.rank==0:
5  print 'total max=',max
$ mpiexec -np 4 python ./mpimax.py
total max= 3

Problems

  1. What's wrong with this code?

    bad50.py
    1import mpi 
    2 
    3sum = mpi.allreduce(mpi.SUM, mpi.rank)
    4if mpi.rank==0:
    5  print 'total sum=',sum
  2. What's wrong with this code?

    bad51.py
    1import mpi 
    2 
    3sum = allreduce(mpi.rank, mpi.SUM)
    4if mpi.rank==0:
    5  print 'total sum=',sum
  3. What's wrong with this code?

    bad52.py
    1import random
    2 
    3sum = mpi.allreduce(mpi.rank, mpi.SUM)
    4if mpi.rank==0:
    5  print 'total sum=',sum
  4. Instead of looking at rank 0, this program checks rank 2 before printing. Does that matter?

    rank2.py
    1import mpi 
    2 
    3sum = mpi.allreduce(mpi.rank, mpi.MAX) 
    4if mpi.rank==2:
    5  print 'total sum=',sum
    $ mpiexec -np 4 python ./rank2.py
    total sum= 3
    
  5. What is the maximum number this program can print out? What is the minimum?

    rand4.py
    1import random 
    2import mpi 
    3 
    4n = random.randint(1,6)
    5sum = mpi.allreduce(n, mpi.MAX) 
    6if mpi.rank==0:
    7  print 'total sum=',sum
    $ mpiexec -np 4 python ./rand4.py
    total sum= 6