View on GitHub

Pwrake

Parallel workflow extension for Rake

Download this project as a .zip file Download this project as a tar.gz file

Pwrake

Parallel workflow extension for Rake

(日本語README), (GitHub Repository)

Features

Installation

Download source tgz/zip and expand, cd to subdir and install:

$ ruby setup.rb

Or, gem install:

$ gem install pwrake

Usage

Parallel execution using 4 cores at localhost:

$ pwrake -j 4

Parallel execution using all cores at localhost:

$ pwrake -j

Parallel execution using total 2*2 cores at remote 2 hosts:

  1. Share your directory among remote hosts via distributed file system such as NFS, Gfarm.
  2. Allow passphrase-less access via SSH in either way:
    • Add passphrase-less key generated by ssh-keygen. (Be careful)
    • Add passphrase using ssh-add.
  3. Make hosts file in which remote host names and the number of cores are listed:

    $ cat hosts
    host1 2
    host2 2
    
  4. Run pwrake with an option --hostfile or -F:

    $ pwrake --hostfile=hosts
    

Options

Command line option

    -F, --hostfile FILE              [Pw] Read hostnames from FILE
    -j, --jobs [N]                   [Pw] Number of threads at localhost (default: # of processors)
    -L, --logfile [FILE]             [Pw] Write log to FILE
        --ssh-opt, --ssh-option OPTION
                                     [Pw] Option passed to SSH
        --filesystem FILESYSTEM      [Pw] Specify FILESYSTEM (nfs|gfarm)
        --gfarm                      [Pw] FILESYSTEM=gfarm
    -A, --disable-affinity           [Pw] Turn OFF affinity (AFFINITY=off)
    -S, --disable-steal              [Pw] Turn OFF task steal
    -d, --debug                      [Pw] Output Debug messages
        --pwrake-conf [FILE]         [Pw] Pwrake configuation file in YAML
        --show-conf, --show-config   [Pw] Show Pwrake configuration options
        --report LOG                 [Pw] Report profile HTML from LOG and exit.

pwrake_conf.yaml

Note for Gfarm

Tested Platform

Acknowledgment

This work is supported by