OpenMPI needs to have an rsh binary. Even if you are using shared memory for openmpi, and openmpi does not use rsh, it still looks for the binary and fails if it cannot find it.
Chroots (used on HCC machines for grid jobs) do not support pty's. OpenMPI has a compile option to turn off pty support.
Once these issues where fixed, we were able to submit QE jobs to the OSG using Condor's partitionable slots on 8 cores.
Before submitting our first QE job, we had to compile OpenMPI and QE. Since we are an HPC center, we had OpenMPI compiled for our Infiniband, therefore it would always fail on the OSG where there is no Infiniband (let alone our brand and drivers).
After compiling, we created compressed files that contained the required files to run QE:
bin.tar.gz - Only includes the cp.x file, specific to our run. It could have well included much more common pw.x.
lib.tar.gz - Includes the Intel math libraries and libgfortran.
openmpi.tar.gz - Includes the entire openmpi install directory (make install)
Additionally, we wrote a wrapper script, run_espresso_grid.sh, that unpacks the required files and sets the environment.
#!/bin/bash tar xzf bin.tar.gz tar xzf lib.tar.gz tar xzf pseudo.tar.gz tar xzf openmpi.tar.gz mkdir tmp