MTL_OFI: Changed Recv cancel to be non-blocking

Updated the OFI MTL's Recv cancel to be a non-blocking call to match
the MPI spec. Given fi_cancel succeeded, then it is expected that the
user will wait on the request to read the result of if the cancel has
completed.

Signed-off-by: Spruit, Neil R <neil.r.spruit@intel.com

Author: matcabral

ompi/mca/mtl/ofi/mtl_ofi.h

@@ -1197,14 +1197,8 @@ ompi_mtl_ofi_cancel(struct mca_mtl_base_module_t *mtl,
                 ret = fi_cancel((fid_t)ompi_mtl_ofi.ofi_ctxt[ctxt_id].rx_ep,
                                &ofi_req->ctx);
                 if (0 == ret) {
-                    /**
-                     * Wait for the request to be cancelled.
-                     */
-                    while (!ofi_req->super.ompi_req->req_status._cancelled) {
-                        opal_progress();
-                        if (ofi_req->req_started)
-                            goto ofi_cancel_not_possible;
-                    }
+                    if (ofi_req->req_started)
+                        goto ofi_cancel_not_possible;
                 } else {
 ofi_cancel_not_possible:
                     /**