IB/cma: cma_match_net_dev needs to take into account port_num