[media] v4l: vsp1: Don't configure RPF memory buffers before calculating offsets