greybus: interface: prevent reactivation during removal