workqueue: destroy worker directly in the idle timeout handler