time: Avoid making adjustments if we haven't accumulated anything