I just think when it looks like it has been knocked around a bit, or if it is a bit faded from UV, I put it into semi-retirement.
By semi-retirement, i mean that it becomes a travel helmet that I use for trips. In general terms I assume that I probably am more likely to damage my helmet when it is in vehicles, hanging from my backpack as I walk through airports, wind blowing it off of the picnic table in a campground, etc. And I would rather not damage a newer helmet that way so I use my older helmets on trips. Then after a few trips, it goes in the trash.
The one I wore on my tour in June and July this past summer went into the trash at the Hostel in Halifax before I flew home.
This is not the ideal way to treat a helmet, under a pannier when I stacked up the luggage on the cart at the airport.