As Rand explains in this Whiteboard Friday, most sites do not need to worry about this. However, for sites that have trouble getting full indexing and have a high number of pages, this can be a handy method to follow.
Faceted Navigation Architecture
Once the engine has gotten down to the product level, you can use search demand to determine what to break, using keyword research. By doing this at the product level, you don’t have to worry about the spiders reaching all your pages while allowing users multiple navigational methods.