Funny thing in all of this, I used to watch Letterman nightly (back in the early 1990s). I haven't really seen his show in a long time, so can't say what has or has not happened.
In addition to your tastes changing saz, it's also possible the show itself has changed (and perhaps definitely since I last saw it). Back then, some of these other comedians were the new kids on the block, and Letterman's main contender was Johny Carson (if I remember correctly), who has long since not been around...whereas Letterman's show had been around a bit longer.