Just looking at one of the factors in natural selection backs this up, stating: Individuals that are better able to cope with the challenges presented by their environment tend to leave more offspring than those individuals less suited to the environment do.
Basically, unfavorable genes are meant to be stopped before they are able to spread.
Sure, but you're reading into that a bit much. Individuals who are better able to cope with challenges presented by their environment are individuals who live longer lives, whether from being hardier to survive adverse weather, or faster to be able to escape predators, et cetera, and in living longer are able to leave more offspring than those who are not as well suited to the environment and end up dying earlier.
Of course, humans are very much different than every wild lifeform. We're above survival of the fittest, our gene pool is full of worthless genes that should never have been able to breed in the first place, and we poison ourselves on a daily basis with the garbage we eat and all the nasty pollutants and chemicals we come in contact with daily. Infertile people aren't necessarily infertile because they should be, they may be that way only because of their environment (and obviously people naturally get infertile in middle age but for strange reason think that they should still be allowed to have children).