The infertility factors could be diverse in both men and women. Below is the list of the problems that could be possibly hindering conception:
Image source - www.psdgraphics.com
The most common factor is difference in the count / motility / morphology from normal attributes of sperm. The following link gives the detailed explanation on the same - http://community.askgargi.com/159/normal-sperm-various-defective-conditions-associated-sperms
Apart from this, there could be other reasons involving genetic disorders, retrograde ejaculation (semen is ejaculated within the bladder rather than outside), structural problems associated with testicles etc.
With the growing number of infertility cases, the common ones attributing to female infertility are PCOS, Endometriosis, ovarian cysts, abnormalities in the physiology of uterus and cervix, antisperm antibodies, thyroid, chemotherapy and radiation.