Not just the fertile ones, but also the men that are doctors, nurses, and other medical staff as well as the man/men who may own the building that the abortions take place in, the man who drives the pregnant woman to the clinic, the men that made the laws allowing abortions and the men that elected those men, etc.Abortion involves pregnancy. Pregancy involves men (or at least their seed). So now you have to include men (OK, maybe just the fertile ones) in the ability to vote.
Like I said, what goes on inside a society is the business of everyone in that society.