I'm talking about my vote as a citizen having an influence over legislation.
A4NoOb, here we are not talking of legislation, we are talking of fundamental human rights. You are right when you say that your vote should count, have an influence over legislation, that is how democracy works.
However, your vote (or mine for that matter), should not have any influence over whether slavery is legal, whether women have the right to vote, whether we have freedom of speech, freedom of worship etc. These are issues of fundamental human rights and as such proper purview of courts and legislatures.
SirJoseph thinks that as a citizen I don't have that right because politicians are morally superior or have some element that grants them that authority.
Again, you have the right to influence the legislation (e.g. what should be the driving age, what should be the income tax level etc.). You have no right to strip somebody of freedom of speech, freedom of religion etc. by a majority vote. Similarly, you have no right to bring back slavery, take vote away from women, or forbid gays to marry.
Human rights are properly decided by courts and by legislatures, and not by mob rule. If we went by mob rule, majority rule, slavery would still be legal today.