Nope, spot on. And recognized in the law. One has to have "scienter" to commit a criminal act, that is, knowing it's illegal (or "wrong," if you choose) and doing it anyway. If Town A were to pass a local ordinance that you couldn't wear orange on Thursday, and you, a passer-through, happened...