Yes, those are more complex scenarios for sure and people covered up the consequences for them pretty effectively over time.
But again, utilitarianism naturally reveals why they needed change.
Civil liberties exist because of us acknowledging legitimate and significant human suffering.