Well my father, and fathers father as far back as we can trace did none of this. I am second generation American. Mom's parents were from Newfoundland (via Ireland in the late 1700's), Dad's were from County Cork Ireland. The color of my skin does not make me guilty.
Neither did my personal family to the best of my knowledge. But if my neighbour gets robbed, and the police go to help her, they do not bring an interac machine with them. It's a free service out of compassion for what that person suffered. When a person suffers a crime, if we can make the criminal pay, we should. But when it's not possible, then society ought to make some kind of compensation out of sheer compassion.
We can't necessarily go out and start picking whose father did what. But it's pretty clear that the indigenous languages, cultures, and consequently peoples are suffering today for these crimes, and society ought to show some compassion with a desire to rebuild these cultures.
Sweden has never been involved in colonialism, yet it gives more of its GDP to develping countries every year than do ex-imperial countries such as France (e.g. Zaire), the UK (e.g. the North America), the US (e.g. the Phillipines), etc.
An injustice has been made, and society ought to be compassionate enough to try to rectify it. If the perpetrators can't rectify it, then society as a whole ought to bear the burden for such serious crimes as colonialism and genocide, be it bodily or cultural.
Instead, we always hear this attitude of privilege, that English, French, Spanish and other major world languages and cultures should be exploited to our advantage with no regard for the damage they are causing to smaller languages and especially with no regard for how they became so powerful in the first place. This thsu allows us to cash in on the interest accrued through imperialism, thus adding insult to injury.