I'm skeptical about that 50 thousand figure, but for the sake of argument, 50 thousand is nothing considering that the population of the colonies in 1776 was 2.5 million.
Regarding slavery, that's just a red herring. Most of the world had slavery in those days. Britain abolished slavery in 1833...