It will become a derogatory term when enough people use it in a derogatory manner. That is the problem with political correctness, how it is used not with it's intent. Intentions do, or rather they should, count for a lot but the over use of political correctness in society negates and downright ignores the intent. Personally, I use or don't use words, phrases or terms only when I know, or sincerely believe, that there is genuine offense taken and not simply because it's politically incorrect to say them.Colored people started as a derogatory term people of color wasn't but I'm sure will be considered derogatory in due time.
I wasn't kidding when I said this topic is a tar baby.
BTW, you should feed your baby, he's looking a little malnourished.