Which is exactly why this does smack of racism.
Why would someone even say this? It is in fact demeaning and Clinton was mocking the future President.
He could have been saying, 'look how far we've come, not that many years ago, he'd be serving us coffee'. It's not necessarily demeaning at all.
Since apparently you were there and were part of the conversation, perhaps you can fill us in on the nuances and body language, as well as the tone of the conversation.