You're assuming that there is only one universe.
No, I'm not. I'm talking about our universe. To travel from Earth to the edge of the OBSERVABLE universe AT THE SPEED OF LIGHT (186,000 miles per second) would take you 46 billion years, over three times the length of time that the universe has existed for.
So we can assume that there's a fair amount of stuff out there that alien spaceships can gather up as fuel.