The Nazi emblems are on an anchor that is on display in Pointe-des-Cascades, about 50 miles west of Montreal in Quebec.
Corey Fleischer, founder of a group called Erasing Hate, was trying to paint over the symbols last Thursday when he was stopped by the local mayor, Gilles Santerre, who called police to have him removed from the park.
Stressing it was part of the area's local history, he said the anchor, which belonged to a merchant vessel, predated the Second World War and was discovered by divers 25 years ago.
He cited a Radio Canada article that said the swastika was a symbol of peace before 1920.
The mayor sad the symbols would not be removed, but promised to place a more descriptive plaque next to the anchor to clarify its history.
Mr Fleischer, who believes the anchor should be in a museum, acknowledged the swastika was originally a religious icon that represented good luck.
But ever since the Nazis subverted its meaning he said there is no place for it in a public park.
"It is no longer a sign of peace. It is no longer a sign of joy," he told CBC.
"There is no ifs ands or buts about it," he told the BBC. "It is a place where people come to feel safe and this is being displayed for everybody to see."
David Ouellette, of the Centre for Israel and Jewish Affairs, said Mr Fleischer should not have acted without consulting the village but said it was important the anchor's history was made clear to the public.
"If you're going to display publicly the swastika, it's important to give the full context," Mr Ouellette told the CBC.
picture of anchor
Canadian town refuses to remove swastikas from park as mayor defends local history