The deaths occurred Monday when the unit came into contact with enemy forces, the coalition headquarters in Kabul said in a statement. The deaths are under investigation.
"Tragically, there is the possibility that fratricide may have been involved," the coalition statement said.
The incident comes as U.S. and allied forces are drawing down and turning the fight over to Afghan security forces.
Monday's deaths rank among of the most serious cases involving coalition-on-coalition friendly fire during the nearly 14-year Afghan war. One of the worst came in April 2002 when four Canadian soldiers were killed when an American F-16 dropped a bomb on them near a night firing exercise in the southern Kandahar.
Several news outlets, including the Associated Press and Afghanistan's Pajhwok Afghan News, reported that Monday's event took place in Arghandab district when troops called for air support during an operation in Gizee area. An Afghan National Army member and an interpreter also were killed, Pajhwok reported.
Taliban spokesman Qari Yusuf Ahmadi told Pajhwok that Taliban fighters were locked in a gun battle with foreign troops when coalition planes bombed their own troops.
The deaths bring to 36 the number of NATO soldiers killed so far this year in Afghanistan, with eight service members killed in June.
Friendly fire kills 5 NATO troops in Afghanistan (external - login to view)