Jimmy Tayse is expected to be extradited first to Pittsburgh, where police say he kidnapped a woman at a Giant Eagle by holding a knife to her toddler's throat.
Police said Tayse told the woman to drive to Ohio, stopping at ATMs along the way to get cash, before allegedly raping her in a motel room.
Police said he held the toddler in his arms the entire time.
In a news conference, police praised the woman for complying with her attacker and said she put her child's life ahead of her own.
"In the way she responded, she probably saved her life and the baby's life. She tried to reason with him until he let her get out of the vehicle," said Assistant Fire Chief Maurita Bryant.
Police said it does not appear that Tayse knew the woman.
Officials said Tayse will eventually be returned to Cambria County on a bench warrant for a number of charges including false imprisonment and driving under the influence.