"We have no information that the family knew this individual or that the individual knew any members of the family," Los Angeles Police Deputy Chief Kirk Albanese said.
About 40 detectives have been investigating since the girl was abducted from her home at around 3 a.m. Wednesday.
"They're working around the clock until this guy gets captured," Officer Norma Eisenman, a police spokeswoman, said Sunday. "As of right now, there are no new developments on the case."
The girl was found about 12 hours after she was abducted, wandering near a Starbucks several miles away.
She was barefoot, had bruises and scratches, and wasn't wearing the same clothes she had on when she vanished. The girl told police two men she didn't recognize had taken her from her home.
Police initially said they were looking for two suspects but now are focusing their efforts on Summers.
"This is the only person we are looking for right now," Albanese said Saturday.
Investigators have said they believe the girl was driven around the San Fernando Valley in a couple of cars and taken to at least two locations, including a storage facility, before she was released.
A passer-by who recognized her picture from media reports saw her outside the Starbucks and called police. The girl had wandered there from the hospital where she was dropped off.
The Los Angeles Times reported law enforcement sources said the girl was sexually assaulted. The Associated Press does not identify victims of sexual assault. Summers isn't a registered sex offender, police said.
Summers, who has a distinctive tattoo of a ghoulish face on his right arm, has arrests dating back to 2002, police said. Among them are robbery, grand theft auto, possession of explosives and kidnapping.
Police said they had no details on the prior kidnapping case.
Summers was released from prison in July after serving his sentence on a petty theft conviction. His supervised release was being handled by county probation instead of state parole as part of a California law designed to ease crowding in state prisons. He also spent six days behind bars in January on a probation violation.
Summers last checked in with his probation officer earlier this month and had been complying with his release terms, police said. He is known to frequent the area where the 10-year-old's kidnapping took place.
Albanese said Summers had been arrested four years ago for investigation of battery that involved "child annoyance." Court records show Summers was convicted of battery in September 2009 but the child annoyance charge was either dismissed or not prosecuted.
Summers has family in Southern California, according to police, and the FBI said it will obtain a warrant for unlawful flight to avoid prosecution if it determines he left the state.