"Those (actions) were, I think, as everyone can agree, if not criminal, they were certainly outrageous and unacceptable," Holder said. "But we are examining the facts to see if there were criminal violations."
Holder is scheduled to testify Wednesday before the House Judiciary Committee.
Three congressional committees already are investigating the IRS for singling out tea party and other conservative groups during the 2010 congressional elections and the 2012 presidential election. But Holder's announcement would take the matter to another level if investigators are able to prove that laws were broken.
Ineffective management at the IRS allowed agents to improperly target tea party groups for more than 18 months, said a report released Tuesday by the Treasury inspector general for tax administration. The report lays much of the blame on IRS supervisors in Washington who oversaw a group of specialists in Cincinnati who screened applications for tax exempt status.
It does not indicate that Washington initiated the targeting of conservative groups. But it does say a top supervisor in Washington did not adequately supervise agents in the field even after she learned the agents were acting improperly.
"The report's findings are intolerable and inexcusable," President Barack Obama said in a statement. "The federal government must conduct itself in a way that's worthy of the public's trust, and that's especially true for the IRS. The IRS must apply the law in a fair and impartial way, and its employees must act with utmost integrity. This report shows that some of its employees failed that test."
The agency started targeting groups with "Tea Party," ''Patriots" or "9/12 Project" in their applications for tax exempt status in March 2010, the inspector general's report said. By August 2010, it was part of the written criteria used to flag groups for additional scrutiny.
Lois Lerner, who heads the IRS division that oversees tax exempt organizations, had been briefed on the matter in June 2011. She ordered the initial tea party criteria to be scrapped, but it later evolved to include groups that promoted the Constitution and the Bill of Rights. The practice was ended in May 2012, the report said.
IRS agents were trying to determine whether the political activities of such groups disqualified them for tax exempt status. These groups were claiming tax exempt status as organizations promoting social welfare. Unlike other charitable groups, they can engage in political activity. But politics cannot be their primary mission.
It is up to the IRS to make the determination.
But by using improper criteria, the IRS targeted some groups, even though there were no indications that they engaged in significant political activities, the report said. Other non-tea party groups that had significant political activities were not screened, the report said.
"The criteria developed by the Determinations Unit gives the appearance that the IRS is not impartial in conducting its mission," the report said.
The additional screening resulted in long delays as IRS agents asked intrusive, sometimes inappropriate questions, or merely let applications languish, the report said. Inappropriate questions included requests for lists of donors and the political affiliation of officers.
"Unfortunately, the report raises more questions than it answers," said House Oversight Committee Chairman Darrell Issa, R-Calif. "What we do know for sure is that the IRS personnel responsible for granting tax exemptions systematically targeted conservative groups for extra scrutiny, and that officials in Washington, D.C., were aware of this practice, even while publicly claiming that it never happened."
The IRS on Friday apologized for singling out tea party and other conservative groups.
On Tuesday, the agency said, "After seeing issues with particular cases, inappropriate shortcuts were used to determine which cases may be engaging in political activities. It is important to note that the vast majority of these cases would still have been centralized based on the general criteria used for other cases."
Acting IRS Commissioner Steven Miller is scheduled to testify before the House Ways and Means Committee at a hearing Friday. Miller became acting commissioner in November, after Commissioner Douglas Shulman completed his five-year term. Shulman had been appointed by President George W. Bush.