Removal of an Unprofessional ALJ from an SSA Disability Case

Imagine the following scenario. A disability case is returned to a representative after a lawsuit in federal court. The representative is crestfallen when she reads the remand order and sees the name of the the administrative law judge (ALJ) who is assigned to hear the case because it is the same ALJ who denied the claim in the first place, and because he has an egregiously low rate of approving disability claims. Her unpleasant situation invites the question, Can the representative successfully request that a different ALJ be assigned to the case? The answer is probably not.

According to HALLEX, the Social Security Administration (SSA) generally assigns the first remand⏤whether initiated by the Appeals Council or a federal court⏤to the same ALJ who initially heard the claim. So, in our scenario, the assignment of the case to the original ALJ is consistent with agency policy.

Still, the representative might ask about exceptions to the general rule. Disqualification is one such exception. The regulations permit disqualification for judicial bias or inadequate due process, but obviously not for a high denial rate.  

In our scenario, even if the claimant “did not receive a full and fair hearing,” it would undoubtedly be too late for the representative to request disqualification. HALLEX requires that a representative object to the ALJ’s assignment “at the earliest opportunity,” and presumably, she would have been aware of any unprofessional conduct prior to the federal court appeal.  

But if a disability claim is pending at the hearing level, and a representative is contemplating making a timely recusal request, she might wonder, When does an ALJ’s unprofessional conduct rise to the level of disqualifying conduct? A review of select federal court cases offers insight on this question.

 In Ventura v. Shalala (1995), the Third Circuit ordered a new hearing with a different ALJ because the original ALJ’s “offensive and unprofessional conduct” had undermined the claimant’s right to a full and fair hearing. Specifically, the Court noted that the ALJ had accused the claimant of lying about a matter that was irrelevant to the disability inquiry, and that the ALJ had largely ignored the claimant’s testimony about his back pain. Additionally, the Court noted the ALJ had rebuked the representative for introducing evidence that contradicted the ALJ’s assertion that the claimant’s back pain was caused by a psychological disorder and for asking the medical expert about a physical origin of the claimant’s back pain.

Miles v. Chater (1996) was decided by the Eleventh Circuit. The ALJ in Miles alleged without evidence that the opinion of the medical expert who had been hired by the claimant’s attorney was unreliable because he had a pro-claimant bias. That unsubstantiated allegation of bias led the Court to disqualify the ALJ for bias, explaining that because ALJs have significant power to determine the outcomes of the claims that come before them, their impartiality “is … integral to the integrity of the system.” 

The Seventh Circuit’s opinion in Sarchet v. Chater (1996) shows that case law limits the authority of federal judges to order SSA to disqualify ALJs, but that it does not limit their ability to recommend their removal from cases. 

Writing for the Court, Judge Richard Posner, the celebrated jurist, catalogued “a substantial number of illogical or erroneous statements” in the ALJ’s decision. Specifically, he noted the ALJ’s decision revealed that she was unaware of fibromyalgia’s classification as a rheumatic disease and of how the disease is diagnosed. He also noted she misunderstood the claimant’s testimony about her pain, as well as the testimony of two expert witnesses. Furthermore, he called attention to comments of hers that indicated a class bias. For example, he noted that she had belittled the claimant’s testimony as “melodramatic,” and that she had made “unfounded sociological speculations” about the claimant’s limited work history.  

Judge Posner ruled that a remand was warranted because the ALJ’s limited command of the facts made her analysis so “unreliable” as to constitute reversible error. While he expressed concern that the “tone” of her decision betrayed “an unshakeable commitment” to denying the claim, he held that there was insufficient evidence of bias to disqualify her; so he instead “merely recommend[ed], that the case be transferred” to another ALJ.

While a judicial recommendation does not carry the legal force of a judicial order, a representative should be encouraged if she has one in hand. According to Marcus and Bush, the authors of Social Security Disability Practice (2024), SSA usually follows the case reassignment recommendations of the federal courts.