Algorithmic sentencing: the black box of justice

The intersection of algorithmic efficiency and the pursuit of justice has created a systemic friction that threatens to dismantle the foundations of transparent rehabilitation. Algorithmic sentencing refers to the integration of data-driven formulas and risk assessment tools - frequently incorporating machine learning and artificial intelligence - to influence the most critical junctions of the criminal justice system. From the initial determination of bail to the finality of sentence length and the conditions of parole, these mathematical models now serve as silent advisors to the bench.

Proponents argue these tools increase consistency, efficiency, and objectivity by removing human emotion from the equation. That argument is not without merit. But its rapid, largely unregulated adoption has triggered a profound crisis around transparency, constitutional due process, and the historical mission of the court to rehabilitate the individual.

The black box problem and the erosion of due process

A central critique of modern judicial technology is the "black box" problem - a term describing the fundamental opacity of high-level AI systems. In the context of sentencing, this refers to the inscrutable nature of internal reasoning within advanced machine learning models. Even the developers who write the initial code may find it difficult to explain why a specific defendant received a particular risk score once a neural network has processed thousands of variables. This opacity makes it functionally impossible for defendants, their legal counsel, or the presiding judge to understand the logic behind a recommendation.

The Rise of Big Data Policing by Andrew Guthrie Ferguson

The definitive legal analysis of predictive policing technology, surveillance risk and civil liberties.

Buy now

The proprietary nature of these tools deepens the problem considerably. Because private corporations develop many of these algorithmic systems, they often claim trade secret protection over their source code and weighting methodologies. When a defense attorney attempts to challenge the validity of a recidivism risk score, they frequently hit a wall of intellectual property law. That corporate shield prevents public scrutiny of the data and methodology used to deprive individuals of their liberty.

In a system built on the right to confront evidence and cross-examine witnesses, the introduction of an unchallengeable mathematical oracle represents a radical departure from traditional legal standards.

Some legal scholars and data scientists have proposed addressing this through Explainability Scores - a metric assessing how easily a human can interpret an AI system's decision-making process. As a proposed framework, this would allow courts to set minimum interpretability thresholds for tools used in criminal proceedings: systems that cannot be explained to a meaningful standard would be prohibited from influencing bail or sentencing decisions. For cases where life and liberty are at stake, proponents suggest mandating high minimum explainability standards. Without such a requirement, the court risks delegating its moral and legal authority to an automated system that cannot account for its own reasoning. This push forms part of the broader and rapidly growing field of Explainable AI (xAI) - a discipline sitting at the intersection of computer science, ethics, and law.

Legal precedent under scrutiny: the Loomis case

The legal system's failure to resolve these tensions is perhaps best illustrated by State v. Loomis, decided by the Wisconsin Supreme Court in 2016. The case involved Eric Loomis, sentenced in part because a COMPAS risk assessment flagged him as high risk. His attorneys argued this violated his due process rights - specifically, his right to be sentenced on accurate information and his right to challenge the evidence used against him. The algorithm's methodology was a trade secret; Loomis had no ability to inspect or contest the basis for his score.

The Wisconsin Supreme Court upheld the sentence. But the ruling was far from a clean endorsement. The court placed explicit restrictions on COMPAS use: the score could not be the sole or determinative factor in a sentencing decision, it could not be used to establish sentence length, and any presentence investigation report containing a COMPAS score had to include a written advisement for the judge. That advisory was required to state that the tool had not been cross-validated for Wisconsin's specific population, that studies had raised concerns about its accuracy for minority defendants, and that it was designed to assess group risk - not the risk posed by any specific individual.

When Loomis petitioned the U.S. Supreme Court, certiorari was denied in 2017 - leaving the Wisconsin framework intact but the core constitutional question unresolved at the federal level.

What the case exposed was a structural gap that persists today. The legal system has no reliable mechanism to compel disclosure of proprietary algorithmic logic, even when that logic influences a prison sentence. The Loomis framework asks judges to treat AI-generated risk scores with caution while simultaneously denying defendants any meaningful tool for investigating the score's legitimacy. That is not a balanced solution. It is a contradiction coded into judicial practice.

Weapons of Math Destruction by Cathy O'Neil -

Bestselling investigation into how opaque algorithms reinforce inequality across sentencing, hiring, and finance.

Buy now

Algorithmic bias and the amplification of systemic inequality

Algorithms are not neutral entities - they are reflections of the data used to train them. Because these systems are fed historical data from a criminal justice system already shaped by societal biases, they risk codifying and amplifying those disparities. The most prominent example of this failure is COMPAS - the Correctional Offender Management Profiling for Alternative Sanctions tool used extensively across the United States to predict the likelihood of a defendant re-offending.

A landmark 2016 analysis by ProPublica, examining COMPAS scores for more than 7,000 defendants in Broward County, Florida, revealed deep-seated racial inequities in the tool's outputs. Black defendants who did not eventually recidivate were misclassified as high risk at nearly twice the rate of white defendants - 45% compared to 23%. Conversely, white defendants who did re-offend within a two-year window were mistakenly labelled as low risk almost twice as often as their Black counterparts - 48% compared to 28%. Even when controlling for prior criminal history, future recidivism, age, and gender, the algorithm was 77% more likely to assign higher violent risk scores to Black defendants.

The developer, Northpointe (since rebranded as Equivant), disputed the analysis. Their core counter-argument: COMPAS achieves predictive parity - meaning a given score corresponds to roughly the same actual probability of reoffending, regardless of race. Both positions can be simultaneously true. It is mathematically provable that a predictive tool cannot simultaneously achieve equal false positive rates across groups and equal predictive accuracy when the underlying base rates of the measured outcome differ between those groups. When ProPublica and Northpointe argued over competing definitions of fairness, they were not simply disagreeing about one algorithm - they were revealing a fundamental constraint embedded in the very mathematics of risk prediction.

That mathematical problem has no clean resolution. And it is being applied, right now, to decisions about human freedom.

Furthermore, many risk assessment tools use socioeconomic proxies for race that maintain the appearance of neutrality while delivering discriminatory outcomes. Variables such as ZIP codes, the number of friends with arrest records, or housing stability are frequently embedded in these calculations. These factors are often direct results of historical redlining and decades of over-policing in specific neighborhoods, meaning the algorithm effectively punishes a defendant for their environment rather than their individual conduct.

The Tulane University findings and the limits of judicial discretion

Recent research continues to paint a complex picture of AI in the courtroom. A 2024 Tulane University study analyzed over 50,000 drug, fraud, and larceny convictions in Virginia to determine whether AI tools could help correct human bias in sentencing decisions. The data was genuinely encouraging in parts. AI recommendations increased the likelihood that low-risk offenders would avoid incarceration by 16% for drug crimes, 11% for fraud, and 6% for larceny - suggesting real potential to reduce unnecessary jail time.

But the same study uncovered a troubling pattern in how judges actually interact with the tools. Even when the AI suggested leniency or alternative punishments, judges were found to disproportionately decline those suggestions for defendants of color. The technology operated as a one-way ratchet: invoked to justify harsher sentences when the score was high, set aside when it pointed toward rehabilitation.

This selective application transforms an ostensibly objective tool into a mechanism for selectively endorsing pre-existing preferences. The consistency that algorithmic sentencing was designed to deliver collapses entirely when the human element can override "objective" data - but only in one direction. What results is arguably more corrosive than unmediated bias: discrimination laundered through the credibility of mathematics.

The end of transparent rehabilitation and the rise of risk management

The shift toward algorithmic sentencing represents more than a technological change. It signals a fundamental reorientation of what sentencing is for.

Traditional sentencing, while deeply imperfect, allows for a transparent discussion of a defendant's life, potential for change, and criminogenic needs - the specific factors driving criminal behavior that can be addressed through targeted social intervention. Algorithmic tools, by contrast, focus almost exclusively on static, historical factors to produce a prediction of future conduct. This turns the sentencing hearing into a data-driven rubber stamp rather than a holistic evaluation of a human being.

When a judge leans too heavily on a recidivism score, the broader context - a defendant's social environment, family support, mental health history, and genuine mitigating circumstances - becomes secondary to a numerical output. The concern is not abstract. Evidence consistently shows that educational programs, vocational training, stable housing, and substance abuse treatment can meaningfully reduce reoffending over time. If the system only registers a high risk score, it is less likely to invest in the very interventions that could lower that risk.

The result is a self-defeating logic: tools designed to manage risk end up entrenching it.

Predictive policing and the cycle of over-enforcement

The issues of bias and transparency extend well beyond the courtroom. In the streets, they take the form of predictive policing - and the feedback loops it creates.

Tools like PredPol and Chicago's Strategic Subject List - commonly known as the "heat list" - have been criticized for generating self-fulfilling prophecies. The Chicago Police Department's algorithm, active from 2013, labelled hundreds of young Black men as likely perpetrators or victims of gun violence, despite many having no prior criminal record. A RAND Corporation evaluation found that individuals on the so-called heat list were no more likely to be involved in a shooting than matched individuals who were not flagged - a stark verdict on the tool's predictive validity. The Chicago program was eventually shut down by 2019 following sustained criticism from civil rights groups, the ACLU, and city council members.

The mechanism that makes these tools particularly dangerous is the feedback loop they create. By directing policing resources toward algorithmically identified individuals and neighborhoods, the system increases the likelihood of arrests in those areas. Those arrests then feed back into the dataset as new "evidence," reinforcing the perceived need for more policing in the same locations - independent of whether crime has actually increased. The data learns to confirm its own assumptions.

A parallel cautionary tale emerged in Europe. The OxRec algorithm, used by Dutch probation services approximately 44,000 times per year to assess recidivism risk, was investigated and found to have misjudged risk scores in roughly a quarter of cases. Investigators found it relied on outdated data and posed a real risk of discrimination. The Dutch Ministry of Justice and Security directed the probation service to adjust or cease using the tool - one of the first direct regulatory interventions of its kind on the continent.

The global regulatory response

The European Union has moved furthest in establishing a legal framework for these tools. The EU AI Act - the world's first comprehensive legal framework for artificial intelligence, which entered into force in August 2024 - draws a deliberate line around the use of AI in criminal justice.

Under Article 5, AI systems used to assess or predict the risk of a person committing a criminal offence based solely on profiling or personality traits are classified as prohibited practices. These prohibitions became enforceable in February 2025. Separately, the Act classifies AI tools deployed in law enforcement and the administration of justice as high-risk, meaning providers must demonstrate compliance with strict requirements around risk management, transparency, human oversight, and post-market monitoring before deployment. Full application of the high-risk AI regime is expected by August 2026.

The Act's extraterritorial scope is significant: if an AI system's outputs are used within the EU, the Act applies regardless of where the developer is headquartered. US-based vendors whose risk assessment products are deployed in European jurisdictions face concrete compliance obligations as a result. For algorithmic tools that currently operate behind a wall of trade secret protection, this represents a structural shift in what governments can demand.

Outside the EU, the regulatory picture remains fragmented. Canada has led on one front, pioneering Algorithmic Impact Assessments (AIAs) that require federal agencies to evaluate the potential for bias and the severity of the impact on individual rights before any AI tool is implemented in public decision-making. The United States, by contrast, has no comparable federal framework. The constitutional questions exposed by State v. Loomis remain unresolved at the national level, leaving whatever protections exist to operate through individual state courts and voluntary policy choices by agencies.

Path toward accountability: xAI and the future of just sentencing

To prevent the total erosion of transparency in the justice system, a robust accountability movement has emerged around a clear set of demands.

Private companies providing algorithms for public judicial functions should be subject to the same transparency standards as government agencies. This means making their methodology available to independent third-party audits, and - in the view of many reformers - to freedom of information requests where public judicial functions are concerned. The logic is straightforward: a tool that influences liberty cannot be shielded from the scrutiny that liberty demands.

The push for Explainable AI (xAI) sits at the core of this agenda. The goal is not to ban algorithmic tools from courtrooms. It is to ensure that the logic can be inspected, contested, and corrected. A judge must be able to state exactly which factors led to a particular recommendation. A defendant must have the right to challenge the accuracy of those factors - and that requires knowing what they are in the first place.

Ultimately, AI should serve as a diagnostic tool to assist the judiciary - not a replacement for the nuanced, contextual, moral reasoning that only a human can apply to another human's circumstances.

A system that assigns a number to a person's future, conceals the methodology behind that number, and deploys it in proceedings where the individual has no mechanism to challenge it is not a justice system. It is a sorting mechanism. The "black box" of justice will continue to obscure the path to rehabilitation for the most vulnerable defendants until regulatory standards catch up with the technology - and until courts are willing to insist that evidence means evidence, not an algorithm's unverifiable conclusion.