E-Paper
Urgent need for explainable AI
This article is authored by Tauseef Alam, legislative associate and research lead to Sujeet Kumar, Member of Parliament (Rajya Sabha).
Artificial Intelligence (AI) systems are increasingly being deployed across critical domains such as governance, finance, policing, healthcare, and education. From automating loan approvals to facial recognition in law enforcement and algorithmic triaging in hospitals, these technologies are shaping decisions that significantly impact human lives. While AI promises efficiency and scalability, its rapid adoption has outpaced our ability to ensure transparency, accountability, and fairness, raising serious legal, ethical and psychological concerns.
AI systems are not inherently fair, as their algorithms depend on mathematical models and data, while fairness is a subjective, context-specific concept that resists simple quantification, making bias blindness a core limitation.
At the heart of the problem lies the phenomenon of ‘black box’ algorithms, AI systems whose internal logic is either too complex for human understanding or deliberately opaque due to proprietary restrictions. These systems often process vast amounts of data and generate outcomes without offering comprehensible reasons for their decisions. This lack of transparency makes it difficult to audit or challenge outcomes, especially when decisions are discriminatory or erroneous.
Consider the case of Apple Card (2019), where customers reported that women were being granted lower credit limits than men with similar profiles. Similarly, the COMPAS algorithm used in US courts to predict recidivism has been shown to exhibit racial bias against Black defendants. In India, automated systems in welfare schemes like Aadhaar-linked PDS have erroneously excluded beneficiaries due to algorithmic errors, with limited recourse available.
The principle of explainability in algorithmic decision-making is closely tied to the foundational legal doctrines of due process, natural justice, and non-discrimination, all of which require that individuals impacted by a decision must be informed of the reasons behind it. Without such transparency, affected individuals cannot contest or seek remedies against unjust or biased decisions made by AI systems.
The European Union’s General Data Protection Regulation (GDPR), under Article 22, provides individuals with a “right not to be subject to a decision based solely on automated processing” and, in some interpretations, a “right to explanation”. This has become a reference point for global AI regulation debates. In India, however, such a right is absent from current law.
The recently enacted Digital Personal Data Protection Act, 2023, does not explicitly confer a right to explanation in algorithmic decisions. While it emphasises consent and data processing principles, it stops short of addressing automated decision-making, a critical gap given the growing use of AI in welfare distribution, education, recruitment, and policing.
From an ethical standpoint, explainability is essential to preserve informed consent, personal autonomy, and dignity, core values under Articles 14 and 21 of the Indian Constitution.
There was an incident in 2015 when Amazon discontinued an experimental AI-based recruiting tool due to its discriminatory bias against women. Developed to automate resume screening, the system assigned scores based on patterns in predominantly male resumes submitted over a decade, penalising terms like ‘women’s’ and graduates from women’s colleges. Despite attempts to neutralise specific biases, the tool’s reliance on historical data perpetuated gender disparities, leading to its termination. This incident underscores the challenges of embedding fairness in AI systems, highlighting the need for robust data scrutiny and ethical considerations in algorithmic hiring processes to mitigate unintended discrimination.
The possible solution to this development is the incorporation of Explainable AI (XAI), which refers to a suite of methods and tools aimed at making AI systems’ decisions more transparent and understandable to humans. XAI enables stakeholders to comprehend how an AI model arrived at a particular decision, making it possible to detect errors, bias, or unintended consequences.
XAI techniques can be broadly divided into model-specific and model-agnostic approaches. Model-specific methods apply to inherently interpretable algorithms like decision trees or linear regression, where the logic behind predictions is relatively transparent. In contrast, model-agnostic techniques can be applied to any algorithm, including complex models like deep neural networks, which are notoriously opaque. Two widely used model-agnostic tools are LIME (Local Interpretable Model-Agnostic Explanations), which explains individual predictions by creating simpler models around each decision, and SHAP (SHapley Additive exPlanations), which attributes a value to each input feature based on its contribution to the output, using concepts from game theory.
However, deploying XAI at scale comes with challenges. There is often a trade-off between accuracy and explainability; transparent models are usually less powerful than black-box models. Additionally, explaining complex outputs to non-technical stakeholders, especially in India’s multilingual and socio-economically diverse population, makes the task even more difficult. Despite these barriers, XAI remains essential for building trust and accountability in AI systems.
While explainability is vital for accountability, full transparency can raise serious concerns. In India, tech companies may resist disclosing algorithmic details due to intellectual property or security risks, especially in sectors like fintech or healthtech.
Similarly, public agencies using AI for surveillance or fraud detection risk making their systems vulnerable to manipulation if models are too transparent. There is also a trade-off between accuracy and explainability. Complex models like neural networks often outperform simpler, interpretable ones. Replacing them solely for transparency may reduce effectiveness in critical applications.
A major grey area lies in defining who the explanation is for. Should it be understandable to experts, regulators, or the general public? In a linguistically and educationally diverse country like India, this becomes even more complex. Thus, while explainability is necessary, it must be balanced with security, performance, and context-specific needs. Clear regulations are essential to avoid both oversimplification and opacity.
Addressing the black box problem in AI requires collaboration across sectors, including technologists, policymakers, ethicists, and civil society, who must work together to ensure responsible AI deployment.
A key step is mandating Algorithmic Impact Assessments (AIAs) before public use of AI, especially in sensitive sectors like welfare, policing, or hiring. These assessments can help identify bias, risks, and unintended harms early on. Additionally, public audits, transparency registers, and clear redress mechanisms should be institutionalised to ensure accountability. AI literacy among citizens is equally important. People must understand how algorithmic systems affect their rights and be empowered to question them.
For India, this is an opportunity to lead with a rights-based, inclusive AI framework. A balanced approach by prioritising innovation while safeguarding democratic values will be key to building public trust in AI systems.
This article is authored by Tauseef Alam, legislative associate and research lead to Sujeet Kumar, Member of Parliament (Rajya Sabha).
