The AI that knows everything about you will ask you to click a link

The debate on AI is fixated on how smart the machines are becoming. The more urgent question is how we will tell a trustworthy one from a perilous one

A friend of mine recently got hold of an invite-only app called Instinct. It sits inside an ordinary messaging app on his phone, and you talk to it as you would to a colleague. Curious, he began testing it with questions.

For business leaders, the lesson is to treat every AI tool as a vendor with access to your people and customers, and to ask the same hard questions you would ask of a bank or an auditor. For the rest of us, it is simpler. Be most careful with the tool that seems to know you best. (Reuters)

The answers unsettled him. It gave him a dizzying amount of personal detail, including his in-laws’ residential address and his national ID and tax ID numbers. As a final test, he asked it to move money from one of his bank accounts to another. It replied that it would need his authorisation, and helpfully supplied a link in the chat where he could give it.

He did not click. He has been thinking about that link ever since, and so have I.

A good design that looks like a bad one: From what my friend and I have been able to establish, Instinct is a legitimate, invite-only product, not a scam. That is what makes the episode worth telling.

In its best light, this is a well-designed assistant. It is capable, but it stops at the moment of consequence and asks the human to authorise the transfer. That is what most of us say we want from artificial intelligence (AI).

Now consider that moment from the user’s side. We have spent years teaching people that this exact sequence — a familiar sender, personal details and a link — is the signature of fraud. A legitimate product and a convincing fraud now use the same mechanism, the same wording and the same urgency. Nothing on the screen tells you which one you are dealing with.

I am not suggesting Instinct has done anything wrong. My point is that it does not matter. When good products adopt the habits of bad actors, the public loses the signals it relies on to protect itself. Scammers will copy the behaviour precisely, and ordinary people will have no reliable way to tell the difference.

There is a question that deserves more attention than it usually gets. How did an app learn the address of someone’s in-laws, or his tax number? Perhaps from accounts and permissions the user had granted, perhaps from data already scattered across the digital economy. My friend was not sure, and the app did not volunteer an explanation. Scattered data was once protected by inconvenience of gathering and organising it. A model removes that inconvenience, and the user is often the last to learn what it has gathered.

The wrong debate in all this? We spend a great deal of time asking whether AI will take jobs, whether it will become conscious, whether India will build the biggest model. These are legitimate questions, but they have crowded out a more immediate one. In the next few years, most of us will not be harmed by a superintelligence. We will be harmed, or helped, by an assistant we cannot properly vet.

The scarce resource in the AI age is not intelligence. It is verifiable trust. When a system can write, speak, reason and act on our behalf, the question that matters is whether we can confirm who built it, what it can see, what it keeps, and who answers when it goes wrong.

India is unusually well-placed to lead on this. We have built our digital public infrastructure (DPI) on the principle that people should control access to their own data, with consent built directly into the software architecture rather than bolted on as a regulatory afterthought. The account aggregator framework in finance is one example. Our data protection law adds a further safeguard, though a law is only as good as its institutional enforcement.

The next step is to extend that thinking to AI agents. I would suggest three rules.

First, consent must happen somewhere the agent cannot control. If the same system that asks for your permission also supplies the link to grant it, you are being asked to trust the asker. Authorisation should happen inside your bank’s own app or a regulator-approved channel, never through a link delivered in a chat.

Second, an agent should be able to prove who it is. Just as we expect a bank to be registered, an AI that touches money, health or identity should be traceable to an accountable entity.

Third, an agent should be able to explain where its knowledge of you came from. If it cannot, that should end the conversation.

What can individuals and leaders can do? For business leaders, the lesson is to treat every AI tool as a vendor with access to your people and customers, and to ask the same hard questions you would ask of a bank or an auditor. For the rest of us, it is simpler. Be most careful with the tool that seems to know you best. Never authorise a financial action through a link a chat hands you. And ask what a convenient service is really being paid with.

The pattern to watch for is a familiar one. A system earns your attention by showing you how much it knows, and then asks for your trust. Sometimes that system will be honest. Often it will not be. The machines will keep getting smarter. Our task is to get harder to fool.

Bharat Joshi is director, ACTL (a logistics company), and CEO, J Curve Ventures (an advisory firm). The views expressed are personal

(This is a Hindustan Times digital exclusive)