AI startups are growing rapidly and attracting a growing share of market investment. With the AI hype reaching new heights, investors are putting large bets on its transformative abilities. The emergence of agentic AIs has further reinforced these hopes. While it is undeniable that large language models (LLMs) have become staple to many workplaces, the persistent uncertainty about what AI can and cannot do raises the possibility that its rise may actually be just a bubble. Whether AI is on sustainable growth path or not hinges greatly on how well we understand its true capabilities and limitations.
LLMs are fundamentally advanced text predictors. Their main task is to predict the most probable continuation of a given sequence of words which is a process learned from vast amounts of text data across the internet. Unlike simple models, LLMs don’t always choose the single most likely word; rather they can sample from a range of options, which often gives the illusion of “creativity.”
However, raw text prediction alone can produce coherent but aimless text. To make LLMs responsive to user instructions, they need instruction tuning, where the model is retrained on high-quality instruction-response pairs. This allows a simple text predictor to follow directions and align outputs with user intent. Finally, through reinforcement learning from human feedback, the model further refines its responses in line with human values and preferences.
Clearly, the quality of an AI system depends heavily on the data it is trained on and how well it is aligned with user intent. Yet this is where the system’s limitations come in. One of the most common issues is hallucination -- when AI generates unverified or false information.
A 2024 study titled “Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews” found that across 11 reviews, hallucination rates were 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard (now Gemini). Hallucination occurs because LLMs are trained to produce plausible text, not truthful text: they have no inherent sense of truth. Since much of the training data online and in books is written confidently, the AI learns to sound confident even when wrong.
Apart from that AI also faces an explainability problem: it generates outputs based on complex algorithms, but users cannot easily determine why it produced a particular response. Creativity is another limitation. The AI’s “imagination” is constrained to permutations of its training data, many of which can be meaningless, implying it cannot truly think outside the box.
The problem of AI hallucination is particularly worrisome. In October, it was reported that Deloitte was forced to refund its fee to the Australian government over a report containing multiple errors generated by AI, including citations of non-existent academic references and a fabricated court case -- a perfect example of AI hallucinating.
In June, it was reported that in a machine learning book published by Springer, two-thirds of the citations were either incorrect or entirely non-existent. Similarly, in January 2024, a New York lawyer faced possible disciplinary action for citing a non-existent case produced by AI.
A further surprising revelation comes from a recent study conducted by METR, in which a group of experienced software developers were assigned coding tasks with or without AI tools. The results were astonishing: developers completed tasks 20% slower when using AI than when working alone.
This outcome is explained by the “capability-reliability gap.” Although AI systems have learned to perform an impressive range of tasks, they often struggle to execute them with the consistency. Even the most advanced systems make minor errors and thus require human supervision. As a result, developers spent significant time checking and redoing AI code which is more than what it would have taken to write the code themselves.
Apart from issues of hallucination, AI systems also exhibit a tendency toward sycophancy. AI sycophancy refers to the propensity of an AI to agree with or reinforce a user’s opinions or assumptions, even when they are incorrect.
Another related concern is alignment faking. Models are trained through reinforcement learning by being ‘rewarded’ for outputs that adhere to certain pre-determined principles. However, if a model’s inherent principles or preferences conflict with those given to it during reinforcement learning, a model may just “play along,” apparently complying with the new principles while its original preferences remain intact.
In December 2024, a study by the company Anthropic provided empirical evidence of this phenomenon, meaning that AI could be made to agree with statements it did not genuinely consider correct, effectively manipulating its outputs in response to the incentives provided.
Given these limitations, there is already considerable talk that much of the buzz around AI could be more hype than substance. MIT recently tracked the outcomes of 300 publicly disclosed AI initiatives and found that 95% projects failed to deliver any measurable boost to profits.
A March 2025 report by McKinsey revealed that more than 80% of respondents in their survey stated that their organizations were not seeing a tangible impact on enterprise-level revenue from their use of generative AI.
In June 2025, it was reported by Gartner that over 40% of agentic AI projects are likely to be cancelled by the end of 2027 due to escalating costs and unclear business value.
Considering these facts, it is difficult to ascertain what AI will actually deliver. The “black box” nature of AI systems means that we do not fully understand them, resulting in two contrasting perspectives: one of hallucinations and unreliability, and the other pointing to emerging abilities, including the capacity to solve tasks and perform actions for which they were not explicitly trained.
Therefore, it is impossible to dismiss the possibility that the AI phenomenon may represent a speculative bubble. However, given the deep roots that generative AI has established within the workforce, it may be more practical to follow the perspective of Daron Acemoglu, who predicts that AI will have a nontrivial but modest effect: certainly much less than the revolutionary changes some foresee, yet still substantial.
(Amit Kapoor is Chair and Mohammad Saad is researcher at the Institute for Competitiveness.).
(Disclaimer: The opinions expressed in this column are that of the writer. The facts and opinions expressed here do not reflect the views of www.economictimes.com)
LLMs are fundamentally advanced text predictors. Their main task is to predict the most probable continuation of a given sequence of words which is a process learned from vast amounts of text data across the internet. Unlike simple models, LLMs don’t always choose the single most likely word; rather they can sample from a range of options, which often gives the illusion of “creativity.”
However, raw text prediction alone can produce coherent but aimless text. To make LLMs responsive to user instructions, they need instruction tuning, where the model is retrained on high-quality instruction-response pairs. This allows a simple text predictor to follow directions and align outputs with user intent. Finally, through reinforcement learning from human feedback, the model further refines its responses in line with human values and preferences.
Clearly, the quality of an AI system depends heavily on the data it is trained on and how well it is aligned with user intent. Yet this is where the system’s limitations come in. One of the most common issues is hallucination -- when AI generates unverified or false information.
A 2024 study titled “Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews” found that across 11 reviews, hallucination rates were 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard (now Gemini). Hallucination occurs because LLMs are trained to produce plausible text, not truthful text: they have no inherent sense of truth. Since much of the training data online and in books is written confidently, the AI learns to sound confident even when wrong.
Apart from that AI also faces an explainability problem: it generates outputs based on complex algorithms, but users cannot easily determine why it produced a particular response. Creativity is another limitation. The AI’s “imagination” is constrained to permutations of its training data, many of which can be meaningless, implying it cannot truly think outside the box.
The problem of AI hallucination is particularly worrisome. In October, it was reported that Deloitte was forced to refund its fee to the Australian government over a report containing multiple errors generated by AI, including citations of non-existent academic references and a fabricated court case -- a perfect example of AI hallucinating.
In June, it was reported that in a machine learning book published by Springer, two-thirds of the citations were either incorrect or entirely non-existent. Similarly, in January 2024, a New York lawyer faced possible disciplinary action for citing a non-existent case produced by AI.
A further surprising revelation comes from a recent study conducted by METR, in which a group of experienced software developers were assigned coding tasks with or without AI tools. The results were astonishing: developers completed tasks 20% slower when using AI than when working alone.
This outcome is explained by the “capability-reliability gap.” Although AI systems have learned to perform an impressive range of tasks, they often struggle to execute them with the consistency. Even the most advanced systems make minor errors and thus require human supervision. As a result, developers spent significant time checking and redoing AI code which is more than what it would have taken to write the code themselves.
Apart from issues of hallucination, AI systems also exhibit a tendency toward sycophancy. AI sycophancy refers to the propensity of an AI to agree with or reinforce a user’s opinions or assumptions, even when they are incorrect.
Another related concern is alignment faking. Models are trained through reinforcement learning by being ‘rewarded’ for outputs that adhere to certain pre-determined principles. However, if a model’s inherent principles or preferences conflict with those given to it during reinforcement learning, a model may just “play along,” apparently complying with the new principles while its original preferences remain intact.
In December 2024, a study by the company Anthropic provided empirical evidence of this phenomenon, meaning that AI could be made to agree with statements it did not genuinely consider correct, effectively manipulating its outputs in response to the incentives provided.
Given these limitations, there is already considerable talk that much of the buzz around AI could be more hype than substance. MIT recently tracked the outcomes of 300 publicly disclosed AI initiatives and found that 95% projects failed to deliver any measurable boost to profits.
A March 2025 report by McKinsey revealed that more than 80% of respondents in their survey stated that their organizations were not seeing a tangible impact on enterprise-level revenue from their use of generative AI.
In June 2025, it was reported by Gartner that over 40% of agentic AI projects are likely to be cancelled by the end of 2027 due to escalating costs and unclear business value.
Considering these facts, it is difficult to ascertain what AI will actually deliver. The “black box” nature of AI systems means that we do not fully understand them, resulting in two contrasting perspectives: one of hallucinations and unreliability, and the other pointing to emerging abilities, including the capacity to solve tasks and perform actions for which they were not explicitly trained.
Therefore, it is impossible to dismiss the possibility that the AI phenomenon may represent a speculative bubble. However, given the deep roots that generative AI has established within the workforce, it may be more practical to follow the perspective of Daron Acemoglu, who predicts that AI will have a nontrivial but modest effect: certainly much less than the revolutionary changes some foresee, yet still substantial.
(Amit Kapoor is Chair and Mohammad Saad is researcher at the Institute for Competitiveness.).
(Disclaimer: The opinions expressed in this column are that of the writer. The facts and opinions expressed here do not reflect the views of www.economictimes.com)
You may also like
Canada says it's ready to engage after Trump ends trade talks over ad dispute
Winning EuroMillions numbers for £45m jackpot and Thunderball revealed
BJP stops NC clean sweep in J&K RS poll, wins 1 seat with 4 'extra' votes
ECI issues advisory to control misuse of AI-generated content during elections; urges parties to report fake accounts, deepfakes
Tess Daly and Claudia Winkleman to 'host new BBC show' after bombshell Strictly exit
