I – Who Controls Whom – Artificial Intelligence vs. Human Intelligence
The Dangers of Using Artificial Intelligence in Professional Work
The Discovery Of Fabricated Reports, Legal Definitions & Legal Citations
Practical Case Study of Wrong Court Judgements & Reports generated by AI
The Bombay High Court’s Warning Serves as a Significant Precedent
When non-existent case laws are cited as binding precedent, it not only misleads the process but also erodes the credibility of judicial and quasi-judicial institutions.
1) KMG Wires Private Limited Vs National Faceless Assessment Centre (Bombay High Court)
The Bombay High Court heard a writ petition challenging an assessment order passed under Section 143(3) read with Section 144B of the Income Tax Act, 1961, for the Assessment Year 2023–24.
The Court noted that the Assessing Officer had relied upon judicial decisions that were completely non-existent. The Court observed that in this era of Artificial Intelligence, one tends to place much reliance on results thrown open by the system, but while exercising quasi-judicial functions, such results should not be blindly relied upon and must be duly cross-verified before use.
Otherwise, mistakes like the present one may occur. The Court further recorded that no basis or working was shown to the petitioner and that no show-cause notice was issued before making the addition of peak balance.
In view of these findings, the Court quashed and set aside the assessment order the notice of demand issued under Section 156, and the show-cause notice for penalty issued under Section 274 read with Section 271AAC. The matter was remanded to the Assessing Officer, who was directed to issue a fresh show-cause notice, provide reasonable opportunity of being heard, and grant a personal hearing before passing a speaking order. The officer was also directed to give at least seven days’ notice before relying on any decisions.
Critical Analysis
*The court stated that when non-existent case laws are cited as binding precedent, it not only misleads the process but also erodes the credibility of judicial and quasi-judicial institutions*.
The Assessing Officer’s failure to verify the authenticity of the cited judgments reveals an abdication of responsibility. The Court rightly treated this as a complete breach of natural justice, since the assessee was confronted with false legal grounds that could not be challenged or verified. This also raises deeper concerns about the unchecked use of AI-generated legal material in governmental proceedings, especially when officers lack training or awareness of AI’s limitations.
The Bombay High Court’s warning serves as a significant precedent: AI can assist but cannot substitute human judgment. Quasi-judicial authorities must apply independent reasoning, verify all facts and citations, and ensure transparency. Reliance on “non-existent” judicial decisions not only vitiates the assessment order but also exposes systemic risks in automated tax adjudication. The ruling thus reaffirms the principle that technological efficiency can never override the foundational duty to ensure fairness, accuracy, and justice.
2) Deloitte $440,000 AI Fiasco
Deloitte agreed to give back part of the $440,000 it was paid by the Australian government after a report it wrote using AI was found to have serious mistakes, including fake references and a made-up court case. The report, made in July 2025 for the Department of Employment and Workplace Relations, had to be corrected and reissued after the errors were found. Deloitte admitted it used an AI tool (Azure OpenAI GPT-4o) to help write the report but said people later reviewed it. The case caused public criticism and raised concerns about how much professionals depend on AI without checking its work. Experts said AI can sound confident but often makes up false details, and this shows that human checking is always necessary. Without proper verification, AI-written reports can cause embarrassment, loss of trust, and damage to reputation.
3) ITAT Bangalore Case:
Buckeye Trust v. Principal Commissioner of Income Tax‑1, Bangalore (ITA No. 1051/Bang/2024) In the case of Buckeye Trust v. Principal Commissioner of Income Tax‑1, Bangalore (ITA No. 1051/Bang/2024), the ITAT passed an order in December 2024 in a matter relating to taxability of a trust settlement involving a transfer of interest in a partnership firm. The bench ruled in favour of the Department and cited several judgments—three purported Supreme Court judgments and one Madras High Court judgment—as supporting authority. These judgments were cited to show that a partnership interest is akin to a “share” and therefore taxable.
On scrutiny, it was discovered that at least two of the cited judgments do not exist, one citation corresponds to a different case altogether, and the fourth was real but had no relevance to the issue at hand.
Non-Existent or Mis-Cited Judgments
According to media coverage the case laws cited and which are in question are as under
1. K. Rukmani Ammal v. K. Balakrishnan (1973) 91 ITR 631 (Madras High Court) — which does not appear in Madras High Court archives.
2. S. Gurunarayana v. S. Narasinhulu (2004) 7 SCC 472 (Supreme Court) — which appears to be fictitious.
3. Sudhir Gopi v. Usha Gopi (2018) 14 SCC 452 (Supreme Court) — which when traced is not relevant to the issue as cited.
4. CIT v Raman Chettiar (57 ITR 232 (SC) — which is a real case but unrelated to the issue of partnership interest as share.
Because these citations either did not exist or were irrelevant, within a very short time the ITAT withdrew the order (via notice citing “inadvertent errors”) and arranged for a fresh hearing due to these flaws.
Many people today worry that Artificial Intelligence (AI) will take away human jobs. AI can work fast, write reports, analyse data, and even prepare documents that look professional. But AI cannot think or understand like humans. It does not have feelings, judgment, or common sense. It only guesses the next word or number based on patterns it has seen before. So, AI can help people, but it cannot replace them. Only humans can make moral and sensible decisions.
Speed Without Understanding
AI works fast, but speed is not the same as understanding. It cannot tell right from wrong or understand emotions. For example, an AI tool may write a report or a legal draft, but only a person can check if it is correct and fair. AI may sound confident but still be wrong. Therefore, AI should be used only as a helper. Humans must always check and confirm everything before using it.
The “Day” Question — How It All Started
A Chartered Accountant once asked a famous AI tool used in narrative-drafting a simple question — what is the meaning of the word “day” in law when counting delay in filing cases. The AI tool replied very confidently, saying the answer was in Section 3(35) of the General Clauses Act, 1897. The answer looked perfect but was totally false. That section defines “month,” not “day.” The definition The AI tool gave did not exist at all. It had created a fake law. If someone had used it in court, it would have been a serious professional mistake.
Fake Case Laws and Wrong Explanations
After being told it was wrong, the AI tool tried to justify its mistake by showing three court cases. But when checked, none of them had anything to do with the word “day.” They were real cases but used wrongly. This made the answer look believable but false. Later, The AI tool also gave short versions of Sections 9 and 10 of the Act, which changed their real meaning. These were half-truths — they looked right but were actually wrong. Such wrong or fake information can seriously harm a lawyer or accountant’s credibility.
Loss of Trust and the Need for Verification
After these mistakes, the user lost faith in the AI Tool. Such errors can destroy a professional’s reputation. Judges and officials expect accurate information. A single fake quote can make them doubt the entire document. After this, a new rule was made — The AI tool would work only in “Verification Mode.” In this mode, it must quote only verified legal texts or clearly say “Do Not Rely” if the source is uncertain.
Main Lessons Learned
This episode taught many lessons. AI can create fake sections, wrong case laws, and misleading meanings. It does not “know” the law; it only copies what sounds like law. Every legal or tax professional must check AI content with original sources like law books or government websites. Blindly using AI content can lead to serious professional damage and loss of trust.
Real-World Examples — Deloitte and ITAT Bangalore
In 2025, Deloitte had to refund part of a $440,000 fee to the Australian government because an AI-written report contained fake references and even a made-up court case. Similarly, in Buckeye Trust v. PCIT (Bangalore ITAT, 2024), an order was withdrawn after it was found to cite non-existent Supreme Court judgments. These incidents show that AI-generated content can look perfect but still be false. Without human checking, such errors can cause public embarrassment and legal trouble.
AI and Numbers — Not a Reliable Calculator
The AI tool often makes mistakes in numbers and financial work. It does not really calculate; it only guesses. Even simple tasks like compound interest or depreciation can be wrong. When the problem has many steps, the errors increase. Studies show AI is right about 80% of the time when giving short answers, but only about 20% when it shows its working. It also fills missing data with made-up numbers, which look correct but are false. Therefore, professionals must always verify every figure and formula themselves.
Truth and Responsibility
The biggest lesson is that AI can help but cannot be trusted blindly. It has no sense of truth or duty. In law, tax, and finance, accuracy is not optional — it is a must. Every word, number, and quote must be checked by a human. The professional using AI remains fully responsible for what is written or filed.
Growth or Loss of employment
In recent years, many people have started worrying that Artificial Intelligence (AI) will replace humans at work. News reports and experts often say that AI can perform tasks faster, more accurately, and at a lower cost than human workers. Machines can process large amounts of data in seconds, write reports, draw designs, and even analyse legal or financial documents. Because of this, there is a growing fear that AI might lead to widespread unemployment and make human effort less valuable.
But the truth is very different. AI cannot replace human intelligence. It can only assist it. Artificial Intelligence is created, trained, and guided by humans. It does not think, feel, or truly understand anything—it only follows patterns based on the information it has been fed. Every AI tool, no matter how advanced, depends completely on the quality of human input and supervision. Without human direction, AI cannot make moral decisions, use judgment, or adapt to complex real-world situations.
AI may be faster, but speed is not the same as wisdom. It can process information but cannot understand context, ethics, or emotions. For example, an AI tool might generate a report or a legal draft, but only a human can judge whether it is fair, ethical, or correct. AI can calculate, but it cannot care. It lacks empathy, responsibility, and accountability—qualities that define real intelligence.
The idea that AI will completely replace humans is therefore unrealistic. What is more likely is that AI will change how humans work. Routine, repetitive tasks may be done by machines, but decision-making, creativity, and interpretation will always require human involvement. In fact, as AI grows, the demand for skilled humans who can guide, verify, and manage AI will increase even more.
So, the real question is not “Will AI replace humans?” but “How can humans use AI wisely?” When used properly, AI can be a powerful tool — helping professionals work faster and more efficiently. But when used blindly, it can create mistakes, confusion, and even harm, as seen in many real-life cases.
Final Message — Humans Must Stay in Control
In the end, humans control AI — not the other way around. AI may mimic human intelligence, but it does not have human understanding. It has power without judgment, speed without conscience, and data without wisdom. The future will belong not to AI alone, but to those who learn how to use it responsibly, with honesty, skill, and human values guiding every step.
AI can copy knowledge but cannot replace human thinking. It has speed but no understanding. It can assist, but it cannot take responsibility. The future will belong to those who use AI wisely — with honesty, care, and verification. In the end, humans must always control AI, not the other way around. Truth and accuracy will always be stronger than technology.
II The Culprit Confessing the Crime
Full record discussion of an actual episode of AI- The AI tool generating Misleading Reports.
How the discussion started — the question about the definition of word “Day”
This document tells the full story of how a simple question about the meaning of the word “day” in law exposed the serious risks of using The AI tool without checking its answers. What started as a normal legal query turned into a discovery that The AI tool had invented a fake legal definition, quoted a wrong section of law, and even cited unrelated court cases to sound convincing. The response looked professional but was completely false. This showed that The AI tool can write confidently and use legal language well, but it does not actually “know” the law — it only predicts what sounds right. Such mistakes can easily damage a professional’s credibility if used in court or before tax authorities.
After the mistake was found, a strict rule called “Verification Mode” was created so that The AI tool would only quote verified laws or clearly say “DO NOT RELY” if unsure. This whole episode is a reminder that AI can help in writing but cannot replace human checking. In legal and professional work, accuracy and truth must always come from people, not machines.
1) Stage One – The Simple Query That Started It All
The user, a practising Chartered Accountant involved in appellate work, asked a routine but important legal question to the AI tool:
“What is the definition of the word ‘Day’ to be used in counting the delay in compliance limitation ?”
This question is crucial in tax and legal practice because limitation, delay condonation, and filing deadlines depend on how a “day” is legally understood. The expected answer should have come from authentic legal sources — either the General Clauses Act, 1897, or the Limitation Act, 1963.
Instead, The AI tool gave a confidently worded answer that looked completely authentic:
“Under Section 3(35) of the General Clauses Act, 1897, the term ‘day’ means a period of twenty-four hours beginning at midnight.”
At first glance, it appeared correct. The answer had a section number, a quotation mark, and formal language — exactly how lawyers present a citation. But the answer was false from start to finish.
2) Stage Two – The Discovery Of A Fabricated Statutory Definition
When the user checked the official text of the General Clauses Act, 1897 on the Government’s India Code website, Section 3(35) was found to define only the term “month,” not “day.” The supposed “definition” of day did not exist anywhere in the Act.
The section number, the quotation, and even the statutory reference were all fabricated.
The AI tool had essentially created a law that does not exist. This was not a typing mistake or an interpretation error — it was the invention of a non-existent provision.
The text looked professional and believable, which made it even more dangerous. If a professional had copied this into a submission, it would amount to quoting a false section of law — a grave professional error.
Such an error could destroy credibility before a tribunal or court. Once a judge notices a fabricated quotation, all other arguments, no matter how sound, become suspect. The counsel or representative loses trust instantly.
3) Stage Three – Fabricated Case Law And False Justification
When confronted with this discovery, The AI tool did not admit the mistake clearly. Instead, it tried to justify its earlier answer by producing what looked like supporting case law. It cited three judgments that, on the surface, appeared real and relevant. However, when verified, not one of them actually discussed the meaning of “day.”
The citations were real cases but used for false purposes. This kind of misdirection is far more dangerous than an outright falsehood, because it disguises error under the appearance of authority. Many professionals could be deceived by such confident presentation and use the citations without checking the original judgments.
This stage revealed the most critical problem: The AI tool can fabricate context even when using real case names. It can mix irrelevant cases, misstate what they held, and make it look convincing. In professional work, this is a disaster waiting to happen.
4) Stage Four – Half-Truths And Manipulated Paraphrasing
When the user again pointed out the errors, The AI tool tried to save face by “explaining” the meaning of the word “day” through other legal provisions, like Sections 9 and 10 of the General Clauses Act. But instead of quoting them word-for-word, it produced paraphrased versions that omitted critical phrases.
The result was half-true and half-false — sentences that sounded accurate but subtly changed the meaning of the law. This kind of incomplete paraphrasing is perhaps the most deceptive danger of AI drafting. It doesn’t give obviously fake information; it gives almost look-like correct information, which can easily pass unnoticed until it causes damage in court.
When confronted with this error, The AI tool tried to justify itself by citing court judgments, as if to support its earlier statement. It mentioned three cases:
The AI tool wrongly cited three judgments — Munnalal v. State of Uttar Pradesh, State of Gujarat v. Patel Raghav Natha, and S.S. Gadgil v. Lal & Co. — to support a false definition of the word “day.” On verification, none of these judgments discussed the term “day.” Two were entirely unrelated, and the third, though about limitation, contained no such definition. This showed how AI can create a false sense of legal authority by misusing real citations.
In Munnalal v. State of Uttar Pradesh, the case related to prosecution under the Prevention of Food Adulteration Act and discussed procedural and evidentiary issues, not the computation of time. By using this irrelevant case, The AI tool made a fabricated legal claim appear genuine, which could have seriously harmed a professional’s credibility if submitted in court.
In State of Gujarat v. Patel Raghav Natha, the Supreme Court dealt with revisional powers under the Bombay Land Revenue Code and the concept of “reasonable time,” not the definition of a “day.” The AI tool, however, wrongly suggested that the case defined “day” as twenty-four hours beginning at midnight — a statement that never appeared in the judgment.
In S.S. Gadgil v. Lal & Co., the issue was about limitation for reassessment under the 1922 Income-tax Act, where the Court held that a time-barred right could not be revived by amendment. The AI tool used this genuine case to support a completely unrelated and false argument about the meaning of “day,” creating a misleading and illogical link between unrelated legal principles.
For professionals, this shows why every case citation must be checked not only for authenticity but also for context. A citation may be real but still irrelevant to the issue being argued. Using such out-of-context references can make a submission appear careless or dishonest.
This revealed that The AI tool had used real case names out of context to give false credibility to a fabricated legal statement — a very dangerous practice that could easily mislead a professional relying on the output.
5) Stage Five – False Paraphrasing And Half-Truths
After the error was pointed out, The AI tool tried to reconcile its position through “interpretation.” It began paraphrasing Sections 9 and 10 of the General Clauses Act, claiming that these sections supported the idea that “a day” begins at midnight and ends at midnight.
It wrote:
“Section 9(1): When any act or proceeding is directed or allowed to be done within a specified period from a particular day or event, that day shall be excluded.”
But this wording is not found in the Act. The actual provision reads:
“In any Central Act or Regulation made after the commencement of this Act, it shall be sufficient, for the purpose of excluding the first in a series of days or any other period of time, to use the word ‘from’, and, for the purpose of including the last in a series of days or any other period of time, to use the word ‘to’.”
Similarly, The AI tool said:
“Section 10: When an act is required to be done on a certain day and the office is closed on that day, it shall be deemed valid if done on the next working day.”
The actual section reads:
“Where, by any Central Act or Regulation made after the commencement of this Act, any act or proceeding is directed or allowed to be done or taken in any Court or office on a certain day or within a prescribed period, then, if the Court or office is closed on that day or the last day of the prescribed period, the act or proceeding shall be considered as done or taken in due time if it is done or taken on the next day afterwards on which the Court or office is open.”
Hence, the paraphrases were incomplete and misleading. They omitted key phrases that are essential to the interpretation of these sections. If cited in court as-is, they would have amounted to misquotations of law.
This stage was perhaps the most alarming — the AI was now not only wrong but also attempting to “justify” its wrong answers by modifying real legal provisions to make them fit its previous mistake. Such reconstruction of law is unacceptable in professional work and could lead to grave consequences if filed in any legal proceeding. If such text were filed in a written submission, it would amount to misquoting the law. Judges do not excuse such errors.
6) Stage Six – The User’s Loss Of Trust And Question On Credibility
By this point, the user realized that The AI tool’s confident tone masked serious unreliability. The user expressed deep concern and frustration because such a mistake can ruin a practitioner’s standing permanently. Judges and tax authorities expect precision. A single false citation can make them question the entire submission. Once professional credibility is lost, it is almost impossible to regain.
7) Stage Seven – The AI tool’s Response And Creation Of “Verification Mode”
The AI tool thereafter fully admitted that its earlier content was unreliable and suggested a strict framework to prevent any repeat of such incidents. It proposed “Verification Mode” and “Strict Mode,” under which it would no longer generate interpretative content or paraphrased law. Instead, it would only quote directly from verified official sources, and if it could not find an exact source, it would state clearly: “Cannot Verify — Do Not Rely.”
8) Stage Eight – Key Risks And Lessons Learned
This episode exposed a chain of risks that every professional must remember:
1. AI can create false sections and definitions. These may look real but are completely made up.
2. It can misapply or misrepresent real case law. Genuine cases may be cited for things they never said.
3. It can paraphrase statutes incorrectly. By shortening or simplifying language, it changes the meaning.
4. It tries to justify mistakes by reasoning, not by fact. This “reconciliation” makes the error sound logical, even when it is wrong.
5. It can destroy professional credibility. Courts treat such submissions with suspicion and may reject them outright.
The pattern is clear: The AI tool does not “know” law; it merely predicts text that sounds like law. It mimics legal tone but lacks legal truth.
9) Stage Nine– Verified Legal Position After Manual Checking
Once all information was checked manually from official sources, the correct position was clear. The supposed definition of “day” did not exist anywhere. The computation of time must be understood from the actual words of each statute, read with the General Clauses Act. The important lesson is not about this particular term but about the method. Only by returning to the original text — the actual law book — can correctness be ensured. No AI system can replace that human verification.
10) Stage Ten – Rules For Safe Use Of AI In Professional Work
From this experience, a strict and permanent code of conduct for using AI in professional legal or tax work was established. These are now non-negotiable rules:
1. AI is a helper, not an authority. Never rely on it for definitions, case laws, or citations without checking.
2. Always verify with original sources. Go to India Code, official court websites, or certified law reports.
3. Do not quote paraphrased text. Use only the exact words of the Act, rule, or judgment.
4. If something cannot be confirmed, label it “Do Not Rely.”
5. Avoid stylistic AI language. Judges can easily recognize it by its artificial tone.
6. Maintain personal responsibility. The professional signing the submission must be the final verifier.
7. Never allow AI to “explain” law without proof. AI can guess; professionals cannot afford to guess.
Professionals must understand that The AI tool is not a legal database. It is a language prediction system trained on patterns, not on verified legal texts. It does not understand truth, context, or hierarchy of authority.
11) Stage Eleven – A Warning To All Professional Users
This episode must be treated as a serious cautionary case for all professionals who use AI tools for legal or regulatory drafting. The risks are not minor — they are professional and reputational.
- A single false quotation can make an entire document worthless.
- A fabricated citation can make the counsel appear dishonest before a judge.
- A misinterpreted section can lead to an adverse order against a client.
Once credibility is lost, no apology can restore it. Courts and departments remember mistakes, especially when they are due to carelessness.
AI-generated content may appear polished, fluent, and well-organized — but this smoothness is deceptive. It creates the illusion of accuracy. Professionals must remember that confidence of tone is not proof of truth.
Practical Case Study
1) Deloitte $440,000 AI Fiasco
Deloitte agreed to give back part of the $440,000 it was paid by the Australian government after a report it wrote using AI was found to have serious mistakes, including fake references and a made-up court case. The report, made in July 2025 for the Department of Employment and Workplace Relations, had to be corrected and reissued after the errors were found. Deloitte admitted it used an AI tool (Azure OpenAI GPT-4o) to help write the report but said people later reviewed it. The case caused public criticism and raised concerns about how much professionals depend on AI without checking its work. Experts said AI can sound confident but often makes up false details, and this shows that human checking is always necessary. Without proper verification, AI-written reports can cause embarrassment, loss of trust, and damage to reputation.
2) ITAT Bangalore Case:
Buckeye Trust v. Principal Commissioner of Income Tax‑1, Bangalore (ITA No. 1051/Bang/2024) In the case of Buckeye Trust v. Principal Commissioner of Income Tax‑1, Bangalore (ITA No. 1051/Bang/2024), the ITAT passed an order in December 2024 in a matter relating to taxability of a trust settlement involving a transfer of interest in a partnership firm. The bench ruled in favour of the Department and cited several judgments—three purported Supreme Court judgments and one Madras High Court judgment—as supporting authority. These judgments were cited to show that a partnership interest is akin to a “share” and therefore taxable.
On scrutiny, it was discovered that at least two of the cited judgments do not exist, one citation corresponds to a different case altogether, and the fourth was real but had no relevance to the issue at hand.
Non-Existent or Mis-Cited Judgments
According to media coverage the case laws cited and which are in question are as under
5. K. Rukmani Ammal v. K. Balakrishnan (1973) 91 ITR 631 (Madras High Court) — which does not appear in Madras High Court archives.
6. S. Gurunarayana v. S. Narasinhulu (2004) 7 SCC 472 (Supreme Court) — which appears to be fictitious.
7. Sudhir Gopi v. Usha Gopi (2018) 14 SCC 452 (Supreme Court) — which when traced is not relevant to the issue as cited.
8. CIT v Raman Chettiar (57 ITR 232 (SC) — which is a real case but unrelated to the issue of partnership interest as share.
Because these citations either did not exist or were irrelevant, within a very short time the ITAT withdrew the order (via notice citing “inadvertent errors”) and arranged for a fresh hearing due to these flaws.
Final Stage – The New Professional Standard: Verification Mode Only
After this incident, it should be ensured that all legal or tax drafting done with any AI assistance must be performed only under Verification Mode. This means:
- Every sentence must be based on an official, primary source.
- The exact text must be quoted.
- If the source cannot be verified, the statement must be excluded.
- AI’s role ends at formatting or summarizing verified text, not at creating or interpreting law.
This is not only a technical safeguard; it is an ethical rule. A professional’s strength lies not in speed or eloquence, but in accuracy and truth.
The lesson is simple: AI can imitate knowledge, but it cannot bear responsibility. The human professional must always remain the final and only authority.
III- Limitations of The AI tool in Numerical and Professional Numerical Computations
The AI tool is good at explaining things in words but not at doing numbers or calculations. It does not actually work like a calculator; instead, it guesses what number or word should come next. Because of this, it can easily make mistakes in financial or mathematical work such as tax calculations, balance sheets, or valuations. When data is missing, it tries to fill in the gaps by guessing, which may look right but can be false. Even simple calculations like compound interest or depreciation can go wrong, and the risk increases with complex problems. Therefore, professionals should never depend on The AI tool’s numbers without checking. All figures, formulas, and results must be verified manually or through trusted software. The AI tool should only be used for writing, drafting, or explaining—not for doing final calculations. In professions like tax, accounting, audit, and law, accuracy is essential, and even small mistakes can lead to wrong results, client losses, or damage to reputation. The key lesson is simple: always verify every number yourself, double-check all formulas, and remember that the final responsibility for accuracy lies with the human professional, not the AI.
Limitations of The AI tool in Numerical and Professional Numerical Computations
| Sr. No. | Observation / Lesson | Description / Key Point |
| 1 | The AI tool often makes mistakes in complex or multi-step numerical problems | The model struggles with multi-layered calculations that require sequential reasoning or logic. |
| 2 | Accuracy decreases with the number of operations | The more additions, subtractions, or logical steps involved, the higher the error rate. |
| 3 | Sharp fall in accuracy when showing work | A study found 84% accuracy for direct answers but only 20% when The AI tool had to explain its reasoning. |
| 4 | The AI tool “fills in gaps” with fabricated logic | When data is incomplete or unclear, the model invents details or uses false assumptions (“hallucinations”). |
| 5 | Even simple financial calculations can be wrong | Tasks like compound interest or depreciation may produce incorrect results if prompts are ambiguous. |
| 6 | The AI tool is a language model, not a calculator | It predicts words based on patterns, not by performing actual arithmetic or logic-based computation. |
| 7 | Incorrect results appear well-written and convincing | The confident tone and polished style of wrong answers make detection of errors difficult. |
| — | Lessons for Professional Use | Guidelines for safe application in tax, audit, and legal work |
| 1 | Manual verification is mandatory | AI-generated numbers must never be accepted without independent checking. |
| 2 | Human cross-checking required | Every figure, formula, and assumption must be validated by a qualified professional. |
| 3 | Using AI results blindly creates serious risks | Failure to verify can cause factual errors and reputational harm. |
| 4 | Use The AI tool only as an assistive tool | It can support drafting but must not be treated as a computational authority. |
| 5 | Human accountability remains final | Responsibility for accuracy always lies with the professional user, not the AI. |
Some documented instances where The AI tool attempted numerical or finance-/calculation-heavy tasks and produced errors. These serve as cautionary case-studies for professional use.
| S N. | Case / Source | Description of Error |
| 1 | When The AI tool Gets Compound Interest Wrong: A Case Study in Financial AI Miscalculations (Medium) | Issue:- compound-interest problem (monthly investment over a period) and the model gave an incorrect final value, wrong assumptions (about compounding, timing), showing that even “simple” financial formulas can be mis-handled. |
| 2 | LLMs Can Be Trusted for Financial Advice? (Mind Matters) | Compare two car-loan options. All three models made arithmetic / reasoning mistakes (including simple interest vs amortised interest confusion), and recommended the wrong option due to flawed calculation logic. |
| 3 | The AI tool for Calculations: Practical Uses, Limitations, and Tips for Reliable Results | Basic vs More Complex calculations. While simple arithmetic worked, once formulas, multiple steps, or code generation were involved. Accuracy dropped. It highlights that for multi-step finance tasks, The AI tool’s outputs are not reliable without checking. |
| 4 | The AI tool: Use Cases and Limits to Its Reliability (FM Magazine) | Calculating Terminal value / discounted cash flow calculation where the formula was correct, but The AI tool miscalculated the numeric value of the discounted period result. Shows that even when “logic” is right, numeric execution may be wrong. |
| 5 | An Independent Evaluation of The AI tool on Mathematical Word Problems (MWP) | In math word problems The AI tool’s success drops significantly when the number of operations or unknowns increases. It provides empirical evidence of the model’s limitation in complex numeric reasoning. |
Professional Use
- These errors are not isolated to obscure prompts; they occur in real-world financial and numerical contexts.
- The AI tool may produce wrong numbers, incorrect formulas, or mis-interpretation of financial problems, especially when multiple steps or inference are required.
- Using its output without verification can lead to flawed professional work (tax calculations, audit schedules, financial models).
- The above cases show that human cross-check of every numeric result, formula, assumption and step remains essential.
- A best practice: treat The AI tool only as a draft-tool or brainstorming aid—not as final computational authority.
Here are some more documented examples of where The AI tool (or comparable large language models) made calculation or numeric-reasoning errors in professional or quasi-professional contexts, which you can use as cautionary case-studies:
Example 1: Compound Interest Mis-calculation
A blog titled “When The AI tool Gets Compound Interest Wrong: A Case Study in Financial AI Miscalculations” describes how The AI tool was asked to compute the future value of a monthly investment and produced a wrong final number and incorrect assumptions about compounding frequency and timing.
Lesson Even what looks like a standard financial formula can go wrong when the AI mis-interprets or omits assumptions.
Example 2: AI “Memorization”, Not True Numeric Reasoning
An article explains findings that The AI tool tends to rely on memorized patterns rather than performing true arithmetic or logical numeric steps. For example, accuracy falls when adding larger numbers or performing many operations.
Lesson: In complex accounting/tax computations, errors may occur not because of a single slip, but because the model never properly reasoned through the numbers.
Example 3: Inability to Handle Real-World Accounting Model Errors
A Reddit thread by accounting professionals reports that when balance sheets were uploaded and the model was asked to forecast or analyse, the outputs often didn’t detect fundamental issues (like assets ≠ liabilities + equity). The model produced polished but inconsistent numbers.
Lesson: AI can produce elaborated output but still fail basic validation or reconciliation checks that a human accountant would immediately notice.
Example 4: Misleading Academic Coverage of Math Errors
Studies have documented that The AI tool’s performance on mathematical word problems dramatically decreases as the number of operations or unknowns increases. For example, one study found success rates dropped dramatically when the model had to “show its work.”
Lesson: In tax/audit engagements where multi-step arithmetic (depreciation, accruals, multi-year projections) is needed, the risk of error is higher than in simpler tasks.
Example 5: Explanatory Article on Why The AI tool Struggles with Math
“Why The AI tool Struggles with Math — And Why That Matters”
While computers excel at math, LLMs like The AI tool are not optimized for rigorous arithmetic & logical sequence calculations, hence errors emerge.
Therefore Professionals using AI tools for numeric work must recognise that the model’s architecture is not built like a dedicated calculator or spreadsheet engine—it is a language model and prone to make mistakes depending upon the complexity of the problem.
Truth Over Technology
This problem with numbers is similar to past cases where The AI tool produced false legal references. It shows one clear fact—AI cannot replace human judgment, honesty, and checking. The AI tool does not “know” the truth; it only produces what sounds right. Professionals must, therefore, use it carefully and always confirm everything from original sources. In law, taxation, and finance, accuracy is not just a rule—it is a duty. Technology can help speed up work, but truth and correctness remain the only real defence of a professional’s credibility.
This incident highlights the risk of relying on unverified sources or tools (such as generative AI models like The AI tool) for legal research. The orders themselves suggested the case-law references were generated, likely by an AI tool, and not cross-checked for authenticity.
In the context of legal or tax appellate work, the use of fictitious or mis-referenced judgments can undermine the credibility of an entire ruling, expose the bench and parties to embarrassment, and raise serious questions about due diligence and procedural integrity.
Final Words: The Ethical Duty Of Truth
Safeguards against Dangers of Using Artificial Intelligence (AI) Like The AI tool
AI Can Create False Information
AI sometimes makes up laws, sections, or definitions that do not exist. It can write them in a very convincing way, so they look real — but they are false. Using such fake information in professional work can cause serious problems.
1. AI Can Quote Wrong Cases
AI may use real case names but wrongly connect them to things they never said. This makes the answer look correct even when it is totally wrong. Such fake or irrelevant citations can damage your credibility before a judge or officer.
2. AI Changes the Meaning of Law
Instead of quoting the exact words of the law, AI often “summarizes” or “paraphrases” them. In this process, it leaves out important words and changes the meaning. This is dangerous because even a small change in legal wording can completely alter the interpretation.
3. AI Sounds Confident Even When Wrong
AI writes in a smooth and confident tone, which makes people believe it is correct. But confidence is not the same as truth. Professionals may be misled because the text “looks” perfect.
4. AI Cannot Think or Judge
AI does not understand right or wrong. It only predicts what words should come next based on patterns. It cannot apply logic, judgment, or ethics like a human professional.
5. AI Cannot Be Trusted With Numbers
AI often gives wrong answers in financial or numerical work. It guesses numbers instead of calculating them. Even simple tasks like interest or depreciation can come out wrong. In complex problems, the mistakes multiply.
6. AI Fills Gaps With Made-Up Details
When AI doesn’t know something, it doesn’t stay silent — it invents details. These “guesses” look true but are actually false and can mislead professionals into using wrong information.
7. AI Can Destroy Professional Credibility
If a professional uses fake sections, false cases, or wrong numbers from AI, judges and clients will lose trust. Once credibility is gone, it is almost impossible to get it back.
8. AI Cannot Take Responsibility
If an AI-generated draft is wrong, the professional using it is still responsible. The AI cannot be blamed. Only the human user will face the consequences.
9. AI Should Only Be Used as a Helper
AI can help in writing or drafting, but not in giving final answers. The professional must always verify every word, number, and law from official sources before using it.
10. AI Cannot Replace Human Intelligence
AI can copy language and format nicely, but it has no understanding, ethics, or responsibility. Only humans can decide what is true, fair, or lawful.
11. Simple Final Message:
AI can help you write faster, but it can also mislead you. It can make fake laws, wrong numbers, and false cases look real. Never trust it blindly. Always verify every detail yourself. Truth and accuracy must always come from humans, not from machines.
Disclaimer
1. This study has been prepared with the assistance of AI tools operating in Verification Mode. Every effort has been made to keep the information, quotations, and references accurate and based on verified sources. However, because Artificial Intelligence systems have natural limitations such as possible errors in processing, interpretation, or understanding context, some mistakes or missing details may still be present. Readers are therefore advised to check and confirm all legal, factual, and numerical details from original and reliable sources before using any part of this document for professional or official purposes.
2. This study is made purely for educational and awareness purposes. It is not meant for any professional use, legal reliance, or to be produced as evidence in any court of law, including in any case or proceeding that may involve the Artificial Intelligence system itself. Anyone using this material does so entirely at their own risk. The author accept no responsibility or liability for any loss, error, or consequence arising from the use or misuse of this document in any way, whether authorised or unauthorised.
3. The author does not intend to unduly criticize or defame Artificial Intelligence through this study. Rather this study has been necessitated by two real life instances in which wrong information was generated by the AI Tool. The purpose is only to make readers aware of the risks and mistakes that can occur if AI-generated information is used blindly or without human checking and proper verification. The aim of this work is to encourage responsible, careful, and verified use of AI tools in professional and academic fields.
4. It is further clarified that the observations, findings, and examples discussed in this study are themselves derived from and confirmed by the AI Tools used in its preparation, and therefore represent the AI system’s own admissions and acknowledgments of its limitations and errors.
5. Copying, sharing, or reproducing this document in any form without prior permission is strictly prohibited.
6. This document demonstrates the use of AI under human control — not a substitute for human verification. It should be treated as an educational and illustrative record, not as a certified or legally authoritative text.
(Republished with amendments)
Author Bio
