Written by Claude 3 dukes it out with the latest from Open AI GPT-40 in Internal Control and Audits
Are the Numbers Right? Will the Auditors Frown?
It’s a big deal to have confidence in the numbers and to make sure that internal controls are tight to show auditors that you know what you are doing. We are after all safeguarding the numbers and want to ensure that stakeholders do not doubt our financial accuracy.
We often use Antropic’s Claude and ChatGPT to ask questions but which one is better when it comes to specific questions related to internal controls and audits? Now both are getting better and Claude 3 has improved its reading of images and PDFs while ChatGPT 40 is getting smarter and faster.
We put both to the test in our real world of finance and accounting to determine who is better. In our prior A.I. battle, ChatGPT beat Claude(but it was close) and Google’s Gemini…well it was the loser. Now let’s find out who reigns supreme when it comes to specific nitty-gritty details of accounting.
Problems We See in Internal Controls & Audit
Getting flagged for internal controls during an audit is not fun. For rapidly growing startups which now need to add the next round of equity or even debt financing, investors and banks will want an audit.
G&A and the finance department is usually understaffed in the early stages while systems are not totally deployed so we often see common issues such as:
- Manual processes which lead to inefficiencies and errors.
- Difficulty in identifying emerging fraud risks and control deficiencies.
- Limited scalability to handle increasing volumes of data and transactions.
- Inability to adapt quickly to changing regulatory requirements and technological advancements.
- Lack of real-time insights and proactive risk management capabilities.
AI Can Help But Who Gets the Better Grade?
Some ways in which automation and AI can assist in internal control and audit include:
- Automated data analysis to identify anomalies and potential fraud risks.
- Natural Language Processing (NLP) capabilities to understand and analyze unstructured data, such as emails and documents.
- Predictive analytics to forecast future risks and optimize resource allocation.
- Machine learning algorithms for continuous monitoring and adaptive control testing.
- Prompts for generating recommendations and insights based on specific audit objectives and requirements to help you move into the right direction
Duking it out: ChatGPT vs Antropic’s Claude for Internal Control & Audit Efficiency
Round 1: Role of Technology in Audit Efficiency
The outline for a typical audit is shown below and we’ve fed this diagram into ChatGPT and Claude to see who can make better recommendations when navigating through an audit. One note, while the initial responses were very detailed, we asked ChatGPT and Claude to summarize the responses for better readability.
Round 1: Improving the Audit Process.
Our question:
“How can I leverage technology to improve the audit process? I have external auditors that I work with and I want to ensure the process is smooth and doesn’t take too much time or expense. What can I do from a technology point of view to improve the audit process for my software company? “
ChatGPT’s Response:
Claude’s Response:
The Difference Between the Two
Well it seems ChatGPT in finance had a broader response and said to use AI, Blockchain, and Robotic Process Automation (RPA) for advanced data analysis and automation.
It seems it was more focused on efficiency, accuracy, and tools like cloud platforms for real-time collaboration, which can make communications a little smoother with external auditors.
Meanwhile Claude did mention specific audit management systems and said to use the block chain as well but we would give the edge to ChatGPT in this round.
Winner of Round 1: ChatGPT
Round 2
Question: “What are some typical areas of concern from an auditor’s point of view? We do not have inventories because we are a software company. We do not capitalize R&D but we do have complicated revenue contracts with customers as we have multi-year subscription deals.”
ChatGPT’s Response
Claude’s Response:
The Difference Between the Two
The responses were long so we went back and asked them both to condense the response. Both said revenue and deferred revenues as the number one priority but then it changed a little from there. ChatGPT is very specific with liabilities and assets, and intangible assets while Claude was a little more nuanced and mentioned related party transactions and management estimates and judgements.
The numbers are the numbers but the reasoning and judgment and rationale behind the numbers are important and tie in with the tone at the top. So for this, we give the edge to Claude.
Winner of Round 2: Claude
Round 3
Evaluating Risk Management Practices specific to quote to cash: “Discuss how a company can evaluate its risk management practices within its internal control system. What are the key indicators of an effective risk management strategy?” I have included a diagram of how we collect cash for example, tell me what I can do to strengthen internal controls without hampering efficiency. “
ChatGPT’s Response:
Claude’s Response:
The Difference Between the Two
Pretty similar here with both highlighting internal controls for the cash collection process. ChatGPT has a more structured approach in its five key areas and highlights the importance of fostering a risk-aware culture and a supportive governance structure.
There are some qualitative differences as well as ChatGPT said risk management as to be adaptable. But what does that mean? What do I do with this?
Anyway, Claude mentioned that its important to have the set tone at the top and the critical role of leadership for setting this tone and getting risk management right.
We would say that ChatGPT was a more strategic and had a more future proofed approach as it relates to risk managed so we would give it the edge.
Winner of Round 3: ChatGPT
Round 4
Improving Audit Efficiency through Data Analytics: “Discuss how data analytics can be used to improve the efficiency and effectiveness of audit processes. What types of data analytics tools are most useful for auditors? Please recommend a tool that I can use to graphically represent areas of improvement.”
Chatgpt’s Response:
Claude’s Response:
The Difference Between the Two
This one is a little weird as both agree that data analytics is helpful for audit efficiency and effectiveness. Both highlight key use cases like risk assessment, testing controls, and substantive procedures. And both said use Tableau which isn’t really used for auditing visualization, I guess you can use Tableau but its just easier to use flow diagrams instead.
If we had to choose, ChatGPT provides a clearer roadmap for integrating data analytics throughout various audit phases (risk assessment, testing, reporting, etc.). It did mention the value of continuous auditing tools for ongoing monitoring and ensuring recommendation implementation.
Meanwhile, The benefit of using Claude is that it offers a wider range of specific use cases for data analytics, including benchmarking and trend analysis.
Also what I liked is that it mentioned using other visuals such as risk heatmaps, a control-effective dashboard, and process flow diagrams (which is super helpful to auditors). Claude’s response seems to make more sense and fit a wider range of use cases.
Winner Round 4: Claude
Round 5
Internal Control Deficiencies and Remediation: “What steps should an organization take upon identifying a significant deficiency or material weakness in its internal control system? Discuss the process from detection to remediation.”
ChatGPT’s Response
Claude’s Response:
The Difference Between the Two
ChatGPT recommended identifying and addressing potential problems early on by employing a wide array of detection techniques. Documentation is important and it also spoke of the remediation process.
On the other hand, Claude went further, ensuring clear and effective communication throughout all phases, which helps keep all stakeholders on the same page. Claude also prioritizes understanding the potential ramifications of identified issues, which aids in focusing efforts where they are most needed. And Claude mentioned the necessity of thorough testing and validation of any corrective measures to ensure their effectiveness.
Winner of Round 5: Claude
Round 6
This is where we asked about different industries. “What are the typical recommendations that auditors make to i) a large manufacturing company, ii) a non-profit organization, and iii) an enterprise software company. I would like to know so that I can understand what my auditors will recommend in their final report. “
ChatGPT’s Response:
Claude’s Response:
The Difference Between the Two
Both gave a clear response for each vertical. ChatGPT mentioned that manufacturers should tighten up on inventory and cost control. For non-profits, ChatGPT stressed the need to be extra careful with donor funds and grant management, along with having strong governance (Elon Musk’s foundation). For software companies, data protection, revenue recognition, and intellectual property were key areas auditors might focus on.
Claude sounded more seasoned going deeper into the “why” behind each recommendation. Claude suggested specific measures manufacturers could take, like regular inventory checks and better tracking systems. Claude also emphasized the importance of having clear divisions of responsibility within the company to prevent fraud. For non-profits, Claude highlighted the need for robust systems to track donations and grants, along with proper volunteer management. Data security and strong software development processes were crucial, yes makes sense
Winner Round 6: Claude
Round 7: Fraud Detection
“Some companies can get flagged by auditors on internal controls. What can I do considering the diagram attached detailing our process, to prevent and detect fraud?”
ChatGPT’s Response:
Claude’s Response:
Claude sometimes has a hard time with attachments. We could not upload a pdf so we tried adding it as text.
The Difference Between the Two
ChatGPT’s response was broad not just SaaS companies, and mentioned core principles like segregation of duties and access controls, pretty obvious stuff. It also highlights the importance of a well-informed workforce as a first line of defence.
But it was pretty generic response and didnt really give us granular specific details for our SaaS company.
Claude was more SaaS-focused: Offers targeted recommendations relevant to SaaS companies. But its detailed response to document management and user activity monitoring is not really SaaS-centric. It also mentioned the importance of continuous monitoring and adapting controls based on identified risks.
Sometimes the wording is very specific, both had trouble reading the pdf attachment and both ChatGPT and Claude it seems needed more specific prompting to get the granular details specific to a particular industry.
Winner of Round 7: Tie
The Grand Champion is…
Both AI tools and technologies are getting better and even Google’s Gemini is catching up. We fed ChatGPT and Claude a lot of data in our contest and both are pretty good with answers and guidance in enhancing audit and internal control processes.
ChatGPT excels in providing broad, efficiency-focused recommendations across various audit stages, Claude proves superior in offering detailed, context-specific insights, particularly in specialized audit scenarios. Specific details in the prompts will give you specify answers.
For us, ChatGPT gave long detailed yet general application responses while Claude excelled in depth and detail for finance and accounting.
The Winner: Claude 3
