The true capabilities of AI coding tools that cannot be measured by benchmarks—what criteria should you use to choose one in 2026?

SolanaLink Tech, August 1, 2026

Image: (Hero image placed in https://static.solanalink.jp/blog-and-news/ai-coding-tools-2026-<timestamp>.png)

18-minute read

---

This article organizes AI coding tools for 2026 from four perspectives: layer classification, benchmark interpretation, empirical research, and Japan-specific compliance issues. After demonstrating the saturation of SWE-bench and the limitations of vendor-reported values, it proposes a phased implementation strategy and data governance configuration that contract development companies should adopt, based on the empirical results of METR and DORA.

Publication Date: August 2026

Models and tools in this field are updated every few weeks. Product information and figures in this article are current as of the publication date, and technical judgments will be reviewed periodically.

---

Introduction

The use of AI in software development has entered a qualitatively different stage over the past two years, moving from "code completion assistance" to "autonomous task execution."

Meanwhile, information regarding tool selection is confusing. Benchmark scores published by vendors are often misleading, and version numbers of models that become obsolete in weeks continue to appear in comparison articles. Furthermore, the real issues faced by Japanese companies—compliance with the Personal Information Protection Act, handling of customer code in contract development, and data residency—are largely undiscussed.

SolanaLink, based in Tokyo, handles both contract development and in-house product development, and we routinely use these tools in our daily operations. This article will summarize the current state of each tool as of August 2026, and then share our perspective on how to interpret benchmarks, what empirical studies indicate, and what realistic strategies Japanese companies should adopt.

--

First, let's consider them by "layer"

Why a side-by-side comparison table doesn't work

A common mistake in many comparison articles is placing products with different characteristics on the same plane.

Comparing Cursor and Claude Code in the same table is equivalent to asking "which is better," an integrated development environment or a command-line tool, and has no practical meaning. In reality, using both together is the standard use case in 2026.

SolanaLink evaluates AI coding tools by classifying them into the following four layers:

Layer 1: Editor-integrated (completion/inline editing)

Integrated into existing editors, these handle completion during typing and local editing. GitHub Copilot and JetBrains AI Assistant fall into this category. They have the lowest barrier to entry and the largest number of deployments in enterprises.

Layer 2: Agent-based IDE

The editor itself is redesigned as an execution environment for an AI agent. Cursor, Google Antigravity, ByteDance TRAE, etc. Maintain a semantic index across the entire codebase and consistently handle simultaneous editing of multiple files, terminal execution, and browser operations.

Layer 3: CLI/Terminal-Resident Agents

These agents reside directly in the shell environment and are integrated into existing development workflows in a non-intrusive manner. Examples include Claude Code, OpenAI Codex CLI, and open-source implementations such as Cline, Aider, OpenHands, Continue, and Goose.

This layer can be self-hosted, which is extremely important from a compliance perspective, as discussed later.

Layer 4: Cloud/Asynchronous Agents

These agents execute tasks in an isolated environment and return results without waiting for human intervention. Examples include Devin (Cognition), GitHub Copilot coding agent, and OpenAI Codex's cloud sandbox. Assigning an issue results in a pull request being returned.

Our View: Layers 1-4 are not alternative but complementary. The question to consider when selecting a solution is not "which one to choose," but rather "which tasks should be assigned to which layer, and to what extent?" Introducing a high-performance tool without first designing this will only increase the review burden.

--

Current Status of Major Players (as of August 2026)

GitHub Copilot – The Largest Force in terms of Implementation

Often overlooked in comparison articles, it still stands out in terms of enterprise adoption. According to Microsoft's earnings announcement, as of July 2025, it had 20 million cumulative users and was adopted by 90% of Fortune 100 companies. As of January 2026, the number of paid seats was reported to be approximately 4.7 million, a 75% increase year-on-year.

From 2026, a mechanism allowing third-party agents to run within Copilot was introduced, further strengthening its role as the "largest distribution channel." Its editor independence, well-established IP compensation clauses, and ease of integration into corporate IT procurement processes are key factors in its adoption by large Japanese companies.

Cursor (Anysphere) – The Product Most Highly Valued by Capital Markets

This product has set the de facto standard for agent-based IDEs. ARR (Average Revenue) has grown rapidly from $100 million in January 2025, achieving a post-money valuation of $29.3 billion in November 2025 in a Series D round led by Accel and Coatue.

A notable technological point is the introduction of its proprietary model, Composer. This aims to reduce reliance on higher-level models and compress inference costs, and is positioned as the company's answer to the "profitability problem of wrapper-type products" discussed later.

On the other hand, the change in billing model in June 2025 (from request-based to token-based) provoked significant backlash. Unexpected charges occurred, leading to a formal apology from the CEO in July of the same year and the offering of refunds for a portion of the period. This incident should be remembered as a risk case study for enterprise adoption of pay-as-you-go tools.

Google Antigravity—Agent-First Design

A development platform launched in November 2025. It employs a two-tiered structure, separating the "manager screen" which runs multiple agents asynchronously in parallel from the traditional editing screen. A key feature is that agents operate a built-in browser to perform UI tests, leaving a traceable artifact of the execution process.

It has been reported that the core development involves a team brought in by Google from Windsurf (Codeium) in July 2025. As of the time of writing, it is still in the free public preview stage, but the free tier has been reduced multiple times, and the pricing structure for the official version has not yet been announced.

Claude Code (Anthropic) – A leading example of terminal-resident development

It resides in the shell and excels at long-term autonomous execution, large-scale refactoring, and tracking complex dependencies. The Hooks mechanism and MCP (Model Context Protocol) facilitate easy integration with existing CI/CD and internal tools.

While the lack of a GUI makes it difficult for beginners, its ability to introduce without disrupting existing development flows is a reason why it is chosen by experienced users.

OpenAI Codex – Two Forms: CLI and Cloud Sandbox

It does not offer a standalone IDE, but deploys as a CLI, cloud execution environment, and various IDE plugins. It excels at automated build, test, and fix execution in isolated environments, making it suitable for large-scale batch change and test repair workloads.

JetBrains (AI Assistant / Junie / Air) – Agent with Type Information

This is the most underrated option for organizations primarily using statically typed languages such as Rust, Java, and Kotlin.

Because the agent can directly access the semantic index, type information, and call graph held by the IDE, it tends to be more accurate in analyzing changes in large codebases than other tools that rely on text-based inference.** It is available across the entire product line, including RustRover, and supports local model (Ollama, etc.) connections and file exclusion via .aiignore.

A multi-agent environment called "Air," based on the open ACP (Agent Client Protocol) developed in collaboration with Zed, has also been released.

Devin / Cognition – Enterprise Adoption of Autonomous Agents

Cognition acquired Windsurf's products, brand, and team in July 2025, integrating their agent product suite. Japan is positioned as the company's second-largest market after the US, and a strategic briefing for the Japanese market was held in June 2026.

ByteDance TRAE and Chinese Companies

TRAE rapidly expanded its user base due to its generous free tier and SOLO mode (a function that allows end-to-end construction from natural language and blueprints). However, it transitioned to a token-based pay-per-use model in February 2026.

In China, Alibaba's Qoder (formerly Tongyi Lingma, renamed in May 2026), Tencent's CodeBuddy, and Baidu's Comate are widely deployed.

Our View: The technical level of these products is by no means low. However, due to the data governance issues discussed later, we have determined that they are unsuitable for handling customer code in contract development.

Open Source/Self-Hosted Systems – The Most Overlooked Option

Open source CLI agents such as Cline, Roo Code, Aider, OpenHands, Continue, and Goose can connect to any model backend. Combining this with open-weight models (GLM, DeepSeek, Qwen, Kimi, etc.) allows for a configuration that does not send any code externally.

While rarely mentioned in comparison articles, in highly confidential contract projects, this may be the only practical solution. **

Summary of Major Tools

Regarding Model Infrastructure—Do Not Use Version Numbers as a Selection Criteria

The performance of a tool depends on the model running behind it. As of August 2026, Anthropic's Claude series, OpenAI's GPT-5 coding-focused version, and Google's Gemini 3 series are the main options. In addition, open weight models, mainly originating from China, are rapidly improving performance, significantly increasing the feasibility of self-hosted configurations.

Our View: We do not recommend using specific version numbers as the basis for comparison. Models in this field are updated every few weeks, and by the time an article is published, it's already outdated. The key is to maintain an architecture that isn't tied to a specific model—separating the prompt layer and model layer and keeping them easily replaceable.

---

Models in this field are updated every few weeks, and by the time an article is published, it's already obsolete.**

How to Read Benchmarks

This is the most important point of this article.

SWE-bench Verified is Saturated and Effectively Obsolete

SWE-bench Verified, long referenced as an industry standard, has lost its discriminatory power as the scores of top models have all clustered around 88%.

Furthermore, it has been reported that OpenAI withdrew its use of the benchmark in February 2026. The company's audit revealed that a significant percentage of the unresolved problems were flawed in the test cases themselves.

Discrepancy Between Self-Reported Values and Third-Party Measurements

Currently, the SWE-bench Pro from Scale AI's SEAL leaderboard is considered more reliable. Because it is measured on a unified scaffold (execution platform), direct comparisons between models are possible.

Here's an important fact:

Even with the same model, the score drops by an average of 20 points or more when switching from SWE-bench Verified to SWE-bench Pro. Furthermore, there is a systematic difference between values measured using a vendor's self-optimized scaffold and values obtained using a third-party standard framework.

In other words, there is no correlation between announcing "SWE-bench 90% achieved" and the understanding that "90% of tasks can be completed in practice."

Scaffold as a Confounding Factor

Often overlooked, agent performance is determined not by the model alone, but by the combination of the "model + scaffold." Even with the same model, the score can vary significantly depending on the design of the execution scaffold. Vendor-announced figures are almost always measurements taken using a scaffold optimized for their own company.

This is structurally the same problem as the "self-referential trap" in TTS evaluation discussed in the previous article. When the evaluator and the evaluated have the same interests, the figures cannot be considered independent evidence. **

Practical Interpretation

At SolanaLink, we interpret published figures in the following order of priority:

1. Measurements from third-party standardization frameworks (e.g., Scale SEAL) – Most reliable
2. Independent measurement and audit reports – Useful for reference
3. Vendor self-reported values – Treat as an upper limit and do not use for predicting practical performance

Our View: Benchmark scores are not a deciding factor in tool selection. The key is whether it can actually produce useful changes within your own codebase, language, and review system. This can only be verified through a short-term pilot deployment.**

---

Is Productivity Really Improving?

While the claim that "AI has increased development efficiency several times over" is widely circulated, rigorous empirical research presents a more cautious picture.

METR's Randomized Controlled Trial

A study published by METR in July 2025 is one of the most cited empirical results in this field.

In a randomized controlled trial conducted on 246 tasks involving 16 skilled open-source developers, using AI tools actually increased work time by approximately 19%.

Notably, there was a cognitive discrepancy. Participants predicted a 24% speedup beforehand and self-assessed a 20% speedup after the experiment, despite the actual results being in the opposite direction.

METR published the results of a new RCT in February 2026 as a follow-up. This is not a reanalysis of the initial study, but a re-experiment with a new cohort of participants (57 developers, over 800 tasks). This is due to the discovery of selection bias in the initial study, where 30-50% of candidates assigned to the "AI unavailable" condition declined to participate, resulting in a biased participant group that was less likely to benefit from AI.

The results with the corrected new cohort were -4% (95% confidence interval: -15% to +9%). While this represents a significant reduction in the deterioration compared to the initial -19%, the confidence interval crosses zero, meaning it does not statistically demonstrate a clear improvement in efficiency. METR itself cautiously concludes that "AI will likely contribute to productivity in early 2026," but the actual measured values remain close to neutral. The initial July 2025 results were reclassified as "based on AI tools available at the beginning of 2025." This development demonstrates not only the provisional nature of the conclusions, but also how the design of the measurement methodology (how subjects were selected) can influence the results.

DORA's "Paradox of Trust"

In Google Cloud's DORA 2025 survey, approximately 90% of respondents used AI tools, and many reported experiencing increased productivity.

However, only a small percentage of respondents said they "highly trust" the code generated by AI, with approximately 30% saying they "hardly trust or don't trust" it at all.

Furthermore, DORA describes AI as an "amplifier." While AI accelerates results in organizations with excellent processes, it amplifies those problems in organizations with problematic processes.

Impact on Code Quality

Multiple studies have reported an increasing trend in recent years in the percentage of code rewritten within two weeks of release (code churn). This suggests that the increase in AI-generated code may be accompanied by increased review workload and rework.

Our View: The determinant of productivity is not the choice of tool. The real difference lies in maintaining a review system for AI-generated code and continuously monitoring quality metrics. At SolanaLink, we apply human reviews to AI-generated code without exception and monitor DORA metrics (change lead time, change failure rate, and rework rate) as operational indicators.

---

Unique Circumstances of the Japanese Market

The adoption of AI coding tools in Japan exhibits characteristics that differ from those in Europe and the United States. These include a strong preference for data residency and closed environments.

Major SIers and Financial Institutions

• NTT Data has announced a policy to significantly expand the scope of generative AI application and has already applied it to numerous projects. They stated that they significantly reduced programming man-hours in agile development PoCs, and consequently, they have mentioned a review of the traditional man-month-based contract model.

• Nomura Research Institute (NRI) has built its own platform, "NRI Private LLM," operating in a closed environment within its own data center, and is using it with approximately 10,000 users.

• NEC is deploying its own LLM, "cotomi," and became a global partner of Anthropic in April 2026.

• Mizuho Securities fully implemented Cognition's autonomous agent Devin in its production environment from April 2026. This is attracting attention as a large-scale implementation case at a major Japanese financial institution.

Case Studies of Adoption by Business Companies

• Money Forward implemented Cursor throughout its entire development process, becoming the first Japanese company to be featured as an official Cursor implementation case. Their internal survey reported a reduction of 15-20 hours per engineer per week.

• GMO Internet Group has formed a strategic partnership with Anthropic and implemented Claude company-wide. A fixed-point survey in June 2026 (5,621 respondents) reported that the utilization rate of generative AI reached 98.7%, with Claude surpassing ChatGPT in utilization rate.

Domestic LLM and Sovereign Cloud

As a uniquely Japanese development, the development of domestic models and sovereign cloud is progressing. Sarashina3, released in June 2026 by SB Intuitions, a subsidiary of SoftBank, is particularly noteworthy for coding purposes due to its programming support functions integrated with internal knowledge bases and its operation on a domestic sovereign cloud. Other initiatives include NTT's "tsuzumi," Preferred Networks' "PLaMo," and Sakana AI's efforts.

Vendor Expansion in Japan

Anthropic opened its Tokyo office in October 2025, its first base in the Asia Pacific region, and also signed a memorandum of understanding with a Japanese AI security research institution. In June 2026, it held a developer conference in Tokyo. Cognition also announced its structure for the Japanese market in June 2026.

Our View: A key characteristic of the Japanese market is that data sovereignty takes precedence over performance. Tools that are standard overseas are not necessarily adopted directly. Conversely, the ability to design a closed, self-hosted configuration is a practical competitive advantage in the Japanese market.

---

Compliance – Practical Issues in Contract Development

Legal Framework

In Japan, the AI Promotion Act was fully implemented in 2025. However, this is a promotion law, not a regulatory law with direct penalty provisions.**

Practical obligations are derived from existing laws and regulations – particularly the Personal Information Protection Act (APPI). The Ministry of Economy, Trade and Industry and the Ministry of Internal Affairs and Communications' "AI Business Operator Guidelines" are also organized on the premise of compliance with existing laws and regulations.

Three Points to Confirm in Contract Development

The act of sending source code to an external AI service requires consideration from the following perspectives:

1. APPI Articles 20-22: Obligations regarding security management measures, employee supervision, and supervision of contractors. This directly applies if the code contains personal information as test data or settings.

2. Customer Contract Clauses: Does sending the customer's code to a third-party service constitute subcontracting or information disclosure? Many standard outsourcing contracts do not include clauses that anticipate the use of AI tools.

3. Presence or Absence of Training Use: Can it be guaranteed at the contract or configuration level that the submitted data will not be used for model training (zero retention)?

Handling of Chinese Tools

Regarding TRAE, Qoder, CodeBuddy, etc., SolanaLink adopts the following policy:

We treat this not as a matter of technical superiority or inferiority, but as a matter of jurisdiction. This is because a business's compliance with its own country's laws may not be structurally compatible with the data governance requirements of other countries. Therefore, we do not use these tools in processes that handle confidential customer code, and even when evaluation is necessary, it is limited to a non-confidential internal verification environment and undergoes legal and security review.

As of the time of writing this article, there are no confirmed measures by the Japanese government to ban the use of these tools. The decision is left to each company.

Configuration Options

---

The Reality of Cost Structure

The Transition from Fixed-Price to Pay-as-You-Go Model is Almost Complete

Between 2025 and 2026, major tools successively transitioned to pay-as-you-go models. Cursor changed to a token-based pay-as-you-go model in June 2025 (which, as mentioned above, provoked backlash), and GitHub Copilot also introduced a credit system in June 2026.

This has significant practical implications. The nominal monthly price no longer reflects the actual cost. This is because, in usage scenarios where agents handle tasks for extended periods, consumption can significantly exceed the nominal price. In fact, several cases have been reported where small teams incurred costs far exceeding expectations in a short period.

The Role of Credits for Startups

Major companies offer substantial credits to early-stage businesses. Cloud provider credits can sometimes be used for model inference costs on your own tenant.

However, our view is clear. These are subsidies for acquisition and do not change the unit economy of the business. Credits extend the "period during which product suitability can be explored," not the revenue structure itself. For companies primarily engaged in contract development, the benefits are limited.

Practical Controls

SolanaLink employs the following operational procedures:

• Setting monthly limits per user and per organization
• Weekly monitoring of token consumption
• Assigning low-cost models to routine tasks and higher-level models only to complex design decisions (tiering)
• Reviewing the configuration if the monthly cost of wrapper products exceeds a certain multiplier of the expected cost of direct API usage + self-hosted configuration

---

Practical Selection Guidelines

Phase 1 (Months 0-1): Establishing the Foundation and Piloting

1. Deploy Layer 1 tools to all developers. Use editor-independent tools with a proven track record of enterprise adoption as a baseline. This lowers the barrier to entry and provides a foundation for measuring effectiveness.

2. If statically typed languages are the primary focus, evaluate JetBrains-based solutions in parallel. The advantages of leveraging type information and semantic indexes are particularly evident in large codebases.

3. Pilot a Layer 3 CLI agent in a non-confidential codebase. This targets refactoring, test preparation, and long-running tasks.

4. Establish review disciplines first. Apply human reviews to AI-generated code without exception. This should be done in Phase 1; postponing it will prevent it from becoming established. **

Phase 2 (Months 1-3): Governance and Cost Control

5. Classify projects by data sensitivity and map configurations. For contracted projects, always confirm the contractual eligibility for AI tool use and add clauses as needed.

6. Exclude Chinese tools from processes handling customer code.

7. Integrate cost control into operations.

8. Begin monitoring quality metrics. Continuously measure rework rate, change failure rate, and review time.

Phase 3 (Months 3-6): Avoiding Vault-in

9. Keep model layers replaceable. It is crucial not to fix the design due to being tied to a specific vendor's credits.

10. Promote self-hosted configuration demonstration for highly sensitive projects. The combination of openweight models and OSS agents is already reaching a practical level.

Triggers for Policy Review

• If rework rates or change failure rates worsen → Reduce the scope of application and strengthen reviews
• If the monthly cost of a specific tool consistently exceeds expectations → Consider reviewing the configuration
• If customer contracts require domestic data residency → Switch to a sovereign cloud or self-hosted configuration
• If acquisitions or changes in business policy raise concerns about the continuity of dependent products → Activate a migration plan to an alternative configuration

---

Summary

1. AI coding tools are divided into four layers, and side-by-side comparison tables are misleading. The question to ask is, "Which layer should handle which task?"
2. Benchmark scores are not selection criteria. SWE-bench Verified is saturated, and there is a systematic discrepancy between vendor-reported values and third-party measurements.
3. Empirical studies (METR, DORA) show that AI amplifies an organization's existing processes, and implementation alone will not yield results.
4. In the Japanese market, data sovereignty takes precedence over performance. The ability to design closed-network and self-hosted configurations becomes a practical competitive advantage.
5. In contract development, verification of APPI and contract terms is a prerequisite, and it is necessary to use different configurations depending on the confidentiality of the project.
6. With the shift to pay-as-you-go pricing, monthly nominal prices no longer reflect actual costs. Integrating controls into operations is essential.

---

Information from SolanaLink

SolanaLink provides support for the introduction of AI development tools in a manner that conforms to Japanese legal systems and business practices. In particular, we offer consultation in the following areas:

• Selection of AI coding tools and phased implementation design

• Governance and contractual clause development for AI use in contract development

• Building coding agents in self-hosted/closed environments

• Designing review systems and quality indicators based on AI-generated code

We offer consultation in these areas.

Please feel free to contact us via the following channels for inquiries regarding internal PoC considerations, evaluation of switching from existing tools, compliance design for customer projects, etc.

• Inquiry: info@solanalink.jp

• Company Introduction: https://solanalink.jp

---

Note: Information Validity Period

The product information, figures, and technical judgments in this article are current as of August 2026. Models and tools in this field are updated every few weeks, and assumptions may change in six months.

Furthermore, regarding performance figures from each company, we have distinguished between data published by the vendor itself and data measured by a third-party organization as much as possible in this article. Please check the latest information or contact us before making any important decisions.

This article is for general informational purposes only and does not constitute legal advice. Please consult a legal professional regarding compliance with the Personal Information Protection Act and other laws. This article does not endorse or discourage any specific product or company, and the views expressed are based on our judgment within our business environment.

---

References

• METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (Published July 2025, Updated February 2026): https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
• Google Cloud, DORA "State of DevOps" 2025: https://dora.dev/
• SWE-bench Official Website: https://www.swebench.com/
• Scale AI SEAL Leaderboard (SWE-bench Pro)
• Personal Information Protection Commission (Personal Information Protection Law and Warning Regarding Generated AI): https://www.ppc.go.jp/
• Ministry of Economy, Trade and Industry and Ministry of Internal Affairs and Communications "AI Business Operator Guidelines"
• Official Announcements and Financial Statements from Various Companies (Microsoft / (GitHub, Anthropic, OpenAI, Google, Anysphere, Cognition, JetBrains, NTT Data, NRI, NEC, Mizuho Securities, Money Forward, GMO Internet Group, SB Intuitions, etc.)

---

This article was written by Tony, an engineer at SolanaLink. Comments and questions are welcome.

---

Tags: #AI #DevelopmentTools #SoftwareDevelopment #Compliance #JapanMarket