PENTESTY · EDITORIAL ANALYSIS · · 4 min read
Chinese AI models: evaluate cost and security before switching
Lower AI prices create more options. An enterprise decision needs evidence about outcomes, data exposure and the ability to change providers safely.
By Pentesty
The market signal and its limits
Cyber Security Brazil reports competitive pressure from Chinese models on the AI market. In its official Kimi K3 introduction, Moonshot describes an open-weight model and acknowledges that its overall performance still trails the strongest proprietary models in its comparison. This supports technical evaluation, but establishes neither savings nor equivalent quality for an individual enterprise. The report behind this analysis Official Kimi K3 introduction
Compare the cost of an accepted decision
Start with a bounded task: classify internal requests, find a contract clause or propose a code correction. Define an acceptable output and calculate spending per approved result. Include review, additional attempts, document retrieval, infrastructure and integration maintenance. A lower unit price may matter less when the task requires more intervention from an employee.
Give every candidate the same evaluation cases and reserve a portion outside prompt tuning. Separate correct answers, incomplete answers and errors with meaningful consequences. Record latency and unavailability as well. The decision proposed here is to buy a measured capability for a particular workflow, with explicit limits on where it may be used.
Draw the data boundary before the pilot
Document the difference between calling an external API and operating model weights on your own infrastructure. For each option, request evidence about retention, subcontractors, processing location, deletion and administrative access. A provider’s geographic origin belongs in a dependency assessment, alongside concrete examination of the contract, architecture and actual exposure.
Prefer synthetic data or authorized, minimized samples during the pilot. Give the model separate credentials and access only to necessary tools. An output requesting broader privileges should trigger an independent decision. Review who can submit context to the model, because material from documents and integrations can cross the same trust boundary as a user request.
A model change should leave an evidence trail
Prepare a short approval package: identified model version, configuration, evaluation set, results by task, data map and acceptance owner. Include an authorized attempt to access another user’s records and a simulation of an unavailable tool. Keep the expected and observed responses, while removing sensitive information from retained records.
Propose stop conditions before the first production call. For example, suspend automation when a required reference is missing, requested permissions exceed the approved scope or one execution exceeds its spending limit. Values should reflect the product’s consequences and tolerance for failure. Exercise the stop mechanism and recovery path to demonstrate that the policy can operate.
The fallback path belongs in procurement
Design a return path to the previous provider or a human process. Prevent automatic switching from sending information to an unapproved provider. Run a simulated failure and record the destination, reason, restoration time and transmitted data. A strategy with several models remains controllable only when each possible path has its own authorization.
Frequently asked questions
Is an open-weight model automatically more secure?
No. Assess the complete application, including runtime, dependencies, weight distribution, data access and updates. Access to weights can support inspection and operation, but does not replace verification of the deployed system.
What should be the first savings metric?
Use cost per accepted outcome within quality and timing requirements. Compare significant errors and review effort too. Cost per token remains useful for diagnosis, without making the procurement decision on its own.
Should every task move at the same time?
A gradual migration lets the team validate different consequences and data types. Start with a reversible workflow, retain the previous baseline and expand only after reviewing evidence and assigning accountable owners.
What evidence should a CTO request before approval?
Request reproducible evaluations, a data map, effective permissions, failure handling and a return-path exercise. Each artifact should identify the version and environment so reviewers can understand exactly what was tested.
Sources and references
Cyber Security Brazil: model competition
Original Pentesty analysis, published October 15, 2026. The proposed scenarios are editorial guidance, with scope and validation to be defined for each environment.