AI code rewrites: how to evaluate performance and security controls
A fast benchmark does not demonstrate that an application preserved its access rules. Learn how to assess a migration developed with AI assistance.
Czytaj po portugalskuBy Pentesty
What a language comparison can establish
CyberSec Brazil reported a comparison of Campfire implementations in different languages. Basecamp’s public repositories expose distinct implementations and verification procedures. This article does not repeat the reported performance figures: measurements depend on the tested revision and configuration. The case raises an engineering question: what evidence supports replacing an application in production?
Define parity before measuring speed
Start with observable behaviors the existing application must preserve. Include account creation, access recovery, permissions, attachments, search and administrative operations relevant to the product. For each behavior, record inputs, expected outcomes and persistent effects. A correct HTTP response may conceal an incorrect write or a missing authorization check, so the response alone is insufficient evidence.
Choose business examples that exercise boundaries. Can a person read another group’s conversation? Does a suspended account retain a valid session? Does receiving the same event twice duplicate an operation? Convert these questions into acceptance criteria agreed with the product maintainers. This is a proposed migration assessment, rather than a claim that the cited projects contain those defects.
Compare controls using different identities
Run equivalent scenarios against both implementations using test accounts with different roles. Use recognizable synthetic records to identify an inappropriate response. Capture the initial state, authorized request and observed outcome. The assessment should establish both that an allowed operation succeeds and that an operation outside the account’s scope is denied.
- Authorization: preserve boundaries between organizations, groups and roles.
- Sessions: check expiration, logout and revocation across agreed workflows.
- Data: compare validation, persistence and exposure through responses and logs.
- Integrations: verify identity, permissions and failure handling for each connected service.
Make the benchmark reproducible
Pin the code revision, runtime version, available resources and storage configuration. Document the volume and shape of synthetic data. If one implementation uses a warm cache while another starts empty, record the difference and run comparable scenarios before attributing the result to a programming language. Preserve the configuration alongside the measurement so another maintainer can repeat it.
Describe load as a set of user journeys, with a mix of reads, writes and persistent connections appropriate to the hypothesis being tested. Observe latency, errors and resource use together. A run returning more responses per second might reject work, omit checks or measure a different operation. Investigate these possibilities before accepting the improvement as evidence for migration.
Approve migration with evidence and a recovery path
The NIST SSDF provides practices for integrating security into development. In the approach proposed here, the team maintains a record connecting requirements, tests and results to the candidate revision. That record should let another accountable person repeat the scenario and understand acceptance, without relying on an explanation produced by the AI tool that helped write the implementation.
Separate technical acceptance from the production release decision. Confirm data compatibility and the feasibility of recovery before changing traffic. A schema migration can prevent recovery by simply restoring the previous application version. Rehearse the return path with an authorized, protected copy of data, defining who can interrupt the migration and how the restored service will be validated.
Pentesty can evaluate access paths and controls within an agreed scope. Combine that assessment with functional and performance tests maintained by the product team. The decision needs complementary evidence: business behavior remains valid, access boundaries hold and the implementation meets the operational objective that justified the rewrite. Record unresolved differences explicitly rather than allowing a headline throughput figure to stand in for acceptance.
Who defines the test when AI also helps evaluate the code?
Editorial update: October 16, 2026
In its August 2026 risk report, Anthropic said its models had not met either of its policy criteria for research and development automation, but recorded lower confidence in that assessment. In When AI builds itself, the lab distinguishes progress in task execution from choosing research goals. These are provider assessments within defined scopes, without demonstrating full recursive self-improvement or universal safety.
When reviewing a rewrite, separate the candidate implementation from the contract determining its acceptance. AI may suggest tests, but the responsible team needs to approve requirements and changes to those criteria. Keep essential authorization, persistence and behavior cases outside the set the agent may freely modify. A test changed together with the code can pass while leaving the migration's original question unanswered.
In an authorized exercise using a fictional application, fix the expected behaviors and retain the acceptance suite's revision. Request the change within agreed permissions and execute approved cases from a controlled environment. If a legitimate need to change a criterion arises, record the reasoning and separate approval before updating the baseline. Compare results and limitations without making the model's own assessment the only evidence of success.
Frequently asked questions
Does a benchmark prove Rust is always faster?
No. Its result describes an implementation under particular conditions. Compare revisions, configurations, completed operations, errors and resource use before drawing conclusions for your own application.
Does AI-generated code require different tests?
It must satisfy the same product requirements. Migration additionally needs evidence of parity with the previous implementation, including authorization boundaries and persistent effects, rather than relying on similar-looking output.
Which control should be assessed first?
Select the access boundaries with the greatest impact on the product. Use identities with different permissions and verify both authorized outcomes and the denial of operations outside each identity’s scope.
When can a rewrite be accepted?
When it meets agreed functional, security and operational criteria with reproducible results and a viable recovery path. A speed improvement alone does not establish that the other conditions have been met.
Sources and references
- Cyber Security Brazil: editorial news reference
- Basecamp: Campfire implementation in Rust
- Basecamp: Campfire implementation and verification in Elixir
- NIST SP 800-218: secure software development
Original Pentesty analysis, published October 15, 2026. The proposed scenarios are editorial guidance, with scope and validation to be defined for each environment.