Hakaru Lab | Published: 2026-09-26 | Review
Who this is for: Individual developers and small-scale operators who are considering incorporating a newly released generative AI model (a type of artificial intelligence that handles tasks such as text generation, summarization, and analysis) into their own work, and readers gathering information for a similar decision.
Key points (3 lines)
・Our lab sent the same request to GPT-5.6 Sol (the previous model) and GPT-6 Astra (the new model) at the same time and measured the results. The new model finished in about 1/3.9 the time, and the conclusions on all five discussion points matched between the two models.
・Based on a single comparison run on September 5, 2026, our lab judged that this lets us decide to "start trying it out" but not to "confirm it as the primary model," and we treat the two decisions separately. This gives individual developers and small-scale operators considering adopting a new generative AI model material for thinking about how far a single comparison run can go as a basis for judgment.
・The scope is limited to one comparison run that sent the same request to two models simultaneously, and a flaw in our scoring procedure discovered immediately afterward. The conclusions are based on the state of things as of September 10, 2026.
Basis of this review
・Reviewed on: 2026-09-10
・Conflicts of interest: Our lab is a paying user of both models compared in this article. There are no advertising links in this article. Beyond that, we have no other capital relationship, compensation relationship, or received benefit.
According to OpenAI's official release notes, the new generative AI model "GPT-6 Astra" was released on September 3, 2026 (the time zone listed in the notes is unconfirmed, so in Japan Standard Time this could have been September 3 or 4). Because the rollout was staggered, it did not actually reach our lab's environment until September 5. Our lab normally runs its research and review work by dividing tasks across multiple AI models, and we consider whether to adopt a new model each time one is released.
The decision to start using the new model by default (what we call "adoption" below) had already been made on the operating side before any measurement took place (under the task-division setup mentioned above, we review the assignment of duties each time a new model is released, and this decision was part of that review). The side-by-side measurement numbers described in Section 2 came together only after that. The measurement was not meant to decide whether to adopt the model; it was material for judging how to use it for which purposes, and later, whether it could be confirmed as the primary model. For that, we sent the same request to the previous model (GPT-5.6 Sol) and the new model at the same time and compared consumption, time taken, and the conclusions each one produced.
On September 5, 2026, our lab sent one piece of review work — a fairly heavy request covering multiple discussion points, run as an actual review request rather than a benchmark task built for comparison — to both the old and new models at the same time, using identical request text. The two models started almost simultaneously (6 seconds apart), and both finished normally. The old model's result arrived later because its run took 796 seconds. However, the run log shows one record of an internal attempt to launch a supporting process that failed; the consumption figure below may include this portion (the breakdown is unconfirmed). Consumption is counted in "tokens," the smallest unit an AI uses to process text (token: originally meaning "a mark" or "a token coin"). The results were as follows.
| Model | Reasoning-strength setting | Consumption (token count) | Time taken |
|---|---|---|---|
| Old model (GPT-5.6 Sol) | Default value in the config file (equivalent to the highest tier) | 399,825 | 796 seconds |
| New model (GPT-6 Astra) | Set several tiers below the highest (explicitly specified) | 110,099 | 203 seconds |
(Measurement environment: each was run once in our lab's execution environment, and the consumption and time-taken figures are taken directly from the run logs. The old model's setting was not chosen for the sake of comparison — it was simply the config file's default value in effect at the time. Our lab currently treats this setting as limited to test use only, on the grounds that it "automatically splits work internally, so consumption cannot be read correctly," and the old model's consumption figure here may not be usable as a comparable measurement. The reasoning-strength settings for the two models were also not matched, so not all of this difference can be attributed to the models themselves.)
The new model finished in about 1/3.9 the time. The request contained five discussion points, and the conclusions the two models reached matched on all five points. On consumption, the new model came out to about 1/3.6 of the old model's figure, but our lab's other records from the same day show that the old model's median consumption for similar tasks was far lower than this figure — this single run was the heaviest execution recorded that day. Our records show that consumption for the same model doing the same type of request can vary by a factor of several even under similar conditions, so this ratio means no more than "this is what happened in this one run."
For reference, according to OpenAI's official information, GPT-6 Astra's pricing is $10 per million input tokens and $50 per million output tokens for the short-input tier (inputs of roughly 272,000 tokens or fewer), and $20 input / $75 output above that. Compared to the previous generation GPT-5.6 Sol's rate ($4 per million input tokens, $20 output), the short-input tier is about 2.5 times the unit price (it is unconfirmed whether the old model's pricing has a similar tiered breakdown). Looking at unit price alone, the new model is more expensive, but as shown above, it finished in less time. That said, our lab did not record the input/output breakdown, so we cannot calculate the actual cost per request. Also, after this measurement, we switched to recording processing weight in a different unit rather than total consumption, and these two runs predate that change, so they were not measured in the current unit. Unit price and actual cost are separate metrics, and this distinction matters when reading these numbers.
Even after obtaining the results in Section 2, our lab did not conclude that "the new model is better." There are three reasons.
A single request is only one sample. The fact that the conclusions matched on all five points is encouraging, but the results could differ with a different type, difficulty, or subject area of request. A single measurement can be a clue, but it does not prove superiority across all use cases. And as noted in Section 2, the reasoning settings for the two models were not matched, and the old model's consumption figure itself is questionable as a measurement, so this was not a strict apples-to-apples comparison.
"Fast and cheap" and "correct" are different axes. The five conclusions matched, but the old model's answer contained one factual error regarding usage-quota specifications, where the new model's answer was correct. On the other hand, the new model's answer contained one instance where it asked to extend a general exception beyond the scope under review; after we pointed this out, it was revised to apply only to the specific case at hand. The types of errors differed, but both sides had something to correct. A match in conclusions does not mean a match in the accuracy of the details.
The weight of the decision is different. A decision to "try it out" and a decision to "use this model as the standard for this use case going forward" carry different consequences if wrong. The former can be reversed quickly if it doesn't fit; the latter gets built into other workflow designs and decision criteria, so it needs a stronger basis before being made.
With this in mind, our lab treats "adoption" and "promotion" as two separate decisions.
"Adoption" has been in effect since September 5, 2026, but as noted in Section 1, this was decided on the operating side beforehand. "Promotion" had not yet been confirmed as of when this article was written (September 10, 2026).
```mermaid
flowchart LR
A[New model released] --> B[Side-by-side measurement on the same request, old vs. new]
B --> C[Record results: consumption, time taken, agreement of conclusions]
C --> D[Adoption: start using by default, limited to specific use cases]
D --> F[Continue recording paired comparisons: 4 weeks, 8+ pairs, per use case]
F --> G{Meets minimum conditions for promotion?}
G -->|Yes| H[Promotion: confirmed as the standard]
G -->|No| I[Stays at adoption, or is withdrawn]
```
(Diagram: the flow that keeps the adoption and promotion decisions separate. Adoption was decided independently of the measurement results; the measurement mainly feeds into the ongoing record-keeping and the promotion decision that follow it.)
For the "promotion" decision, we designed a setup that uses a blind test — scoring by a model from a company different from either the old or new model's developer, with the identity of each answer's source hidden (blind test: a method originally used in settings such as clinical trials to prevent the evaluator's own preconceptions from affecting the result). On the same day we obtained the results in Section 2, our lab tried running the first round of this blind test.
But when we actually tried it, the AI model we intended to assign as the scorer had already read the new model's answer as part of the Section 2 review work. Once it had already seen one answer, asking it afterward to "score this with the source hidden" was no longer meaningful — the scorer already knew one side. Both conditions required for a valid blind test, ordering and anonymity, were broken.
Our lab judged this first pair invalid before running the scoring, not after. We did not discard a result after the fact — we determined that the plan to have it score at all could not hold. The formal comparison is planned to begin on or after September 7, 2026, and as of the review date we have not confirmed how many valid comparison pairs have been recorded. From this experience we learned that, when running a blind test, we need to secure a model that has not yet seen either answer before sending out the request. Reflecting this into our operating rules is still ahead of us.
Based on Sections 3 and 4, our lab has set the following minimum conditions for making the "promotion" decision. These conditions were not devised solely by our lab — they were proposed by the two models being compared themselves, then adopted after being challenged by a third model. We note, as a limitation of this standard itself, that the parties being evaluated had a hand in proposing the evaluation criteria.
These conditions are provisional for now, with a review date of October 5, 2026, to be revised based on weekly results. Since the formal comparison is planned to start on or after September 7, 2026, no use case could yet meet the four-week, 8-plus-pair condition.
Adopted (our decision): Our lab has been using the newly released generative AI model (GPT-6 Astra) as the default model since September 5, 2026, limited to research and review work, preliminary investigation before execution, root-cause investigation of problems, and generating material for comparisons. This "default," however, is not unconditional — we check the remaining usage quota on every run, and when it is low, we automatically fall back to the old model or halt execution; in fact, some work on the review date was still run on the old model. Drafting text, sorting content, and write operations remain with a different model for now. The decision to confirm a model as the standard for a specific use case ("promotion") is on hold until the minimum conditions in Section 5 are met — judged per use case, four weeks, at least 8 valid comparison pairs, matched reasoning settings, 3 wins out of 4 weeks, zero critical errors, a completion rate at or above the old model's, improvement in cost or checking time, scoring by a model from a different company, and ties not counted as wins. The baseline date is the review date (September 10, 2026), and this is expected to be revisited based on subsequent results.
Recommended (advice to readers): For individual developers and small-scale operators considering adopting a new generative AI model, we recommend not making the decision to "start trying it out" and the decision to "confirm it as the standard" on the same basis or at the same time. A single comparison measurement can be material for "it doesn't look bad," but it is not, by itself, grounds for confirming superiority. Be careful when the reasoning-strength settings were not matched during comparison, or when the run used for the consumption comparison happened to be an unusually heavy one — in either case it's hard to tell how much that difference drove the result. If you use a scoring model from a different developer than either model being compared, we recommend checking, before sending the request, whether the scorer has already seen one of the answers to be compared — our own lab has experience invalidating the first pair of a blind test by failing to check this. Note also that "a different developer" here refers to the relationship with the two models being compared; it does not mean the scoring model is a third party independent of our own lab's setup.
Method: This article is based on our lab's records of a side-by-side measurement on the same request, conducted on September 5, 2026, and on our review of a flaw in the scoring procedure discovered immediately afterward. The figures (consumption, time taken) are recorded as they appear in the run logs. The execution environment was a command-line interface (CLI: an interface operated by text input rather than on-screen buttons), with one trial run per condition. The model's release date and pricing are based on OpenAI's official information, checked during the 0:00 hour (JST) on September 10, 2026.
Unverified items:
Corrections:
Sources:
Our lab (hakarulab.com) carries out its research, writing, and verification through a division of labor across multiple AI models. This article describes, based on our records, a measurement our lab actually ran when considering whether to incorporate a new AI model, and a procedural flaw we found along the way. Policy decisions and final confirmation are handled by humans at the operating entity.