Introducing SciCode++: Scientific coding data for frontier models

SciCode¹ was introduced in 2024 by a group of scientists from leading US universities and research institutes. The original benchmark included 80 main problems and 338 subproblems across mathematics, physics, chemistry, biology, and materials science. Instead of asking for a written response or a single numerical answer, it asked the model to turn a scientific method into executable code.

That was important because it offered a practical solution to the verification problem for research-grade scientific tasks. Real scientific work is hard to reduce to multiple-choice questions or a value of x, y, or z. SciCode problems look more like something a postdoc might hand to a PhD student:

Postdoc: Hey, could you take a small piece of the neutrino code off my plate? I need a function that gives the oscillation probabilities - how likely each of the three neutrino flavors is to turn into each of the others after traveling a given distance at a given energy.

Start with the standard mixing matrix, use it to write down the Hamiltonian. Here's the part I actually care about: write that Hamiltonian as a combination of the eight Gell-Mann matrices. Rebuild the evolution operator and read off the probabilities from its entries. Return them as a flat list of nine numbers: electron to (e, mu, tau), muon to (e, mu, tau), tau to (e, mu, tau). Tomorrow's fine, coffee's on me.

PhD student: Got it, I'll push it by tomorrow.

By making the answer a well-defined Python function, SciCode reflects how computational scientists actually work, allows several subproblems to feed into a main problem, and makes the result rigorously testable. Test cases can probe subtle cases and corner conditions rather than checking a single output. In effect, the SciCode authors found a way to assign a loss function to a much more complex scientific workflow.

SciCode-Verified² revisited that benchmark problem by problem in 2026. The audit catalogued incorrect reference solutions, conventions that had never been specified, grading that depended on random seeds, tolerances tighter than the numerics supported, and tests too weak to distinguish a correct implementation from a lucky one. Many of those issues required domain expertise to identify.

Figure 1. Scientific evaluation is moving from scientific Q&A, to executable scientific coding, to verified scientific coding, and toward interdependent scientific workflows.

That progression creates a new challenge when the same format is used for training at scale: how do you make sure the difficulty comes from the model itself rather than from the task or verifier?

SciCode-Verified fixed many of the problems in the original benchmark. But the same issues can show up again when new SciCode-style tasks are created for training and evaluation. A task may leave out an important convention, the tests may be too weak or too strict, or an upstream mistake may make later steps look wrong.

This becomes much more important when you are building thousands of tasks. A low pass rate tells you that the model failed, but not why. Before using that failure as a training signal, you need to know whether the problem came from the model or from the task itself. That is the problem SciCode++ is designed to solve.

SciCode++ is designed so that a model failure actually tells you something useful. We combine expert-written scientific tasks, executable verification, sub-problem-level scoring, frontier-model calibration, and multi-stage review to make the signal much more reliable. The failure modes below are exactly the ones this process is designed to catch.

Figure 2. Five broad causes of failure in scientific coding tasks. Four come from the data or evaluation; one reflects a genuine model capability gap.

Model breaking means creating a domain-specific task that a target model reliably fails to solve across several runs, usually 4 to 8. It is a useful signal, but a poor objective on its own. Models can fail for different reasons, and not all of them are legitimate capability failures:

• If a solution is not computable because the question is not well-posed in the first place, the model failure is not what we are after. The task should be fair.
• If the function tests do not catch corner cases or discriminate between subtle but important details, imperfect solutions can pass and contaminate the dataset.
• If an upstream wrong answer propagates and causes a later subproblem to fail, we may record a false negative downstream. The evaluation should distinguish the upstream mistake from the downstream capability.

One common heuristic is to target tasks that frontier models solve on some runs but not all. We still use pass-rate bands as one calibration signal, but newer reasoning models make this less useful as a standalone target. We also look at expert judgment, earlier model generations, and the relationship between subproblem and whole-task performance.

These quality issues are not hypothetical. SciCode-Verified audited the original benchmark and identified 263 defects across 65 test problems. Issues affected 91% of main problems, and 78% of the score-suppressing defects required physics or mathematics expertise to identify. The audit should not be treated as the final scientific judgment on every problem, but the overall signal is clear: the SciCode format is powerful, and the quality of the specification and verifier can materially affect the measured result.

Table 1. SciCode-Verified found 263 defects across 65 test problems, spanning both scientific and surface-level issues. Source: Hu, Z., Huang, X., Deng, Y., & Chen, X. (2026), arXiv:2608.04975.

We care about this because these quality issues separate datasets produced carefully by domain experts from datasets that satisfy a simple metric but fail to provide useful training signals.

Consider a neurophysiology task about how a synapse updates its strength: biology’s version of backpropagation. Two neurons fire a few milliseconds apart, and depending on which fired first and by how much, the connection between them gets a little stronger or a little weaker. The model we use for this (Shouval, Bear and Cooper, 2002) is a chain of coupled differential equations:

1. a chemical signal that decays over time,
2. a voltage pulse that travels back along the neuron,
3. a voltage-dependent gate,
4. a calcium concentration that integrates the current,
5. and a nonlinear learning rule that decides which way the weight moves.

Getting the final answer means integrating all of it numerically, in the right order, with every constant and convention consistent from the first step to the last. One wrong sign anywhere in the chain and the answer flips.

The task follows the standard SciCode format:

• A short scientific background to set context, which may include a scientific article.
• A well-defined main question that can be answered by setting up a Python function with a given signature.
• Two or more standalone sub-problems, each with its own function signature, that break the main question into steps; the main function is built by calling them in order.

The complete task and example test functions are shown below. Every constant, sign, and unit convention has to be pinned down so that the task is well-defined and the solution verifiable. (Usually there are 5 to 10 asserts for each subproblem and main problem; we include one per example here for brevity.)

This task creates a nonlinear multistep flow of information that is very well defined. None of the pieces is exotic. What makes it a good task is that each piece hands a number to the next, each has a convention that could reasonably go the other way, and the final answer only means something if all of them agree. 

Each test looks for one way the science can go wrong: a value at a known point catches a misplaced constant, a sign catches a reversed convention, a limiting case catches a missing term. End-to-end tests then add a sanity check on the qualitative behaviour, the sign or shape of the final result, alongside an exact value at one or two points.

We do not demand arbitrary precision; a different but valid implementation must pass. A later test never inherits an earlier failure. And a model that picks the wrong Python tool or misreads a signature has made a judgement error too, so that failure we count as a genuine failure.

The full dataset contains roughly 6,000 tasks, out of which 4,000 were used in our initial experiment. Each task ran 8 times against a current frontier model. Physics takes the largest share, followed by materials science, chemistry, mathematics, and biology. Mean main-problem pass rates run from 4% in biology to 14% in mathematics, with none above 15%.

Figure 3. Difficulty varies sharply by domain: mean main-problem pass rates range from 4% in biology to 14% in mathematics, with no domain above 15%.

Whatever the domain, about 9 in 10 tasks are general computational tasks: a method is stated, usually the one a published paper used to get its result, and the model is asked to turn it into working code and reproduce that result. Sometimes we go a step further and leave out the paper's exact choices, so the model has to work out the method itself before it can implement it. The remaining problems are optimization, data analysis and a small set of prediction and design tasks.

Figure 4. Most tasks remain beyond the model's reliable reach: 70% of main problems were unsolved across all 8 attempts.

Across this dataset, 70% of main problems were never solved in any of the eight tries, 10% were solved once, 16% two or three times, and under 4% four times or more. Each task is given to a state-of-the-art model (decided at the beginning of the campaign) eight times. Every sample is then run against the test function suite, and we record how many of the eight pass the main problem: 0 of 8, 1 of 8, and so on. That fraction is the task's reported pass rate.

The middle of the distribution is relatively thin. Most tasks are either consistently unsolved or solved only a small number of times, so run-to-run variance alone does not explain the difficulty we see in this dataset.

Composition also lowers whole-task success. If three subproblems each succeed 80% of the time, a task that requires all three to succeed would have an expected success rate of about 51%, even before accounting for any additional difficulty in combining them. This is also what makes the SciCode++ structure interesting beyond coding benchmarks: the model has to carry intermediate results through a multi-step workflow, much like the longer tool-using scientific tasks frontier models are increasingly being asked to handle.

We ran an early post-training experiment to test whether training on our private SciCode++ data improves performance on the public SciCode benchmark. It did: performance improved by 10.3% relative to baseline on SciCode and by 9.1% on SciCode-Verified.

To understand that result, here is how we set up the experiment.

Post-training scope: We selected Qwen 3.5 9B with reasoning disabled and trained it with supervised fine-tuning. The model is small enough to train efficiently while still providing a meaningful baseline. SciCode prompts include previous subproblems and their answers, which can create long contexts and high GPU-memory requirements. Keeping reasoning disabled simplified the training setup because model-provided reasoning traces would require even longer contexts, and gold reasoning traces were not available.

Error analysis: We first measured baseline performance on public SciCode and SciCode-Verified, then reviewed the errors to guide the experiment. Our starting hypothesis was that a substantial share of failures came from incorrect scientific reasoning rather than coding ability alone. The error analysis supported that hypothesis.

Prompt and training data design: To isolate the current subproblem from earlier mistakes, we provided gold solutions for previous subproblems in the prompt. We then selected training and validation data aligned with public SciCode in logical structure and format, while keeping public SciCode and SciCode-Verified as the test sets.

Post-training setup: SciCode has a 2-level hierarchy in which each task contains several subproblems in sequence. We randomized the training data at the task level while preserving subproblem order, and split training and validation at the main-task level so related subproblems stayed together. Maximum context and output lengths were set from the training-data token distributions. We then trained the baseline model with LoRA using Unsloth for 2 epochs while tuning the learning rate and LoRA rank.

Post-training results: Performance improved by 10.3% relative to baseline on SciCode and by 9.1% on SciCode-Verified. Consistent with the SciCode-Verified findings, the model scored higher on SciCode-Verified than on the original SciCode. Error analysis after training also showed fewer failures associated with incorrect scientific reasoning, while ordinary coding and syntax errors became more visible.

Figure 5. Post-training on Turing-designed private SciCode++ data improved performance on both public benchmarks: +10.3% on SciCode and +9.1% on SciCode-Verified.

Future work: The next experiments will test reinforcement learning, additional datasets, and other ways to reduce the remaining failure modes. In this experiment, Turing-designed private SciCode++ data improved performance on both public SciCode benchmarks after post-training.

Meeting the standard above at production scale means holding the same checks across thousands of tasks and a range of customer requirements.

At Turing, we have built scientific coding data for several frontier labs, and the acceptance criteria are rarely the same twice. Some may want a target failure rate benchmarked against their own model. Others may want every subproblem scored separately, or a spread of difficulty rather than a single narrow band. Working across these requirements has shown us the many ways a task can look hard without actually being useful. Working across these requirements has given us a close view of where frontier models still fail in science and what true difficulty actually looks like.

Our tasks are written by working scientists (PhDs and professors in the relevant fields) then pass through an expert trainer, independent L1 review, independent L2 review, and a frontier-model pass-rate gate. That combination matters because task difficulty depends on the problem, the verifier, and the current model frontier, and all three have to line up for the result to be useful.

For model teams, the real value is in failures that trace back to a specific capability, pointing clearly to what to improve next.

If you are working on scientific reasoning or coding models, get in touch with us. Let’s discuss the right domains and difficulty bands for your models.

1. Tian et al., SciCode: A Research Coding Benchmark Curated by Scientists, arXiv:2407.13168 (2024).

2. Hu, Huang, Deng & Chen, SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models, arXiv:2608.04975 (2026).