Google Research Releases ToolGrad Framework for AI Data

Google Research has developed and released a new framework called ToolGrad, designed to tackle the inefficiencies inherent in conventional tool-use data generation pipelines. Previous pipelines such as ToolBench and ToolACE relied on a query-first recipe where systems sampled a pool of APIs, tasked a large language model to invent plausible user instructions, and dispatched a depth-first search agent to locate a satisfying tool-use path. This traditional search method offered no guarantee of success, resulting in wasted compute and discarded samples whenever dead-ends occurred.

To overcome this limitation, ToolGrad reverses the operational order entirely. It begins by constructing a ground-truth tool-use chain through actual API executions, then annotates that verified chain with a matching user query. Because an explicit, working chain contains far less ambiguity than a hypothetical prompt, the chain-to-query translation step requires only a single LLM call.

Shifting the paradigm from query-first to an answer-first framework is a pivotal advancement in overcoming synthetic data bottlenecks for large language models. Training models to reliably invoke external functions demands high-quality training trajectories, and letting autonomous agents wander blindly often leads to high failure rates. Forcing successful execution upfront before synthesizing instructions drastically reduces data waste and boosts training efficiency.

Stock photo for illustration only, not from the actual event

Each iteration of the system executes four distinct modules in sequence, and repeating this loop yields a complete sample containing a user query, a verified API workflow, and a final response. The repository's default configuration executes 10 iterations across 50 sampled APIs per workflow.

The research team evaluated data generation performance on the ToolBench API database, which encompasses over 16,000 real-world APIs, comparing ToolGrad directly against ToolBench's DFS-based query-first approach. According to the findings, ToolGrad achieved a remarkable 99.8% pass rate. The remaining 0.2% failure cases occurred when the agent failed to retrieve successful responses from three selected APIs across all 10 iterations, resulting in an empty sample being saved.

Furthermore, the researchers generated ToolGrad-500, a 500-sample dataset constructed using Gemini 2.5 Flash-Lite, and utilized it to post-train Gemma-3 models at 1B, 4B, and 12B parameter scales. Evaluation was conducted on the Berkeley Function Calling Leaderboard, which utilizes a distinct tool set compared to ToolBench, creating an out-of-distribution test environment featuring unseen tools. Reproduction scripts target BFCL V1 and V2 via a customized fork, executing inference inside a vLLM Docker image verified on a single NVIDIA A100 40GB GPU.

Source: MarkTechPost