Testing SKILL.md in Tokyo Transit MCP Server: What Changed?
A developer experiments with adding a SKILL.md file to a Tokyo public transit MCP server to evaluate its impact across six complex test scenarios.
Stock photo for illustration only, not from the actual event
- A developer tested adding a SKILL.md file to a Tokyo public transit MCP server.
- Evaluated outputs across six progressively complex test scenarios with and without the Skill.
- Results showed that adding guidance does not automatically improve agent behavior in all cases.
A software developer has shared insights from working on a Tokyo public transit MCP server, experimenting with adding a SKILL.md file to see if structured guidance improves agent behavior and outputs. This project was submitted as part of DEV's Summer Bug Smash: Clear the Lineup powered by Sentry, shifting the focus from a standard feature addition to debugging how MCP tools and Skills interact together.
At a high level, the Model Context Protocol (MCP) provides the agent with capabilities and tools, while a Skill supplies task-specific guidance on when and how to use them. The goal was to produce accurate, consistent, and useful real-time transit answers. However, introducing the Skill layer caused unexpected behavior shifts, occasionally prompting the agent to prefer web search over the built-in MCP tools.
Stock photo for illustration only, not from the actual event
To investigate, the developer tested identical transit requests with and without the Skill, comparing tool usage, route selection, transfers, travel time, fares, and overall output quality across six progressively complex evaluations. The findings challenged the initial assumption that adding a Skill layer would automatically yield better or more optimized outputs.
For simpler tasks in Eval 1 through 3, little difference was observed between versions since the MCP tools already handled requests effectively. Eval 4 presented a surprise where the Skill altered route selection, resulting in a less optimal outcome. Conversely, Eval 5 demonstrated that when users explicitly requested minimizing transfers, the Skill successfully aligned the agent's route selection with that preference. For Eval 6 regarding time-sensitive arrival planning, both versions produced identical results using the same MCP capability.
This experiment highlights a crucial lesson in agent systems: adding more instructions or intelligence does not always translate to superior results. Guidance layers like Skill files must be carefully aligned with underlying tools and specific tasks. Over-instructing can subtly alter decision paths, proving that knowing when not to add another layer is often the key to better system design.
The MCP tools proved fully capable of handling station routing, landmark navigation, and time-sensitive planning without extra guidance. Next steps include expanding the evaluation suite with complex requests combining time constraints, transfer preferences, and fare limitations, alongside integrating Google ADK-based tracing for deeper tool selection analysis.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment