Run baseline evaluations on selected models and task set

[XTOL20C]

Execute the first complete baseline evaluation once the core prerequisites are ready.

This issue is not about designing the infrastructure from scratch. It is about using the selected models, prepared task set, and agreed evaluation workflow to produce the first real baseline results for the project.

This issue should only be started after the following are sufficiently defined:

• selected LLM APIs / models
• chosen task scope
• prepared inputs or question sets
• response logging format
• scoring or evaluation procedure

The purpose of this issue is to move from setup and planning into an actual baseline experimental run.

Deliverable

A completed baseline evaluation run with saved outputs and a short written summary of results.

Acceptance criteria

• The evaluated models are listed explicitly.
• The evaluated task set or question set is identified explicitly.
• The run is executed using the agreed evaluation workflow.
• Outputs are stored in the agreed format.
• Results are scored or otherwise summarized in a consistent way.
• A short summary of findings, issues, and next steps is added to the issue.

[CaptainPawel]

Final Baseline Evaluation Summary
This issue is now being closed following the execution of the first baseline evaluation. The run has been completed using the selected LLM APIs and the provided SSAL network topology.

Evaluation Overview
The evaluation tested Gemini 2.5 Flash, GPT 5.4, and GPT 5.4 Mini. The core task was to calculate the shortest path between specific Origin and Destination nodes using strictly defined constraints, including 1w/2w (one-way/two-way) markers and minimization of the "Length" attribute. All outputs were required to follow a strict JSON format.

Detailed Analysis of Test Results and Complexity
The baseline run revealed a clear correlation between the structural complexity of the network and the models' failure rates:

Linear Navigation (Tests 1 & 2): These tests focused on the Bulevardi corridor. Because the path was primarily a straight sequence of nodes (e.g., 25291537 to 313984198), GPT 5.4 achieved high distance precision (99.1%). This represents the "low difficulty" baseline where the model only needs to follow a single street name.

Multi-Street Transitions (Tests 3, 4 & 5): Difficulty increased significantly when the route required a "turn" or a transition between different street segments (e.g., moving from Bulevardi to Korkeavuorenkatu or Yrjönkatu). In Test 3, accuracy dropped to 8.1%, indicating that the models struggle to identify intersection nodes where one edge name ends and another begins within the SSAL data.

Edge Cases and High Search Depth (Tests 6 & 7): These tests targeted nodes deeper in the adjacency list or at the periphery of the network (e.g., Hietalandenventa). In these scenarios, both models failed even the basic task of maintaining JSON formatting. This suggests that as the "reasoning path" gets longer, the models lose the ability to adhere to structural constraints.

The Token Constraint Breakthrough (Tests 8–12): This phase introduced a critical discovery regarding Gemini 2.5 Flash.

The 1600 Token Threshold: While Gemini failed all JSON formatting tasks between 300 and 1024 tokens, it successfully entered the evaluation in Tests 11 & 12 when the Max Output Tokens were set to exactly 1600.

Hyperparameter Sensitivity: Interestingly, increasing the limit to 2048 (Test 10) caused Gemini to fail again, proving that format compliance in complex navigation tasks is highly sensitive to the specific output buffer size.

Navigation Logic vs. Calculation (Tests 11 & 12): Even after achieving format compliance, a "Hallucination Gap" remains. For example, in Test 11, Gemini reached the destination but calculated a length of 10.8m against an actual algorithmic length of 106.0m, showing that while the path can be found, mathematical precision in graph traversal is still a major hurdle.

Technical Observations

Format Compliance: Gemini 2.5 Flash consistently failed to deliver JSON output across all tests between 1-10 until the Max Input Tokens were set for 1600. GPT 5.4 was stable on simple routes but failed on complex ones.

Logic Inhibitors: The models frequently ignored "1w" (one-way) constraints when the route complexity increased, leading to mathematically invalid paths.

Node Density: Nodes with multiple outgoing connections acted as "confusion points," where the models often picked sub-optimal paths instead of calculating the true shortest route.

Conclusion and Next Steps
The baseline is now firmly established, defining the performance floor for the project. The results definitively prove that a standard zero-shot approach is insufficient for reliable graph navigation, as models struggle with both mathematical summation and adherence to directional constraints ("1w").

Furthermore, the tests identified a critical hyperparameter dependency: Gemini 2.5 Flash’s ability to maintain JSON structure is tied to a specific "output window" (stabilizing at 1600 tokens), whereas GPT 5.4 remains more structurally stable but prone to logic collapse on complex routes.

Next steps will involve:

Standardizing Hyperparameters: Fix the "Max Output Tokens" for each model based on the stability thresholds identified (e.g., 1600 for Gemini) to ensure that future logic failures are not caused by formatting truncation.

Implementing Chain-of-Thought (CoT) Prompting: Forcing the models to "think out loud" by listing available neighbors and their lengths before generating the final JSON. This is crucial to bridge the "Hallucination Gap" in distance calculations.

Utilizing Few-Shot Examples: Introducing 2-3 successful route examples in the prompt to stabilize Gemini’s format compliance across all token ranges and improve the handling of "1w" (one-way) markers.

Intersection "Visibility" Enhancement: Refining the SSAL data representation to explicitly tag nodes where multiple street names meet, reducing the "confusion points" identified in Tests 3, 4, and 5.

Test Results and prompt logs were saved and uploaded to GitHub under LLM Test Results and llm_compare_history.json.