Given the size and complexity of results data generated by the Halo Pilot, we felt a need to create a user-friendly tool to better browse and understand our experimental outcome. Our Results Browser was designed to allow users to browse through the 100 challenge questions and question subparts. The browser presents each question’s designation and text, as well as a scoring matrix with the subject matter experts’ (SMEs) grading of answer correctness and quality of justification for each of the three Halo teams (SRI, Ontoprise and Cycorp).

For multipart questions, the scoring matrix represents the results for that question part only. In this case, the aggregate question scores appear in a matrix below the scoring matrix.

Selecting the Question Encoding icon in the appropriate team row in the scoring matrix pops open that team’s formal encoding of the English text of the question into their logical language for the given question or question subpart.

Selecting the Graded Results icon in the appropriate team row and SME column pops open an image of the graded exam paper of the specified team by the specified SME for the given question or question subpart.

Selecting the Time To Answer icon pops open a frame with CPU time in seconds each team spent on the given question or question subpart.

Selecting the Failure Detail icon pops open a frame with a detailed analysis of points lost on a given question or question subpart in the context of the Project Halo brittleness taxonomy.

For those who are interested in viewing the raw data, related documents can be found in the following sections:

• Challenge Questions



• Question Encoding



• Graded Results



• Failure Details

LAUNCH the Interactive Halo Pilot Results Browser.