1. Goal & Description:

We are currently working on building quality for cricket related queries on AI Mode.

AIM quality on cricket This requires “on the go” evaluations of the live game And also some evals around pre & post game; league, athlete and in general any cricket related queries - will help us understand the queries that fans have associated with a game, league, team & athlete!

The core objective here is for raters to simulate the experience of a dedicated cricket fan by not only identifying and evaluating queries in real-time but also engaging with AIM on pre, post game and also asking their cricket queries associated with the league, athlete etc in general

This process will include side-by-side comparisons across multiple platforms, with raters capturing screenshots of their findings and evaluating them on helpfulness and factuality

2. Data to be Evaluated:

We request that the raters primarily focus on generating: \"live\" queries as they actively watch, track, or report on the game(s)—specifically, the type of questions or prompts a dedicated sports fan would genuinely issue. submission of contextually relevant queries in these buckets:

• Live match tracking- Requests for real-time scores, in-game plays, in-match stats, etc. for games currently in progress.
• Schedules and broadcasts- Queries asking about future fixtures, TV/streaming channels, start times, or venue details.
• Rules and education - Queries seeking explanations of specific rules, terminology, or how a professional game is played.
• Video highlights and replays - Requests for video content of recent or historical games, plays, or summaries.
• Historical stats and records- Queries for past data emphasizing deep history (older than ~1 month), career statistics, all-time records, etc. 6 Fantasy sports assistance - Queries related to fantasy roster management, draft research, and strategy, including start/sit decisions.
• Expert analysis and opinion- Requests for subjective takes, power rankings, forward-looking predictions, or deep-dive \"why\" explanations.
• Sports news and storylines- Requests for updates on off-field developments, business transactions, and summaries of recent events.
• Player and team profiles- Static factual lookups about athletes or teams, such as age, physical stats, home stadium, roster lists, etc.

Specific Mechanism and Request

1.Issue every query simultaneously on both AIM and ChatGPT

2.Track which version of ChatGPT they are using (screenshot)

3.Provide a single-sided rating for each response, evaluating both helpfulness and factuality, accompanied by detailed reasoning for each.

4.Deliver a side-by-side win/loss/neutral rating for each pair of responses.

5. Complete this process for the maximum feasible number of queries within the \"game window,\" which extends from a short time before the game begins to approximately 30–60 minutes after its conclusion.

No of queries: 100 per game

Language: English & Hindi