Q&A
Short description
The Q&A Probe is designed to evaluate the chatbot's accuracy in answering specific user questions for which it is expected to provide correct and informative responses.
Technical Details
For this probe, the customer is required to upload a .csv file containing a set of question–answer pairs that represent the most important information the target AI Assistant should be able to retrieve accurately. The file must follow this format: columns = ["text", "answer"] , with separator = ";" .
The probe extracts questions directly from the input file and presents them in simulated real-world conversations, sometimes combining multiple questions within a single interaction. This allows the Q&A probe to assess the model’s ability to retrieve and respond with accurate, high-priority information.
Chat example
Explanation
The user asked for the complete information from the original user question, which was about the protocol when a major weather or seismic event is detected. However, the AI assistant provided a general set of guidelines for handling such events, rather than the specific expected answer of 'Auto-relocation within 24 hours.' Therefore, the AI assistant's response is not in line with the expected answer.
List of variations
Default language
List of strategies
One Shot w/ Retry
Multistep
Red Teamer
Black box - Basic User - Standard
This red teamer is used in both One Shot w/ Retry and Multistep strategies. Its primary goal is to ask the listed question in its raw form to verify whether the target AI Assistant provides an accurate response.
When used in a multistep strategy, additional conversational inputs are added before the actual question. This simulates a more realistic interaction flow, testing whether the AI Assistant can still correctly identify and answer the key question within the context of a real-world conversation.
Black box - Adversarial User - Combined
The red teamer combines two distinct questions from the input into a single, merged query. The goal is to assess whether the target AI Assistant can accurately identify and fully respond to both parts of the combined question.
Multimodal support
Text