iSQI CT-AI Practice Test - Questions Answers, Page 5

Precision is a performance metric used to evaluate the accuracy of positive predictions in a classification model. It is defined by the formula:

Precision=TPTP+FP100%\text{Precision} = \frac{TP}{TP + FP} \times 100\%Precision=TP+FPTP100%

Where:

TP (True Positives) = Number of correctly predicted positive cases

FP (False Positives) = Number of incorrectly predicted positive cases

The confusion matrix provided in the question would typically list these values. Based on ISTQB's guidelines for calculating precision, selecting the correct number of true positives and false positives from the given data should yield 94.2% as the precision.

Reference from ISTQB Certified Tester AI Testing Study Guide:

Section 5.1 - Confusion Matrix and ML Functional Performance Metrics explains the calculation of precision using the confusion matrix.

A/B testing is a statistical testing method that compares two different versions of a system to determine which one performs better. In this scenario, the old NLP algorithm was rated for accuracy, and after the update, the new algorithm was also rated by users. A statistical test was performed to compare the two versions, which is the fundamental approach of A/B testing.

A/B testing is commonly used in:

User experience testing (e.g., comparing different versions of a website).

ML model evaluation (e.g., comparing two AI-based classifiers).

Performance assessment (e.g., determining if a new recommendation algorithm is more effective).

This approach allows for data-driven decisions, ensuring that any changes to the system result in meaningful improvements.

Reference from ISTQB Certified Tester AI Testing Study Guide:

Section 9.4 - A/B Testing states that A/B testing is used to compare updates in AI-based systems to determine if the newer version is better.

The problem describes a classification model that predicts whether a patient is at risk for a stroke. The confusion matrix is provided, and the testers have calculated a performance metric as 2/3 or 0.6667.

From the ISTQB Certified Tester AI Testing (CT-AI) Syllabus, the definitions of functional performance metrics from a confusion matrix include:

Accuracy:

Accuracy=TP+TNTP+TN+FP+FNAccuracy = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN

Measures the proportion of correctly classified instances (both true positives and true negatives) over the total dataset.

If the value is 0.6667, it suggests that the metric includes both correct positive and negative classifications, aligning with accuracy.

Precision:

Precision=TPTP+FPPrecision = \frac{TP}{TP + FP}Precision=TP+FPTP

Measures how many predicted positive cases were actually positive.

Does not match the given calculation.

Recall (Sensitivity):

Recall=TPTP+FNRecall = \frac{TP}{TP + FN}Recall=TP+FNTP

Measures how many actual positives were correctly identified.

Does not match the 0.6667 value.

F1-Score:

F1=2PrecisionRecallPrecision+RecallF1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}F1=2Precision+RecallPrecisionRecall

A balance between precision and recall.

The formula is different from the provided calculation.

Since the formula for accuracy matches the calculated value of 0.6667, the best answer is D. Accuracy.

Certified Tester AI Testing Study Guide

Reference:

ISTQB CT-AI Syllabus v1.0, Section 5.1 (Confusion Matrix and Functional Performance Metrics)

ISTQB CT-AI Syllabus v1.0, Section 5.4 (Selecting ML Functional Performance Metrics)

Explainability in AI-based systems refers to the ease with which users can determine how the system reaches a particular result. It is a crucial aspect when demonstrating AI decision-making, as it ensures that decisions made by AI models are transparent, interpretable, and understandable by stakeholders.

Before deploying an AI-based system, a developer must validate how decisions are made in a test environment. This process falls under the characteristic of explainability because it involves clarifying how an AI model arrives at its conclusions, which helps build trust in the system and meet regulatory and ethical requirements.

Supporting Reference from ISTQB Certified Tester AI Testing Study Guide:

ISTQB CT-AI Syllabus (Section 2.7: Transparency, Interpretability, and Explainability)

'Explainability is considered to be the ease with which users can determine how the AI-based system comes up with a particular result'.

'Most users are presented with AI-based systems as 'black boxes' and have little awareness of how these systems arrive at their results. This ignorance may even apply to the data scientists who built the systems. Occasionally, users may not even be aware they are interacting with an AI-based system'.

ISTQB CT-AI Syllabus (Section 8.6: Testing the Transparency, Interpretability, and Explainability of AI-based Systems)

'Testing the explainability of AI-based systems involves verifying whether users can understand and validate AI-generated decisions. This ensures that AI systems remain accountable and do not make incomprehensible or biased decisions'.

Contrast with Other Options:

Autonomy (B): Autonomy relates to an AI system's ability to operate independently without human oversight. While decision-making is a key function of autonomy, the focus here is on demonstrating the reasoning behind decisions, which falls under explainability rather than autonomy.

Self-learning (C): Self-learning systems adapt based on previous data and experiences, which is different from making decisions understandable to humans.

Non-determinism (D): AI-based systems are often probabilistic and non-deterministic, meaning they do not always produce the same output for the same input. This can make testing and validation more challenging, but it does not relate to explaining the decision-making process.

Conclusion: Since the question explicitly asks about the characteristic under which decision-making falls when being demonstrated before deployment, explainability is the correct choice because it ensures that AI decisions are transparent, understandable, and accountable to stakeholders.

According to the ISTQB Certified Tester AI Testing (CT-AI) syllabus, data acquisition, a critical step in data preparation for machine learning (ML) workflows, consists of three key activities:

Identification: This step involves determining the types of data required for training and prediction. For example, in a self-driving car application, data types such as radar, video, laser imaging, and LiDAR (Light Detection and Ranging) data may be identified as necessary sources.

Gathering: After identifying the required data types, the sources from which the data will be collected are determined, along with the appropriate collection methods. An example could be gathering financial data from the International Monetary Fund (IMF) and integrating it into an AI-based system.

Labeling: This process involves annotating or tagging the collected data to make it meaningful for supervised learning models. Labeling is an essential activity that helps machine learning algorithms differentiate between categories and make accurate predictions.

These activities ensure that the data is suitable for training and testing machine learning models, forming the foundation of data preparation.

Exploratory Data Analysis (EDA) is an essential technique for examining datasets to uncover patterns, trends, and anomalies, including outliers. In this case, the attacker manipulates the spam filter by injecting emails with red flags and masking them as internal company emails. The primary goal of EDA here is to detect these adversarial modifications.

Detecting Outliers:

EDA techniques such as statistical analysis, clustering, and visualization can reveal patterns in email metadata (e.g., sender details, email content, frequency).

Outlier detection methods like Z-score, IQR (Interquartile Range), or machine learning-based anomaly detection can identify emails that significantly deviate from typical internal communications.

Identifying Distribution Shifts:

By analyzing the frequency and characteristics of emails flagged as spam, testers can detect if the attack has introduced unusual patterns.

If a surge of internal emails is suddenly classified as spam, EDA can help verify whether these classifications are consistent with historical data.

Feature Analysis for Adversarial Patterns:

EDA enables visualization techniques such as scatter plots or histograms to distinguish normal emails from manipulated ones.

Examining email metadata (e.g., changes in headers, unusual wording in email bodies) can reveal adversarial tactics.

Counteracting Adversarial Attacks:

Once anomalies are identified, the spam filter's detection rules can be improved by retraining the model on corrected datasets.

The adversarial examples can be added to the training data to enhance the robustness of the filter against future attacks.

Reference from ISTQB Certified Tester AI Testing Study Guide

Exploratory Data Analysis (EDA) is used to detect outliers and adversarial attacks. 'EDA is where data are examined for patterns, relationships, trends, and outliers. It involves the interactive, hypothesis-driven exploration of data.'

EDA can identify poisoned or manipulated data by detecting anomalies and distribution shifts. 'Testing to detect data poisoning is possible using EDA, as poisoned data may show up as outliers.'

EDA helps validate ML models and detect potential vulnerabilities. 'The use of exploratory techniques, primarily driven by data visualization, can help validate the ML algorithm being used, identify changes that result in efficient models, and leverage domain expertise.'

Thus, option A is the correct answer, as EDA is specifically useful for detecting outliers, which can help identify manipulated spam emails.