ExamGecko
Question list
Search
Search

List of questions

Search

Related questions











Question 260 - MLS-C01 discussion

Report
Export

An insurance company developed a new experimental machine learning (ML) model to replace an existing model that is in production. The company must validate the quality of predictions from the new experimental model in a production environment before the company uses the new experimental model to serve general user requests.

Which one model can serve user requests at a time. The company must measure the performance of the new experimental model without affecting the current live traffic

Which solution will meet these requirements?

A.
A/B testing
Answers
A.
A/B testing
B.
Canary release
Answers
B.
Canary release
C.
Shadow deployment
Answers
C.
Shadow deployment
D.
Blue/green deployment
Answers
D.
Blue/green deployment
Suggested answer: C

Explanation:

The best solution for this scenario is to use shadow deployment, which is a technique that allows the company to run the new experimental model in parallel with the existing model, without exposing it to the end users. In shadow deployment, the company can route the same user requests to both models, but only return the responses from the existing model to the users.The responses from the new experimental model are logged and analyzed for quality and performance metrics, such as accuracy, latency, and resource consumption12. This way, the company can validate the new experimental model in a production environment, without affecting the current live traffic or user experience.

The other solutions are not suitable, because they have the following drawbacks:

A: A/B testing is a technique that involves splitting the user traffic between two or more models, and comparing their outcomes based on predefined metrics.However, this technique exposes the new experimental model to a portion of the end users, which might affect their experience if the model is not reliable or consistent with the existing model3.

B: Canary release is a technique that involves gradually rolling out the new experimental model to a small subset of users, and monitoring its performance and feedback.However, this technique also exposes the new experimental model to some end users, and requires careful selection and segmentation of the user groups4.

D: Blue/green deployment is a technique that involves switching the user traffic from the existing model (blue) to the new experimental model (green) at once, after testing and verifying the new model in a separate environment.However, this technique does not allow the company to validate the new experimental model in a production environment, and might cause service disruption or inconsistency if the new model is not compatible or stable5.

References:

1:Shadow Deployment: A Safe Way to Test in Production | LaunchDarkly Blog

2:Shadow Deployment: A Safe Way to Test in Production | LaunchDarkly Blog

3:A/B Testing for Machine Learning Models | AWS Machine Learning Blog

4:Canary Releases for Machine Learning Models | AWS Machine Learning Blog

5:Blue-Green Deployments for Machine Learning Models | AWS Machine Learning Blog

asked 16/09/2024
Sriharsha Janga
43 questions
User
Your answer:
0 comments
Sorted by

Leave a comment first