1. Launch the Model Dashboard of Taxi fare model where you can see a dashboard with different sections including Latest Performance, Prediction Metrics, Alert Summary, Alerts list, Policy list and Segment list.
2. Check out the Latest Performance on the upper left corner and notice the drop in performance that occurred a few days earlier.
3. Change the date range selector on upper right corner to Last 30 Days to check whether any performance drop occurred in the past month as well. Change back to Last 7 Days.
4. Explore the inference traffic in Prediction Metrics over time and note that the average daily traffic in the last 7 days is fairly consistent.
Performance and Prediction Metrics for steps 2-4
5. Go to Alert Summary, check the distribution of alerts across 4 different perspectives: severity, policy type, policy, and segment. Mouse over the donut charts to see more details.
6. Go to Alerts list, sort the alerts by Severity to view all critical alerts and you will see two alerts from the MAE performance policy as well as several critical alerts in the Week-to-week drift w/segments policy (Policy Name column) and Williamsburg-Manhattan Segment (Segment column). Review the Target window and Baseline window to see the difference in date range.
7. Scroll down to the Policy List to see the 4 policies created for this model; Recent Runs provides users quick insight into whether any critical (red dot) or warning alert (orange dot) was triggered in the last runs. For example, you may notice the Week-to-week drift w/segments policy has many alerts in the recent runs, but Month-to-month drift policy has neither warnings nor critical alerts.
8. Click the MAE performance policy to explore model performance further.
9. Expand the Policy Information section to see the policy type is “performance drift” and it compares the current day’s performance to the prior day’s performance, and the threshold for alerts is set at 20 for warnings and 50 for critical alerts.
10. Hide Policy Information. Scroll down to the Percentage change in mae chart, click on the circle where you see the biggest drop in performance. This updates other charts on the page: the alert on the left becomes highlighted, the Performance Details table underneath displays the actual performance of the selected date and performance change compared to the prior day, and Williamsburg-Manhattan segment is highlighted.
11. On the Performance Trend chart, you can see the actual performance trend for all data as well as for the two segments – Williamsburg-Manhattan and Brooklyn-Manhattan. Notice Williamsburg-Manhattan has a significant performance drop while the performance is stable in the Brooklyn-Manhattan segment. The performance in the Williamsburg-Manhattan segment is significant enough to have an impact to All Data .
12. Now let’s find out what potentially caused performance drop. Scroll down to the Related Policies at the bottom and open the Week-to-week w/segment policy which has many critical alerts. (Note that other policies can also be accessed by clicking the Policies link on the left navigation bar of the Model Dashboard.)
13. In the Week-to-week drift w/segment policy, expand Policy Information, you can see this policy type is feature drift, the Target window is week to date, and the Baseline window is past 2 weeks.
14. Go to Overall Feature Drift chart, you can see the drift of Williamsburg-Manhattan segment continues to increase while Brooklyn-Manhattan segment and All Data keeps relatively flat.
15. Click the circle of Williamsburg-Manhattan segment which has a high drift value and note that the charts below are refreshed based on the selection to reflect the corresponding target and baseline window in this segment as well as the level of drift for each feature in this policy.
16. Click feature est_trip_distance, and view the Value Distribution chart which shows the value distribution comparison of the target (current week to date) and baseline (last 2 weeks) for this feature. You will notice the est_trip_distance in the target window shows the majority of trips were longer in distance while baseline window has a more equal distribution across short and long distances.
17. Scroll back to the Overall Feature Drift chart, click the circle on the Williamsburg-Manhattan segment for the day prior to when the big drift was first observed; select est_trip_distance feature again in the feature list, notice the Value Distribution chart shows a similar distribution for the feature between target and baseline window.
18. Select other circles for All Data or Brooklyn-Manhattan segment, repeat the same steps as above by selecting one feature in the list and view the comparison of the value distribution between target and baseline window to observe the difference.
19. Drill down further to understand when drift occurred, and identify factors that point to why, open the Day-to-day drift w/segment policy by clicking this policy in Related policies.
20. Notice the spike when drift first occurs; note the downward line the next day that indicates the value distribution of features continues with the new normal (i.e., the level of drift remained at the same high level as the prior day.)
21. Select a circle in the Overall Feature Drift and perform the same observations as you did in the Week-to-week drift w/segment policy; select a feature and check out the value distribution of this feature in the target window in comparison to baseline window.
Based on all the metrics we have observed, we know there was a significant change in the Williamsburg–Manhattan segment days ago, and the new pattern continues. A quick internet search reveals that a one week construction project began on the day drift was first observed, and the closure of the Williamsburg Bridge resulted in much longer trip distance and trip time from Williamsburg to Manhattan. As this is a temporary problem, you do not need to retrain your model and instead simply add a courtesy pop-up message on the taxi app to alert customers to higher travel times during the next few days due to construction.
We hope this tutorial was helpful. If you have any questions, please don’t hesitate to reach out! You can reach us at email@example.com