Popular Products

You are a data engineer at Delta Retail, an online retail platform that processes billions of daily events across search, clicks, carts, and payments. Recently, several analytics pipelines running on Hadoop MapReduce - including “most viewed products”, “daily active users”, and “conversion-rate by category” - have started missing SLAs. After investigation, the team found that a small number of products and power users generate a disproportionate share of all events, causing strong key skew in intermediate data.

Explain, using the MapReduce model,

1. Why key skew poses a performance bottleneck in Delta Retail’s large-scale analytics jobs?
2. How skew affects the behavior of the map, shuffle, and reduce phases, especially reducer hotspots and long tail completion times?
3. Which mitigation strategies you would apply to solve this issue?

Note: SLA in this context refers to the expected end-to-end completion time of the MapReduce analytics jobs. When key skew slows down a few reducers, the entire job exceeds this completion time, causing an SLA miss.