Anomaly Pipeline Architecture

Let’s say you work as a machine learning engineer at Datadog. Your team is tasked with building a pipeline that processes a continuous stream of server logs from customer environments to enable anomaly detection. A trained anomaly detection model is already available.

How would you design the end-to-end data pipeline to support this use case?