Log Anomaly Detection Model

Let’s say you work as a machine learning engineer at Datadog. Your team is tasked with building a model that detects anomalous patterns in a continuous stream of server logs collected from diverse customer environments.

These logs include system event messages, error reports, performance metrics, authentication records, and application-specific debug information. The logs are semi-structured, vary in format across customers, and are generated at high volume and velocity.

How would you approach building and validating this anomaly detection model?