Simple Platform Engineering and Simply High Cardinality Observability

Platform engineering transforms repetitive Engineering requirements into standardised and dependable services, or "paved roads". These paved roads offer engineering teams a smooth way to develop and deploy business logic layers continuously. Monitoring and Observability is one such platform workflow.

Compared to other components like Kafka, Spark, HDFS, etc., observability and monitoring have not had the luxury of warehouse-grade infrastructure components that can scale with the Engineering needs. In the absence of this "paved road", teams end up with a patchwork of solutions and siloed data or, worse, no solutions at all.

Observability heavily relies on metrics as a crucial aspect, providing a quick and cost-effective approach to addressing software health and SDLC queries. However, as the needs increase, so does the cardinality. This talk discusses building Observability platforms and workflows to manage the pulse of High Cardinality to solve use cases like combating Noisy Neighbors and being able to observe performance across millions of concurrent users.