So you are running systems in production and you want to collect data from your systems. You need to build a monitoring system.
That won’t work and it won’t scale. So please stop for a moment, and think.
What kind of monitoring do you want do build? I know at least three different types of monitoring system, and they have very different objectives, and consequently designs.
Three types of Monitoring Systems
The first and most important system you want to have is checking for incidents. This Type 1 monitoring is basically a transactional monitoring system: