So you are running systems in production and you want to collect data from your systems. You need to build a monitoring system.
That won’t work and it won’t scale. So please stop for a moment, and think.
What kind of monitoring do you want do build? I know at least three different types of monitoring system, and they have very different objectives, and consequently designs.
The first and most important system you want to have is checking for incidents. This Type 1 monitoring is basically a transactional monitoring system:
AMS-IX press release: »After 17.5 years, Job Witteman will leave AMS-IX as of October 1st. Job Witteman, who was the founder of the world’s largest internet exchange, will leave at the peak of development of the company, […]«
»AMS-IX has grown to be the world’s largest internet exchange, having now more than 900 customers, operating 7 internet platforms globally and a peak of internet traffic of 5.5 Tbps. In addition, AMS-IX is considered by the Dutch Government as the third main port, together with Schiphol Airport and the port of Rotterdam. This shows that the ongoing development of the digital sector and infrastructure is taken seriously.«
Two weeks ago, I wrote about The Data Center in the Age of Abundance and claimed that IOPS are – among other things – a solved problem.
What does a solved problem look like? Here is a benchmark running 100k random writes of 4K per second, with zero Jitter, at 350µs end-to-end write latency across six switches.
Databases really like reliably timed writes like these.
Maximum queue depth would be 48, the system is not touching that.
So I have been testing, again.
My hapless test subject this time is a Dell Box, an R630.
It has a comfortable 384GB of memory, one of two 25 GBit/s ports active, and it comes with two E5-2690v4 CPUs. That gives it 14 cores per die, 28 cores in total, or with hyperthreading, 56 threads.
$ cat /proc/cpuinfo | grep 'model name' | uniq -c 56 model name : Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz $ ./mprime -v Mersenne Prime Test Program: Linux64,Prime95,v28.10,build 1
Over at Google plus, Jan Walzer made a photo of a few Thomas Krenn blades. The blades are water cooled, hot-swappable, and the waste heat is not actually wasted, but being used for heating (prototype for a DLR supercomputing project).
In the Optane article I have been writing about how persistent bit-addressable memory will be changing things, and how network latencies may becoming a problem.
The ACM article Attack of the Killer Microseconds has another, more general take on the problem. It highlights how we are prepared in our machines to deal with very short delays such as nanoseconds, and how we are also prepared to deal with very long delays such as milliseconds. It’s the waits inbetween, the network latencies, sleep state wakeups and SSD access waits, that are too short to do something else and too long to busy wait in a Spinlock.
The multiqueue block layer finally has support for I/O scheduling. That is useful in its own right, but the real news is that it enables the merging of the long-awaited BFQ I/O scheduler. That, says block maintainer Jens Axboe, “should be ready for 4.12”.
Of course, if you are on a LTS release of a Linux kernel, it’s unlikely that you will profit from this any time soon.
Following a great idea from their friends at GitLab, Soup.io loses all postings since 2015 because of malfunctioning backups. They write:
We had a big database crash, and the backups we had were corrupted.
The only working backup was from 2015.
Improving Recovery Procedures
9. Automated testing of recovering PostgreSQL database backups (#1102)
Does your database backup successfully restore? Are you sure? Are you testing this?
Remember these words of wisdom:
Nobody wants backup.
Everybody wants restore.
— Martin Seeger
I had the opportunity to play with a Blade Center Chassis with 16 Blades, each of them a Dual-E5 2690v4, so 56 threads (28 cores) times 16.
$ mkdir mprime; \ ] cd mprime; \ ] wget http://www.mersenne.org/ftp_root/gimps/p95v2810.linux64.tar.gz; \ ] tar xvzf p95v2810.linux64.tar.gz; \ ] ./mprime -m
running with “stress test only”, “mode 1 – small FFT” and 56 cores gets me quite a bit of power consumption.
Idle Blades is being reported as 140W, busy blades are 400W.
Images below the fold.