Die wunderbare Welt von Isotopp

Trying lvmraid for real

Kristian Köhntopp - December 3, 2019
So after testing LVM Raid in princple, I have been trying it on some real hardware to see what happens. The idea was to estimate if it scales and if not, how it doesn’t. I was expecting to run into all kinds of obscure problems in my testing, but in fact, it was a quick and short death. Here is my box: QuantaGrid D42A-2U with an AMD EPYC 7551P CPU (32C, HT off), 1024 GB of memory, a boot disk and 12x Micron_9200_MTFDHAL11TATCW (PDF ) for 120TB of disk storage.

Cloning and splitting logical volumes

Kristian Köhntopp - December 2, 2019
Where I work, we routinely run our databases on XFS on LVM2. The setup Each database has their database on /mysql/schemaname, with the subdirectories /mysql/schemaname/{data,log,tmp}. The entire /mysql/schemname tree is a LVM2 Logical Volume mysqlVol on the Volume Group vg00, which is then formatted as an XFS filesystem. # pvcreate /dev/nvme*n1 # vgcreate vg00 /dev/nvme*n1 # lvcreate -n mysqlVol -L...G vg00 # mkfs -t xfs /dev/vg00/mysqlVol # mount -t xfs /dev/vg00/mysqlVol /mysql/schemaname Basic Ops You can grow an existing LVM Logical Volume with lvextend -L+50G /dev/vg00/mysqlVol or similar, and then xfs_grow /dev/vg00/myqlVol.

Fertig gelesen: Exit Strategy

Kristian Köhntopp - November 22, 2019
Exit Strategy by Martha Wells is the fourth book in the the Murderbot Diaries (All Systems Red , Artificial Condition, Rogue Protocol, Exit Strategy, Network Effect). This novella is the culmination of the Murderbot Novellas, in which our nameless murderbot cyborg tries to nail the evil GrayCris corporation for the series of crimes against settlers and researchers, has to rescue it’s “owner”, sponsor and friend Dr. Mensah from the claws of GrayCris, and make friends on the way.

Fertig gelesen: Fashion and Its Social Agendas: Class, Gender, and Identity in Clothing

Kristian Köhntopp - November 21, 2019
Fashion and Its Social Agendas: Class, Gender, and Identity in Clothing by Diane Crane is an analysis of the changes in society during the various phases of the industrial revolution through the lens of clothing in various societal strata. The book studies fashion and clothing choices in three countries - France, the United States, and England - over a period of 150 years. France and England are examples of class societies in the nineteenth century.

Fertig gelesen: The Wiz Biz

Kristian Köhntopp - November 20, 2019
The Wiz Biz by Rick Cook is a book in the genre of ‘magical realism’. Some real-world person is being transported into a universe with predictable magic in a classical fantasy setting and is applying scientic method and industrial production methods to the setting. In this book, the real-world person is Walter Zumwalt, a stereotypical hacker, which as part of a hail-mary summoning rital is being transported into a magical war between the good guys and the bad guys.

When a file changes, do a thing

Kristian Köhntopp - November 19, 2019
When developing there is often an edit-compile-test cycle, or an edit-distribute-changes cycle or a similar repetetive task. You could poll changes, for example with cron every minute or similarly, but that is wasteful and slow. All modern operating systems have mechanisms for processes to subscribe to file or directory changes. In MacOS, we do have the File System Events API since 10.5, in Linux we got three different implementations (as described in LWN ): The original dnotify, its replacement inotify and the even more recent fanotify (which got its own LWN article ).

A blast from the past

Kristian Köhntopp - November 18, 2019
TL:DR: If you have long running transactions, MySQL does not deal well with this, and it will slow down the box. That’s okay as long as you are basically alone on your box, but if you aren’t, the others will hate you. The database machine ‘somehierarchy-02’ in a general purpose load balancer pool for somehierarchy had replication delay. It’s a MySQL replica and is receiving the same write workload than all the other boxen in that pool.

Fertig gelesen: Change Agent

Kristian Köhntopp - November 18, 2019
Change Agent is Daniel Suarez fifth book. As usual, he builds his stories around one technology or technological change and constructs an action-rich storyline around it. For “Change Agent”, this is CRISPR genetic modification of living beings. He basically imagines a programmable genetic agent that rewrites living beings including humans on the go - the eponymous change agent. It’s 2045, and an Interpol Agent Kenneth Durand is trying to pursue the Huli Jing cartel, a criminal black market corporation that produces enhanced designer babies for rich prospectice parents.

Fertig gelesen: Infinity Engine

Kristian Köhntopp - November 18, 2019
Infinity Engine is the third and final book in Neal Asher’s Transformation Sequence , a trilogy set in the Polity Universe . This is the conclusion of the three book arc about the redemption of the mis-manufactured and temporarily mad AI Penny Royal. Asher takes us through a set of encounters he prepared two other books for, contrasting the reconstruction and the redemption of Penny Royal with the descend into madness of the forensic AI Brockle, and weaving together all the other story-strands that he set up earlier into a literally star-shattering finale.

Everything was a file, but we got better

Kristian Köhntopp - November 14, 2019
I fell into the Twitters again. @CarrickDB joked about Unix, Files and Directories: And that is a case of “Haha, only serious”. Because directories used to be files, and that was a bad time. Check out the V7 Unix mkdir command. At this point in history we do not have a mkdir(2) syscall, yet, so we need to construct the entire directory in multiple steps. mknod(2) an inode that has the S_IFDIR flag set , even if that macro does not even exist yet.

Filling disk space fast

Kristian Köhntopp - November 11, 2019
Some of the databases at work are a tad on the large side, in the high 2-digit terabytes of size. Copying these to new machines at the moment takes a rather long time, multiple days, up to a week. Speeding it up pays twice, because with shorter copy times there is also less binlog to catch up. I have been looking into disk copy speeds in order to better understand the limits.

Climate change happened 30 years ago

Kristian Köhntopp - October 15, 2019
Water is a very weird substance. Melting ice at 0C to water at 0C takes a fantastic amount of energy: Water has a latent heat of 384J/gram. Compared to other materials that is pretty much at the upper end of the spectrum. Warming up 1g of liquid water by 1C is the original definition of 1 Calorie, or 4.184 Joules in todays units. The water on our planet is a buffer that can store energy if there is more energy incoming that the earth can dissipate, or release energy if the bilancing is negative.

Ignoring Catalina nags

Kristian Köhntopp - October 15, 2019
Note to self: Ignoring the Catalina Nagscreens is done like this: $ sudo softwareupdate --ignore "macOS Catalina" $ defaults write com.apple.systempreferences AttentionPrefBundleIDs 0 && killall Dock And that should make MacOS ignore Catalina until further notice. Further notice does look like this: $ /usr/sbin/softwareupdate --reset-ignored

Catalina: Upgrade carefully.

Kristian Köhntopp - October 8, 2019
Catalina is here. It fixes many things, among them iTunes. On the other hand, it drops support for 32-Bit MacOS applications. Consider the upgrade carefully. $ system_profiler SPApplicationsDataType … Steam: Version: 1.5 Obtained from: Identified Developer Last Modified: 23.08.16, 20:32 Kind: Intel 64-Bit (Intel): No Signed by: Developer ID Application: Valve Corporation, Developer ID Certification Authority, Apple Root CA Location: /Applications/Steam.app … Anything listed with 64-Bit (Intel): No will stop working after the upgrade.

Data Centers and Energy

Kristian Köhntopp - October 5, 2019
Deutsche Welle is shocked: Generation Greta is watching Netflix (Article in German Language), Netflix runs on computers, and apparently computers are using power. Let’s have a look. EDIT: Second part Streaming and Energy now available. End-User Device. It’s 2019. End user devices are using Wi-Fi, and are running on Batteries. They are Cellphones, Tablets or Laptops. Devices that are not connected to grid are more usable if they are trying to save energy, and so all modern devices are full of special purpose hardware that allows them to fulfill their main functions more energy efficient.

MySQL Does Disk I/O

Kristian Köhntopp - September 27, 2019
We had a discussion at work about MySQL doing Disk I/O to a NVME disk, and if it is valid to turn off the doublewrite buffer when using XFS. TL;DR: It’s not, you can turn off the doublewrite buffer only on filesystems that never do in-place updates (ZFS, btrfs), or that have their own doublewrite buffer (ext4 with journal=data). A flash layer underneath the actual filesystem is likely not going to help you without additional measures.

The Rockforth Fertilizer Incident

Kristian Köhntopp - September 26, 2019
So I have been playing this Internet Spaceships Game. No, the other one . It also has NPC Backstory, and on 13-Sep-3305 , a multiplanetary food crisis was predicted, as a prep for the introduction of a new tradeable good on 18-Sep-3005 . Distribution of the fertiliser has started at Marshall Dock in the Riedquat system. We encourage traders to take advantage of our introductory prices.” That went wrong.

MySQL Performance Limits

Kristian Köhntopp - September 6, 2019
The last time I saw a MySQL server operating at a performance limit was in 2012. Back them we had a production master on (then) current hardware, running stable at about 21000 QPS. At 24000 QPS it tended to become unstable and fall over, dying in global locks on the InnoDB Adaptive Hash Index or other global locks. I need to better understand how MySQL works today, and what the limits are on a box that is considered large in 2019.


Kristian Köhntopp - September 3, 2019
Ein Thread über Konsenssysteme aus Twitter Mehr als ein Key-Value Store Heise schreibt: Verteilter Key-Value-Store: etcd erreicht Version 3.4 . etcd ist auch ein Key-Value-Store, aber das ist nur ein Nebendetail. Die Beschreibung der neuen Funktionen im Artikel macht auch schon keinen Sinn für einen KV-Store. etcd ist ein Konsenssystem. Es realisiert Clustermitgliedschaft, verteilte Locks, und darauf aufbauende Dienste wie Service Discovery und Loadbalancing. Es gibt drei von Kyle Kingsbury getestete Implementierungen von Konsenssystemen, die funktionieren: Zookeeper, etcd und consul.

Ceph and NVME - not a good combination?

Kristian Köhntopp - July 16, 2019
Saving an older twitter thread on Ceph . I just have finished reading Part 4: RHCS 3.2 Bluestore Advanced Performance Investigation and now I do not know what to say. Ceph set out with the idea to make storage a commodity by using regular PCs + a lot of local hard disks + software to make inexpensive storage. For that, you ran a number of control processes and an OSD process per disk (“object storage demon”).