Systemd Service and Socket Activation
In today’s Yak Shaving session I needed to understand how to expose the docker socket of a remote machine over the network. You should not do that, it is totally insecure, but I needed to do that to test something.
Socket Activation
I discovered that dockerd is running with -H fd://.
# ps axuwww | grep docker[d]
root 1616732 0.5 0.1 2930892 52168 ? Ssl 15:32 2:25 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
That is happening in the docker.service definition for Docker:
# systemctl cat docker.service | egrep '(service$|ExecStart)'
# /lib/systemd/system/docker.service
After=network-online.target firewalld.service containerd.service
Wants=containerd.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
That is, /lib/systemd/system/docker.service defines activation through the fd:// file descriptor.
That descriptor is in turn provided by the docker.socket unit, which looks like this:
root@server:~# systemctl cat docker.socket
# /lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
Both units are in /lib/systemd/system, which means they are OS-provided and should not be directly edited.
Overriding
I could override the service with systemctl edit docker.service, and provide a different ExecStart.
Since that is a list, I need to empty it first, and then put a new definition in.
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd
This starts docker without options, and I could define one or more sockets in /etc/docker/daemon.json.
It would also drop systemd socket activation, though.
That led me to the question “How does one actually write a daemon that cooperates with systemctl socket activation?” and also to the question “Can I have socket activation listen to more than one port, for example a Unix Domain Socket and a TCP-Socket?”
Writing a Python daemon with socket activation
Let’s write a simple daemon:
#! /usr/bin/env python3
from socketserver import TCPServer, StreamRequestHandler
import socket
import logging
class MyDaemon(StreamRequestHandler):
def handle(self):
logging.info(f"Connection from {self.client_address}.")
self.data = self.rfile.readline().strip()
self.data = self.data.decode("utf-8")
response = str(self.data[::-1])
logging.info(f"Data: {self.data} Response: {response}")
self.wfile.write(response.encode("utf-8"))
class Server(TCPServer):
def __init__(self, server_address, hnd):
# call superclass, but bind_and_activate is done by systemd.
# If we set that to True, it could run as a standalone program.
TCPServer.__init__(self, server_address, hnd, bind_and_activate=False)
# take socket passed on from systemd (3, first after stdin, stdout, stderr)
self.socket = socket.fromfd(self.3, self.address_family, self.socket_type)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
HOST, PORT = "127.0.0.1", 12345
server = Server((HOST, PORT), Handler)
server.serve_forever()
If we set bind_and_activate= to True, that thing would run from the command line as a standalone server.
$ ./demoserver.py # now telnet 127.0.0.1 12345 to test
Instead, we need to set this up with systemd.
A service and a socket unit
We need to set up a service definition unit. Since we chose a high port number (>1024), we can run it as a user without privileges.
Systemd is actually running a copy of itself for each logged-in user:
$ ps axuwww | grep system[d] | grep user
kris 327983 0.1 0.0 20144 11948 ? Ss Nov06 31:11 /lib/systemd/systemd --user
gdm 2419957 0.1 0.0 18768 10884 ? Ss Nov21 13:03 /lib/systemd/systemd --user
So we define a service unit using systemctl --user --full --force kris.service and then do the same for a socket unit:
$ systemctl --user cat kris.service
# /home/kris/.config/systemd/user/kris.service
[Unit]
Description=Kris Service
After=network.target kris.socket
Requires=kris.socket
[Service]
Type=simple
ExecStart=/usr/bin/python3 %h/Python/systemd-socketserver/demoserver.py
TimeoutStopSec=5
[Install]
WantedBy=default.target
Our service is declared as having a Requires= relationship to its socket.
And because relationships do not define ordering, we also put the sockets into the After= relationship.
The service is a simple service, so systemd will simply fork off the defined binary.
Ths service definition uses %h, which is replaced by the users home directory.
We also define a timeout.
Now the socket, on which we depend:
$ systemctl --user cat kris.socket
# /home/kris/.config/systemd/user/kris.socket
[Unit]
Description=Kris Socket
PartOf=kris.service
[Socket]
ListenStream=127.0.0.1:12345
[Install]
WantedBy=default.target
The PartOf= makes sure the socket is started and stopped together with the server.
The ListenStream= is the port systemd will open, bind to and listen on.
The socket produced by systemd will them be handed off to us on FD 3.
Starting the service
It is sufficient to work on kris.service to start and stop things, because of the relationships we defined between socket and service.
$ systemctl --user show kris.service| grep kris
ExecStart=...
ExecStartEx=...
WorkingDirectory=!/home/kris
Id=kris.service
Names=kris.service
Requires=kris.socket basic.target app.slice
ConsistsOf=kris.socket
After=kris.socket basic.target network.target app.slice
TriggeredBy=kris.socket
FragmentPath=/home/kris/.config/systemd/user/kris.service
The ConsistsOf= here comes from the PartOf= defined in the socket.
After running systemctl --user start kris.service we will see the service running using lsof:
$ lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 327983 kris 36u IPv4 2390840115 0t0 TCP 127.0.0.1:12345 (LISTEN)
python3 1957657 kris 3u IPv4 2390840115 0t0 TCP 127.0.0.1:12345 (LISTEN)
python3 1957657 kris 5u IPv4 2390840115 0t0 TCP 127.0.0.1:12345 (LISTEN)
Testing the service
We can now test:
$ echo "I am a test" | netcat 127.0.0.1 12345; echo
tset a ma I
And the log shows:
$ journalctl --user-unit kris.service| tail -5
Nov 27 23:58:48 server systemd[327983]: Started Kris Service.
Nov 27 23:59:59 server python3[1957657]: INFO:root:Connection from ('127.0.0.1', 57512).
Nov 27 23:59:59 server python3[1957657]: INFO:root:Data: I am a test Response: tset a ma I
Nov 28 00:00:05 server python3[1957657]: INFO:root:Connection from ('127.0.0.1', 57528).
Nov 28 00:00:05 server python3[1957657]: INFO:root:Data: I am a test Response: tset a ma I
Listening on two ports
We can now change the socket unit:
(systemd-socketserver) kris@server:~$ systemctl --user cat kris.socket
# /home/kris/.config/systemd/user/kris.socket
[Unit]
Description=Kris Socket
PartOf=kris.service
[Socket]
ListenStream=127.0.0.1:12345
ListenStream=127.0.0.1:23456
[Install]
WantedBy=default.target
When we restart the service, this is being reflected in the systemd running, but only partially in the service running:
$ systemctl --user stop kris.service
$ systemctl --user start kris.service
$ lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 327983 kris 35u IPv4 2391279244 0t0 TCP 127.0.0.1:12345 (LISTEN)
systemd 327983 kris 37u IPv4 2391279245 0t0 TCP 127.0.0.1:23456 (LISTEN)
python3 1961235 kris 3u IPv4 2391279244 0t0 TCP 127.0.0.1:12345 (LISTEN)
python3 1961235 kris 4u IPv4 2391279245 0t0 TCP 127.0.0.1:23456 (LISTEN)
python3 1961235 kris 6u IPv4 2391279244 0t0 TCP 127.0.0.1:12345 (LISTEN)
So we also need to adjust our binds (or, in the case of docker, the daemon.json).