spawn
=====

What is spawn?
---------------

``spawn`` is a suite which provides services to the process spawner of
various other daemons, e.g. `beng-proxy
<https://github.com/CM4all/beng-proxy/>`__ and `Workshop
<https://github.com/CM4all/workshop/>`__.


The Accessory Daemon
--------------------

This daemon listens for seqpacket connections on abstract socket
:file:`@cm4all-spawn` and allow clients to create namespaces
(`protocol definition
<https://github.com/CM4all/libcommon/blob/master/src/spawn/accessory/Protocol.hxx>`__).
The client is usually the process spawner of daemons like `beng-proxy
<https://github.com/CM4all/beng-proxy/>`__ and `Lukko
<https://github.com/CM4all/lukko/>`__.


The Reaper Daemon
-----------------

This daemon watches cgroups below certain scopes; once they run empty,
statistics are collected and logged and the cgroup is deleted.


``SIGHUP``
^^^^^^^^^^

On ``systemctl reload cm4all-spawn-reaper`` (i.e. ``SIGHUP``), the
daemon calls the Lua function ``reload`` if one was defined.  It is up
to the Lua script to define the exact meaning of this feature.


Resource Accounting
^^^^^^^^^^^^^^^^^^^

.. highlight:: lua

The file :file:`/etc/cm4all/spawn/reaper.lua` is a `Lua
<http://www.lua.org/>`_ script which is executed at startup.  If it
defines a function called ``cgroup_released``, then this function will
be called every time a cgroup is reaped.  The function may for example
log its resource usage.  Example::

  function cgroup_released(cgroup)
    print(cgroup.memory_peak)
  end

The following attributes of the ``cgroup`` parameter can be queried:

* ``path``: the cgroup path as noted in :file:`/proc/self/cgroup`,
  e.g. :file:`/user.slice/user-1000.slice/session-42.scope`

* ``btime``: The time the cgroup was created as `Lua timestamp
  <https://www.lua.org/pil/22.1.html>`__; may be ``nil`` if the kernel
  does not support ``btime`` on ``cgroupfs``.

* ``age``: The age of this cgroup in seconds.  Only available if
  ``btime`` is.

* ``xattr``: A table containing extended attributes of the control
  group.

* ``parent``: Information about the parent of this cgroup; it is
  another object of this type (or ``nil`` if there is no parent
  cgroup).

* ``cpu_total``, ``cpu_user``, ``cpu_system``: the total,
  userspace-only or kernel-only CPU usage [in seconds].

* ``memory_peak``: the peak memory usage [in bytes].

* ``memory_events_high``: The number of times processes of the cgroup
  are throttled and routed to perform direct memory reclaim because
  the high memory boundary was exceeded.

* ``memory_events_max``: The number of times the cgroup's memory usage
  was about to go over the max boundary.

* ``memory_events_oom``: The number of time the cgroup's memory usage
  was reached the limit and allocation was about to fail.

* ``pids_peak``: the peak number of processes.

* ``pids_forks``: the number of ``fork()`` system calls

* ``pids_events_max``: the number of times the ``pids.max`` setting
  was exceeded.


Addresses
^^^^^^^^^

It is recommended to create all `address` objects during startup, to
avoid putting unnecessary pressure on the Lua garbage collector, and
to reduce the overhead for invoking the system resolver (which blocks
*Passage* execution).  The function `control_resolve()` creates such an
`address` object::

  server1 = control_resolve('192.168.0.2')
  server2 = control_resolve('[::1]:4321')
  server3 = control_resolve('server1.local:1234')
  server4 = control_resolve('/run/server5.sock')
  server5 = control_resolve('@server4')

These examples do the following:

- convert a numeric IPv4 address to an `address` object (port defaults
  to 5478, the *beng-proxy* control standard port)
- convert a numeric IPv6 address with a non-standard port to an
  `address` object
- invoke the system resolver to resolve a host name to an IP address
  (which blocks passage startup; not recommended)
- convert a path string to a "local" socket address
- convert a name to an abstract "local" socket address (prefix '@' is
  converted to a null byte, making the address "abstract")


socket
^^^^^^

A simple low-level networking library.  Example::

  tcp = socket:connect('localhost:1234')
  udp = socket:connect('localhost:4321', {type='dgram'})
  multicast = socket:connect('[ff02::dead:beef]:2345', {type='dgram'})
  unix = socket:connect('/run/test.socket')
  abstract = socket:connect('@test', {type='seqpacket'})
  abstract:send('hello world')

The ``socket`` library has the following methods:

- ``connect(ADDRESS, [OPTIONS])``: Create a new socket connected to
  the specified address.  ``OPTIONS`` may be a table with the
  following keys:

  - ``type``: the socket type, one of ``stream`` (the default),
    ``dgram``, ``seqpacket``.

  Returns a new socket object on success or ``[nil,error]`` on error.

Socket objects have the following methods:

- ``close()``: Close the socket.

- ``send(DATA, [START], [END])``: Send data (i.e. a string) to the
  peer.  ``START`` and ``END`` are start and end position within the
  string with the same semantics as in ``string.sub()``.  Returns the
  number of bytes sent on success or ``[nil,error]`` on error.


control_client
^^^^^^^^^^^^^^

A client for the `beng-proxy control protocol
<https://beng-proxy.readthedocs.io/en/latest/control.html>`__.

During startup, create a ``control_client`` object::

  -- IPv4 (default port)
  c = control_client:new('224.0.0.42')

  -- IPv6 on default port
  c = control_client:new('ff02::dead:beef')

  -- IPv6 on non-default port (requires square brackets)
  c = control_client:new('[ff02::dead:beef]:1234')

  -- local socket
  c = control_client:new('/run/cm4all/workshop/control')

  -- abstract socket
  c = control_client:new('@bp-control')

The ``new()`` constructor returns ``nil,error`` on error (and thus the
call can be wrapped in ``assert()`` to raise a Lua error instead).

The method ``build()`` creates an object which can be used to build a
control datagram with one or more commands.  After that datagram has
been assembled, it can be sent with the ``send()`` method.  Example::

  c:send(c:build():fade_children('foo'):flush_http_cache('bar'))

The ``send()`` method returns ``nil,error`` on error.

The builder implements the following methods:

- ``cancel_job(PARTITION_NAME, JOB_ID)``
- ``discard_session(ID)``
- ``disconnect_database(ACCOUNT)``
- ``fade_children(TAG)``
- ``flush_filter_cache(TAG)``
- ``flush_http_cache(TAG)``
- ``reject_client(ADDRESS)``
- ``reset_limiter(ACCOUNT)``
- ``tarpit_client(ADDRESS)``
- ``terminate_children(TAG)``


libsodium
^^^^^^^^^

There are some `libsodium <https://www.libsodium.org/>`__ bindings.

`Helpers <https://doc.libsodium.org/helpers>`__::

  bin = sodium.hex2bin("deadbeef") -- returns "\xde\xad\xbe\ef"
  hex = sodium.bin2hex("A\0\xff") -- returns "4100ff"

`Generating random data
<https://doc.libsodium.org/generating_random_data>`__::

  key = sodium.randombytes(32)

`Sealed boxes
<https://libsodium.gitbook.io/doc/public-key_cryptography/sealed_boxes>`__::

  pk, sk = sodium.crypto_box_keypair()
  ciphertext = sodium.crypto_box_seal('hello world', pk)
  message = sodium.crypto_box_seal_open(ciphertext, pk, sk)

`Point*scalar multiplication
<https://doc.libsodium.org/advanced/scalar_multiplication>__::

  pk = sodium.crypto_scalarmult_base(sk)


PostgreSQL Client
^^^^^^^^^^^^^^^^^

The Lua script can query a PostgreSQL database.  First, a connection
should be established during initialization::

  db = pg:new('dbname=foo', 'schemaname')

In the handler function, queries can be executed like this (the API is
similar to `LuaSQL <https://keplerproject.github.io/luasql/>`__)::

  local result = assert(db:execute('SELECT id, name FROM bar'))
  local row = result:fetch({}, "a")
  print(row.id, row.name)

Query parameters are passed to ``db:execute()`` as an array after the
SQL string::

  local result = assert(
    db:execute('SELECT name FROM bar WHERE id=$1', {42}))

The functions ``pg:encode_array()`` and ``pg:decode_array()`` support
PostgreSQL arrays; the former encodes a Lua array to a PostgreSQL
array string, and the latter decodes a PostgreSQL array string to a
Lua array.

To listen for `PostgreSQL notifications
<https://www.postgresql.org/docs/current/sql-notify.html>`__, invoke
the ``listen`` method with a callback function::

  db:listen('bar', function()
    print("Received a PostgreSQL NOTIFY")
  end)


Network Namespaces
------------------

The Debian package :file:`cm4all-spawn-netns` contains the systemd
service template :file:`cm4all-spawn-netns@.service` which creates a
new network namespace connected with the current namespace over a pair
of ``veth`` devices.  This requires a script in
:file:`/etc/cm4all/spawn/netns/setup.d` which sets up the ``veth``
device inside the new namespace; its name is passed as command-line
argument.  The other ``veth`` device is expected to be set up with
:file:`systemd-networkd`.


Slice
-----

The Debian package :file:`cm4all-slice` contains the systemd slice
``system-cm4all.slice`` where the scopes of most process spawners
live.


NSS-LogName
-----------

The NSS module was moved to https://github.com/CM4all/nss_logname