The minimal network configuration supported on Scibian HPC clusters consists of two physically separated networks:

For performance reasons with distributed computing, HPC clusters generally have a third low-latency network. It is used for both I/O to the main storage system and distributed computing communications (typically MPI messages) between compute nodes. The hardware technologies of this network may vary upon performance requirements but it generally involves high bandwidth (10+GB/s) and low latency technologies such as InfiniBand, Omni-Path or 10GB Ethernet. In the absence of dedicated low-latency network, the backoffice network is also used for central storage system I/O and distributed computing communications.

It is recommended to split the backoffice network with four VLAN dedicated to the following groups of network interfaces:

In this setup, it is recommended to route IP subnetworks between the VLAN with L3 switche(s) on the backoffice network.

This setup has significant advantages both in terms of reliability and security:

The infrastructure cluster is composed by two types of nodes: the admin node and the generic service nodes.

The admin node is the access node for administrators and the central point of administrative operations. All common administrative actions are performed on this node. It does not run any intensive workloads, just simple short-lived programs and it does not need to be very powerful. It does not store sensible data nor run critical services, so it does not need to be very reliable either. Example of hardware specifications:

The generic service nodes run all critical infrastructure services (within service virtual machines) and manage all production administrative data. Scibian HPC requires a pool from 3 (minimum) to 5 (recommended) generic service nodes. The pool works in active cluster mode, the load is balanced with automatic fail-over. All generic service nodes of a cluster must be fairly identical for efficient load-balancing.

The generic service nodes manage the production data into a distributed object-storage system. It is highly recommended that the nodes have a dedicated block storage device for this purpose. The workload is mostly proportional to the number of compute nodes but the generic service nodes must be quite powerful to comfortably handle load peaks happening during some operations (ex: full cluster reboot). Also, since services are run into virtual machines, a fairly large amount of RAM is required. Services can generate a lot of traffic on the backoffice network, it is relevant to provide a network adapter with high bandwidth. Even though high-availability is ensured at the software level with automatic fail-over between generic service nodes, it is nevertheless recommended to get hardware redundancy on most devices of the generic service nodes to avoid always risky and hazardous service migrations as much as possible. Example of hardware specifications:

All physical nodes must be connected to all three physical networks. There are virtual bridges on the host of the generic service nodes connected to the WAN and backoffice networks. The service virtual machines have connections to the virtual bridges upon their hosted service requirements.

The user-space cluster is composed of frontend nodes and compute nodes.

The nodes of the user-space cluster are deployed with a diskless live system stored in RAM. It implies that, technically speaking, the nodes do not necessarily need to have local block storage devices.

The frontend nodes are the access hosts for users so they must be connected to all three physical networks. It is possible to have multiple frontend nodes in active cluster mode for load-balancing and automatic fail-over. The exact hardware specifications of the frontend nodes mostly depend on user needs and expectations. Users may need to transfer large amount of data to the cluster, it is therefore recommended to provide high-bandwidth network adapters for the WAN network. These nodes can also be designed to compile computational codes and in this case, they must be powerful in terms of CPU, RAM and local storage I/O.

The compute nodes run the jobs so they must provide high performances. Their exact hardware specifications totally depend on user needs. They must be connected to both the backoffice and the low-latency networks.

The storage system is designed to host user data. It provides one or several shared POSIX filesystems. The evolved storage technologies depend on user needs ranging from a simple NFS NAS to a complex distributed filesystem such as Lustre or GPFS with many SAN and I/O servers.

All services running on the cluster should be highly available (HA). Some services not critical for normal cluster operation can be not highly available, but this should be avoided if possible.

The following section lists the different techniques used to achieve high-availability of the cluster services.

By default on Scibian HPC clusters, the DHCP servers sends as filename to iPXE ROM an HTTP URL to a Python CGI script bootmenu.py which is a iPXE bootmenu generator. Optionally, this behaviour can be altered by modifying iscdhcp::bootmenu_url parameter in Hiera repository.

The Python CGI script bootmenu.py is provided by scibian-hpc-netboot-bootmenu package.

On the HTTP server side, this script initially parses the nodes boot parameters configuration YAML file /etc/scibian-hpc-netboot/boot-params.yaml. This file provides all node specific boot parameters, including ethernet boot device, default OS, media and version, etc. When looking for a parameter (ex: os), the script first searches into the nodeset sections whose node is member (ex: node fbcn02 is member of nodeset fbcn[01-10]). If not found, the parameter is finally read into the defaults section.

The /etc/scibian-hpc-netboot/boot-params.yaml file is deployed by Puppet-HPC based on the following inputs:

Then, the script compiles sequentially all the menu entries provided as YAML files in directory /etc/scibian-hpc-netboot/menu/entries.d. The YAML files must respect the following format:

Where:

An <os> can contain multiple <media>, a <media> can contain multiple <version>. An entry is defined by the concatenation of these 3 parameters, ex: scibian9-disk-main. Then, each entry is defined by the following parameters:

The ${base-url} is a iPXE placeholder defined by the CGI script for every entries. Its value mainly depends on the media of the entry:

The <*_server> parameters are defined in nodes boot parameters YAML configuration file /etc/scibian-hpc-netboot/boot-params.yaml.

The parameters of an entry can be templated with all node boot parameters and the OS, media, version and initrd of the entry. As an example, here is a valid entry:

All the parameters between double curly braces (ex: {{boot_dev}}) are dynamically replaced by node boot parameters. This way, entries can be defined in a generic way.

The YAML entries files in directory /etc/scibian-hpc-netboot/menu/entries.d are read sequentially. The entries provided in the next files can override entries defined in previous files. In other words, only the last definition of an entry is considered. As an example, the entry scibian9-disk-main defined in 0_default.yaml can be overriden in 1_other.yaml.

The scibian-hpc-netboot-menu provides default entries with file 0_default.yaml. All the entries defined in this file can be overriden with Puppet-HPC by setting the profiles::bootsystem::menu_entries hash parameter in Hiera repository.

By default on Scibian HPC clusters, the URL provided in the bootmenu entries for the Debian installer preseed (scibian*-disk-* entries) is actually directed to a Python CGI script preseedator.py. This behaviour can be altered by overriding the respective menu entries, please refer to iPXE Bootmenu Generator section for explanations.

This CGI script preseedator.py dynamically generate a preseed for Debian installer for the node given in parameter. This CGI script is provided by the scibian-hpc-netboot-preseedator package.

In the first place, the script reads the nodes boot parameters located in file /etc/scibian-hpc-netboot/boot-params.yaml. Please refer to iPXE Bootmenu Generator section to understand how this file is built.

Then, it parses its YAML configuration file /etc/scibian-hpc-netboot/installer/installer.yaml. This file basically contains all debian installer related parameters such as the URL to the APT mirror/proxy and the list of additional repositories. The content of this file is based on the following inputs:

Finally, the preseedator.py script generates the preseed based on the template file /etc/scibian-hpc-netboot/installer/preseed.jinja2. The template is filled with parameters previously loaded.

The template provides a mechanism to download an external partition schema file from the installation server (diskinstall_server in boot-params.yaml). The URL directs to an another Python CGI script partitioner.py. This script is also provided by scibian-hpc-netboot-preseedator package.

This script searches for a partition schema file in directory /etc/scibian-hpc-netboot/installer/schemas in the following order:

The first found file is returned by the script. By default, only the common file is provided by the package. With Puppet-HPC, it is possible to deploy node or role specific schemas by setting the boothttp::partition_schemas array in Hiera repository.

The frontend nodes offer a virtual IP address on the WAN network that features both an highly-available and load-balanced SSH service for users to access the HPC cluster. The load-balancing feature automatically distributes users on all available frontend nodes. This load-balancing is operated with persistence so that users (based on their source IP address) are always redirected to the same frontend node in a time frame. Behind the virtual IP address, the high-availability of the SSH service is also ensured in case of outage on a frontend node. These load-balancing and high-availability features are ensured by the Keepalived software.

For security reasons, a firewall is also set up on the frontend nodes to control outgoing network traffic. This firewall service is managed by Shorewall, a high-level configuration tool for Linux netfilter. Because of all the various network flows involved in Keepalived, it must be tightly integrated with the firewall rules. The following diagram illustrates both the network principles behind the high-availability/load-balancing mechanisms and the integration with the software components of the firewall:

The Keepalived sofware checks all the frontend nodes using the VRRP^[4] protocol on the WAN network interfaces (purple arrow in the diagram). This protocol must be allowed in the OUTPUT chain of the firewall so that Keepalived can work properly.

On the master frontend node, the HA virtual IP address is set on the network interface attached to the WAN network. The Keepalived software configures the IPVS^[5] Linux kernel load-balancer to redirect new TCP connections with a Round-Robin algorithm. Therefore, a part of the TCP connections is redirected to the sshd daemon of other frontend nodes (orange arrow in the diagram). An exception must be specified in the OUTPUT chain of the firewall to allow these redirected connections.

To perform such redirections, IPVS simply changes the destination MAC address, to set the address of the real destination frontend, in the Ethernet layer of the first packet of the TCP connection. However, the destination IP address does not change: it is still the virtual IP address.

On the slave frontend nodes, the HA virtual IP address is set on the loopback interface. This is required to make the kernel accept the redirected packets from the master frontend node addressed to the virtual IP address. In order to avoid endless loops, the IPVS redirection rules are disabled on slave frontend nodes or else, packets would be redirected endlessly.

By default, the Linux kernel answers the ARP requests coming from any network device for any IP address attached to any network device. For example, on a system with two network devices: eth0 with ip0 and eth1 with ip1, if an ARP request is received for ip1 on eth0, the kernel positively responds to it, with the MAC address of eth0. Though it is convenient in many cases, this feature is annoying on the frontend nodes, since the virtual IP address is set on all of them. Consequently all frontend nodes answer the ARP requests coming from the WAN default gateway. In order to avoid this behaviour, the net.ipv4.conf.<netif>.arp_ignore and net.ipv4.conf.<netif>.arp_announce sysctl Linux kernel parameters, where <netif> is the network interface connected to the WAN network, are respectively set to 1 and 2. Please refer to the Linux documentation for more details on these parameters and their values: http://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

The Keepalived software also checks periodically if the sshd service is still available on all frontend nodes by trying to perform a TCP connection to their real IP addresses on the TCP/22 port (green arrow in the diagram). An exception must be present in the OUPUT chain of the firewall to allow these connections.

There is an unexplained behaviour in the Linux kernel where the Netfilter conntrack module considers that new TCP connections redirected by IPVS to the local sshd daemon have an invalid cstate. This point can be verified with well placed iptable rules using the LOG destination. This causes the TCP SYN/ACK answer from the sshd to be blocked by the OUTPUT chain since it considers the connection is new and not related to any incoming connections. To workaround this annoying behaviour, an exception has been added in the OUTPUT chain of the firewall to accept connections with a source port that is TCP/22 and a source IP address that is the virtual IP address. This is not totally satisfying in terms of security but there is no known easy or obvious way to exploit this security exception from a user perspective for other purposes.

If a slave frontend node becomes unavailable, Keepalived detects it either with VRRP checks, or with TCP checks in case only the sshd daemon is crashed. The IPVS rules are changed dynamically to avoid redirecting new TCP connections to this failing node.

If the master frontend node becomes unavailable, the Keepalived software selects a new master node within the other frontend nodes. Then, on this new master node, Keepalived restores the IPVS redirection rules (since they were previously disabled to avoid loops) and moves the virtual IP address from the loopback interface to the WAN network interface.

If a frontend node is scheduled to be turned of, it is possible to drain it.

This diagram gives an overview of the load-balancing and high-availability mechanisms involved in the DNS service of the Scibian HPC clusters:

On Linux systems, when an application needs to resolve a network hostname, it usually calls the gethostbyname*() and getaddrinfo() functions of the libc. With a common configuration of the Name Service Switch (in the file /etc/nsswitch.conf), the libc searches for the IP address in the file /etc/hosts and then fallbacks to a DNS resolution. The DNS solver gathers the IP address by sending a request to the DNS nameservers specified in the file /etc/resolv.conf. If this file contains multiple nameservers, the solver sends the request to the first nameserver. If it does not get the answer before the timeout, it sends the request to the second nameserver, and so on . If the application needs another DNS resolution, the solver will follow the same logic, always trying the first nameserver in priority. It implies that, with this default configuration, as long as the first nameserver answers the requests before the timeout, the other nameservers are never requested and the load is not balanced.

This behavior can be slightly altered with additional options in the file /etc/resolv.conf

On Scibian HPC clusters, Puppet manages the file /etc/resolv.conf and ensures these two options are present. It also randomizes the list of nameservers with the fqdn_rotate() function of the Puppet stdlib community module. This function randomizes the order of the elements of an array but uses the fqdn fact to ensure the order stays the same for a node with a given FQDN. That is, each node will get a different random rotation from this function, but a given node’s result will be the same every time unless its hostname changes. This prevents the file content from changing with every Puppet runs. With this function, all the DNS nameservers are equivalently balanced on the nodes. Combined with the options rotate, it forms an efficient load-balancing mechanism.

The DNS servers are managed with the bind daemon on the generic service nodes. Each generic service nodes has a virtual IP address managed by a keepalived daemon and balanced between all the generic service nodes. The IP addresses of the nameservers mentioned in the file /etc/resolv.conf on the nodes are these virtual IP addresses. If a generic service node fails, its virtual IP address is automatically routed to another generic service node. In combination with options timeout:1, this constitutes a reliable failover mechanism and ensures the high-availability of the DNS service.

This diagram illustrates how Consul and the DNS servers integrate to provide load-balanced and horizontally scaled network services with high-availability:

The Consul agent daemon can run in two modes: server and client. The cluster of Consul servers maintains the state of the cluster using the raft protocol. The clients communicate with the servers to detect failures using the gossip protocol. Both agents expose the data of the Consul cluster through a HTTP REST API. On Scibian HPC clusters, the Consul servers run on the generic service nodes while the admin node runs a client agent.

As explained in the Software architecture section, Consul discovers network services on a pool of nodes. The services discovered by Consul on Scibian HPC clusters are hosted on the generic service nodes. Each Consul server is responsible for checking its locally running services, such as an HTTP server for example. The state being constantly shared by all Consul agents, every agent is actually able to tell where the services are available. Consul notably provides a DNS interface. Given a particular virtual hostname referring to a service, Consul can give the IP addresses of the servers currently running this service.

Consul is not designed to operate as a full DNS server. It listens for incoming requests on an alternative UDP port for a particular sub-domain virtual.<domain>, where <domain> is configurable and depends on the cluster.

On the nodes, the clients are configured to connect to services in this particular sub-domain, for example http.virtual.<domain> for the HTTP service. The DNS requests sent by the clients are received by the bind daemon through the virtual IP addresses of the generic service nodes, as explained in DNS Load-balancing and High-availability section. The DNS bind daemon is configured to forward the requests on the virtual sub-domain to the local Consul agent. The Consul agent answers the DNS request with the static IP address of the generic service nodes running this service, in random order.

In this architecture, both the DNS requests to the Consul servers and the services (eg. HTTP) requests are load-balanced on all the generic service nodes in high-availability mode. The same mechanism also applies to APT proxies, Ceph RADOS gateways, and so on.

The Consult utility is installed on the admin node to request the current status of the Consul cluster. It connects to the REST API of the Consul client running locally and prints the status on the standard output.

Install Debian 9 Stretch base system using any of the official Debian installation media (CD, DVD, USB key, PXE server, etc) at your convenience. Configure the network interfaces with static IP addresses in compliancy with the cluster IP adressing plan. Set the hostname following the architecture conventions, for example: fbservice1.

Once the node has rebooted on freshly installed system, add the Scibian 8 APT repositories to the configuration:

Download and enable Scibian repository keyring:

Update the packages repositories local database:

Install the following Scibian HPC administration node meta-package:

All the files manipulated during the installation process will be placed into a dedicated working directory. The location of this directory is arbitrary, for example: ~root/install. This directory will be designated as $ADMIN in the following section of the installation procedure documentation.

Clone both Puppet-HPC and internal configuration repositories into this dedicated working directory:

At this stage, the internal repository can be populated with all files and data initially required to install the cluster.

If the internal configuration repository is fully empty and is initialiazed from scratch, a few base directories must be created under its root.

Some settings are common to all HPC clusters of an organization, in particular settings regarding the external services. To avoid duplication of these settings in all HPC cluster configurations, they are defined once in the organization layer of the hiera repository shared by all HPC clusters.

Initialize the file $ADMIN/hpc-privatedata/hieradata/org.yaml with the following content:

The examples values must be replaced with the settings corresponding to your organization environment.

Some directories are required to store cluster specific file and settings inside the internal configuration repository. Create these directories with the following command:

The hpc-config-push Puppet-HPC utility expects to find a Puppet and Hiera configuration files for the cluster under the puppet-config directory of the internal configuration repository. Simply copy examples configuration files provided with Puppet-HPC:

The hiera.yaml file notably specifies the layers of YAML files composing the hiera repository. It can eventually be tuned for additional layer to fit your needs.

Puppet-HPC requires the cluster name and prefix to be a declared a YAML file cluster-nodes.yaml. Technically speaking, this YAML file is deployed by hpc-config utilities on every nodes in /etc/hpc-config directory. It is then used as aconfiguration input file for the external node classifer (ENC) cluster-node-classifier provided with hpc-config. Define the file $ADMIN/hpc-privatedata/puppet-config/$CLUSTER/cluster-nodes.yaml with the following content:

If the cluster is composed of multiple areas, they must also be declared in this YAML file with their associated roles. For example:

In this declaration, the admin and service roles are membered of the infra area, the front and cn roles are membered of the user area.

The cluster specific layers of the Hiera repository must be initialized with a sufficient description of the HPC cluster. This description is the cluster definition.

The Puppet role service associated to the generic service nodes must be defined with the corresponding profiles. This is achieved by initializing file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/service.yaml with the following content:

The first profiles (below the common comment) are common to all nodes of the cluster. The profiles after the HW host comment are common to all bare metal nodes. The last profiles, after the service comment, carry the base services hosted by the generic service nodes.

The last parameter profiles::network::gw_connect defines on which network’s gateway the nodes use as their default route.

Cluster configurations comprises many sensitive data such as passwords, private keys, confidential files, and so on. The Puppet-HPC stack provides an integrated mechanism for storing these data securily in the internal configuration repository. This mechanism is fully explained in the Puppet-HPC Reference Documentation (chapter Software Architecture, section Sensitive Data Encryption). Basically, these data are encrypted using two keys:

These keys are also used to decrypt data on nodes of the cluster main area. If the cluster is composed of only one area (ex: default), only these two keys are involved on the cluster. Otherwise, additional and dedicated keys are used by the other areas to decrypt their sensitive data.

The goal of this section is to configure the Temporary Installation Services on the Temporary Installation Node. This process is done in two steps:

Consul is not available because the consul cluster needs quorum to work. Quorum can only be achieved when more than half of the generic service nodes are configures. The DNS server is therefore configured to only returns the temporary installation node for all requests on the consul domain. This is done simply by adding temporarily the following parameters in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

Technically speaking, these parameters makes bind authorative on the virtual DNS zone before Consul service discovery utility is available. The virtual zone contains all the symbolic names to the network services (ex: http.virtual). This way, all services will be directed to the temporary installation node with the IP address provided in install_server_ip parameter.

The first run also needs to work properly without a local DNS server and without a local repository cache proxy. These services will be configured during this first run. Local repositories must also be disabled during the first run.

Where <AREAS> must be replaced with the comma separated list of areas on the cluster (ex: infra,user or default).

The configuration will be pushed on local files while the temporary installation is used. The settings above configures this, but the first push must use a configuration that will be created manually in the file: /etc/hpc-config/push.conf.

The directory where the keys were generated cannot be used as a key source for apply because it will be overwritten during the apply. So it must be copied before doing the apply. To deploy the configuration of the temporary installation node, run the following commands:

The area parameter is required if the service node is not in default area.

If the run returned no error, there is some checks to do before proceeding. In the following commands IP1 is the IP address of the current node. VIP[1-4] are the IP addresses of the VIP for the service nodes.

You should check the following commands return no errors:

With these commands we are now sure that:

The goal of this run is to switch hpc-config-apply to download files through apache and not just get them locally. We also change the local DNS client configuration to use the newly configured local DNS server.

To change the hpc-config-apply source, do these changes in cluster.yaml:

To switch to the local DNS server, remove the profiles::dns::client::nameservers added for the first run and uncomment the normal one that was commented out. Also remove the temporary apt::proxy_host setting to use the configured apt-cacher-ng.

Do the actual run:

If the two commands run without error, the initial setup succeeded.

At this stage, the temporary installation service are fully configured and available to install other generic service nodes.

The other generic service nodes must now be rebooted in PXE mode to run the Debian installer and configure the base system:

Replace the BMC credentials with the appropriate values.

Once the base system is fully installed, the nodes reboot and become available with SSH. Check this with:

At this stage, all generic services nodes are available to host the configuration environments. The parameters of the hpc-config-push utility can be updated to switch from posix to sftp. In this mode, the utility will upload the configuration environment on all generic service nodes. Edit $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml file to update the hpconfig::push::config_options hash with the following changes:

Then push and apply the configuration on the first service node:

This will update /etc/hpc-config/push.conf configuration file.

Then run this command again to upload the configuration environment on all service nodes:

Starting from now, all generic service nodes can be used as a valid source for the configuration environments.

Deployment is based on a tool called ceph-deploy. This tool performs the steps on a node to setup a ceph component. It is only used for the initial setup of the Ceph cluster. Once the cluster is running, the configuration is reported in the Puppet configuration in case it is re-deployed.

The reference configuration uses one disk (or hardware RAID LUN) to hold the system (/dev/sda) and another to hold the Ceph OSD data and journal (/dev/sdb). Three or five nodes must be chosed to setup the MON and MDS services, the remaining nodes are used only as OSD and RadosGW nodes.

The ceph-deploy utility generates authentication keys for Ceph. Once the cluster is running, theses keys are manually collected and encrypted with eyaml to be included in the hiera configuration.

In the following example MONs/MDS are installed on nodes fbservice[2-4] while the node fbservice1 only has OSD and RGW.

All the base services are now deployed on all the generic service nodes. It is time to enable load-balancing and high-availability with Consul service discovery tool.

Consul needs a shared secret key to encrypt communication between its distributed agents. Generate this key with:

The output of this command must be reported in the area layer of the hiera repository $ADMIN/hpc-privatedata/hieradata/$CLUSTER/areas/$AREA.yaml whose generic service nodes are members (ex: default or infra) using eyaml:

Add consul::server profile to the service role:

Then, run Puppet on all services nodes:

Check that all the generic service nodes are members of the Consul cluster with this command:

The output should report that all the services nodes are members and alive.

Remove dns::server::virtual_relay and install_server_ip parameters from $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

With this new configuration, Bind DNS server relays all DNS requests on the virtual zone to Consul DNS interface.

Push and the apply the new configuration:

Finally, check DNS requests on virtual zone are managed by Consul with:

The output must report multiple generic service nodes static IP addresses in random order.

Since the beginning of the installation process, the temporary installation node hosts installation files and services required to install the other generic service nodes. Now, all the other generic service nodes host the same files and services. Finally, the temporary installation node must be re-installed to be strictly identical to the other generic service nodes in terms of configuration.

Reboot the node in PXE mode through its BMC:

Wait for the network installation to proceed and the node to reboot on the system freshly installed on its disks.

Add the admin role by creating the file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/admin.yaml with the following content:

The profiles listed after the admin comment carry the software required on the admin node. The profiles::environment::service::packages has a specific value for this role in order to install the admin meta-package.

Append the node definition in the master_network hash, for example:

Optionally, adjust the node boot parameters in the boot_params hash, for example:

Synchronize SSH host keys:

Push and apply the new configuration:

And reboot the node in PXE mode to proceed the network installation:

Wait for the network installation to proceed. Once the installation is over, the node reboot on its freshly installed system on its disks and it becomes available through SSH. Starting from this point, all the following operations of the installation process are realized from this admin node.

The administration environment must be re-created following the same instructions given in the temporary installation node administration environmnet section.

The Clara utility is available on the admin node. Its ipmi plugin can be configured with this small snippet added with eyaml to the cluster specific layer of the hiera repository:

Then add the IPMI identifiers to the admin node area layer (ex: default or infra) of the Hiera repository using eyaml:

Push and apply configuration on the admin node:

Then, the clara ipmi plugin can be used as explained in its documentation (man clara-ipmi (1)).

The Libvirt service must create various virtual networks to connect the virtual machines to the HPC cluster and a storage pool on Ceph RDB interface to store the virtual disks of the virtual machines. These virtual resources are setup with the following configuration in the cluster specific layer of the hiera repository:

The <uuid> is an arbitrary UUID^[6] to identify uniquely the secret. For example, it can be generated with this command:

Add the libvirt Ceph client identifiers with the following hash into the generic service nodes area layer (ex: default or infra) of the Hiera repository using eyaml:

The <key> is given by the following command:

The profile profiles::virt::host must be added to service nodes role definition.

Push and apply configuration on the generic service nodes:

Clara has dedicated configuration for its virt plugin. This configuration is set with the following two hashes in the cluster specific layer of the hiera repository:

The clara::virt_options hash notably specifies the list of generic services nodes that hosts the virtual machines and the domain templates and parameters associated to each service virtual machine. For the moment, only the default domain template and parameters are set. The second hash clara::virt_tpl_hpc_files defines the templates of Libvirt XML domains definitions. In this example, there is one default domain XML template for all virtual machines which should be fine for most Scibian HPC clusters.

The domain XML template must be located in $ADMIN/hpc-privatedata/files/$CLUSTER/$AREA/virt/domain_default_template.xml, where $AREA is the area of the generic service nodes. Here is a full example of this file:

In this example, the following values must be replaced:

Deploy these new settings by pushing and applying the configuration on the admin node:

Now that Libvirt and Clara virt plugin are properly setup, the various service virtual machines can be defined. The steps to define the service virtual machines are mostly generic and common to all of them. As an example for this documentation, the two service virtual machines fbdoe[1-2] will be defined.

Optionally, define specific boot_params for the virtual machines in $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml if the defaults parameters are not appropriate:

Also, in the same file, an additional domain template and parameters association can be appended to the clara::virt_options for these new virtual machines, if the default domain parameters are not appropriate:

In this example, the following settings are overriden from the defaults:

Then, the new role doe must be defined in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/doe.yaml with all the appropriate profiles.

Push and apply configuration on admin node:

Extract MAC address of the virtual machine on the administration network:

Then add the network settings of the virtual machines in the master_network hash with their MAC addresses:

Eventually, virtual IP addresses can also be defined for the virtual machines in the vips hash of the same file.

Generate the SSH host keys in synchronization with the master_network:

Push and apply the new configuration on the generic service nodes:

Define the new virtual machines with Clara on the generic service node of your choice, for example fbservice1:

Then start the virtual machine by wiping its virtual block storage devices and boot in PXE mode:

Eventually, watch the serial console with:

You are free to define the service virtual machines you want on Scibian HPC clusters. The service virtual machines can run any software services you would like. However, some specific generic virtual machines are required in the reference architecture to run some mandatory additional services.

The required service virtual machines are:

User authentication on Scibian HPC clusters is based on LDAP directory using ldaps protocol (LDAP over SSL/TLS). This protocol requires the LDAP replica to have valid SSL certificate and asymmetric keys.

For production use, it is recommended to obtain a certificate signed by a valid PKI CA^[7], either a public CA on the Internet or a CA internal to your organization. Otherwise, it is possible to use self-signed certificates.

Copy the private key and the certificate under the following paths:

Where $AREA is the area of the LDAP replica nodes (ex: default or infra).

Encrypt these files with clara enc plugin:

Remove the unencrypted files:

Then, append the auth::replica profile and set certificate owner to openldap in the proxy role:

Push and apply the configuration on the proxy nodes:

Finally, follow the steps documented in Chapter 17, LDAP bootstrap.

Once the LDAP replica are bootstrapped and operational, it is possible to setup NSS LDAP backend and PAM LDAP authentication on the nodes.

On Scibian HPC clusters, NSS LDAP backend and PAM authentication over LDAP are both setup with the same auth::client profile. This profile must be used in combination with the access::base profile. This profile controls the remote access rules to the nodes. By default, the profile prevents remote access to the nodes with LDAP accounts. The access rules must explicitely whitelist users and/or administrators to allow remote access with SSH.

There are two main access whitelist parameters:

The administrators related access rules must be listed in the base_options while the users related access rules must only be present in the production_options list. This way, only administrators can access the HPC cluster in maintenance mode. For example:

These parameters must be set in the roles specific layer of the hiera repository as access rules depends on the role of the nodes. For example, users may access the frontend nodes but not the admin node.

Additionally, it is also possible to setup sudo rules with the sudo::base profile and the sudo::sudo_config_opts list. This parameter is basically a list of sudo rules. For example, to allow the group of administrator to sudo any command on the admin node, add the following excerpt to file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/admin.yaml:

By default, the PAM and NSS LDAP backend connect to the HPC cluster internal LDAP replica. This replica is hosted by service virtual machine. In order to make LDAP authentication on the admin nodes and generic service nodes possible for the administrators when the virtual machines are offline (typically during maintenances), it is possible to add the following parameter in the associated roles:

This way, the nodes will connect to the organization reference LDAP directory instead of the internal LDAP replica.

Push and apply the configuration on all the affected nodes with:

Slurm communications between nodes are secured using Munge which is based on a secret shared key. Generate this munge key with the following command:

Encrypt the key using Clara:

Remove the unencrypted key:

Setup the nodes and partitions managed by Slurm in the slurm::partitions_options hash in the cluster specific layer of the Hiera repository. For example:

Please refer to Slurm documentation for more details about these settings.

In the same, setup the LDAP/SlurmDBD users synchronization utility, for example:

Please refer to the example configuration file for more details.

Still in the cluster specific layer of the Hiera repository, setup the shared storage directory.

Eventually, it is possible to tune Slurm, GRES, SlurmDBD, job submit LUA script with the following parameters:

Some software components need to be manually bootstrapped on the batch nodes before being started:

The shared storage can be on CephFS or on NFS HA, the suitable bootstrap procedure must be performed:

Please refer to the Bootstrap procedure chapter of this document for all details.

Once the configuration is set in the Hiera repository, push and apply the configuration on the admin and batch nodes:

Check Slurm is available by running the sinfo command on the admin node. If the command report the nodes and partitions state without error, Slurm is properly running.

The diskless image is generated by the Clara images plugin. This plugin need some configuration in the cluster specific layer of the Hiera repository. Here is an example of such configuration:

The clara::live_files parameter contains a list of files deployed under the configuration directory of Clara. Their files are:

This script can notably be used to customize the image or set files and directories that are required very early in the live boot process before Puppet run.

All the files under the files_dir directory are copied without modification into the image. The required files are:

Once all these files have been added to the cluster specific files directory, push and apply the configuration on the admin node:

Now that Clara is setup, the image can be created with the following command:

Also create the associated initrd environment:

Deploy the generate image and initrd to the p2p nodes with:

Restart peer-to-peer services to load new files:

The diskless environment is finally ready and available to frontend and compute nodes.

Before booting the frontend and compute nodes, they must be declared in the internal configuration repository in the first place. Append the nodes to the boot_params hash in $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

The cowsize must be increased to 8GB from default 2GB on frontend and graphical nodes because these nodes need much more packages to be installed at boot time.

Then define the roles associated to the frontend and the compute nodes, for example front, cn and gn. For these roles definitions, keep in mind the following rules:

The nodes must be added into the master_network hash in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/network.yaml with all their network interfaces and the MAC addresses of their network interface connected to the administration and their BMC.

Generate all the SSH host keys:

Push and apply the configuration to the admin and generic service nodes:

Finally, boot all the nodes in PXE mode with Clara:

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

Slurm-web is both a web interface and REST API service to get and visualize the current status of the jobs and ressources managed by Slurm.

Puppet-HPC is able to deploy the REST API component of Slurm.

Where $AREA is the area of the nodes hosting the REST API.

Then, define XML cluster racking description file hpc-privatedata/files/$CLUSTER/$AREA/slurmweb/racks.xml according to Slurm-web documentation.

Add profiles::http::slurmweb profile in the role of the nodes hosting the REST API.

Finally, push and apply the new configuration on the admin node and on the nodes hosting the profile:

TBD

Generate and encrypt the SSH key used to poweroff the nodes from the batch nodes:

Where $AREA is the area of the batch nodes.

Then add those settings in the cluster specific layer of the hiera repository:

Where <PUBKEY> is the public key in file hpc-privatedata/files/$CLUSTER/$AREA/pwmgt/id_rsa_slurm.pub.

Finally, apply the new configuration on the batch nodes and all the compute nodes:

The ceph-deploy directory is created during the initial ceph installation, to use the ceph-deploy again or from another service or admin node, it must be recreated.

After the reinstallation of one of the generic service nodes with a mon, it must be re-initialized. This procedure only works on a running cluster, the initial mon creation uses another command.

From an admin node:

This procedure only applies if the content of an OSD volume is lost. If the node is reinstalled without erasing the content of the OSD volume, the configuration in puppet will be enough to start the osd volume again.

The relevant OSD ID can be retrieved with:

Before doing this procedure, make sure the OSD is really down and not mounted on the OSD node.

CephFS filesystem is used between the batch nodes to shared Slurm controller state. The filesystem must be initialized before being used by Slurm.

First, mount temporarily the CephFS filesystem:

Create a subdirectory for Slurm controller, set its ownership and restrict its mode:

Finally, umount it:

Puppet-HPC is now able to use this filesystem for Slurm on batch nodes.

The hashed root password is stored in the variable profiles::cluster::root_password_hash in yaml files. The value must be encrypted using eyaml. It can be simply changed using the eyaml command.

Once changed, the new configuration must be applied on all the machines of the cluster.

The root SSH keys are stored in the internal repository. The privates keys must be encrypted. The SSH public rsa key is also in the variable openssh::server::root_public_key. It is necessary to change the files and the value of the variable at the same time. To avoid connections problems, it is necessary to follow these steps in this order:

The SSH host keys are stored, encrypted, in the internal repository. To avoid connections problems, it is necessary to follow these steps in this order:

Replacing the eyaml PKCS7 key pair consist in reality of two actions:

These steps must be followed in order to safely change the eyaml keys:

Save the old keys:

Copy the new keys in /etc/puppet/secure/keys/.

Decrypt all the yaml files encoded using the old keys:

The decrypt.yaml contains all the secret in plain text. It should be removed as soon as possible.

Encrypt the files with the new keys:

Remove the old saved keys from the admin node:

Create a tarball, encode it with clara enc and add it to the files directory of the internal repository:

Where:

At this stage, the keys are now stored encrypted in the internal repository and are available locally in the standard eyaml paths.

In the default Scibian-HPC configuration, the PKCS7 keys propagation service runs on all the generic service nodes. First, the encoded tarball must be manually copied on the nodes:

Where <generic server X> is the hostname of the generic service node.

Then apply the configuration using the new keys:

This will copy the eyaml PKCS7 key pair in the right directory to be serviced by the propagation service to all others nodes when applying the puppet configuration. These last two operations must be executed on all the generic service nodes.

Don’t forget to remove the keys from the /tmp directory on the admin node and on all the service nodes.

Replacing the AES key used to encode files in the internal repository consist in several steps.

Generate a new AES key:

For each encoded file in the internal repository, it is necessary to decode it with the old key and re-encode it with the new one.

Where:

Using clara for both operations, decode and encode, is not possible as it support only one AES key.

This re-encryption step can be automated with the reencode-file.sh script in the puppet-hpc scripts dir:

The files /tmp/oldkey and /tmp/newkey are files with just the old and new AES key respectively. This script does not depend on clara but basically performs the same steps as above.

The AES key must be placed in cluster_decrypt_password in the cluster layer of the Hiera repository:

Replace the key:

Apply the new configuration on the admin node, to update clara configuration:

The steps to change these credentials are described here:

The certificates used for monitoring are stored, encrypted, in the internal repository in <internal repository>/files/<cluster>/icinga2/certs/. Each host has a certificate and a key. The steps to follow to change them are:

The cluster must use a private cluster keyring. This keyring is used to sign the local packages repository.

It is stored in the internal repository: <internal repository>/files/<cluster>/repo/

Here are the steps to follow to change it:

Generate passwords conform with your organization policy and edit the following parameters with eyaml in the hiera repository:

These parameters correspond to the passwords of the MariaDB having respectively R/W and R/O grants on the SlurmDBD database.

Once modified, push and apply the configuration with the following commands:

The hpc-config-apply command will perform the following steps, on each batch node:

The --fanout=1 parameter of the clush command makes sure the configuration is not applied simultaneously on both batch nodes. This could cause the SlurmDBD daemon to be restarted at the same time and make this service unavailable for a short period of time.

On Scibian HPC clusters, the default iPXE ROM is provided by the ipxe package. Alternatively, you can build a custom ROM following the instructions available on iPXE website and deploy it with Puppet-HPC.

First, copy the custom ROM (ex: custom.kpxe) in the $ADMIN/hpc-privatedata/files/$CLUSTER/cluster/boot/ipxe/ directory.

Then, define the boottftp::hpc_files hash in the cluster layer of the Hiera repository to declare the file to deploy:

Then, set the ipxebin parameter accordingly in the boot_params hash of the cluster layer of the Hiera repository, for example:

Finally, deploy configuration changes on DHCP and boot servers:

As explained in Section 5.2, “iPXE Bootmenu Generator”, the bootmenu entries available in iPXE are declared in YAML files. Puppet-HPC provides a mechanism to deploy custom entries and optionally override the defaults provided by scibian-hpc-netboot-bootmenu package.

For this purpose, edit the cluster layer of the Hiera repository to declare the bootsystem::menu_entries hash profile parameter, for example:

This declares an additional scibian9-ram-test entry. Optionally, it is also possible to set this entry as the default for some nodes in the boot_params hash of the cluster layer of the Hiera repository, for example:

This way, the fbcn04 node will boot this new entry by default.

Finally, deploy the configuration changes on boot servers:

As explained in Section 5.1.2, “Disk installation”, the Debian installer environment is installed with debian-install-*-netboot-amd64 packages. These packages are designed to work on most hardware, however it may be required to use alternate environment in some cases, notably if the hardware needs special non-free modules or firmwares during the installation.

For this purpose, Puppet-HPC lets the ability to deploy custom Debian Installer environment within an archive.

For information, it is possible to build a base archive using the packages, for example:

Starting from this point, the archive can be tuned upon your needs.

To deploy this archive on the boot servers, the boothttp::archives hash parameter must be defined accordingly in the cluster specific layer of the Hiera repository:

Then, define a bootmenu entry, following the procedure in Section 26.2, “Bootmenu Entries”, to network boot this custom environment.

Finally, deploy the new configuration on the boot servers:

As explained in Section 5.3, “Debian Installer Preseed Generator”, the preseed generator provides a link to a CGI script that generates dynamically for the node a partition schema (aka. recipe) for Debian installer partman utility.

By default, this script sends a partition schema common to all nodes. The default common partition schema is provided by scibian-hpc-netboot-preseedator package. It configures the /dev/sda disk with LVM physical volume and creates dedicated logical volumes for /, /var, /tmp and swap partitions. However, the script is able to send specific partitions schemas for a given host or role.

Puppet-HPC gives the ability to override the default common partition schema provided by the package and to deploy these specific partition schemas and

Once the alternate partman partition recipe is defined, copy the file into $ADMIN/hpc-privatedata/files/$CLUSTER/cluster/boot/disk-installer/schemas/ directory.

Then, define the boothttp::partition_schemas hash parameter in cluster layer of the Hiera repository to declare the partition schemas to deploy, for example:

In this example, the following partition schemas are deployed:

Finally, deploy the new configuration on the boot servers:

To perform a scheduled reboot of a frontend it is better to avoid new connection going to the frontend node that will be rebooted. The new connections are highly available and load balanced with IPVS.

It is possible to remove a frontend from the pool of node accepting new connections without killing active connections with the ipvsadm command by setting the weight of a node to 0.

To list the current weight, on a frontend:

To avoid a frontend node being attributed to new sessions, the weight of the node can be manually set to 0. This setting does not completely forbid new connection to go to the node, if a user already has a session, new session will go to the same node regardless of the weight. This setting also does not block connections made directly to the node and not the virtual IP address.

The modification can be reversed by setting the weight back to 1 manually.

When a node start is should not start the keepalived service automatically. This permits a failed node to be started without it becoming master with an remaining problem.

Before starting the keepalived service, the following conditions must be met:

When these conditions are met, the service can be started:

If the node is to become master (master node in the VIP configuration or other node is down), check that the first check goes well. It runs every minutes and logs are in /var/log/user.log. The message following message must appear:

If the master node disappears, because it is turned off or because the keepalived service is stopped, the failover will happen, but it will take a bit of time (a little more than a minute). This timeout can be entirely avoided by doing a manual failover of the master node before cutting the keepalived service.

To do this, the keepalived configuration must be changed manually on the node. Edit the file /etc/keepalived/keepalived.conf. Find the configuration for the NFS VIP and change the priority to 40, and the role to BACKUP. The service must be reloaded:

The failover should happen quickly. Once the node failed over, stop the keepalived service:

The original configuration must be restored before starting the service again. This will happen if you launch a hpc-config-apply manually or if you reboot the node.