The cluster is connected to three physically separated networks:

It is recommended to split the administration Ethernet network with a VLAN dedicated to all management devices (BMC [1], CMC [2], etc). This has significant advantages:

The administration cluster is composed by two types of nodes: the admin node and the generic service nodes.

The admin node is the access node for administrators and the central point of administrative operations. All common administrative actions are performed on this node. It does not run any intensive workloads, just simple short-lived programs and it does not need to be very powerful. It does not store sensible data nor run critical services, so it does not need to be very reliable either. Example of hardware specifications:

The generic service nodes run all critical infrastructure services (within service virtual machines) and manage all production administrative data. Scibian HPC requires a pool from 3 (minimum) to 5 (recommended) generic service nodes. The pool works in active cluster mode, the load is balanced with automatic fail-over. All generic service nodes of a cluster must be fairly identical for efficient load-balancing.

The generic service nodes manage the production data into a distributed object-storage system. It is highly recommended that the nodes have a dedicated block storage device for this purpose. The workload is mostly proportional to the number of compute nodes but the generic service nodes must be quite powerful to comfortably handle load peaks happening during some operations (ex: full cluster reboot). Also, since services are run into virtual machines, a fairly large amount of RAM is required. Services can generate a lot of traffic on the administration network, it is relevant to provide a network adapter with high bandwidth. Even though high-availability is ensured at the software level with automatic fail-over between generic service nodes, it is nevertheless recommended to get hardware redundancy on most devices of the generic service nodes to avoid always risky and hazardous service migrations as much as possible. Example of hardware specifications:

All physical nodes must be connected to all three physical networks. There are virtual bridges on the host of the generic service nodes connected to the WAN, administration (and eventually management) networks. The service virtual machines have connections to the virtual bridges upon their hosted service requirements.

The user-space cluster is composed of frontend nodes and compute nodes.

The nodes of the user-space cluster are deployed with a diskless live system stored in RAM. It implies that, technically speaking, the nodes do not necessarily need to have local block storage devices.

The frontend nodes are the access hosts for users so they must be connected to all three physical networks. It is possible to have multiple frontend nodes in active cluster mode for load-balancing and automatic fail-over. The exact hardware specifications of the frontend nodes mostly depend on user needs and expectations. Users may need to transfer large amount of data to the cluster, it is therefore recommended to provide high-bandwidth network adapters for the WAN network. These nodes can also be designed to compile computational codes and in this case, they must be powerful in terms of CPU, RAM and local storage I/O.

The compute nodes run the jobs so they must provide high performances. Their exact hardware specifications totally depend on user needs. They must be connected to both the administration and the low-latency networks.

The storage system is designed to host user data. It provides one or several shared POSIX filesystems. The evolved storage technologies depend on user needs ranging from a simple NFS NAS to a complex distributed filesystem such as Lustre or GPFS with many SAN and I/O servers.

All services running on the cluster should be highly available (HA). Some services not critical for normal cluster operation can be not highly available, but this should be avoided if possible.

The following section lists the different techniques used to achieve high-availability of the cluster services.

The frontend nodes offer a virtual IP address on the WAN network that features both an highly-available and load-balanced SSH service for users to access the HPC cluster. The load-balancing feature automatically distributes users on all available frontend nodes. This load-balancing is operated with persistence so that users (based on their source IP address) are always redirected to the same frontend node in a time frame. Behind the virtual IP address, the high-availability of the SSH service is also ensured in case of outage on a frontend node. These load-balancing and high-availability features are ensured by the Keepalived software.

For security reasons, a firewall is also set up on the frontend nodes to control outgoing network traffic. This firewall service is managed by Shorewall, a high-level configuration tool for Linux netfilter. Because of all the various network flows involved in Keepalived, it must be tightly integrated with the firewall rules. The following diagram illustrates both the network principles behind the high-availability/load-balancing mechanisms and the integration with the software components of the firewall:

The Keepalived sofware checks all the frontend nodes using the VRRP[4] protocol on the WAN network interfaces (purple arrow in the diagram). This protocol must be allowed in the OUTPUT chain of the firewall so that Keepalived can work properly.

On the master frontend node, the HA virtual IP address is set on the network interface attached to the WAN network. The Keepalived software configures the IPVS[5] Linux kernel load-balancer to redirect new TCP connections with a Round-Robin algorithm. Therefore, a part of the TCP connections is redirected to the sshd daemon of other frontend nodes (orange arrow in the diagram). An exception must be specified in the OUTPUT chain of the firewall to allow these redirected connections.

To perform such redirections, IPVS simply changes the destination MAC address, to set the address of the real destination frontend, in the Ethernet layer of the first packet of the TCP connection. However, the destination IP address does not change: it is still the virtual IP address.

On the slave frontend nodes, the HA virtual IP address is set on the loopback interface. This is required to make the kernel accept the redirected packets from the master frontend node addressed to the virtual IP address. In order to avoid endless loops, the IPVS redirection rules are disabled on slave frontend nodes or else, packets would be redirected endlessly.

By default, the Linux kernel answers the ARP requests coming from any network device for any IP address attached to any network device. For example, on a system with two network devices: eth0 with ip0 and eth1 with ip1, if an ARP request is received for ip1 on eth0, the kernel positively responds to it, with the MAC address of eth0. Though it is convenient in many cases, this feature is annoying on the frontend nodes, since the virtual IP address is set on all of them. Consequently all frontend nodes answer the ARP requests coming from the WAN default gateway. In order to avoid this behaviour, the net.ipv4.conf.<netif>.arp_ignore and net.ipv4.conf.<netif>.arp_announce sysctl Linux kernel parameters, where <netif> is the network interface connected to the WAN network, are respectively set to 1 and 2. Please refer to the Linux documentation for more details on these parameters and their values: http://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

The Keepalived software also checks periodically if the sshd service is still available on all frontend nodes by trying to perform a TCP connection to their real IP addresses on the TCP/22 port (green arrow in the diagram). An exception must be present in the OUPUT chain of the firewall to allow these connections.

There is an unexplained behaviour in the Linux kernel where the Netfilter conntrack module considers that new TCP connections redirected by IPVS to the local sshd daemon have an invalid cstate. This point can be verified with well placed iptable rules using the LOG destination. This causes the TCP SYN/ACK answer from the sshd to be blocked by the OUTPUT chain since it considers the connection is new and not related to any incoming connections. To workaround this annoying behaviour, an exception has been added in the OUTPUT chain of the firewall to accept connections with a source port that is TCP/22 and a source IP address that is the virtual IP address. This is not totally satisfying in terms of security but there is no known easy or obvious way to exploit this security exception from a user perspective for other purposes.

If a slave frontend node becomes unavailable, Keepalived detects it either with VRRP checks, or with TCP checks in case only the sshd daemon is crashed. The IPVS rules are changed dynamically to avoid redirecting new TCP connections to this failing node.

If the master frontend node becomes unavailable, the Keepalived software selects a new master node within the other frontend nodes. Then, on this new master node, Keepalived restores the IPVS redirection rules (since they were previously disabled to avoid loops) and moves the virtual IP address from the loopback interface to the WAN network interface.

If a frontend node is scheduled to be turned of, it is possible to drain it.

This diagram gives an overview of the load-balancing and high-availability mechanisms involved in the DNS service of the Scibian HPC clusters:

On Linux systems, when an application needs to resolve a network hostname, it usually calls the gethostbyname*() and getaddrinfo() functions of the libc. With a common configuration of the Name Service Switch (in the file /etc/nsswitch.conf), the libc searches for the IP address in the file /etc/hosts and then fallbacks to a DNS resolution. The DNS solver gathers the IP address by sending a request to the DNS nameservers specified in the file /etc/resolv.conf. If this file contains multiple nameservers, the solver sends the request to the first nameserver. If it does not get the answer before the timeout, it sends the request to the second nameserver, and so on . If the application needs another DNS resolution, the solver will follow the same logic, always trying the first nameserver in priority. It implies that, with this default configuration, as long as the first nameserver answers the requests before the timeout, the other nameservers are never requested and the load is not balanced.

This behavior can be slightly altered with additional options in the file /etc/resolv.conf

On Scibian HPC clusters, Puppet manages the file /etc/resolv.conf and ensures these two options are present. It also randomizes the list of nameservers with the fqdn_rotate() function of the Puppet stdlib community module. This function randomizes the order of the elements of an array but uses the fqdn fact to ensure the order stays the same for a node with a given FQDN. That is, each node will get a different random rotation from this function, but a given node’s result will be the same every time unless its hostname changes. This prevents the file content from changing with every Puppet runs. With this function, all the DNS nameservers are equivalently balanced on the nodes. Combined with the options rotate, it forms an efficient load-balancing mechanism.

The DNS servers are managed with the bind daemon on the generic service nodes. Each generic service nodes has a virtual IP address managed by a keepalived daemon and balanced between all the generic service nodes. The IP addresses of the nameservers mentioned in the file /etc/resolv.conf on the nodes are these virtual IP addresses. If a generic service node fails, its virtual IP address is automatically routed to another generic service node. In combination with options timeout:1, this constitutes a reliable failover mechanism and ensures the high-availability of the DNS service.

This diagram illustrates how Consul and the DNS servers integrate to provide load-balanced and horizontally scaled network services with high-availability:

The Consul agent daemon can run in two modes: server and client. The cluster of Consul servers maintains the state of the cluster using the raft protocol. The clients communicate with the servers to detect failures using the gossip protocol. Both agents expose the data of the Consul cluster through a HTTP REST API. On Scibian HPC clusters, the Consul servers run on the generic service nodes while the admin node runs a client agent.

As explained in the Software architecture section, Consul discovers network services on a pool of nodes. The services discovered by Consul on Scibian HPC clusters are hosted on the generic service nodes. Each Consul server is responsible for checking its locally running services, such as an HTTP server for example. The state being constantly shared by all Consul agents, every agent is actually able to tell where the services are available. Consul notably provides a DNS interface. Given a particular virtual hostname referring to a service, Consul can give the IP addresses of the servers currently running this service.

Consul is not designed to operate as a full DNS server. It listens for incoming requests on an alternative UDP port for a particular sub-domain virtual.<domain>, where <domain> is configurable and depends on the cluster.

On the nodes, the clients are configured to connect to services in this particular sub-domain, for example http.virtual.<domain> for the HTTP service. The DNS requests sent by the clients are received by the bind daemon through the virtual IP addresses of the generic service nodes, as explained in DNS Load-balancing and High-availability section. The DNS bind daemon is configured to forward the requests on the virtual sub-domain to the local Consul agent. The Consul agent answers the DNS request with the static IP address of the generic service nodes running this service, in random order.

In this architecture, both the DNS requests to the Consul servers and the services (eg. HTTP) requests are load-balanced on all the generic service nodes in high-availability mode. The same mechanism also applies to APT proxies, Ceph RADOS gateways, and so on.

The Consult utility is installed on the admin node to request the current status of the Consul cluster. It connects to the REST API of the Consul client running locally and prints the status on the standard output.

Install Debian 8 Jessie base system using any of the official Debian installation media (CD, DVD, USB key, PXE server, etc) at your convenience. Configure the network interfaces with static IP addresses in compliancy with the cluster IP adressing plan. Set the hostname following the architecture conventions, for example: fbservice1.

Once the node has rebooted on freshly installed system, add the Scibian 8 APT repositories to the configuration:

Download and enable Scibian repository keyring:

Update the packages repositories local database:

Install the following Scibian HPC administration node meta-package:

All the files manipulated during the installation process will be placed into a dedicated working directory. The location of this directory is arbitrary, for example: ~root/install. This directory will be designated as $ADMIN in the following section of the installation procedure documentation.

Clone both Puppet-HPC and internal configuration repositories into this dedicated working directory:

At this stage, the internal repository can be populated with all files and data initially required to install the cluster.

If the internal configuration repository is fully empty and is initialiazed from scratch, a few base directories must be created under its root.

Some settings are common to all HPC clusters of an organization, in particular settings regarding the external services. To avoid duplication of these settings in all HPC cluster configurations, they are defined once in the organization layer of the hiera repository shared by all HPC clusters.

Initialize the file $ADMIN/hpc-privatedata/hieradata/org.yaml with the following content:

The examples values must be replaced with the settings corresponding to your organization environment.

Some directories are required to store cluster specific file and settings inside the internal configuration repository. Create these directories with the following command:

The hpc-config-push Puppet-HPC utility expects to find a Puppet and Hiera configuration files for the cluster under the puppet-config directory of the internal configuration repository. Simply copy examples configuration files provided with Puppet-HPC:

The hiera.yaml file notably specifies the layers of YAML files composing the hiera repository. It can eventually be tuned for additional layer to fit your needs.

Puppet-HPC requires the cluster name to be a fact (according to Facter). For this purpose, define the file $ADMIN/hpc-privatedata/puppet-config/$CLUSTER/hpc-config-facts.yaml with the following content:

This YAML file is deployed by Puppet-HPC hpc-config-{push,apply} on every nodes in /var/lib/puppet/facts.d directory. The content of the YAML files located this directory are considered as valid custom facts.

Eventually, additional arbitrary custom facts can be defined for the cluster in this file at your convenience.

The cluster specific layers of the Hiera repository must be initialized with a sufficient description of the HPC cluster. This description is the cluster definition.

The Puppet role service associated to the generic service nodes must be defined with the corresponding profiles. This is achieved by initializing file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/service.yaml with the following content:

The first profiles (below the common comment) are common to all nodes of the cluster. The profiles after the HW host comment are common to all bare metal nodes. The last profiles, after the service comment, carry the base services hosted by the generic service nodes.

The last parameter profiles::network::gw_connect defines on which network’s gateway the nodes use as their default route.

There are few files that are required in the cluster’s file directory of the internal configuration repository to deploy the generic service nodes.

First, create the needed sub-directories:

Download the Debian network installer archive:

The iPXE ROM must be built from sources downloaded on iPXE website:

The ipxe_serial.bin variant must be generated after modifying hard-coded serial console parameters in iPXE source code as documented in iPXE official documentation.

The iPXE menu boot entries are generated dynamically, depending on the source node name, by the CGI Python script $ADMIN/hpc-privatedata/files/$CLUSTER/boot/cgi/bootmenu.py:

Debian installer partman utility requires a partition schema with rules and constraints to create the disk partitions. This file must is located in $ADMIN/hpc-privatedata/files/$CLUSTER/boot/disk-installer/scibian8/partition-schema. Here is a complete example for this file:

Finally, the version bind9 DNS server available in Debian Jessie is not able to automatically listen for incoming connections to IP addresses configured after the daemon has started. This is annoying when virtual IP addresses are gathered to failover a failing generic service node. This behaviour is workaround with the following script located in $ADMIN/hpc-privatedata/files/$CLUSTER/ha/rndc_reload.sh:

This script is run when the virtual IP addresses move from one generic service node to another. It asks bind9 daemon to reload its configuration which also makes it listen on new IP addresses.

The goal of this section is to configure the Temporary Installation Services on the Temporary Installation Node. This process is done in two steps:

The reference architecture (with Ceph/S3) does not uses the local apache server in the final configuration. Some other architectures uses a local apache Virtual Host instead of Ceph/S3 (profiles::http::system). If this is the case, /var/www/html can be replaced by hiera('profiles::http::system::docroot') and http.service by web-system.service in the private_files_dir URL.

Consul is not available because the consul cluster needs quorum to work. Quorum can only be achieved when more than half of the generic service nodes are configures. The DNS server is therefore configured to only returns the temporary installation node for all requests on the consul domain. This is done simply by adding temporarily the following parameters in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

Technically speaking, these parameters makes bind authorative on the virtual DNS zone before Consul service discovery utility is available. The virtual zone contains all the symbolic names to the network services (ex: http.virtual). This way, all services will be directed to the temporary installation node with the IP address provided in install_server_ip parameter.

The first run also needs to work properly without a local DNS server and without a local repository cache proxy. These service will be configured during this first run. Local repositories must also be disabled during the first run.

The configuration will be pushed on local files while the temporary installation is used. The settings above configures this, but the first push must use a configuration that will be created manually in the file: /etc/hpc-config/push.conf.

The directory where the keys where generated cannot be used as a key source for apply because it will be overwritten during the apply. So it must be copied before doing the apply. To deploy the configuration of the temporary installation node, run the following commands:

At this stage the unique temporary installation node should work, the configuration is modified to use it. The following hiera keys must be changed in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

Check the new settings with:

If the two commands run without error, the initial setup succeeded.

At this stage, the temporary installation service are fully configured and available to install other generic service nodes.

The other generic service nodes must now be rebooted in PXE mode to run the Debian installer and configure the base system:

Replace the BMC credentials with the appropriate values.

Once the base system is fully installed, the nodes reboot and become available with SSH. Check this with:

Deployment is based on a tool called ceph-deploy. This tool performs the steps on a node to setup a ceph component. It is only used for the initial setup of the Ceph cluster. Once the cluster is running, the configuration is reported in the Puppet configuration in case it is re-deployed.

The reference configuration uses one disk (or hardware RAID LUN) to hold the system (/dev/sda) and another to hold the Ceph OSD data and journal (/dev/sdb). Three or five nodes must be chosed to setup the MON and MDS services, the remaining nodes are used only as OSD and RadosGW nodes.

The ceph-deploy utility generates authentication keys for Ceph. Once the cluster is running, theses keys are manually collected and encrypted with eyaml to be included in the hiera configuration.

In the following example MONs/MDS are installed on nodes fbservice[2-4] while the node fbservice1 only has OSD and RGW.

At this stage, the configuration was published on the HTTP server of the temporary installation node. Now that Ceph cluster is available, the configuration can be migrated to the S3 backend of Ceph RadosGW.

Edit the cluster specific layer of the hiera repository $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml to make the hpc-config utilities push and download configuration in dedicated Ceph S3 bucket:

Push and apply configuration:

At this stage, the hpc-config utilities are configured to use Ceph. Try this new configuration by running them once again:

Finally, tune the private_files_dir in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml to use Ceph S3 as well:

Run a final check:

Starting from this moment, all the cluster configuration is hosted in the Ceph distributed filesystem.

All the base services are now deployed on all the generic service nodes. It is time to enable load-balancing and high-availability with Consul service discovery tool.

Consul needs a shared secret key to encrypt communication between its distributed agents. Generate this key with:

The output of this command must be reported in the cluster’s specific layer of the hiera repository $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml with eyaml:

Add consul::server profile to the service role:

Then, run Puppet on all services nodes:

Check that all the generic service nodes are members of the Consul cluster with this command:

The output should report that all the services nodes are members and alive.

Remove dns::server::virtual_relay and install_server_ip parameters from $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

With this new configuration, Bind DNS server relays all DNS requests on the virtual zone to Consul DNS interface.

Push and the apply the new configuration:

Finally, check DNS requests on virtual zone are managed by Consul with:

The output must report multiple generic service nodes static IP addresses in random order.

Since the beginning of the installation process, the temporary installation node hosts installation files and services required to install the other generic service nodes. Now, all the other generic service nodes host the same files and services. Finally, the temporary installation node must be re-installed to be strictly identical to the other generic service nodes in terms of configuration.

Reboot the node in PXE mode through its BMC:

Wait for the network installation to proceed and the node to reboot on the system freshly installed on its disks.

Add the admin role by creating the file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/admin.yaml with the following content:

The profiles listed after the admin comment carry the software required on the admin node. The profiles::environment::service::packages has a specific value for this role in order to install the admin meta-package.

Append the node definition in the master_network hash, for example:

Adjust the node boot parameters in the boot_params hash, for example:

Synchronize SSH host keys:

Push and apply the new configuration:

And reboot the node in PXE mode to proceed the network installation:

Wait for the network installation to proceed. Once the installation is over, the node reboot on its freshly installed system on its disks and it becomes available through SSH. Starting from this point, all the following operations of the installation process are realized from this admin node.

The administration environment must be re-created following the same instructions given in the temporary installation node administration environmnet section.

The Clara utility is available on the admin node. Its ipmi plugin can be configured with this small snippet added with eyaml to the cluster specific layer of the hiera repository:

Push and apply configuration on the admin node:

Then, the clara ipmi plugin can be used as explained in its documentation (man clara-ipmi (1)).

The Libvirt service must create various virtual networks to connect the virtual machines to the HPC cluster and a storage pool on Ceph RDB interface to store the virtual disks of the virtual machines. These virtual resources are setup with the following configuration in the cluster specific layer of the hiera repository:

In this example, the following values must be replaced:

Push and apply configuration on the generic service nodes:

Clara has dedicated configuration for its virt plugin. This configuration is set with the following two hashes in the cluster specific layer of the hiera repository:

The clara::virt_options hash notably specifies the list of generic services nodes that hosts the virtual machines and the domain templates and parameters associated to each service virtual machine. For the moment, only the default domain template and parameters are set. The second hash clara::virt_tpl_hpc_files defines the templates of Libvirt XML domains definitions. In this example, there is one default domain XML template for all virtual machines which should be fine for most Scibian HPC clusters.

The domain XML template must be located in $ADMIN/hpc-privatedata/files/$CLUSTER/virt/domain_default_template.xml. Here is a full example of this file:

In this example, the following values must be replaced:

Deploy these new settings by pushing and applying the configuration on the admin node:

Now that Libvirt and Clara virt plugin are properly setup, the various service virtual machines can be defined. The steps to define the service virtual machines are mostly generic and common to all of them. As an example for this documentation, the two service virtual machines fbdoe[1-2] will be defined.

The first step is to define the boot_params of the virtual machines in $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

Eventually, in the same file, an additional domain template and parameters association can be appended to the clara::virt_options for these new virtual machines, if the default domain parameters are not appropriate:

In this example, the following settings are overriden from the defaults:

Then, the new role doe must be defined in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/doe.yaml with all the appropriate profiles.

Push and apply configuration on admin node:

Extract MAC address of the virtual machine on the administration network:

Then add the network settings of the virtual machines in the master_network hash with their MAC addresses:

Eventually, virtual IP addresses can also be defined for the virtual machines in the vips hash of the same file.

Generate the SSH host keys in synchronization with the master_network:

Push and apply the new configuration on the generic service nodes:

Define the new virtual machines with Clara on the generic service node of your choice, for example fbservice1:

Then start the virtual machine by wiping its virtual block storage devices and boot in PXE mode:

Eventually, watch the serial console with:

You are free to define the service virtual machines you want on Scibian HPC clusters. The service virtual machines can run any software services you would like. However, some specific generic virtual machines are required in the reference architecture to run some mandatory additional services.

The required service virtual machines are:

User authentication on Scibian HPC clusters is based on LDAP directory using ldaps protocol (LDAP over SSL/TLS). This protocol requires the LDAP replica to have valid SSL certificate and asymmetric keys.

For production use, it is recommended to obtain a certificate signed by a valid PKI CA[7], either a public CA on the Internet or a CA internal to your organization. Otherwise, it is possible to use self-signed certificates.

Copy the private key and the certificate under the following paths:

Encrypt these files with clara enc plugin:

Remove the unencrypted files:

Then, append the auth::replica profile and set certificate owner to openldap in the proxy role:

Push and apply the configuration on the proxy nodes:

Finally, follow the steps documented in LDAP bootstrap procedure.

Once the LDAP replica are bootstrapped and operational, it is possible to setup NSS LDAP backend and PAM LDAP authentication on the nodes.

On Scibian HPC clusters, NSS LDAP backend and PAM authentication over LDAP are both setup with the same auth::client profile. This profile must be used in combination with the access::base profile. This profile controls the remote access rules to the nodes. By default, the profile prevents remote access to the nodes with LDAP accounts. The access rules must explicitely whitelist users and/or administrators to allow remote access with SSH.

There are two main access whitelist parameters:

The administrators related access rules must be listed in the base_options while the users related access rules must only be present in the production_options list. This way, only administrators can access the HPC cluster in maintenance mode. For example:

These parameters must be set in the roles specific layer of the hiera repository as access rules depends on the role of the nodes. For example, users may access the frontend nodes but not the admin node.

Additionally, it is also possible to setup sudo rules with the sudo::base profile and the sudo::sudo_config_opts list. This parameter is basically a list of sudo rules. For example, to allow the group of administrator to sudo any command on the admin node, add the following excerpt to file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/roles/admin.yaml:

By default, the PAM and NSS LDAP backend connect to the HPC cluster internal LDAP replica. This replica is hosted by service virtual machine. In order to make LDAP authentication on the admin nodes and generic service nodes possible for the administrators when the virtual machines are offline (typically during maintenances), it is possible to add the following parameter in the associated roles:

This way, the nodes will connect to the organization reference LDAP directory instead of the internal LDAP replica.

Push and apply the configuration on all the affected nodes with:

Slurm communications between nodes are secured using Munge which is based on a secret shared key. Generate this munge key with the following command:

Encrypt the key using Clara:

Remove the unencrypted key:

Setup the nodes and partitions managed by Slurm in the slurm::partitions_options hash in the cluster specific layer of the Hiera repository. For example:

Please refer to Slurm documentation for more details about these settings.

In the same, setup the LDAP/SlurmDBD users synchronization utility, for example:

Please refer to the example configuration file for more details.

Still in the cluster specific layer of the Hiera repository, setup the shared storage directory.

Eventually, it is possible to tune Slurm, GRES, SlurmDBD, job submit LUA script with the following parameters:

By default, the MariaDB server is setup with parameters to harden its security. Notably, the following settings are deployed by default:

Obviously, these settings can be altered in the hiera repository. Here is an example yaml excerpt to change these default values:

Some software components need to be manually bootstrapped on the batch nodes before being started:

The shared storage can be on CephFS or on NFS HA, the suitable bootstrap procedure must be performed:

Please refer to the Bootstrap procedure chapter of this document for all details.

Once the configuration is set in the Hiera repository, push and apply the configuration on the admin and batch nodes:

Check Slurm is available by running the sinfo command on the admin node. If the command report the nodes and partitions state without error, Slurm is properly running.

The diskless image is generated by the Clara images plugin. This plugin need some configuration in the cluster specific layer of the Hiera repository. Here is an example of such configuration:

The clara::live_files parameter contains a list of files deployed under the configuration directory of Clara. Their files are:

This script can notably be used to customize the image or set files and directories that are required very early in the live boot process before Puppet run.

All the files under the files_dir directory are copied without modification into the image. The required files are:

Once all these files have been added to the cluster specific files directory, the Clara p2p plugin must be configured in the cluster specific layer of the Hiera repository with the following parameter:

Push and apply the configuration on the admin node:

Now that Clara is setup, the image can be created with the following command:

Also create the associated initrd environment:

Deploy the generate image and initrd to the _p2p nodes with!

Restart peer-to-peer services to load new files:

The diskless environment is finally ready and available to frontend and compute nodes.

Before booting the frontend and compute nodes, they must be declared in the internal configuration repository in the first place. Append the nodes to the boot_params hash in $ADMIN/hpc-privatedata/hieradata/$CLUSTER/cluster.yaml:

The cowsize must be increased to 8GB from default 2GB on frontend and graphical nodes because these nodes need much more packages to be installed at boot time.

Then define the roles associated to the frontend and the compute nodes, for example front, cn and gn. For these roles definitions, keep in mind the following rules:

The nodes must be added into the master_network hash in file $ADMIN/hpc-privatedata/hieradata/$CLUSTER/network.yaml with all their network interfaces and the MAC addresses of their network interface connected to the administration and their BMC.

Generate all the SSH host keys:

Push and apply the configuration to the admin and generic service nodes:

Finally, boot all the nodes in PXE mode with Clara:

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

TBD

Generate and encrypt the SSH key used to poweroff the nodes from the batch nodes:

Then add those settings in the cluster specific layer of the hiera repository:

Where <PUBKEY> is the public key in file hpc-privatedata/files/$CLUSTER/pwmgt/id_rsa_slurm.pub.

Finally, apply the new configuration on the batch nodes and all the compute nodes:

After the reinstallation of one of the generic service nodes with a mon, it must be re-initialized. This procedure only works on a running cluster, the initial mon creation uses another command.

From an admin node:

This procedure only applies if the content of an OSD volume is lost. If the node is reinstalled without erasing the content of the OSD volume, the configuration in puppet will be enough to start the osd volume again.

The relevant OSD ID can be retrieved with:

Before doing this procedure, make sure the OSD is really down and not mounted on the OSD node.

CephFS filesystem is used between the batch nodes to shared Slurm controller state. The filesystem must be initialized before being used by Slurm.

First, mount temporarily the CephFS filesystem:

Create a subdirectory for Slurm controller, set its ownership and restrict its mode:

Finally, umount it:

Puppet-HPC is now able to use this filesystem for Slurm on batch nodes.

The hashed root password is stored in the variable profiles::cluster::root_password_hash in yaml files. The value must be encrypted using eyaml. It can be simply changed using the eyaml command.

Once changed, the new configuration must be applied on all the machines of the cluster.

The root SSH keys are stored in the internal repository. The privates keys must be encrypted. The SSH public rsa key is also in the variable openssh::server::root_public_key. It is necessary to change the files and the value of the variable at the same time. To avoid connections problems, it is necessary to follow these steps in this order:

The SSH host keys are stored, encrypted, in the internal repository. To avoid connections problems, it is necessary to follow these steps in this order:

Replacing the eyaml PKCS7 key pair consist in reality of two actions:

These steps must be followed in order to safely change the eyaml keys:

Save the old keys:

Copy the new keys in /etc/puppet/secure/keys/.

Decrypt all the yaml files encoded using the old keys:

The decrypt.yaml contains all the secret in plain text. It should be removed as soon as possible.

Encrypt the files with the new keys:

Remove the old saved keys from the admin node:

Create a tarball, encode it with clara enc and add it to the files directory of the internal repository:

Where:

At this stage, the keys are now stored encrypted in the internal repository and are available locally in the standard eyaml paths.

In the default Scibian-HPC configuration, the PKCS7 keys propagation service runs on all the generic service nodes. First, the encoded tarball must be manually copied on the nodes:

Where <generic server X> is the hostname of the generic service node.

Then apply the configuration using the new keys:

This will copy the eyaml PKCS7 key pair in the right directory to be serviced by the propagation service to all others nodes when applying the puppet configuration. These last two operations must be executed on all the generic service nodes.

Don’t forget to remove the keys from the /tmp directory on the admin node and on all the service nodes.

Replacing the AES key used to encode files in the internal repository consist in several steps.

Generate a new AES key:

For each encoded file in the internal repository, it is necessary to decode it with the old key and re-encode it with the new one.

Where:

Using clara for both operations, decode and encode, is not possible as it support only one AES key.

This re-encryption step can be automated with the reencode-file.sh script in the puppet-hpc scripts dir:

The files /tmp/oldkey and /tmp/newkey are files with just the old and new AES key respectively. This script does not depend on clara but basically performs the same steps as above.

The AES key must be placed in cluster_decrypt_password in the cluster layer of the Hiera repository:

Replace the key:

Apply the new configuration on the admin node, to update clara configuration:

The steps to change these credentials are described here:

The certificates used for monitoring are stored, encrypted, in the internal repository in <internal repository>/files/<cluster>/icinga2/certs/. Each host has a certificate and a key. The steps to follow to change them are:

The cluster must use a private cluster keyring. This keyring is used to sign the local packages repository.

It is stored in the internal repository: <internal repository>/files/<cluster>/repo/

Here are the steps to follow to change it:

Generate passwords conform with your organization policy and edit the following parameters with eyaml in the hiera repository:

These parameters correspond to the passwords of the MariaDB having respectively R/W and R/O grants on the SlurmDBD database.

Once modified, push and apply the configuration with the following commands:

The hpc-config-apply command will perform the following steps, on each batch node:

The --fanout=1 parameter of the clush command makes sure the configuration is not applied simultaneously on both batch nodes. This could cause the SlurmDBD daemon to be restarted at the same time and make this service unavailable for a short period of time.

To perform a scheduled reboot of a frontend it is better to avoid new connection going to the frontend node that will be rebooted. The new connections are highly available and load balanced with IPVS.

It is possible to remove a frontend from the pool of node accepting new connections without killing active connections with the ipvsadm command by setting the weight of a node to 0.

To list the current weight, on a frontend:

To avoid a frontend node being attributed to new sessions, the weight of the node can be manually set to 0. This setting does not completely forbid new connection to go to the node, if a user already has a session, new session will go to the same node regardless of the weight. This setting also does not block connections made directly to the node and not the virtual IP address.

The modification can be reversed by setting the weight back to 1 manually.

When a node start is should not start the keepalived service automatically. This permits a failed node to be started without it becoming master with an remaining problem.

Before starting the keepalived service, the following conditions must be met:

When these conditions are met, the service can be started:

If the node is to become master (master node in the VIP configuration or other node is down), check that the first check goes well. It runs every minutes and logs are in /var/log/user.log. The message following message must appear:

If the master node disappears, because it is turned off or because the keepalived service is stopped, the failover will happen, but it will take a bit of time (a little more than a minute). This timeout can be entirely avoided by doing a manual failover of the master node before cutting the keepalived service.

To do this, the keepalived configuration must be changed manually on the node. Edit the file /etc/keepalived/keepalived.conf. Find the configuration for the NFS VIP and change the priority to 40, and the role to BACKUP. The service must be reloaded:

The failover should happen quickly. Once the node failed over, stop the keepalived service:

The original configuration must be restored before starting the service again. This will happen if you launch a hpc-config-apply manually or if you reboot the node.

In order to invalidate and purge the packages caching service apt-cacher-ng cache content, run the following commands consecutively:

A Virtual Machine is composed of two mostly independant objects:

The two objects must be deleted separately.

The first step is to stop the Virtual Machine:

Once it is in the state SHUTOFF you can undefine it:

The VM will still appear on clara virt list with the state: MISSING. It means clara still sees the disk image but not the Virtual Machine definition.

You can then proceed with deleting the disk image, by checking the full disk image name with clara virt list --details, you must find the volume name and the pool name.

On a physical host:

On all other physical hosts: