Set up Kubernetes

Prerequisites

  • Ensure that k8s entry is present in the softwares list in software_config.json, as mentioned below:
    "softwares": [
                {"name": "k8s", "version":"1.29.5"},
             ]

  • Ensure to run local_repo.yml with the k8s entry present in software_config.json, to download all required Kubernetes packages and images.

  • Once all the required parameters in omnia_config.yml are filled in, omnia.yml can be used to set up Kubernetes.

  • Ensure that k8s_share is set to true in storage_config.yml, for one of the entries in nfs_client_params.

Inventory details

  • For Kubernetes, all the applicable inventory groups are kube_control_plane, kube_node, and etcd.

  • The inventory file must contain:

    1. Exactly 1 kube_control_plane.

    2. At least 1 kube_node.

    3. Odd number of etcd nodes.

Note

Ensure that the inventory includes an [etcd] node. etcd is a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data. For more information, click here.

Sample inventory

[kube_control_plane]

10.5.1.101

[kube_node]

10.5.1.102

[etcd]

10.5.1.101

To install Kubernetes

Run either of the following commands:

  1. ansible-playbook omnia.yml -i inventory

  2. ansible-playbook scheduler.yml -i inventory

Note

  • To run the scheduler.yml playbook independently from the omnia.yml playbook on Intel Gaudi nodes, start by executing the performance_profile.yml playbook. Once that’s done, you can run the scheduler.yml playbook separately.

  • To add new nodes to an existing cluster, click here.

Additional installations

Omnia installs the following packages on top of the Kubernetes stack:

  1. amdgpu-device-plugin (ROCm device plugin)

This is a Kubernetes device plugin implementation that enables the registration of AMD GPU in a container cluster for compute workload. Click here for more information.

  1. mpi-operator

The MPI Operator makes it easy to run allreduce-style distributed training on Kubernetes. Click here for more information.

  1. xilinx device plugin

The Xilinx FPGA device plugin for Kubernetes is a Daemonset deployed on the Kubernetes (k8s) cluster which allows you to:

  1. Discover the FPGAs inserted in each node of the cluster and expose information about FPGA such as number of FPGA, Shell (Target Platform) type and etc.

  2. Run FPGA accessible containers in the k8s cluster

Click here for more information.

  1. nfs-client-provisioner

  • NFS subdir external provisioner is an automatic provisioner that use your existing and already configured NFS server to support dynamic provisioning of Kubernetes Persistent Volumes via Persistent Volume Claims.

  • The NFS server utilised here is the one mentioned in storage_config.yml.

  • Server IP is <nfs_client_params.server_ip> and path is <nfs_client_params>.<server_share_path> of the entry where k8s_share is set to true.

Click here for more information.

  1. nvidia-device-plugin

For the NVIDIA device plugin to function seamlessly, Omnia installs the “nvidia-container-toolkit” as part of the omnia.yml or scheduler.yml playbook execution. The NVIDIA device plugin for Kubernetes is a “DaemonSet” that allows you to automatically:

  1. Expose the number of GPUs on each nodes of your cluster

  2. Keep track of the health of your GPUs

  3. Run GPU enabled containers in your Kubernetes cluster

Click here for more information.

  1. gaudi-device-plugin

    The Gaudi device plugin is a Kubernetes device plugin implementation that enables the registration of Intel Gaudi AI accelerators in a container cluster. This plugin enables the efficient utilization of Gaudi accelerators for compute workloads within the cluster. For the gaudi-device-plugin to function seamlessly, Omnia installs the “habanalabs-container-runtime” as part of the omnia.yml or scheduler.yml playbook execution.

    The Gaudi device plugin for Kubernetes is a “DaemonSet” that allows you to automatically:

    1. Enable the registration of Intel Gaudi accelerators in your Kubernetes cluster.

    2. Keep track of device health.

    3. Run jobs on the Intel Gaudi accelerators.

    Click here for more information.

Optional installation

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.