Troubleshooting Guide
Control plane logs
All log files can be viewed via the Dashboard tab ( ). The Default Dashboard displays omnia.log
and syslog
. Custom dashboards can be created per user requirements.
Below is a list of all logs available to Loki and can be accessed on the dashboard:
Name |
Location |
Purpose |
Additional Information |
---|---|---|---|
Omnia Logs |
/var/log/omnia.log |
Omnia Log |
This log is configured by Default. This log can be used to track all changes made by all playbooks in the omnia directory. |
Accelerator Logs |
/var/log/omnia/accelerator.log |
Accelerator Log |
This log is configured by Default |
Monitor Logs |
/var/log/omnia/monitor.log |
Monitor Log |
This log is configured by Default |
Network Logs |
/var/log/omnia/network.log |
Network Log |
This log is configured by Default |
Platform Logs |
/var/log/omnia/platforms.log |
Platform Log |
This log is configured by Default |
Provision Logs |
/var/log/omnia/provision.log |
Provision Log |
This log is configured by Default |
Scheduler Logs |
/var/log/omnia/scheduler.log |
Scheduler Log |
This log is configured by Default |
Security Logs |
/var/log/omnia/security.log |
Security Log |
This log is configured by Default |
Storage Logs |
/var/log/omnia/storage.log |
Storage Log |
This log is configured by Default |
Telemetry Logs |
/var/log/omnia/telemetry.log |
Telemetry Log |
This log is configured by Default |
Utils Logs |
/var/log/omnia/utils.log |
Utils Log |
This log is configured by Default |
Cluster Utilities Logs |
/var/log/omnia/utils_cluster.log |
Cluster Utils Log |
This log is configured by Default |
syslogs |
/var/log/messages |
System Logging |
This log is configured by Default |
Audit Logs |
/var/log/audit/audit.log |
All Login Attempts |
This log is configured by Default |
CRON logs |
/var/log/cron |
CRON Job Logging |
This log is configured by Default |
Pods logs |
/var/log/pods/ * / * / * log |
k8s pods |
This log is configured by Default |
Access Logs |
/var/log/dirsrv/slapd-<Realm Name>/access |
Directory Server Utilization |
This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’) |
Error Log |
/var/log/dirsrv/slapd-<Realm Name>/errors |
Directory Server Errors |
This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’) |
CA Transaction Log |
/var/log/pki/pki-tomcat/ca/transactions |
FreeIPA PKI Transactions |
This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’) |
KRB5KDC |
/var/log/krb5kdc.log |
KDC Utilization |
This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’) |
Secure logs |
/var/log/secure |
Login Error Codes |
This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’) |
HTTPD logs |
/var/log/httpd/ * |
FreeIPA API Calls |
This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’) |
DNF logs |
/var/log/dnf.log |
Installation Logs |
This log is configured on Rocky OS |
Zypper Logs |
/var/log/zypper.log |
Installation Logs |
This log is configured on Leap OS |
BeeGFS Logs |
/var/log/beegfs-client.log |
BeeGFS Logs |
This log is configured on BeeGFS client nodes. |
Provisioning logs
Logs pertaining to provisioning can be viewed in /var/log/xcat/xcat.log
on the target nodes.
Logs of individual containers
A list of namespaces and their corresponding pods can be obtained using:
kubectl get pods -A
Get a list of containers for the pod in question using:
kubectl get pods <pod_name> -o jsonpath='{.spec.containers[*].name}'
Once you have the namespace, pod and container names, run the below command to get the required logs:
kubectl logs pod <pod_name> -n <namespace> -c <container_name>
Connecting to internal databases
- TimescaleDB
Go inside the pod:
kubectl exec -it pod/timescaledb-0 -n telemetry-and-visualizations -- /bin/bash
Connect to psql:
psql -U <postgres_username>
Connect to database:
< timescaledb_name >
- MySQL DB
Go inside the pod:
kubectl exec -it pod/mysqldb-n telemetry-and-visualizations -- /bin/bash
Connect to psql:
psql -U <mysqldb_username> -p <mysqldb_password>
Connect to database:
USE <mysqldb_name>
Checking and updating encrypted parameters
Move to the filepath where the parameters are saved (as an example, we will be using
provision_config.yml
):cd input/
To view the encrypted parameters:
ansible-vault view provision_config.yml –vault-password-file .provision_vault_key
To edit the encrypted parameters:
ansible-vault edit provision_config.yml –vault-password-file .provision_vault_key
Checking pod status on the control plane
Select the pod you need to troubleshoot from the output of
kubectl get pods -A
Check the status of the pod by running
kubectl describe pod <pod name> -n <namespace name>