Troubleshooting

Known Issues

Workload Name Length Limitation

This will lead to a constant scrolling grey bar for workload health, where workloads never show a health status.

This is due to a known issue with Console that restricts the length of workload names. If your deployment name exceeds 57 (47 in 1.1.0 and earlier) characters, Console will be unable to determine the workload name, and be unable to render the health of the workload.

This is because Kubernetes truncates the pod name to keep it under 64 characters, while allowing room for the pod hash - and Console relies on the pod name to determine the workload name.

Until this is resolved in Console, the workaround is to limit the length of deployment names.

lightbend_monitor_api_errors

An alert from lightbend_monitor_api_errors indicates that the console failed to process one or more monitors. This can occur, for example, when you upgrade the console and it cannot handle an existing monitor. This alert warns when the metric is greater than 0.

If this alert fires, please perform a debug dump with the installer script and open a support ticket with the dump file attached:

./lbc.py debug-dump --namespace=<console namespace>

OOMKilled Containers and Pods

It is possible that your use of Console requires more resources than those specified in the defaults. If you have containers and pods terminated with an OOMKilled reason, try to increase their memory limit. For example, you can increase the memory limit for the Console Prometheus container by adding the following setting to your values.yaml file:

prometheusMemoryRequest: 512Mi

See more details about how to customize Console installation in the install guide.

Make sure to check the memory available for the Kubernetes cluster node. If the sum of all memory limits is higher than node capacity, it can exhaust the memory, causing some pods to terminate to free some memory.