Runner Disconnected
Docker
1
Find the runner containerFilter to just the runner:You should see output like:
2
Check the logs
3
Diagnose the error
- Timeout
Network communication to Blink is down or limited — likely due to a firewall.
Refer to the outbound traffic documentation for the required firewall rules.
Verify Blink is up and running:
Check the #p1-production-alert Slack channel for any active production alerts.
Kubernetes
1
Check pod status
2
Get runner logs
Note: This is valid only if you have a single
blink-runner instance. If HPA (Horizontal Pod Autoscaling) is enabled, retrieve each pod’s logs individually:3
Diagnose the error
- Timeout
Network communication to Blink is down or limited — likely due to a firewall.
Refer to the outbound traffic documentation.
Verify Blink is up:
Check #p1-production-alert for active production alerts.
Runner Upgrade Failure
Docker
Note: Docker handles runner upgrades differently and does not use the
blink-operator. 1
Get the runner logs
2
Look for upgrade indicatorsSearch for one of the following messages:Expected messages:
Creating new container with name: blink-runner-{version} and configuration from container with id: {some-container-id}Found existing container with name: blink-runner-{version} already running with id: {some-container-id}
3
Debug the failed runnerUsing the
{some-container-id} from the log output, locate and inspect the existing runner containers. Their logs will contain the details of the failure — review them to determine whether you can resolve the issue or need to escalate to R&D.4
Recovery stepsTry restarting the container:If that doesn’t work, perform a manual upgrade through the UI, then re-check from the top of the Docker logs section.
Kubernetes
1
Check pod status
2
Get runner logs
If HPA is enabled, check each pod individually:
3
Check for the blink-operator
The operator will automatically roll back after 5 minutes if the upgrade fails. If you don’t see it running, check the runner logs for:If you see this message and the runner is still running but the operator is no longer alive, the upgrade has failed.
4
Validate operator deployment
Debug before escalating to R&D
- Operator is running → Do nothing, wait for it to complete.
- Operator exists but is not running → Wait one minute, then check again. If still not running, delete it:
5
Restart the runner
6
Trigger a manual upgrade via the UI
In the Blink UI, navigate to the runner group, click the 3-dot menu, and select Update.
7
Re-run the full process
Go through the entire Kubernetes upgrade process again and monitor the
blink-operator logs closely.
Based on those logs, determine whether you can resolve the issue with the customer or whether R&D escalation is needed.