# PrivateLink Status Abnormal Alert

The PrivateLink Status Abnormal alert indicates that a PrivateLink connection between EMQX Cloud and an external service is in an abnormal state, typically `Failed`.

This usually means that EMQX Cloud cannot reach the customer service through the interface endpoint (ENI). Data integration actions or authentication and authorization requests may also fail.

## PrivateLink Network Path

A complete PrivateLink network path contains the following components:

```text
EMQX VPC
  └─ Endpoint
        ↓
PrivateLink internal channel
        ↓
Endpoint service (customer side)
        ↓
Load balancer
        ↓
Target group
        ↓
Backend server (EC2 / ECS / Pod)
```

This alert means that the EMQX Cloud endpoint cannot reach a healthy, available backend service through this path.

## Load Balancer Health Check Failed

### Symptom

The endpoint service's load balancer has no healthy backend instances in its target group, so PrivateLink cannot establish a working connection.

PrivateLink or the endpoint service may not show an obvious error, but traffic cannot be forwarded.

### Common Causes

- Every backend instance in the target group is unhealthy.
- Backend instances fail the load balancer health check.

### Resolution

- Check the health of the load balancer's backend instances.
- Use the health check results in the load balancer console to investigate each unhealthy backend.

## Backend Service Not Listening or Port Misconfigured

### Symptom

The backend service is not listening on the port or network interface configured for the load balancer. Health checks continue to fail and PrivateLink remains unavailable.

The target group is usually unhealthy, and the backend logs contain no requests from the load balancer.

### Common Causes

- The backend service is stopped or its process exited unexpectedly.
- The backend service listens on a different port from the target group configuration.
- The service listens only on `127.0.0.1`, not `0.0.0.0`.
- The health check uses TCP, but the port is closed or blocked by a firewall.

### Resolution

- Verify that the backend service is running and listening on the correct interface and port.
- Make sure the target group port matches the backend service's listening port.
- Do not bind the service only to the loopback address. The load balancer must be able to establish a TCP connection.

## Backend Security Group Blocks Load Balancer Subnet Traffic

### Symptom

With an NLB, the load balancer itself has no security group. Traffic reaching a backend instance originates from the CIDR of the NLB subnet.

If the backend security group does not allow inbound traffic from the NLB subnet CIDR to the health check port, the health check fails and PrivateLink enters an abnormal state.

### Common Causes

- Backend security group rules do not allow the NLB subnet CIDR.
- The health check port is missing from the allowed inbound rules.

### Resolution

- Review and update the backend server's security group rules.
- Allow inbound traffic from the NLB subnet CIDR on the health check port.

## Endpoint, Endpoint Service, or Load Balancer Changed or Deleted

### Symptom

PrivateLink depends on the endpoint service and load balancer configuration. If a previously working PrivateLink connection suddenly enters an abnormal state without a change in EMQX Cloud, a critical resource on the peer side was likely changed.

### Common Causes

- The load balancer (NLB) was deleted or recreated.
- The target group configuration changed or its backend servers were replaced.
- Backend instances were stopped or removed.
- A subnet in an availability zone (AZ) was disabled.
- The AZs allowed by the endpoint service were changed.
- The endpoint was deleted manually.

### Resolution

- Review recent changes to the endpoint, endpoint service, load balancer, target group, and backend resources.
- Establish a network resource change review process to prevent accidental changes from breaking the PrivateLink dependency chain.

## Troubleshooting

1. Log in to the cloud provider console and check whether **PrivateLink** is `Failed` or in another abnormal state.
2. Verify that the **load balancer** still exists and that its status and metrics are normal.
3. Check the **target group** and verify that its backend instances are healthy.
4. Verify that the AZs enabled for the **load balancer** match the AZs that contain the backend services.
5. Check the backend server **security group** and verify that it allows inbound traffic from the load balancer subnet CIDR.
6. Work with EMQX Technical Support to use `telnet <endpoint-ip> <port>` or `nc` to test connectivity from the endpoint to the backend service port.