Improve Log Metadata For Bot Unable To Join The Cluster

Apr 23, 2025 by ADMIN 56 views

Problem Statement

As a Teleport Cluster administrator, it can be challenging to identify the bot that is failing to join the cluster, especially when the log data does not provide sufficient information. The current log entry only captures the IP address of the remote host, but not the bot name. This makes it difficult to determine which bot is experiencing issues.

Current Log Entry

The current log entry is as follows:

Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]: 2025-04-23T12:28:42.675Z WARN [AUTH]      "" can not join the cluster with role Bot, token error: token expired or not found auth/join.go:68
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]: 2025-04-23T12:28:42.675Z WARN [AUTH]      Failure to join cluster occurred error:[
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]: ERROR REPORT:
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]: Original Error: *trace.AccessDeniedError "" can not join the cluster with role "Bot", token expired or not found
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]: Stack Trace:
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/lib/auth/join.go:75 github.com/gravitational/teleport/lib/auth.(*Server).checkTokenJoinRequestCommon
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/lib/auth/join.go:325 github.com/gravitational/teleport/lib/auth.(*Server).RegisterUsingToken
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/lib/joinserver/joinserver.go:401 github.com/gravitational/teleport/lib/joinserver.(*JoinServiceGRPCServer).RegisterUsingToken
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/api@v0.0.0/client/proto/joinservice.pb.go:1114 github.com/gravitational/teleport/api/client/proto._JoinService_RegisterUsingToken_Handler.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/lib/auth/middleware.go:558 github.com/gravitational/teleport/lib/auth.(*Middleware).withAuthenticatedUserUnaryInterceptor
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1211.golang.org/grpc.getChainUnaryHandler.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/lib/limiter/limiter.go:152 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*Limiter).UnaryServerInterceptorWithCustomRate.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/api@v0.0.0/metadata/metadata.go:76 github.com/gravitational/teleport/api/metadata.UnaryServerInterceptor
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:76 github.com/gravitational/teleport/api/utils/grpc/interceptors.GRPCServerUnaryErrorInterceptor
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:22 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func2
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.55.0/interceptor.go:316 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1202 google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         github.com/gravitational/teleport/api@v0..0/client/proto/joinservice.pb.go:1116 github.com/gravitational/teleport/api/client/proto._JoinService_RegisterUsingToken_Handler
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1394 google.golang.org/grpc.(*Server).processUnaryRPC
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1805 google.golang.org/grpc.(*Server).handleStream
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         google.golang.org/grpc@v1.66.3/server.go:1029 google.golang.org/grpc.(*Server).serveStreams.func2.1
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]:         runtime/asm_amd64.s:1700 runtime.goexit
Apr 23 12:28:42 ip-xyx.xy.xy.xyz.sa-east-1.compute.internal teleport[358372]: User Message: "" can not join the cluster with role "Bot", token expired or not found] host_id: node_name: remote_addr:xy.xyz.xyz.xy role:Bot auth/join.go:150

Additional Information in Audit Event

The same information should be available in the corresponding Teleport Audit event. The following information is captured:

{
  "addr.remote": "xy.xyz.xyz.xy",
  "cluster_name": "cvasquez-teleport-cluster",
  "code": "TJ001E",
  "ei": 0,
  "error": "\"\" can not join the cluster with role \"Bot\", token expired or not found",
  "event": "bot.join",
  "success": false,
  "time": "2025-04-23T13:11:44.294Z",
  "uid": "5d07af5d-5f9c-45e5-adf1-ceaeb778215f"
}

Problem Solved

This solution helps Teleport Cluster administrators more easily identify the bot failing to join the cluster.

Workaround

The addition of addr.remote helped, but was described as being a long process to narrow down the bot with that information.

Teleport Cluster Version

The Teleport Cluster version is 17.2.7, 16.5.0.

Solution

Q: What is the current issue with the log metadata?

A: The current log entry only captures the IP address of the remote host, but not the bot name. This makes it difficult to determine which bot is experiencing issues.

Q: What information should be captured in the log entry?

A: The log entry should capture the bot name, which is failing to join the cluster.

Q: Why is it important to capture the bot name in the log entry?

A: Capturing the bot name in the log entry helps Teleport Cluster administrators more easily identify the bot failing to join the cluster.

Q: What is the current workaround for identifying the bot?

A: The current workaround is to use the addr.remote information, but it is described as being a long process to narrow down the bot with that information.

Q: What is the recommended solution for improving the log metadata?

A: The recommended solution is to capture the bot name in the log entry.

Q: How can the bot name be captured in the log entry?

A: The bot name can be captured in the log entry by modifying the Teleport Cluster configuration to include the bot name in the log entry.

Q: What are the benefits of capturing the bot name in the log entry?

A: Capturing the bot name in the log entry provides several benefits, including:

Improved identification of the bot failing to join the cluster
Reduced time and effort required to troubleshoot issues
Enhanced overall cluster management and administration

Q: What is the recommended Teleport Cluster version for implementing this solution?

A: The recommended Teleport Cluster version is 17.2.7, 16.5.0.

Q: Are there any additional steps required to implement this solution?

A: Yes, additional steps may be required to implement this solution, including modifying the Teleport Cluster configuration and updating the log entry format.

Q: What is the estimated time and effort required to implement this solution?

A: The estimated time and effort required to implement this solution will depend on the specific requirements and complexity of the implementation.

Q: Who should implement this solution?

A: This solution should be implemented by Teleport Cluster administrators or system administrators with experience in managing and troubleshooting Teleport Cluster configurations.