S3: You're Speaking Plain HTTP To An SSL-enabled Server Port

by ADMIN 61 views

Bug Description

When using Mimir on Canonical's Private Openstack Cloud (Prodstack 6), a peculiar error occurs on juju debug-log. The error message indicates that the service is unable to successfully send a request to object storage due to a plain HTTP request being sent to an SSL-enabled server port. This issue arises despite the explicit configuration of using the HTTPS protocol on port 443.

The error message:

2025-04-21T21:24:23.084Z [mimir] ts=2025-04-21T21:24:23.080326537Z caller=mimir.go:902 level=error msg="module failed" module=sanity-check err="starting module sanity-check: invalid service state: Failed, expected: Running, failure: blocks storage: unable to successfully send a request to object storage: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br />\nReason: You're speaking plain HTTP to an SSL-enabled server port.<br />\n Instead use the HTTPS scheme to access this URL, please.<br />\n</p>\n<hr>\n<address>Apache/2.4.52 (Ubuntu) Server at radosgw.ps6.canonical.com Port 80</address>\n</body></html>"

The configuration:

- `juju config s3-integrator-loki bucket='k8s-stg-is-tests-cos_loki_data'`
- `juju config s3-integrator-loki endpoint='https://radosgw.ps6.canonical.com:443/'`
- `juju config s3-integrator-loki path='loki'`
- `juju config s3-integrator-loki region='prodstack6'`
- `juju config s3-integrator-loki s3-uri-style='path'`
- `juju config s3-integrator-loki trust='True'`

As evident from the configuration, the HTTPS protocol is explicitly specified on port 443. However, the issue persists even when the port is not specified, and the protocol is left as is.

Inspection of units:

k8s-stg-is-tests-cos@is-bastion-ps6:~/plan$ juju show-unit mimir-worker/0 | grep radosgw -C 2
        /data/tsdb-sync\n  storage_prefix: blocks\n  tsdb:\n    dir: /data/tsdb\ncommon:\n  storage:\n    backend:
        s3\n    s3:\n      access_key_id: REDACTED\n      bucket_name:
        k8s-stg-is-tests-cos_mimir_data\n      endpoint: radosgw.ps6.canonical.com:443/\n      insecure:
        true\n      region: prodstack6\n      secret_access_key: REDACTED\ncompactor:\n  data_dir:
        /data/data-compactor\ningester:\n  ring:\n    replication_factor: 3\nmemberlist:\n  cluster_label:

The insecure: true flag is present in configuration, which is not explicitly set by the user. This might be the root cause of the issue.

To Reproduce

  1. Deploy Mimir Coordinator
  2. Deploy Mimir Worker
  3. Integrate Mimir Coordinator to Mimir Worker
  4. Deploy S3 Integrator
  5. Configure S3 Integrator using HTTPS
  6. Integrate S3 Integrator to Mimir Coordinator
  7. See the error

Environment

k8s-stg-is-tests-cos@is-bastion-ps6

Relevant log output

2025-04-21T21:24:23.084Z [mimir] ts=2025-04-21T21:24:23.080326537Z caller=mimir.go:902 level=error msg="module failed" module=sanity-check err="starting module sanity-check: invalid service state: Failed, expected: Running, failure: blocks storage: unable to successfully send a request to object storage: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br />\nReason: You're speaking plain HTTP to an SSL-enabled server port.<br />\n Instead use the HTTPS scheme to access this URL, please.<br />\n</p>\n<hr>\n<address>Apache/2.4.52 (Ubuntu) Server at radosgw.ps6.canonical.com Port 80</address>\n</body></html>"

Additional context

We have a live environment with this issue at the moment, if you'd like to debug or if there's more information we can provide.

Possible Solutions

  1. Verify the configuration: Double-check the configuration of the S3 Integrator to ensure that the HTTPS protocol is correctly specified on port 443.
  2. Inspect the units: Use juju show-unit to inspect the units and verify that the insecure: true flag is not present in the configuration.
  3. Check the logs: Review the logs to see if there are any other errors or warnings that might be related to this issue.
  4. Debug the issue: Use a debugger or a logging tool to step through the code and identify the exact location where the plain HTTP request is being sent to the SSL-enabled server port.

Q&A

Q: What is the issue with the S3 Integrator? A: The S3 Integrator is configured to use the HTTPS protocol on port 443, but it is still sending plain HTTP requests to the SSL-enabled server port.

Q: Why is this happening? A: The insecure: true flag is present in the configuration, which is not explicitly set by the user. This flag allows the S3 Integrator to send plain HTTP requests to the SSL-enabled server port.

Q: How can I resolve this issue? A: To resolve this issue, you need to remove the insecure: true flag from the configuration. You can do this by setting the insecure flag to false in the S3 Integrator configuration.

Q: What are the consequences of sending plain HTTP requests to an SSL-enabled server port? A: Sending plain HTTP requests to an SSL-enabled server port can lead to security vulnerabilities and data breaches. It is essential to use the HTTPS protocol to ensure the security and integrity of the data.

Q: How can I verify that the S3 Integrator is using the HTTPS protocol? A: You can verify that the S3 Integrator is using the HTTPS protocol by checking the logs and the configuration. You can also use a tool like curl to test the connection and ensure that it is using the HTTPS protocol.

Q: What are the benefits of using the HTTPS protocol? A: The HTTPS protocol provides several benefits, including:

  • Security: HTTPS encrypts the data transmitted between the client and the server, ensuring that it remains confidential and secure.
  • Integrity: HTTPS ensures that the data is not tampered with during transmission, ensuring that it remains intact and accurate.
  • Authentication: HTTPS authenticates the identity of the server, ensuring that it is a trusted and legitimate server.

Q: How can I ensure that the S3 Integrator is configured correctly? A: To ensure that the S3 Integrator is configured correctly, you need to:

  • Verify the configuration: Double-check the configuration of the S3 Integrator to ensure that it is correctly specified.
  • Inspect the units: Use juju show-unit to inspect the units and verify that the insecure: true flag is not present in the configuration.
  • Check the logs: Review the logs to see if there are any other errors or warnings that might be related to this issue.
  • Debug the issue: Use a debugger or a logging tool to step through the code and identify the exact location where the plain HTTP request is being sent to the SSL-enabled server port.

By following these steps, you can ensure that the S3 Integrator is configured correctly and that it is using the HTTPS protocol to ensure the security and integrity of the data.

Troubleshooting Tips

  • Check the configuration: Verify that the configuration of the S3 Integrator is correct and that the HTTPS protocol is specified.
  • Inspect the units: Use juju show-unit to inspect the units and verify that the insecure: true flag is not present in the configuration.
  • Check the logs: Review the logs to if there are any other errors or warnings that might be related to this issue.
  • Debug the issue: Use a debugger or a logging tool to step through the code and identify the exact location where the plain HTTP request is being sent to the SSL-enabled server port.

By following these troubleshooting tips, you can identify and resolve the issue with the S3 Integrator and ensure that it is using the HTTPS protocol to ensure the security and integrity of the data.