S3 SSL Error Creating a Diff
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bromanko
	Jul 22 2014, 2:05 AM

Description

I have started receiving errors from the Phabricator S3 file engine when uploading to S3. This happens at both the /file endpoint and when creating diffs, pasting files, etc. The error states:

PhabricatorS3FileStorageEngine: S3Exception: S3::putObject(): [51] SSL peer certificate or SSH remote key was not OK

This is running on an Ubuntu instance on EC2. Aside from Ubuntu security updates nothing has changed on this box. All S3 related commands fail.

I have tried curl'ing to various S3 URLs from the box (as ubuntu/root/www-data) and do not receive certificate errors.

Any thoughts on how to debug this or fix the issue would be greatly appreciated.

Event Timeline

bromanko created this task.Jul 22 2014, 2:05 AM

bromanko raised the priority of this task from to High.

bromanko updated the task description. (Show Details)

bromanko added projects: Phabricator, Files.

bromanko added a subscriber: bromanko.

You can try setting this to false near line 112 of S3.php:

public static $useSSLValidation = true;

This will disable validation, however.

You can check if curl.cainfo is set to something weird in your php.ini.

Beyond this, I'm not sure what could be wrong.

I confirmed that disabling $useSSLValidation does work but I would prefer not to do that.

There is nothing related to curl in php.ini.

Below is a stacktrace from the nginx error log related to the error. Do you know if this is an SSL issue connecting to our Phabricator instance vs connecting to S3? That might help me debug.

2014/07/22 15:38:56 [error] 6210#0: *250 FastCGI sent in stderr: "PHP message: [2014-07-22 08:38:56] EXCEPTION: (S3Exception) S3::putObject(): [51] SSL peer certificate or SSH remote key was not OK at [<phabricator>/externals/s3/S3.php:654]
PHP message:   #0 S3::__triggerError(string, string, integer) called at [<phabricator>/externals/s3/S3.php:654]
PHP message:   #1 S3::putObject(string, string, string, string) called at [<phabricator>/src/applications/files/engine/PhabricatorS3FileStorageEngine.php:47]
PHP message:   #2 PhabricatorS3FileStorageEngine::writeFile(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:342]
PHP message:   #3 PhabricatorFile::writeToEngine(PhabricatorS3FileStorageEngine, string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:227]
PHP message:   #4 PhabricatorFile::buildFromFileData(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:304]
PHP message:   #5 PhabricatorFile::newFromFileData(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:790]
PHP message:   #6 PhabricatorFile::loadBuiltins(PhabricatorUser, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:816]
PHP message:   #7 PhabricatorFile::loadBuiltin(PhabricatorUser, string) called at [<phabricator>/src/applications/project/query/PhabricatorProjectQuery.php:197]
PHP message:   #8 PhabricatorProjectQuery::didFilterPage(array) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:252]
PHP message:   #9 PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/phid/type/PhabricatorPHIDType.php:95]
PHP message:   #10 PhabricatorPHIDType::loadObjects(PhabricatorObjectQuery, array) called at [<phabricator>/src/applications/phid/query/PhabricatorObjectQuery.php:130]
PHP message:   #11 PhabricatorObjectQuery::loadObjectsByPHID(array, array) called at [<phabricator>/src/applications/phid/query/Ph
2014/07/22 15:38:56 [error] 6210#0: *250 FastCGI sent in stderr: "/src/applications/files/storage/PhabricatorFile.php:816]
PHP message:   #3 PhabricatorFile::loadBuiltin(PhabricatorUser, string) called at [<phabricator>/src/applications/project/query/PhabricatorProjectQuery.php:197]
PHP message:   #4 PhabricatorProjectQuery::didFilterPage(array) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:252]
PHP message:   #5 PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/phid/type/PhabricatorPHIDType.php:95]
PHP message:   #6 PhabricatorPHIDType::loadObjects(PhabricatorObjectQuery, array) called at [<phabricator>/src/applications/phid/query/PhabricatorObjectQuery.php:130]
PHP message:   #7 PhabricatorObjectQuery::loadObjectsByPHID(array, array) called at [<phabricator>/src/applications/phid/query/PhabricatorObjectQuery.php:63]
PHP message:   #8 PhabricatorObjectQuery::loadPage() called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:219]
PHP message:   #9 PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/phid/query/PhabricatorHandleQuery.php:25]
PHP message:   #10 PhabricatorHandleQuery::loadPage() called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:219]
PHP message:   #11 PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/policy/query/PhabricatorPolicyQuery.php:98]
PHP message:   #12 PhabricatorPolicyQuery::loadPage() called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:219]
PHP message:   #13 PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/differential/customfield/DifferentialViewPolicyField.php:34]
PHP message:   #14 DifferentialViewPolicyField::renderEditControl(array) called at [<phabricator>/src/infrastructure/customfield/field/PhabricatorCustomFieldList.php:117]
PHP message:   #15 PhabricatorCustomFieldList::appendFieldsToForm(AphrontFormView)
2014/07/22 15:38:57 [error] 6210#0: *251 FastCGI sent in stderr: "PHP message: [2014-07-22 08:38:57] EXCEPTION: (S3Exception) S3::getObject(phab.ourserver.com, phabricator/i5/wk/omuwilcy2itqrtjk): [51] SSL peer certificate or SSH remote key was not OK at [<phabricator>/externals/s3/S3.php:723]
PHP message:   #0 S3::__triggerError(string, string, integer) called at [<phabricator>/externals/s3/S3.php:723]
PHP message:   #1 S3::getObject(string, string) called at [<phabricator>/src/applications/files/engine/PhabricatorS3FileStorageEngine.php:59]
PHP message:   #2 PhabricatorS3FileStorageEngine::readFile(string) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:475]
PHP message:   #3 PhabricatorFile::loadFileData() called at [<phabricator>/src/applications/files/controller/PhabricatorFileDataController.php:44]
PHP message:   #4 PhabricatorFileDataController::processRequest() called at [<phabricator>/webroot/index.php:95]" while reading response header from upstream, client: 10.10.0.206, server: phab.ourserver.com, request: "GET /file/data/waku7odgujmqqsbq3grq/PHID-FILE-jinep2baztxbz4xkgfuq/profile-github_-profile.jpg HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "phab.ourserver.com"

The issue is with the Phabricator server connecting to S3.

You could maybe try setting curl.cainfo to point at /path/to/libphutil/resources/ssl/default.pem, which is the CA bundle that cURL distributes. If the cURL linked to PHP (but not the curl on the CLI) somehow has broken certs, that might fix it. This is really fishing, though.

That fixed it. Time to figure out what happened to the certs on this box.

Thanks!!

bromanko closed this task as Resolved.Jul 22 2014, 6:43 PM

bromanko claimed this task.

It fixed part of it. Turns out that was not the extent of the issue. We noticed that file uploads were still failing. The actual root cause was that our bucket name contained a dot in it. This AWS forum post pointed us in the right direction:

https://forums.aws.amazon.com/thread.jspa?messageID=366218

We switched to a bucket without a dot in the name and the issues resolved.

I am uncertain what changed to cause the problem to appear in the first place.

Yes, I can confirm this too.

A dot "." in AWS bucket name will trigger this issue.

S3 SSL Error Creating a DiffClosed, ResolvedPublicActions

Description

Event Timeline

S3 SSL Error Creating a Diff
Closed, ResolvedPublic
Actions