5.16.2

S3 Server-Side Notes & Requirements

Using the direct-to-S3 uploader module means that most of the server-side work required to parse upload requests is handled by Amazon for you. However, there is some minimal communication required between Fine Uploader and your local server. This document will outline both required and optional server-side tasks. Note that, at the writing of this document, Fine Uploader developers have provided fully functional server-side examples in PHP, node.js, Python, and Java. Other examples will be created in the future.

Your signature endpoint must, at the very least:

  • Sign non-chunked requests
  • Sign chunked requests

Creating your S3 bucket's CORS configuration

In order to use the upload-to-S3 feature, you MUST properly set the CORS configuration in your S3 bucket(s). Fine Uploader must make cross-origin requests to S3 whenever it communicates with AWS. You can read more about CORS configuration on the AWS developer site. A simple and typical CORS configuration would look like this:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <ExposeHeader>ETag</ExposeHeader>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Some notes on the above configuration:

  • The POST method is required to allow HTML Form uploads by Fine Uploader, as well as other various REST API calls. You will always need to include this.
  • The PUT method is required only if you enable the chunking feature in Fine Uploader.
  • The DELETE method is if you plan on deleting files from your bucket server-side. This would be critical if you enable Fine Uploader's delete file feature. It is also required if you enable the chunking feature, as Fine Uploader sends a DELETE request to S3 when a user cancels an in-progress upload in order to ensure S3 removes all chunks associated with that upload present in your bucket.
  • You can (and probably should) make the AllowedOrigin condition(s) a bit more restrictive. If you know specifically what domains will host your Fine Uploader instance, include these in the AllowedOrigin tag value.
  • You can (and probably should) make the AllowedHeader condition(s) a bit more restrictive. If you do, you will always need to allow the following headers: "origin" and "content-type". If you have the chunking feature enabled, you will also need to include: "x-amz-acl", "x-amz-meta-qqfilename", "x-amz-date", and "authorization". If you have chunking enabled, you will also need to authorize headers for any custom parameters (user metadata) you want to attach to the object in S3. All parameters are sent as headers in the "Initiate Multipart Upload" request sent by Fine Uploader with a prefix of "x-amz-meta-".
  • If you are using the scaling feature, you will need to account for these headers in your CORS configuration as well: "x-amz-meta-qquuid", "x-amz-meta-qqparentuuid", and "x-amz-meta-qqparentsize".
  • If you are using version 4 signature support, you will also need to account for a header of "x-amz-content-sha256" for all chunked uploads.

Signing non-chunked upload requests (required: all browsers)

The only required server-side task for all browsers is an endpoint that provides a version 2 or version 4 signature for the policy document. This endpoint corresponds to the signature.endpoint property. Be sure you set the signature.version configuration option to 4 if you are sending files to a bucket in a region that only supports version 4 signatures.

Fine Uploader will construct a policy document (required for securely uploading the file to your S3 bucket) which then must be signed using your AWS secret key. When Fine Uploader requires this signature, it will send a POST request to the endpoint specified in your signature.endpoint option, passing the JSON policy document in the request payload with a Content-Type of "application/json". You can read more about policy documents on Amazon's developer site.

For version 4 signatures, a query parameter of "v4" will be appended to the signature endpoint's URI. This will make it easy for your signature server to determine the correct signing algorithm to use.

Verifying the policy

When your server receives this policy document, it should first verify that the policy contains expected properties. Some common properties to validate on a policy include the "bucket" and the "content-length-range". The PHP server-side example maintained by Fine Uploader developers validates these two properties.

If the policy document appears incorrect (due to client-side tampering), your server should immediately return a response with a status code of 500, a Content-Type of "application/json" and the following payload:

{
    "invalid": true
}

The above response will let Fine Uploader know that the policy document may have been tampered with client-side, and it will not send the file to S3 until the issue is addressed.

If there is any other error generating the signature, you may return a response with a status code of 500, a Content-Type of "application/json", and a payload including an error message that will be displayed to the user:

{
    "error": "There was an error generating the AWS signature"
}

If your server returns an error response with no "error" attribute, a default message will be used.

Signing the policy - version 2 signatures

If your server determines that the policy document is accurate, it should then base-64 encode the policy document, generate a base-64 encoded HMAC SHA1 of the policy document using your AWS secret key, and then return the base-64 encoded policy document and the base-64 encoded signature in the response payload with a status of 200-204 and a Content-Type of "application/json". Your response payload should follow this format:

{
    "policy": "INSERT BASE-64 ENCODED POLICY HERE",
    "signature": "INSERT BASE-64 ENCODED SIGNED POLICY HERE"
}

It's quite simple to sign the policy document server-side. Amazon's developer site also provides some code in several languages illustrating how to do this.

Signing the policy - version 4 signatures

Amazon has made the version 4 signature algorithm quite complicated to implement for unknown reasons. If you'd like to see an example of signing a policy document supplied by Fine Uploader S3 using the version 4 signing process, have a look at the Fine Uploader PHP S3 signature server example. The general process is as follows:

  1. Handle the policy signature request from Fine Uploader S3. Look at the query string to determine if this is a version 4 signature, or not.
  2. Locate the policy document in the request body. Note that the entire contents of the request body represent this JSON policy.
  3. Locate the conditions array in the policy.
  4. Locate the x-amz-credential item in the conditions array.
  5. Generate the signature as outlined in the AWS version 4 signature calculate guide. The relevant section is at the bottom of the linked page. All of the values you need for the "Signing Key" section are included in the value of the x-amz-credential element in the policy document.
  6. Return a JSON response with two properties: "policy" which includes the base-64-encoded policy document, and "signature" which includes the generated signature.

Note that you will have to base64-encode the policy document to generate the "SigningKey", which is needed for the last step in the signature generation process.

Signing chunked uploads (required: modern browsers)

Fine Uploader S3 uses Amazon S3's REST API to initiate, upload, complete, and abort multipart uploads. The REST API handles authentication by signing canonically formatted headers. This signing is something you need to implement server-side.

Providing a chunked upload signature - version 2 signatures

All your server needs to do to authenticate and supported chunked uploads direct to Amazon S3 is sign a string representing the headers of the request that Fine Uploader sends to S3. This string is found in the payload of the signature request:

{ "headers": /* string to sign */ }

The presence of this property indicates to your sever that this is, in fact, a request to sign a REST/multipart request and not a policy document.

This signature for the headers string differs slightly from the policy document signature. You should NOT base64 encode the headers string before signing it. All you must do, server-side, is generate an HMAC SHA1 signature of the string using your AWS secret key and then base64 encode the result. Your server should respond with the following in the body of an 'application/json' response:

{ "signature": /* signed headers string */ }

Providing a chunked upload signature - version 4 signatures

If you'd like to see an example of generating a signature for a chunked upload request created by Fine Uploader S3 using the version 4 signing process, have a look at the Fine Uploader PHP S3 signature server example. The general process is as follows:

  1. Handle a chunked upload signature request from Fine Uploader. Look at the query string to determine if this is a version 4 request, or not. The presence of a "headers" property in the JSON-encoded message body indicates that this is indeed related to a chunked upload request.
  2. Follow the signing process outlined in the diagram on Amazon's version 4 header-based auth documentation page.
  3. The "headers" string provided by Fine Uploader corresponds to the "StringToSign" portion of the diagram on the page referenced in the previous step, with one notable difference. Instead of a hashed "canonical request" at the end of the "StringToSign", Fine Uploader S3 will include the raw newline-delimited canonical request. This allows you to properly inspect the request before signing it.
    1. Examine the raw canonical request string at the end of the headers string to verify that the request is valid before signing it.
    2. Replace the raw canonical request at the end of the "StringToSign" sent by Fine Uploader S3 with a SHA256 hashed version.
    3. Now generate the signature using this constructed "StringToSign" by following the logic in the third step (titled "Signature") of the above-mentioned AWS document.
  4. Return a JSON response with the generated signature as the value of a "signature" property.

Supporting IE9 (and older) and Android 2.3.x (and older)

For browsers that do not support the File API, Fine Uploader must submit selected files inside of form, targeting a specific dynamically-generated iframe. When the response comes in, Fine Uploader will not be able to determine if the upload was successful since the response originated from a domain other than the one hosting the uploader. To get around this, we can ask Amazon to redirect the response, on success, to an endpoint of our choice. This is where the iframeSupport option comes into play.

Your iframeSupport.localBlankPagePath value must point to a page on your server. It can (and probably should) be an empty HTML page, but it MUST reside on the same origin/domain as the one hosting your upload page.

Supporting the upload success POST to your server (optional)

If you would like Fine Uploader to notify your server when any file has been successfully uploaded to S3, you should set the uploadSuccess.endpoint property. If this is set, Fine Uploader will send a POST request to your server with a Content-Type of "application/x-www-form-urlencoded". The payload of this request, by default, will contain the following information:

  • S3 bucket
  • Key name of the associated file in S3
  • UUID of the file
  • Name of the file
  • The ETag of the object in S3 (for non-chunked uploads only)
  • Any parameters/form fields you have associated with the file

An example of the payload for this request sent by Fine Uploader would look like this:

key=f9a922bd-3007-4393-a76e-925fc009639c.txt&uuid=f9a922bd-3007-4393-a76e-925fc009639c&name=rubycsv.txt&bucket=fineuploadertest&etag=123

Parsing url-encoded payloads should be trivial and handled by most web application frameworks.

If you need to perform some specific task to verify the file server-side at this point, you can do so when handling this request and let Fine Uploader know if there is a problem with this file by returning a response with an appropriate (anything other than 200-204) status code. Furthermore, you can include a message to be displayed (FineUploader/default-UI mode) and passed to your onError callback handler via an error property in the payload of your response. In this case, the response payload must be valid JSON.

You can also pass any data to your Fine Uploader complete event handler, client-side, by including it in a valid JSON response to the uploadSuccess.endpoint POST request. In fact, the S3 demo server-side code on FineUploader.com is passing a signed URL to the complete handler which allows you to view the file you've uploaded.

Delete file feature support (optional)

Support for the delete file feature when using the S3 uploader is mostly the same as when using the traditional upload mode. The S3 uploader does add "key" and "bucket" parameters with the request. Otherwise, the request and the server-side code required to handle these requests is the same as when using the traditional uploader. Fine Uploader expects your server-side code to delete the associated file in S3 via Amazon's S3 API, and then return a response to Fine Uploader's delete request when this has task has been handled. See the server-side documentation for the traditional uploader for additional information on handling delete file requests.

CORS support (optional)

Support for CORS exists for the requests sent to the signature.endpoint uploadSuccess.endpoint paths. You will need to set the expected property of the cors option when setting up your Fine Uploader instance. You must also include appropriate headers in your server-response, and possibly handle OPTIONS (pre-flight) requests sent by the browser. Please read the blog post on CORS support for details. Note that you can ignore the "Handling iframe CORS upload requests server-side" section.

Thumbnail generation support (optional)

If you would like to override the client-side generated preview (where supported) or provide a thumbnail for a non-previewable file that you have generated server-side, you can do so by providing an absolute or relative path (URL) to this thumbnail in your response to the uploadSuccess request via a thumbnailUrl property in your JSON response. The URL may be cross-origin as well. See the previews/thumbnails feature page for more information on this feature.