org.jets3t.service.impl.rest.httpclient

Class RestStorageService

java.lang.Object

org.jets3t.service.StorageService

org.jets3t.service.impl.rest.httpclient.RestStorageService

All Implemented Interfaces:

JetS3tRequestAuthorizer

Direct Known Subclasses:

GoogleStorageService, S3Service

public abstract class RestStorageService
extends StorageService
implements JetS3tRequestAuthorizer

Abstract REST/HTTP implementation of an S3Service based on the HttpClient library.

This class uses properties obtained through Jets3tProperties. For more information on these properties please refer to JetS3t Configuration

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	RestStorageService.HTTP_METHOD

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.http.client.CredentialsProvider	credentialsProvider
protected java.lang.String	defaultServerSideEncryptionAlgorithm
protected java.lang.String	defaultStorageClass
protected org.apache.http.client.HttpClient	httpClient
protected RegionEndpointCache	regionEndpointCache
protected boolean	shuttingDown

Fields inherited from class org.jets3t.service.StorageService

BUCKET_STATUS__ALREADY_CLAIMED, BUCKET_STATUS__DOES_NOT_EXIST, BUCKET_STATUS__MY_BUCKET, timeOffset

Constructor Summary

Constructors

Constructor and Description
RestStorageService(ProviderCredentials credentials) Constructs the service and initialises the properties.
RestStorageService(ProviderCredentials credentials, java.lang.String invokingApplicationDescription, org.apache.http.client.CredentialsProvider credentialsProvider) Constructs the service and initialises the properties.
RestStorageService(ProviderCredentials credentials, java.lang.String invokingApplicationDescription, org.apache.http.client.CredentialsProvider credentialsProvider, Jets3tProperties jets3tProperties) Constructs the service and initialises the properties.

Method Summary

Modifier and Type	Method and Description
protected void	addMetadataToHeaders(org.apache.http.client.methods.HttpUriRequest httpMethod, java.util.Map<java.lang.String,java.lang.Object> metadata) Adds all valid metadata name and value pairs as HTTP headers to the given HTTP method.
protected void	addRequestHeadersToConnection(org.apache.http.client.methods.HttpUriRequest httpMethod, java.util.Map<java.lang.String,java.lang.Object> requestHeaders) Adds the provided request headers to the connection.
protected java.lang.String	addRequestParametersToUrlPath(java.lang.String urlPath, java.util.Map<java.lang.String,java.lang.String> requestParameters) Adds all the provided request parameters to a URL in GET request format.
void	authorizeHttpRequest(org.apache.http.client.methods.HttpUriRequest httpMethod, org.apache.http.protocol.HttpContext context, java.lang.String forceRequestSignatureVersion) Authorizes an HTTP/S request by signing it with an HMAC signature compatible with the S3 service and Google Storage (legacy) authorization techniques.
int	checkBucketStatus(java.lang.String bucketName) Find out the status of a bucket with the given name.
protected java.util.Map<java.lang.String,java.lang.Object>	copyObjectImpl(java.lang.String sourceBucketName, java.lang.String sourceObjectKey, java.lang.String destinationBucketName, java.lang.String destinationObjectKey, AccessControlList acl, java.util.Map<java.lang.String,java.lang.Object> destinationMetadata, java.util.Calendar ifModifiedSince, java.util.Calendar ifUnmodifiedSince, java.lang.String[] ifMatchTags, java.lang.String[] ifNoneMatchTags, java.lang.String versionId, java.lang.String destinationObjectStorageClass, java.lang.String destinationObjectServerSideEncryptionAlgorithm) Copy an object within your account.
protected StorageBucket	createBucketImpl(java.lang.String bucketName, java.lang.String location, AccessControlList acl, java.util.Map<java.lang.String,java.lang.Object> headers) Creates a bucket.
protected java.util.Map<java.lang.String,java.lang.Object>	createObjectImpl(java.lang.String bucketName, java.lang.String objectKey, java.lang.String contentType, org.apache.http.HttpEntity requestEntity, java.util.Map<java.lang.String,java.lang.Object> metadata, java.util.Map<java.lang.String,java.lang.String> requestParams, AccessControlList acl, java.lang.String storageClass, java.lang.String serverSideEncryptionAlgorithm)
protected void	deleteBucketImpl(java.lang.String bucketName)
protected void	deleteObjectImpl(java.lang.String bucketName, java.lang.String objectKey, java.lang.String versionId, java.lang.String multiFactorSerialNumber, java.lang.String multiFactorAuthCode)
void	deleteObjectWithSignedUrl(java.lang.String signedDeleteUrl) Deletes an object using a pre-signed DELETE URL generated for that object.
protected void	deleteWebsiteConfigImpl(java.lang.String bucketName)
protected StorageOwner	getAccountOwnerImpl()
protected AccessControlList	getBucketAclImpl(java.lang.String bucketName)
protected java.lang.String	getBucketLocationImpl(java.lang.String bucketName)
protected StorageBucketLoggingStatus	getBucketLoggingStatusImpl(java.lang.String bucketName)
org.apache.http.client.CredentialsProvider	getCredentialsProvider()
org.apache.http.client.HttpClient	getHttpClient()
org.apache.http.conn.ClientConnectionManager	getHttpConnectionManager()
protected AccessControlList	getObjectAclImpl(java.lang.String bucketName, java.lang.String objectKey)
protected AccessControlList	getObjectAclImpl(java.lang.String bucketName, java.lang.String objectKey, java.lang.String versionId)
AccessControlList	getObjectAclWithSignedUrl(java.lang.String signedAclUrl) Gets an object's ACL details using a pre-signed GET URL generated for that object.
protected StorageObject	getObjectDetailsImpl(java.lang.String bucketName, java.lang.String objectKey, java.util.Calendar ifModifiedSince, java.util.Calendar ifUnmodifiedSince, java.lang.String[] ifMatchTags, java.lang.String[] ifNoneMatchTags, java.lang.String versionId)
S3Object	getObjectDetailsWithSignedUrl(java.lang.String signedHeadUrl) Gets an object's details using a pre-signed HEAD URL generated for that object.
protected StorageObject	getObjectImpl(java.lang.String bucketName, java.lang.String objectKey, java.util.Calendar ifModifiedSince, java.util.Calendar ifUnmodifiedSince, java.lang.String[] ifMatchTags, java.lang.String[] ifNoneMatchTags, java.lang.Long byteRangeStart, java.lang.Long byteRangeEnd, java.lang.String versionId)
S3Object	getObjectWithSignedUrl(java.lang.String signedGetUrl) Gets an object using a pre-signed GET URL generated for that object.
RegionEndpointCache	getRegionEndpointCache()
protected WebsiteConfig	getWebsiteConfigImpl(java.lang.String bucketName)
protected org.apache.http.client.HttpClient	initHttpConnection() Initialise HttpClient and HttpConnectionManager objects with the configuration settings appropriate for communicating with S3.
protected void	initializeDefaults()
protected void	initializeProxy()
boolean	isBucketAccessible(java.lang.String bucketName) Indicates whether a bucket exists and is accessible to a service user.
protected boolean	isLiveMD5HashingRequired(StorageObject object)
protected boolean	isRecoverable403(org.apache.http.client.methods.HttpUriRequest httpRequest, java.lang.Exception exception) Determine whether a given 403 Forbidden HTTP error response is recoverable and should be retried.
protected abstract boolean	isTargettingGoogleStorageService()
protected boolean	isXmlContentType(java.lang.String contentType)
protected StorageBucket[]	listAllBucketsImpl(java.util.Map<java.lang.String,java.lang.Object> headers)
protected StorageObjectsChunk	listObjectsChunkedImpl(java.lang.String bucketName, java.lang.String prefix, java.lang.String delimiter, long maxListingLength, java.lang.String priorLastKey, boolean completeListing) Lists objects in a bucket up to the maximum listing length specified.
protected StorageObject[]	listObjectsImpl(java.lang.String bucketName, java.lang.String prefix, java.lang.String delimiter, long maxListingLength) Lists objects in a bucket.
protected StorageObjectsChunk	listObjectsInternal(java.lang.String bucketName, java.lang.String prefix, java.lang.String delimiter, long maxListingLength, boolean automaticallyMergeChunks, java.lang.String priorLastKey)
protected org.apache.http.HttpResponse	performRequest(org.apache.http.client.methods.HttpUriRequest httpMethod, int[] expectedResponseCodes)
protected org.apache.http.HttpResponse	performRequest(org.apache.http.client.methods.HttpUriRequest httpMethod, int[] expectedResponseCodes, org.apache.http.protocol.HttpContext context) Performs an HTTP/S request by invoking the provided HttpMethod object.
protected org.apache.http.HttpResponse	performRestDelete(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.String> requestParameters, java.lang.String multiFactorSerialNumber, java.lang.String multiFactorAuthCode) Performs an HTTP DELETE request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.
protected org.apache.http.HttpResponse	performRestGet(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.String> requestParameters, java.util.Map<java.lang.String,java.lang.Object> requestHeaders) Performs an HTTP GET request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.
protected org.apache.http.HttpResponse	performRestGet(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.String> requestParameters, java.util.Map<java.lang.String,java.lang.Object> requestHeaders, int[] expectedStatusCodes) Performs an HTTP GET request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.
protected org.apache.http.HttpResponse	performRestHead(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.String> requestParameters, java.util.Map<java.lang.String,java.lang.Object> requestHeaders) Performs an HTTP HEAD request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.
protected org.apache.http.HttpResponse	performRestPost(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.Object> metadata, java.util.Map<java.lang.String,java.lang.String> requestParameters, org.apache.http.HttpEntity requestEntity, boolean autoRelease) Performs an HTTP POST request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.
protected org.apache.http.HttpResponse	performRestPostWithXmlBuilder(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.Object> metadata, java.util.Map<java.lang.String,java.lang.String> requestParameters, com.jamesmurty.utils.XMLBuilder builder)
protected HttpResponseAndByteCount	performRestPut(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.Object> metadata, java.util.Map<java.lang.String,java.lang.String> requestParameters, org.apache.http.HttpEntity requestEntity, boolean autoRelease) Performs an HTTP PUT request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.
protected HttpResponseAndByteCount	performRestPutWithXmlBuilder(java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.Object> metadata, java.util.Map<java.lang.String,java.lang.String> requestParameters, com.jamesmurty.utils.XMLBuilder builder)
protected boolean	prepareRESTHeaderAcl(java.util.Map<java.lang.String,java.lang.Object> metadata, AccessControlList acl) Prepares the metadata with the given ACL an ACL is provided and can be represented with a HTTP header.
protected void	prepareServerSideEncryption(java.util.Map<java.lang.String,java.lang.Object> metadata, java.lang.String serverSideEncryptionAlgorithm, java.lang.String objectKey)
protected void	prepareStorageClass(java.util.Map<java.lang.String,java.lang.Object> metadata, java.lang.String storageClass, boolean useDefaultStorageClass, java.lang.String objectKey)
protected void	putAclImpl(java.lang.String bucketName, java.lang.String objectKey, AccessControlList acl, java.lang.String versionId)
protected void	putBucketAclImpl(java.lang.String bucketName, AccessControlList acl)
protected void	putObjectAclImpl(java.lang.String bucketName, java.lang.String objectKey, AccessControlList acl, java.lang.String versionId)
void	putObjectAclWithSignedUrl(java.lang.String signedAclUrl, AccessControlList acl) Sets an object's ACL details using a pre-signed PUT URL generated for that object.
protected StorageObject	putObjectImpl(java.lang.String bucketName, StorageObject object) Beware of high memory requirements when creating large S3 objects when the Content-Length is not set in the object.
protected void	putObjectWithRequestEntityImpl(java.lang.String bucketName, StorageObject object, org.apache.http.HttpEntity requestEntity, java.util.Map<java.lang.String,java.lang.String> requestParams)
S3Object	putObjectWithSignedUrl(java.lang.String signedPutUrl, S3Object object) Puts an object using a pre-signed PUT URL generated for that object.
protected void	setBucketLoggingStatusImpl(java.lang.String bucketName, StorageBucketLoggingStatus status)
void	setCredentialsProvider(org.apache.http.client.CredentialsProvider credentialsProvider) Sets the credentials provider this service will use to authenticate itself.
void	setHttpClient(org.apache.http.client.HttpClient httpClient) Replaces the service's default HTTP client.
void	setRegionEndpointCache(RegionEndpointCache rec)
protected org.apache.http.client.methods.HttpUriRequest	setupConnection(RestStorageService.HTTP_METHOD method, java.lang.String bucketName, java.lang.String objectKey, java.util.Map<java.lang.String,java.lang.String> requestParameters) Creates an HttpRequest object to handle a particular connection method.
protected void	setWebsiteConfigImpl(java.lang.String bucketName, WebsiteConfig config)
protected void	shutdownImpl() Shut down all connections managed by the underlying HttpConnectionManager.
protected void	verifyExpectedAndActualETagValues(java.lang.String expectedETag, StorageObject uploadedObject) Compares the expected and actual ETag value for an uploaded object, and throws an ServiceException if these values do not match.

Methods inherited from class org.jets3t.service.StorageService

assertAuthenticatedConnection, assertValidBucket, assertValidObject, assertValidObject, copyObject, copyObject, createBucket, createBucket, createBucketImpl, deleteBucket, deleteBucket, deleteObject, getAccountOwner, getBucket, getBucketAcl, getCurrentTimeWithOffset, getDisableDnsBuckets, getEnableServerSideEncryption, getEnableStorageClasses, getEndpoint, getHttpPort, getHttpsOnly, getHttpsPort, getInternalErrorRetryMax, getInvokingApplicationDescription, getJetS3tProperties, getObject, getObject, getObjectAcl, getObjectDetails, getObjectDetails, getOrCreateBucket, getProviderCredentials, getResourceParameterNames, getRestHeaderPrefix, getRestMetadataPrefix, getSignatureIdentifier, getVirtualPath, getXmlResponseSaxParser, isAuthenticatedConnection, isHttpsOnly, isObjectInBucket, isShutdown, listAllBuckets, listAllBucketsImpl, listObjects, listObjects, listObjects, listObjectsChunked, listObjectsChunked, moveObject, newBucket, newObject, putBucketAcl, putBucketAcl, putObject, putObjectAcl, putObjectAcl, renameMetadataKeys, renameObject, setProviderCredentials, shutdown, sleepOnInternalError, updateObjectMetadata

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

httpClient

protected org.apache.http.client.HttpClient httpClient

credentialsProvider

protected org.apache.http.client.CredentialsProvider credentialsProvider

regionEndpointCache

protected RegionEndpointCache regionEndpointCache

defaultStorageClass

protected java.lang.String defaultStorageClass

defaultServerSideEncryptionAlgorithm

protected java.lang.String defaultServerSideEncryptionAlgorithm

shuttingDown

protected volatile boolean shuttingDown

Constructor Detail

RestStorageService

public RestStorageService(ProviderCredentials credentials)

Constructs the service and initialises the properties.

Parameters:

credentials - the user credentials to use when communicating with S3, may be null in which case the communication is done as an anonymous user.

RestStorageService

public RestStorageService(ProviderCredentials credentials,
                          java.lang.String invokingApplicationDescription,
                          org.apache.http.client.CredentialsProvider credentialsProvider)

Constructs the service and initialises the properties.

Parameters:

credentials - the S3 user credentials to use when communicating with S3, may be null in which case the communication is done as an anonymous user.

invokingApplicationDescription - a short description of the application using the service, suitable for inclusion in a user agent string for REST/HTTP requests. Ideally this would include the application's version number, for example: Cockpit/0.7.3 or My App Name/1.0

credentialsProvider - an implementation of the HttpClient CredentialsProvider interface, to provide a means for prompting for credentials when necessary.

RestStorageService

public RestStorageService(ProviderCredentials credentials,
                          java.lang.String invokingApplicationDescription,
                          org.apache.http.client.CredentialsProvider credentialsProvider,
                          Jets3tProperties jets3tProperties)

Constructs the service and initialises the properties.

Parameters:

credentials - the S3 user credentials to use when communicating with S3, may be null in which case the communication is done as an anonymous user.

invokingApplicationDescription - a short description of the application using the service, suitable for inclusion in a user agent string for REST/HTTP requests. Ideally this would include the application's version number, for example: Cockpit/0.7.3 or My App Name/1.0

credentialsProvider - an implementation of the HttpClient CredentialsProvider interface, to provide a means for prompting for credentials when necessary.

jets3tProperties - JetS3t properties that will be applied within this service.

Method Detail

initializeDefaults

protected void initializeDefaults()

Overrides:

initializeDefaults in class StorageService

initializeProxy

protected void initializeProxy()

isTargettingGoogleStorageService

protected abstract boolean isTargettingGoogleStorageService()

shutdownImpl

protected void shutdownImpl()
                     throws ServiceException

Shut down all connections managed by the underlying HttpConnectionManager.

Specified by:

shutdownImpl in class StorageService

Throws:

ServiceException

initHttpConnection

protected org.apache.http.client.HttpClient initHttpConnection()

Initialise HttpClient and HttpConnectionManager objects with the configuration settings appropriate for communicating with S3. By default, this method simply delegates the configuration task to RestUtils.initHttpConnection(JetS3tRequestAuthorizer, org.jets3t.service.Jets3tProperties, String, org.apache.http.client.CredentialsProvider).

To alter the low-level behaviour of the HttpClient library, override this method in a subclass and apply your own settings before returning the objects.

Returns:

configured HttpClient library client and connection manager objects.

getHttpConnectionManager

public org.apache.http.conn.ClientConnectionManager getHttpConnectionManager()

Returns:

the manager of HTTP connections for this service.

getHttpClient

public org.apache.http.client.HttpClient getHttpClient()

Returns:

the HTTP client for this service.

setHttpClient

public void setHttpClient(org.apache.http.client.HttpClient httpClient)

Replaces the service's default HTTP client. This method should only be used by advanced users.

Parameters:

httpClient - the client that will replace the default client created by the class constructor.

getCredentialsProvider

public org.apache.http.client.CredentialsProvider getCredentialsProvider()

Returns:

the credentials provider this service will use to authenticate itself, or null if no provider is set.

setCredentialsProvider

public void setCredentialsProvider(org.apache.http.client.CredentialsProvider credentialsProvider)

Sets the credentials provider this service will use to authenticate itself. Changing the credentials provider with this method will have no effect until the initHttpConnection() method is called.

Parameters:

credentialsProvider - Credentials

getRegionEndpointCache

public RegionEndpointCache getRegionEndpointCache()

setRegionEndpointCache

public void setRegionEndpointCache(RegionEndpointCache rec)

isXmlContentType

protected boolean isXmlContentType(java.lang.String contentType)

Parameters:

contentType - HTTP Header

Returns:

true if the given Content-Type string represents an XML document.

performRequest

protected org.apache.http.HttpResponse performRequest(org.apache.http.client.methods.HttpUriRequest httpMethod,
                                                      int[] expectedResponseCodes)
                                               throws ServiceException

Throws:

ServiceException

performRequest

protected org.apache.http.HttpResponse performRequest(org.apache.http.client.methods.HttpUriRequest httpMethod,
                                                      int[] expectedResponseCodes,
                                                      org.apache.http.protocol.HttpContext context)
                                               throws ServiceException

Performs an HTTP/S request by invoking the provided HttpMethod object. If the HTTP response code doesn't match the expected value, an exception is thrown.

Parameters:

httpMethod - the object containing a request target and all other information necessary to perform the request

expectedResponseCodes - the HTTP response code(s) that indicates a successful request. If the response code received does not match this value an error must have occurred, so an exception is thrown.

context - An HttpContext to facilitate information sharing in the HTTP chain

Throws:

ServiceException - all exceptions are wrapped in an ServiceException. Depending on the kind of error that occurred, this exception may contain additional error information available from an XML error response document.

isRecoverable403

protected boolean isRecoverable403(org.apache.http.client.methods.HttpUriRequest httpRequest,
                                   java.lang.Exception exception)

Determine whether a given 403 Forbidden HTTP error response is recoverable and should be retried. Normally 403s should only be retried if we can take some action as a side effect which makes the subsequent request likely to succeed.

Generally, such errors should not be retried since a user's access permissions for an item are unlikely to change, but if a service is using expiring authorization tokens (e.g. OAuth) it may be worthwhile retrying after refreshing those tokens.

Parameters:

httpRequest - Request

exception - Server response

Returns:

true if the request should be retried, otherwise false.

authorizeHttpRequest

public void authorizeHttpRequest(org.apache.http.client.methods.HttpUriRequest httpMethod,
                                 org.apache.http.protocol.HttpContext context,
                                 java.lang.String forceRequestSignatureVersion)
                          throws ServiceException

Authorizes an HTTP/S request by signing it with an HMAC signature compatible with the S3 service and Google Storage (legacy) authorization techniques.

The signature is added to the request as an Authorization header.

Specified by:

authorizeHttpRequest in interface JetS3tRequestAuthorizer

Parameters:

httpMethod - the request object

context -

forceRequestSignatureVersion - request signature/signing version to use for requests, may be null in which case the default signature version is applied.

Throws:

ServiceException

addRequestParametersToUrlPath

protected java.lang.String addRequestParametersToUrlPath(java.lang.String urlPath,
                                                         java.util.Map<java.lang.String,java.lang.String> requestParameters)
                                                  throws ServiceException

Adds all the provided request parameters to a URL in GET request format.

Parameters:

urlPath - the target URL

requestParameters - the parameters to add to the URL as GET request params.

Returns:

the target URL including the parameters.

Throws:

ServiceException

addRequestHeadersToConnection

protected void addRequestHeadersToConnection(org.apache.http.client.methods.HttpUriRequest httpMethod,
                                             java.util.Map<java.lang.String,java.lang.Object> requestHeaders)

Adds the provided request headers to the connection.

Parameters:

httpMethod - the connection object

requestHeaders - the request headers to add as name/value pairs.

addMetadataToHeaders

protected void addMetadataToHeaders(org.apache.http.client.methods.HttpUriRequest httpMethod,
                                    java.util.Map<java.lang.String,java.lang.Object> metadata)
                             throws ServiceException

Adds all valid metadata name and value pairs as HTTP headers to the given HTTP method. Null metadata names are ignored, as are metadata values that are not of type string.

The metadata values are verified to ensure that keys contain only ASCII characters, and that items are not accidentally duplicated due to use of different capitalization. If either of these verification tests fails, an ServiceException is thrown.

Parameters:

httpMethod - Request

metadata - Metadata request headers

Throws:

ServiceException

verifyExpectedAndActualETagValues

protected void verifyExpectedAndActualETagValues(java.lang.String expectedETag,
                                                 StorageObject uploadedObject)
                                          throws ServiceException

Compares the expected and actual ETag value for an uploaded object, and throws an ServiceException if these values do not match.

Parameters:

expectedETag - Expected checksum

uploadedObject - Uploaded key

Throws:

ServiceException

performRestHead

protected org.apache.http.HttpResponse performRestHead(java.lang.String bucketName,
                                                       java.lang.String objectKey,
                                                       java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                       java.util.Map<java.lang.String,java.lang.Object> requestHeaders)
                                                throws ServiceException

Performs an HTTP HEAD request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.

Parameters:

bucketName - the bucket's name

objectKey - the object's key name, may be null if the operation is on a bucket only.

requestParameters - parameters to add to the request URL as GET params

requestHeaders - headers to add to the request

Returns:

the HTTP method object used to perform the request

Throws:

ServiceException

performRestGet

protected org.apache.http.HttpResponse performRestGet(java.lang.String bucketName,
                                                      java.lang.String objectKey,
                                                      java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                      java.util.Map<java.lang.String,java.lang.Object> requestHeaders)
                                               throws ServiceException

Performs an HTTP GET request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.

Parameters:

bucketName - the bucket's name

objectKey - the object's key name, may be null if the operation is on a bucket only.

requestParameters - parameters to add to the request URL as GET params

requestHeaders - headers to add to the request

Returns:

The HTTP method object used to perform the request.

Throws:

ServiceException

performRestGet

protected org.apache.http.HttpResponse performRestGet(java.lang.String bucketName,
                                                      java.lang.String objectKey,
                                                      java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                      java.util.Map<java.lang.String,java.lang.Object> requestHeaders,
                                                      int[] expectedStatusCodes)
                                               throws ServiceException

Performs an HTTP GET request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.

Parameters:

bucketName - the bucket's name

objectKey - the object's key name, may be null if the operation is on a bucket only.

requestParameters - parameters to add to the request URL as GET params

requestHeaders - headers to add to the request

expectedStatusCodes - HTTP status response codes expected, anything else causes an exception.

Returns:

The HTTP method object used to perform the request.

Throws:

ServiceException

performRestPut

protected HttpResponseAndByteCount performRestPut(java.lang.String bucketName,
                                                  java.lang.String objectKey,
                                                  java.util.Map<java.lang.String,java.lang.Object> metadata,
                                                  java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                  org.apache.http.HttpEntity requestEntity,
                                                  boolean autoRelease)
                                           throws ServiceException

Performs an HTTP PUT request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.

Parameters:

bucketName - the name of the bucket the object will be stored in.

objectKey - the key (name) of the object to be stored.

metadata - map of name/value pairs to add as metadata to any S3 objects created.

requestParameters - parameters to add to the request URL as GET params

requestEntity - an HttpClient object that encapsulates the object and data contents that will be uploaded. This object supports the resending of object data, when possible.

autoRelease - if true, the HTTP Method object will be released after the request has completed and the connection will be closed. If false, the object will not be released and the caller must take responsibility for doing this.

Returns:

a package including the HTTP method object used to perform the request, and the content length (in bytes) of the object that was PUT to S3.

Throws:

ServiceException

performRestPost

protected org.apache.http.HttpResponse performRestPost(java.lang.String bucketName,
                                                       java.lang.String objectKey,
                                                       java.util.Map<java.lang.String,java.lang.Object> metadata,
                                                       java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                       org.apache.http.HttpEntity requestEntity,
                                                       boolean autoRelease)
                                                throws ServiceException

Performs an HTTP POST request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.

Parameters:

bucketName - the name of the bucket the object will be stored in.

objectKey - the key (name) of the object to be stored.

metadata - map of name/value pairs to add as metadata to any S3 objects created.

requestParameters - parameters to add to the request URL as GET params

requestEntity - an HttpClient object that encapsulates the object and data contents that will be uploaded. This object supports the re-sending of object data, when possible.

autoRelease - if true, the HTTP Method object will be released after the request has completed and the connection will be closed. If false, the object will not be released and the caller must take responsibility for doing this.

Returns:

a package including the HTTP method object used to perform the request, and the content length (in bytes) of the object that was POSTed to S3.

Throws:

ServiceException

performRestDelete

protected org.apache.http.HttpResponse performRestDelete(java.lang.String bucketName,
                                                         java.lang.String objectKey,
                                                         java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                         java.lang.String multiFactorSerialNumber,
                                                         java.lang.String multiFactorAuthCode)
                                                  throws ServiceException

Performs an HTTP DELETE request using the performRequest(org.apache.http.client.methods.HttpUriRequest, int[]) method.

Parameters:

bucketName - the bucket's name

objectKey - the object's key name, may be null if the operation is on a bucket only.

Returns:

The HTTP method object used to perform the request.

Throws:

ServiceException

performRestPutWithXmlBuilder

protected HttpResponseAndByteCount performRestPutWithXmlBuilder(java.lang.String bucketName,
                                                                java.lang.String objectKey,
                                                                java.util.Map<java.lang.String,java.lang.Object> metadata,
                                                                java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                                com.jamesmurty.utils.XMLBuilder builder)
                                                         throws ServiceException

Throws:

ServiceException

performRestPostWithXmlBuilder

protected org.apache.http.HttpResponse performRestPostWithXmlBuilder(java.lang.String bucketName,
                                                                     java.lang.String objectKey,
                                                                     java.util.Map<java.lang.String,java.lang.Object> metadata,
                                                                     java.util.Map<java.lang.String,java.lang.String> requestParameters,
                                                                     com.jamesmurty.utils.XMLBuilder builder)
                                                              throws ServiceException

Throws:

ServiceException

setupConnection

protected org.apache.http.client.methods.HttpUriRequest setupConnection(RestStorageService.HTTP_METHOD method,
                                                                        java.lang.String bucketName,
                                                                        java.lang.String objectKey,
                                                                        java.util.Map<java.lang.String,java.lang.String> requestParameters)
                                                                 throws ServiceException

Creates an HttpRequest object to handle a particular connection method.

Parameters:

method - the HTTP method/connection-type to use, must be one of: PUT, HEAD, GET, DELETE

bucketName - the bucket's name

objectKey - the object's key name, may be null if the operation is on a bucket only.

Returns:

the HTTP method object used to perform the request

Throws:

ServiceException

isBucketAccessible

public boolean isBucketAccessible(java.lang.String bucketName)
                           throws ServiceException

Description copied from class: StorageService

Indicates whether a bucket exists and is accessible to a service user.

Caution: This check started to cause issues in situations where you need to immediately create a bucket when it does not exist. To conditionally create a bucket, use the StorageService.getOrCreateBucket(String) method instead.

This method can be performed by anonymous services.

Implementation notes

This method can be implemented by attempting to list the objects in a bucket. If the listing is successful return true, if the listing failed for any reason return false.

Specified by:

isBucketAccessible in class StorageService

Parameters:

bucketName - the bucket to check.

Returns:

true if the bucket exists and is accessible to the service user, false otherwise.

Throws:

ServiceException

checkBucketStatus

public int checkBucketStatus(java.lang.String bucketName)
                      throws ServiceException

Description copied from class: StorageService

Find out the status of a bucket with the given name.

Caveats:

• If someone else owns the bucket but has made it public, this method will mistakenly return StorageService.BUCKET_STATUS__MY_BUCKET.

• S3 can act strangely when you use this method in some circumstances. If you check the status of a bucket and find that it does not exist, then create the bucket, the service will continue to tell you the bucket does not exists for up to 30 seconds. This problem has something to do with connection caching (I think).

  This S3 quirk makes it a bad idea to use this method to check for a bucket's existence before creating that bucket. Use the StorageService.getOrCreateBucket(String) method for this purpose instead.

Specified by:

checkBucketStatus in class StorageService

Returns:

StorageService.BUCKET_STATUS__MY_BUCKET if you already own the bucket, StorageService.BUCKET_STATUS__DOES_NOT_EXIST if the bucket does not yet exist, or StorageService.BUCKET_STATUS__ALREADY_CLAIMED if someone else has already created a bucket with the given name.

Throws:

ServiceException

listAllBucketsImpl

protected StorageBucket[] listAllBucketsImpl(java.util.Map<java.lang.String,java.lang.Object> headers)
                                      throws ServiceException

Specified by:

listAllBucketsImpl in class StorageService

Parameters:

headers - Additional metadata to send with request

Returns:

the buckets in your account.

Throws:

ServiceException

getAccountOwnerImpl

protected StorageOwner getAccountOwnerImpl()
                                    throws ServiceException

Specified by:

getAccountOwnerImpl in class StorageService

Returns:

the owner of an account.

Throws:

ServiceException

listObjectsImpl

protected StorageObject[] listObjectsImpl(java.lang.String bucketName,
                                          java.lang.String prefix,
                                          java.lang.String delimiter,
                                          long maxListingLength)
                                   throws ServiceException

Description copied from class: StorageService

Lists objects in a bucket. Implementation notes

The implementation of this method is expected to return all the objects in a bucket, not a subset. This may require repeating the list operation if the first one doesn't include all the available objects, such as when the number of objects is greater than maxListingLength.

Specified by:

listObjectsImpl in class StorageService

prefix - only objects with a key that starts with this prefix will be listed, may be null.

delimiter - only list objects with key names up to this delimiter, may be null.

Returns:

the objects in a bucket.

Throws:

ServiceException

listObjectsChunkedImpl

protected StorageObjectsChunk listObjectsChunkedImpl(java.lang.String bucketName,
                                                     java.lang.String prefix,
                                                     java.lang.String delimiter,
                                                     long maxListingLength,
                                                     java.lang.String priorLastKey,
                                                     boolean completeListing)
                                              throws ServiceException

Description copied from class: StorageService

Lists objects in a bucket up to the maximum listing length specified.

Implementation notes The implementation of this method returns only as many objects as requested in the chunk size. It is the responsibility of the caller to build a complete object listing from multiple chunks, should this be necessary.

Specified by:

listObjectsChunkedImpl in class StorageService

prefix - only objects with a key that starts with this prefix will be listed, may be null.

delimiter - only list objects with key names up to this delimiter, may be null.

Throws:

ServiceException

listObjectsInternal

protected StorageObjectsChunk listObjectsInternal(java.lang.String bucketName,
                                                  java.lang.String prefix,
                                                  java.lang.String delimiter,
                                                  long maxListingLength,
                                                  boolean automaticallyMergeChunks,
                                                  java.lang.String priorLastKey)
                                           throws ServiceException

Throws:

ServiceException

deleteObjectImpl

protected void deleteObjectImpl(java.lang.String bucketName,
                                java.lang.String objectKey,
                                java.lang.String versionId,
                                java.lang.String multiFactorSerialNumber,
                                java.lang.String multiFactorAuthCode)
                         throws ServiceException

Specified by:

deleteObjectImpl in class StorageService

Throws:

ServiceException

getObjectAclImpl

protected AccessControlList getObjectAclImpl(java.lang.String bucketName,
                                             java.lang.String objectKey)
                                      throws ServiceException

Throws:

ServiceException

getObjectAclImpl

protected AccessControlList getObjectAclImpl(java.lang.String bucketName,
                                             java.lang.String objectKey,
                                             java.lang.String versionId)
                                      throws ServiceException

Specified by:

getObjectAclImpl in class StorageService

Throws:

ServiceException

getBucketAclImpl

protected AccessControlList getBucketAclImpl(java.lang.String bucketName)
                                      throws ServiceException

Specified by:

getBucketAclImpl in class StorageService

Throws:

ServiceException

putObjectAclImpl

protected void putObjectAclImpl(java.lang.String bucketName,
                                java.lang.String objectKey,
                                AccessControlList acl,
                                java.lang.String versionId)
                         throws ServiceException

Specified by:

putObjectAclImpl in class StorageService

Throws:

ServiceException

putBucketAclImpl

protected void putBucketAclImpl(java.lang.String bucketName,
                                AccessControlList acl)
                         throws ServiceException

Specified by:

putBucketAclImpl in class StorageService

Throws:

ServiceException

putAclImpl

protected void putAclImpl(java.lang.String bucketName,
                          java.lang.String objectKey,
                          AccessControlList acl,
                          java.lang.String versionId)
                   throws ServiceException

Throws:

ServiceException

createBucketImpl

protected StorageBucket createBucketImpl(java.lang.String bucketName,
                                         java.lang.String location,
                                         AccessControlList acl,
                                         java.util.Map<java.lang.String,java.lang.Object> headers)
                                  throws ServiceException

Description copied from class: StorageService

Creates a bucket. Implementation notes

The implementing method must populate the bucket object's metadata with the results of the operation before returning the object. It must also apply any AccessControlList settings included with the bucket.

Specified by:

createBucketImpl in class StorageService

Parameters:

bucketName - the name of the bucket to create.

location - the geographical location where the bucket will be stored (if applicable for the target service). A null string value will cause the bucket to be stored in the default location.

acl - an access control object representing the initial acl values for the bucket. May be null, in which case the default permissions are applied.

headers - Additional metadata to add to request

Returns:

the created bucket object, populated with all metadata made available by the creation operation.

Throws:

ServiceException

deleteBucketImpl

protected void deleteBucketImpl(java.lang.String bucketName)
                         throws ServiceException

Specified by:

deleteBucketImpl in class StorageService

Throws:

ServiceException

isLiveMD5HashingRequired

protected boolean isLiveMD5HashingRequired(StorageObject object)

getBucketLocationImpl

protected java.lang.String getBucketLocationImpl(java.lang.String bucketName)
                                          throws ServiceException

Throws:

ServiceException

getBucketLoggingStatusImpl

protected StorageBucketLoggingStatus getBucketLoggingStatusImpl(java.lang.String bucketName)
                                                         throws ServiceException

Throws:

ServiceException

setBucketLoggingStatusImpl

protected void setBucketLoggingStatusImpl(java.lang.String bucketName,
                                          StorageBucketLoggingStatus status)
                                   throws ServiceException

Throws:

ServiceException

putObjectImpl

protected StorageObject putObjectImpl(java.lang.String bucketName,
                                      StorageObject object)
                               throws ServiceException

Beware of high memory requirements when creating large S3 objects when the Content-Length is not set in the object.

Specified by:

putObjectImpl in class StorageService

Throws:

ServiceException

putObjectWithRequestEntityImpl

protected void putObjectWithRequestEntityImpl(java.lang.String bucketName,
                                              StorageObject object,
                                              org.apache.http.HttpEntity requestEntity,
                                              java.util.Map<java.lang.String,java.lang.String> requestParams)
                                       throws ServiceException

Throws:

ServiceException

createObjectImpl

protected java.util.Map<java.lang.String,java.lang.Object> createObjectImpl(java.lang.String bucketName,
                                                                            java.lang.String objectKey,
                                                                            java.lang.String contentType,
                                                                            org.apache.http.HttpEntity requestEntity,
                                                                            java.util.Map<java.lang.String,java.lang.Object> metadata,
                                                                            java.util.Map<java.lang.String,java.lang.String> requestParams,
                                                                            AccessControlList acl,
                                                                            java.lang.String storageClass,
                                                                            java.lang.String serverSideEncryptionAlgorithm)
                                                                     throws ServiceException

Throws:

ServiceException

prepareRESTHeaderAcl

protected boolean prepareRESTHeaderAcl(java.util.Map<java.lang.String,java.lang.Object> metadata,
                                       AccessControlList acl)

Prepares the metadata with the given ACL an ACL is provided and can be represented with a HTTP header.

Parameters:

metadata - Metadata request headers

acl - Access control list

Returns:

true if no acl or standard, false otherwise false if an ACL was provided but it could not be applied as a canned ACL.

prepareStorageClass

protected void prepareStorageClass(java.util.Map<java.lang.String,java.lang.Object> metadata,
                                   java.lang.String storageClass,
                                   boolean useDefaultStorageClass,
                                   java.lang.String objectKey)

prepareServerSideEncryption

protected void prepareServerSideEncryption(java.util.Map<java.lang.String,java.lang.Object> metadata,
                                           java.lang.String serverSideEncryptionAlgorithm,
                                           java.lang.String objectKey)

copyObjectImpl

protected java.util.Map<java.lang.String,java.lang.Object> copyObjectImpl(java.lang.String sourceBucketName,
                                                                          java.lang.String sourceObjectKey,
                                                                          java.lang.String destinationBucketName,
                                                                          java.lang.String destinationObjectKey,
                                                                          AccessControlList acl,
                                                                          java.util.Map<java.lang.String,java.lang.Object> destinationMetadata,
                                                                          java.util.Calendar ifModifiedSince,
                                                                          java.util.Calendar ifUnmodifiedSince,
                                                                          java.lang.String[] ifMatchTags,
                                                                          java.lang.String[] ifNoneMatchTags,
                                                                          java.lang.String versionId,
                                                                          java.lang.String destinationObjectStorageClass,
                                                                          java.lang.String destinationObjectServerSideEncryptionAlgorithm)
                                                                   throws ServiceException

Description copied from class: StorageService

Copy an object within your account. Copies within a single bucket or between buckets, and optionally updates the object's metadata at the same time. An object can be copied over itself, allowing you to update the metadata without making any other changes.

Specified by:

copyObjectImpl in class StorageService

Parameters:

sourceBucketName - the name of the bucket that contains the original object.

sourceObjectKey - the key name of the original object.

destinationBucketName - the name of the destination bucket to which the object will be copied.

destinationObjectKey - the key name for the copied object.

acl - the access control settings that will be applied to the copied object. If this parameter is null, the default (private) ACL setting will be applied to the copied object.

destinationMetadata - metadata items to apply to the copied object. If this parameter is null, the metadata will be copied unchanged from the original object. If this parameter is not null, the copied object will have only the supplied metadata.

Returns:

a map of the header and result information returned after the object copy. The map includes the object's MD5 hash value (ETag), its size (Content-Length), and update timestamp (Last-Modified).

Throws:

ServiceException

getObjectDetailsImpl

protected StorageObject getObjectDetailsImpl(java.lang.String bucketName,
                                             java.lang.String objectKey,
                                             java.util.Calendar ifModifiedSince,
                                             java.util.Calendar ifUnmodifiedSince,
                                             java.lang.String[] ifMatchTags,
                                             java.lang.String[] ifNoneMatchTags,
                                             java.lang.String versionId)
                                      throws ServiceException

Specified by:

getObjectDetailsImpl in class StorageService

Throws:

ServiceException

getObjectImpl

protected StorageObject getObjectImpl(java.lang.String bucketName,
                                      java.lang.String objectKey,
                                      java.util.Calendar ifModifiedSince,
                                      java.util.Calendar ifUnmodifiedSince,
                                      java.lang.String[] ifMatchTags,
                                      java.lang.String[] ifNoneMatchTags,
                                      java.lang.Long byteRangeStart,
                                      java.lang.Long byteRangeEnd,
                                      java.lang.String versionId)
                               throws ServiceException

Specified by:

getObjectImpl in class StorageService

Throws:

ServiceException

putObjectWithSignedUrl

public S3Object putObjectWithSignedUrl(java.lang.String signedPutUrl,
                                       S3Object object)
                                throws ServiceException

Puts an object using a pre-signed PUT URL generated for that object. This method is an implementation of the interface SignedUrlHandler.

This operation does not required any S3 functionality as it merely uploads the object by performing a standard HTTP PUT using the signed URL.

Parameters:

signedPutUrl - a signed PUT URL generated with S3Service.createSignedPutUrl(String, String, java.util.Map, org.jets3t.service.security.ProviderCredentials, java.util.Date).

object - the object to upload, which must correspond to the object for which the URL was signed. The object must have the correct content length set, and to apply a non-standard ACL policy only the REST canned ACLs can be used (eg AccessControlList.REST_CANNED_PUBLIC_READ_WRITE).

Returns:

the S3Object put to S3. The S3Object returned will represent the object created in S3.

Throws:

ServiceException

deleteObjectWithSignedUrl

public void deleteObjectWithSignedUrl(java.lang.String signedDeleteUrl)
                               throws ServiceException

Deletes an object using a pre-signed DELETE URL generated for that object. This method is an implementation of the interface SignedUrlHandler.

This operation does not required any S3 functionality as it merely deletes the object by performing a standard HTTP DELETE using the signed URL.

Parameters:

signedDeleteUrl - a signed DELETE URL generated with S3Service.createSignedDeleteUrl(java.lang.String, java.lang.String, java.util.Date, boolean).

Throws:

ServiceException

getObjectWithSignedUrl

public S3Object getObjectWithSignedUrl(java.lang.String signedGetUrl)
                                throws ServiceException

Gets an object using a pre-signed GET URL generated for that object. This method is an implementation of the interface SignedUrlHandler.

This operation does not required any S3 functionality as it merely uploads the object by performing a standard HTTP GET using the signed URL.

Parameters:

signedGetUrl - a signed GET URL generated with S3Service.createSignedGetUrl(String, String, org.jets3t.service.security.ProviderCredentials, java.util.Date).

Returns:

the S3Object in S3 including all metadata and the object's data input stream.

Throws:

ServiceException

getObjectDetailsWithSignedUrl

public S3Object getObjectDetailsWithSignedUrl(java.lang.String signedHeadUrl)
                                       throws ServiceException

Gets an object's details using a pre-signed HEAD URL generated for that object. This method is an implementation of the interface SignedUrlHandler.

This operation does not required any S3 functionality as it merely uploads the object by performing a standard HTTP HEAD using the signed URL.

Parameters:

signedHeadUrl - a signed HEAD URL generated with S3Service.createSignedHeadUrl(String, String, org.jets3t.service.security.ProviderCredentials, java.util.Date).

Returns:

the S3Object in S3 including all metadata, but without the object's data input stream.

Throws:

ServiceException

getObjectAclWithSignedUrl

public AccessControlList getObjectAclWithSignedUrl(java.lang.String signedAclUrl)
                                            throws ServiceException

Gets an object's ACL details using a pre-signed GET URL generated for that object. This method is an implementation of the interface SignedUrlHandler.

Parameters:

signedAclUrl - a signed URL generated with S3Service.createSignedUrl(String, String, String, String, java.util.Map, org.jets3t.service.security.ProviderCredentials, long, boolean).

Returns:

the AccessControlList settings of the object in S3.

Throws:

ServiceException

putObjectAclWithSignedUrl

public void putObjectAclWithSignedUrl(java.lang.String signedAclUrl,
                                      AccessControlList acl)
                               throws ServiceException

Sets an object's ACL details using a pre-signed PUT URL generated for that object. This method is an implementation of the interface SignedUrlHandler.

Parameters:

signedAclUrl - a signed URL generated with S3Service.createSignedUrl(String, String, String, String, java.util.Map, org.jets3t.service.security.ProviderCredentials, long, boolean).

acl - the ACL settings to apply to the object represented by the signed URL.

Throws:

ServiceException

getWebsiteConfigImpl

protected WebsiteConfig getWebsiteConfigImpl(java.lang.String bucketName)
                                      throws ServiceException

Throws:

ServiceException

setWebsiteConfigImpl

protected void setWebsiteConfigImpl(java.lang.String bucketName,
                                    WebsiteConfig config)
                             throws ServiceException

Throws:

ServiceException

deleteWebsiteConfigImpl

protected void deleteWebsiteConfigImpl(java.lang.String bucketName)
                                throws ServiceException

Throws:

ServiceException