MediaResponseInfo

This message is for backends to pass their scotty media specific fields to ESF. Backend will include this in their response message to ESF. Example: ExportFile is an rpc defined for upload using scotty from ESF.

rpc ExportFile(ExportFileRequest) returns (ExportFileResponse)

Message ExportFileResponse will include apiserving.MediaResponseInfo to tell ESF about data like dynamicDropzone it needs to pass to Scotty.

message ExportFileResponse { optional gdata.Media blob = 1; optional apiserving.MediaResponseInfo media_response_info = 2 }

JSON representation 
{
  "dynamicDropzone": string,
  "customData": string,
  "dynamicDropTarget": string,
  "dataStorageTransform": string,
  "trafficClassField": enum (TrafficClass),
  "mediaForDiff": {
    object (Media)
  },
  "verifyHashFromHeader": boolean,
  "requestClass": enum (RequestClass),
  "scottyCustomerLog": string,
  "scottyAgentUserId": string,
  "originalObjectBlobMintIndex": integer,
  "destinationBlobMintIndex": integer
}
Fields
dynamicDropzone

string

Specifies the Scotty dropzone to use for uploads.
customData

string

Data to copy from backend response to the next backend requests. Custom data is returned to Scotty in the agentState field, which Scotty will then provide in subsequent upload notifications.
dynamicDropTarget

string (bytes format)

Specifies the Scotty Drop Target to use for uploads. If present in a media response, Scotty does not upload to a standard drop zone. Instead, Scotty saves the upload directly to the location specified in this drop target. Unlike drop zones, the drop target is the final storage location for an upload. So, the agent does not need to clone the blob at the end of the upload. The agent is responsible for garbage collecting any orphaned blobs that may occur due to aborted uploads. This field will be preferred to dynamicDropzone. If provided, the identified field in the response must be of the type uploader.agent.DropTarget.

A base64-encoded string.
dataStorageTransform

string (bytes format)

Specifies any transformation to be applied to data before persisting it or retrieving from storage. E.g., encryption options for blobstore2. This should be of the form uploader_service.DataStorageTransform.

A base64-encoded string.
trafficClassField

enum (TrafficClass)

Specifies the TrafficClass that Scotty should use for any RPCs to fetch the response bytes. Will override the traffic class GTOS of the incoming http request. This is a temporary field to facilitate whitelisting and experimentation by the bigstore agent only. For instance, this does not apply to RTMP reads. WARNING: DO NOT USE WITHOUT PERMISSION FROM THE SCOTTY TEAM.
mediaForDiff

object (Media)

Diff Updates must respond to a START notification with this Media proto to tell Scotty to decode the diff encoded payload and apply the diff against this field. If the request was diff encoded, but this field is not set, Scotty will treat the encoding as identity. This is corresponding to Apiary's DiffUploadResponse.original_object.
verifyHashFromHeader

boolean

Tells Scotty to verify hashes on the agent's behalf by parsing out the X-Goog-Hash header.
requestClass

enum (RequestClass)

Request class to use for all Blobstore operations for this request.
scottyCustomerLog

string (bytes format)

Customer-specific data to be recorded in the Scotty logs type is logs_proto_scotty.CustomerLog

A base64-encoded string.
scottyAgentUserId

string (int64 format)

Requester ID passed along to be recorded in the Scotty logs
originalObjectBlobMintIndex

integer

For the first notification of a |diff_encoded| HttpRequestInfo, this is the index of the blob mint that Scotty should use when reading the original blob. This field is optional. It's not required ever, even if destinationBlobMintIndex is set. In situations like that, we will use the destination blob's mint for the destination blob and regular blob ACL checks for the original object.

Note: This field is only for use by Drive API for diff uploads.
destinationBlobMintIndex

integer

For the first notification of a |diff_encoded| HttpRequestInfo, this is the index of the blob mint that Scotty should use when writing the resulting blob. This field is optional. It's not required ever, even if originalObjectBlobMintIndex is set. In situations like that, we will use the destination blob's mint for the destination blob and regular blob ACL checks for the original object.

Note: This field is only for use by Drive API for diff uploads.

Media

A reference to data stored on the filesystem, on GFS or in blobstore.

JSON representation 
{
  "contentType": string,
  "timestamp": string,
  "token": string,
  "length": string,
  "filename": string,
  "referenceType": enum (ReferenceType),
  "path": string,
  "blobRef": string,
  "inline": string,
  "mediaId": string,
  "hash": string,
  "algorithm": string,
  "compositeMedia": [
    {
      object (CompositeMedia)
    }
  ],
  "bigstoreObjectRef": string,
  "objectId": {
    object (ObjectId)
  },
  "blobstore2Info": {
    object (Blobstore2Info)
  },
  "diffVersionResponse": {
    object (DiffVersionResponse)
  },
  "diffChecksumsResponse": {
    object (DiffChecksumsResponse)
  },
  "diffDownloadResponse": {
    object (DiffDownloadResponse)
  },
  "diffUploadRequest": {
    object (DiffUploadRequest)
  },
  "diffUploadResponse": {
    object (DiffUploadResponse)
  },
  "contentTypeInfo": {
    object (ContentTypeInfo)
  },
  "downloadParameters": {
    object (DownloadParameters)
  },
  "crc32cHash": integer,
  "md5Hash": string,
  "sha1Hash": string,
  "sha256Hash": string,
  "isPotentialRetry": boolean,
  "cosmoBinaryReference": string,
  "hashVerified": boolean
}
Fields
contentType

string

MIME type of the data
timestamp

string

Time at which the media data was last updated, in milliseconds since UNIX epoch
token

string

A unique fingerprint/version id for the media data
length

string (int64 format)

Size of the data, in bytes
filename

string

Original file name
referenceType

enum (ReferenceType)

Describes what the field reference contains.
path

string

Path to the data, set if referenceType is PATH
blobRef
(deprecated)

string (bytes format)

Blobstore v1 reference, set if referenceType is BLOBSTORE_REF This should be the byte representation of a blobstore.BlobRef. Since Blobstore is deprecating v1, use blobstore2Info instead. For now, any v2 blob will also be represented in this field as v1 BlobRef.

A base64-encoded string.
inline

string (bytes format)

Media data, set if referenceType is INLINE

A base64-encoded string.
mediaId

string (bytes format)

Media id to forward to the operation GetMedia. Can be set if referenceType is GET_MEDIA.

A base64-encoded string.
hash
(deprecated)

string

Deprecated, use one of explicit hash type fields instead. These two hash related fields will only be populated on Scotty based media uploads and will contain the content of the hash group in the NotificationRequest.

Hex encoded hash value of the uploaded media.
algorithm
(deprecated)

string

Deprecated, use one of explicit hash type fields instead. Algorithm used for calculating the hash. As of 2011/01/21, "MD5" is the only possible value for this field. New values may be added at any time.
compositeMedia[]

object (CompositeMedia)

A composite media composed of one or more media objects, set if referenceType is COMPOSITE_MEDIA. The media length field must be set to the sum of the lengths of all composite media objects.

Note: All composite media must have length specified.
bigstoreObjectRef
(deprecated)

string (bytes format)

Use objectId instead.

A base64-encoded string.
objectId

object (ObjectId)

Reference to a TI Blob, set if referenceType is BIGSTORE_REF.
blobstore2Info

object (Blobstore2Info)

Blobstore v2 info, set if referenceType is BLOBSTORE_REF and it refers to a v2 blob.
diffVersionResponse

object (DiffVersionResponse)

Set if referenceType is DIFF_VERSION_RESPONSE.
diffChecksumsResponse

object (DiffChecksumsResponse)

Set if referenceType is DIFF_CHECKSUMS_RESPONSE.
diffDownloadResponse

object (DiffDownloadResponse)

Set if referenceType is DIFF_DOWNLOAD_RESPONSE.
diffUploadRequest

object (DiffUploadRequest)

Set if referenceType is DIFF_UPLOAD_REQUEST.
diffUploadResponse

object (DiffUploadResponse)

Set if referenceType is DIFF_UPLOAD_RESPONSE.
contentTypeInfo

object (ContentTypeInfo)

Extended content type information provided for Scotty uploads.
downloadParameters

object (DownloadParameters)

Parameters for a media download.
crc32cHash

integer

For Scotty Uploads: Scotty-provided hashes for uploads

For Scotty Downloads: (WARNING: DO NOT USE WITHOUT PERMISSION FROM THE SCOTTY TEAM.) A Hash provided by the agent to be used to verify the data being downloaded. Currently only supported for inline payloads. Further, only crc32cHash is currently supported.
md5Hash

string (bytes format)

Scotty-provided MD5 hash for an upload.

A base64-encoded string.
sha1Hash

string (bytes format)

Scotty-provided SHA1 hash for an upload.

A base64-encoded string.
sha256Hash

string (bytes format)

Scotty-provided SHA256 hash for an upload.

A base64-encoded string.
isPotentialRetry

boolean

|isPotentialRetry| is set false only when Scotty is certain that it has not sent the request before. When a client resumes an upload, this field must be set true in agent calls, because Scotty cannot be certain that it has never sent the request before due to potential failure in the session state persistence.
cosmoBinaryReference

string (bytes format)

A binary data reference for a media download. Serves as a technology-agnostic binary reference in some Google infrastructure. This value is a serialized storage_cosmo.BinaryReference proto. Storing it as bytes is a hack to get around the fact that the cosmo proto (as well as others it includes) doesn't support JavaScript. This prevents us from including the actual type of this field.

A base64-encoded string.
hashVerified

boolean

For Scotty uploads only. If a user sends a hash code and the backend has requested that Scotty verify the upload against the client hash, Scotty will perform the check on behalf of the backend and will reject it if the hashes don't match. This is set to true if Scotty performed this verification.

CompositeMedia

A sequence of media data references representing composite data. Introduced to support Bigstore composite objects. For details, visit.

JSON representation 
{
  "length": string,
  "referenceType": enum (ReferenceType),
  "path": string,
  "blobRef": string,
  "inline": string,
  "objectId": {
    object (ObjectId)
  },
  "blobstore2Info": {
    object (Blobstore2Info)
  },
  "cosmoBinaryReference": string,
  "crc32cHash": integer,
  "md5Hash": string,
  "sha1Hash": string
}
Fields
length

string (int64 format)

Size of the data, in bytes
referenceType

enum (ReferenceType)

Describes what the field reference contains.
path

string

Path to the data, set if referenceType is PATH
blobRef
(deprecated)

string (bytes format)

Blobstore v1 reference, set if referenceType is BLOBSTORE_REF This should be the byte representation of a blobstore.BlobRef. Since Blobstore is deprecating v1, use blobstore2Info instead. For now, any v2 blob will also be represented in this field as v1 BlobRef.

A base64-encoded string.
inline

string (bytes format)

Media data, set if referenceType is INLINE

A base64-encoded string.
objectId

object (ObjectId)

Reference to a TI Blob, set if referenceType is BIGSTORE_REF.
blobstore2Info

object (Blobstore2Info)

Blobstore v2 info, set if referenceType is BLOBSTORE_REF and it refers to a v2 blob.
cosmoBinaryReference

string (bytes format)

A binary data reference for a media download. Serves as a technology-agnostic binary reference in some Google infrastructure. This value is a serialized storage_cosmo.BinaryReference proto. Storing it as bytes is a hack to get around the fact that the cosmo proto (as well as others it includes) doesn't support JavaScript. This prevents us from including the actual type of this field.

A base64-encoded string.
crc32cHash

integer

crc32.c hash for the payload.
md5Hash

string (bytes format)

MD5 hash for the payload.

A base64-encoded string.
sha1Hash

string (bytes format)

SHA-1 hash for the payload.

A base64-encoded string.

ObjectId

This is a copy of the tech.blob.ObjectId proto, which could not be used directly here due to transitive closure issues with JavaScript support.

JSON representation 
{
  "bucketName": string,
  "objectName": string,
  "generation": string
}
Fields
bucketName

string

The name of the bucket to which this object belongs.
objectName

string

The name of the object.
generation

string (int64 format)

Generation of the object. Generations are monotonically increasing across writes, allowing them to be be compared to determine which generation is newer. If this is omitted in a request, then you are requesting the live object.

Blobstore2Info

Information to read/write to blobstore2.

JSON representation 
{
  "blobId": string,
  "blobGeneration": string,
  "readToken": string,
  "uploadMetadataContainer": string,
  "downloadReadHandle": string,
  "downloadExternalReadToken": string,
  "uploadFragmentListCreationInfo": string
}
Fields
blobId

string

The blob id, e.g., /blobstore/prod/playground/scotty
blobGeneration

string (int64 format)

The blob generation id.
readToken

string

The blob read token. Needed to read blobs that have not been replicated. Might not be available until the final call.
uploadMetadataContainer

string (bytes format)

Metadata passed from Blobstore -> Scotty for a new Google Cloud Storage upload. This is a signed, serialized blobstore2.BlobMetadataContainer proto which must never be consumed outside of Bigstore, and is not applicable to non-Google Cloud Storage media uploads.

A base64-encoded string.
downloadReadHandle

string (bytes format)

Read handle passed from Bigstore -> Scotty for a Google Cloud Storage download. This is a signed, serialized blobstore2.ReadHandle proto which must never be set outside of Bigstore, and is not applicable to non-Google Cloud Storage media downloads.

A base64-encoded string.
downloadExternalReadToken

string (bytes format)

A serialized External Read Token passed from Bigstore -> Scotty for a Google Cloud Storage download. This field must never be consumed outside of Bigstore, and is not applicable to non-Google Cloud Storage media uploads.

A base64-encoded string.
uploadFragmentListCreationInfo

string (bytes format)

A serialized Object Fragment List Creation Info passed from Bigstore -> Scotty for a Google Cloud Storage upload. This field must never be consumed outside of Bigstore, and is not applicable to non-Google Cloud Storage media uploads.

A base64-encoded string.

DiffVersionResponse

Backend response for a Diff get version response. For details on the Scotty Diff protocol.

JSON representation 
{
  "objectVersion": string,
  "objectSizeBytes": string
}
Fields
objectVersion

string

The version of the object stored at the server.
objectSizeBytes

string (int64 format)

The total size of the server object.

DiffChecksumsResponse

Backend response for a Diff get checksums response. For details on the Scotty Diff protocol.

JSON representation 
{
  "objectVersion": string,
  "objectSizeBytes": string,
  "chunkSizeBytes": string,
  "checksumsLocation": {
    object (CompositeMedia)
  },
  "objectLocation": {
    object (CompositeMedia)
  }
}
Fields
objectVersion

string

The object version of the object the checksums are being returned for.
objectSizeBytes

string (int64 format)

The total size of the server object.
chunkSizeBytes

string (int64 format)

The chunk size of checksums. Must be a multiple of 256KB.
checksumsLocation

object (CompositeMedia)

Exactly one of these fields must be populated.

If checksumsLocation is filled, the server will return the corresponding contents to the user. If objectLocation is filled, the server will calculate the checksums based on the content there and return that to the user. For details on the format of the checksums.
objectLocation

object (CompositeMedia)

If set, calculate the checksums based on the contents and return them to the caller.

DiffDownloadResponse

Backend response for a Diff download response. For details on the Scotty Diff protocol.

JSON representation 
{
  "objectLocation": {
    object (CompositeMedia)
  }
}
Fields
objectLocation

object (CompositeMedia)

The original object location.

DiffUploadRequest

A Diff upload request. For details on the Scotty Diff protocol.

JSON representation 
{
  "objectVersion": string,
  "objectInfo": {
    object (CompositeMedia)
  },
  "checksumsInfo": {
    object (CompositeMedia)
  }
}
Fields
objectVersion

string

The object version of the object that is the base version the incoming diff script will be applied to. This field will always be filled in.
objectInfo

object (CompositeMedia)

The location of the new object. Agents must clone the object located here, as the upload server will delete the contents once a response is received.
checksumsInfo

object (CompositeMedia)

The location of the checksums for the new object. Agents must clone the object located here, as the upload server will delete the contents once a response is received. For details on the format of the checksums.

DiffUploadResponse

Backend response for a Diff upload request. For details on the Scotty Diff protocol.

JSON representation 
{
  "objectVersion": string,
  "originalObject": {
    object (CompositeMedia)
  }
}
Fields
objectVersion

string

The object version of the object at the server. Must be included in the end notification response. The version in the end notification response must correspond to the new version of the object that is now stored at the server, after the upload.
originalObject

object (CompositeMedia)

The location of the original file for a diff upload request. Must be filled in if responding to an upload start notification.

ContentTypeInfo

Detailed Content-Type information from Scotty. The Content-Type of the media will typically be filled in by the header or Scotty's bestGuess, but this extended information provides the backend with more information so that it can make a better decision if needed. This is only used on media upload requests from Scotty.

JSON representation 
{
  "bestGuess": string,
  "fromHeader": string,
  "fromFileName": string,
  "fromUrlPath": string,
  "fromBytes": string
}
Fields
bestGuess

string

Scotty's best guess of what the content type of the file is.
fromHeader

string

The content type of the file as specified in the request headers, multipart headers, or RUPIO start request.
fromFileName

string

The content type of the file derived from the file extension of the original file name used by the client.
fromUrlPath

string

The content type of the file derived from the file extension of the URL path. The URL path is assumed to represent a file name (which is typically only true for agents that are providing a REST API).
fromBytes

string

The content type of the file derived by looking at specific bytes (i.e. "magic bytes") of the actual file.

DownloadParameters

Parameters specific to media downloads.

JSON representation 
{
  "allowGzipCompression": boolean,
  "ignoreRange": boolean
}
Fields
allowGzipCompression

boolean

A boolean to be returned in the response to Scotty. Allows/disallows gzip encoding of the payload content when the server thinks it's advantageous (hence, does not guarantee compression) which allows the server to compress the response to the client.
ignoreRange

boolean

Determining whether or not Apiary should skip the inclusion of any Content-Range header on its response to the upload service.