Working with object storage =========================== Object storage provides a simple way to store and retrieve large amounts of unstructured data over HTTP. Object Storage is also referred to as Blob (Binary Large OBject) Storage by Azure, and Simple Storage Service (S3) by Amazon. Typically, you would store your objects within a Bucket, as it is known in AWS and GCP. A Bucket is also called a Container in OpenStack and Azure. In CloudBridge, we use the term Bucket. Storing objects in a bucket --------------------------- To store an object within a bucket, we need to first create a bucket or retrieve an existing bucket. .. code-block:: python bucket = provider.storage.buckets.create('my-bucket') bucket.objects.list() Next, let's upload some data to this bucket. To efficiently upload a file, simple use the upload_from_file method. .. code-block:: python obj = bucket.objects.create('my-data.txt') obj.upload_from_file('/path/to/myfile.txt') You can also use the upload() function to upload from an in memory stream. Note that, an object you create with objects.create() doesn't actually get persisted until you upload some content. To locate and download this uploaded file again, you can do the following: .. code-block:: python bucket = provider.storage.buckets.find(name='my-bucket')[0] obj = bucket.objects.find(name='my-data.txt')[0] print("Size: {0}, Modified: {1}".format(obj.size, obj.last_modified)) with open('/tmp/myfile.txt', 'wb') as f: obj.save_content(f) To download to a local path, prefer download_to_file(), which fetches large objects as parallel ranged reads. save_content() and iter_content() are the single-stream alternatives, for writing to an arbitrary stream or for handing the content on somewhere else - proxying it to an HTTP response, say. .. code-block:: python for chunk in obj.iter_content(): process(chunk) Content is streamed, never held in memory whole, and chunks are sized by chunk_size rather than by the content - binary data is never split on newlines. Only the last chunk may be short. .. code-block:: python for chunk in obj.iter_content(chunk_size=4 * 1024 * 1024): process(chunk) chunk_size defaults to 1 MiB, which suits most callers. It trades per-chunk overhead against memory and latency: each chunk costs a read from the provider plus whatever your loop does per chunk, so small values get expensive over a large object, while a large value is buffered in full before it is yielded and so costs memory per concurrent stream. To change the default globally, set the iter_chunk_size provider config value or the CB_ITER_CHUNK_SIZE environment variable. save_content() takes the same argument. .. note:: On GCP the content is fetched as successive ranged requests, because the underlying client library has no single-connection streaming read, so chunk_size is also the request size there. Prefer a large chunk_size when streaming a large object from GCP. Using tokens for authentication ------------------------------- Some providers may support using temporary credentials with a session token, in which case you will be able to access a particular bucket by using that session token. .. code-block:: python provider = CloudProviderFactory().create_provider( ProviderList.AWS, {'aws_access_key': 'ACCESS_KEY', 'aws_secret_key': 'SECRET_KEY', 'aws_session_token': 'MY_SESSION_TOKEN'}) .. code-block:: python provider = CloudProviderFactory().create_provider( ProviderList.OPENSTACK, {'os_storage_url': 'SWIFT_STORAGE_URL', 'os_auth_token': 'MY_SESSION_TOKEN'}) Once a provider is obtained, you can access the container as usual: .. code-block:: python bucket = provider.storage.buckets.get(container) obj = bucket.objects.create('my_object.txt') obj.upload_from_file(source) Generating signed URLs ---------------------- Signed URLs are a great way to allow users who do not have credentials for the cloud provider of your choice, to interact with an object within a storage bucket. You can generate signed URLs with ``GET`` permissions to allow a user to get an object. .. code-block:: python provider = CloudProviderFactory().create_provider( ProviderList.AWS, {'aws_access_key': 'ACCESS_KEY', 'aws_secret_key': 'SECRET_KEY', 'aws_session_token': 'MY_SESSION_TOKEN'}) bucket = provider.storage.buckets.get("my-bucket") obj = bucket.objects.get("my-file.txt") url = obj.generate_url(expires_in=7200) You can also generate a signed URL with `PUT` permissions to allow users to upload files to your storage bucket. .. code-block:: python provider = CloudProviderFactory().create_provider( ProviderList.AWS, {'aws_access_key': 'ACCESS_KEY', 'aws_secret_key': 'SECRET_KEY', 'aws_session_token': 'MY_SESSION_TOKEN'}) bucket = provider.storage.buckets.get("my-bucket") obj = bucket.objects.create("my-file.txt") url = obj.generate_url(expires_in=7200, writable=True) With your signed URL, you or someone on your team can upload a file like this .. code-block:: python import requests content = b"Hello world!" # Only Azure requires the x-ms-blob-type header to be present, but there's no harm # in sending this in for all providers. headers = {'x-ms-blob-type': 'BlockBlob'} requests.put(url, data=content, headers=headers) Uploading without credentials ----------------------------- A signed URL from ``generate_url`` needs the object to exist first. To let a client without credentials upload an object that doesn't exist yet, generate a presigned request for it by name. Signing it creates nothing. .. code-block:: python bucket = provider.storage.buckets.get("my-bucket") request = bucket.objects.generate_upload_url( "my-file.txt", expires_in=3600, size=len(content)) The client sends the content as the body of a ``request.method`` request to ``request.url``, with ``request.headers``. Azure, for instance, needs the ``x-ms-blob-type`` header there. .. code-block:: python import requests response = requests.request(request.method, request.url, headers=request.headers, data=content) response.raise_for_status() Where the store can bind the size into the signature, ``request.enforces_size`` is ``True`` and the store refuses a body of any other size. Otherwise, check the object's size once it is uploaded. A large object can be uploaded in parts, each through a request of its own. The upload is begun and completed with credentials, possibly by different processes: ``get_multipart_upload`` reopens an upload from its id. .. code-block:: python upload = bucket.objects.create_multipart_upload("my-file.bin") requests_for_client = [upload.generate_part_url(part_number, 3600) for part_number in (1, 2, 3)] # The client sends each part and reports the ETag header of each # response. upload = bucket.objects.get_multipart_upload("my-file.bin", upload.id) upload.complete([upload.reported_part(part_number, etag) for part_number, etag in reported_etags.items()]) A part's URL works until it expires, so the part it uploaded may have been replaced since its ETag was reported. ``complete`` raises ``InvalidPartException`` if the store doesn't hold a part as reported, where the store can tell. ``upload.list_parts()`` lists the parts the store holds, with their sizes, which can be checked before completing. What each store can check: ====================== ============================ ========================= Provider Size bound into the request Reported ETag checked ====================== ============================ ========================= AWS Yes, with SigV4 signatures Yes, by S3 GCP Yes Yes, against each part's MD5 OpenStack No Yes, by Swift Azure No No: Azure returns no ETag for a part, and ``reported_part`` ignores it ====================== ============================ ========================= On Azure a part's request can't be limited to that part: until it expires, it can write any part of the object, or the whole object. On OpenStack, presigned requests and ``generate_url`` sign with the account's temp-URL key (``X-Account-Meta-Temp-URL-Key``, else ``-Key-2``). On an account without one, cloudbridge sets a random key and keeps it for later requests.