Section Four: Making it Durable¶

Using Object Storage to store fractals¶

The Fractals app currently uses the local filesystem on the instance to store the images it generates. This is not scalable or durable, for a number of reasons.

Because the local filesystem is ephemeral storage, if the instance is terminated, the fractal images will be lost along with the instance. Block based storage, which we’ll discuss in Section Five: Block Storage, avoids that problem, but like local filesystems, it requires administration to ensure that it does not fill up, and immediate attention if disks fail.

The Object Storage service manages many of these tasks that normally would require the application owner to manage them, and presents a scalable and durable API that you can use for the fractals app, without having to be concerened with the low level details of how the objects are stored and replicated, and growing the storage pool. In fact, Object Storage handles replication intrinsicly, storing multiple copies of each object and returning one of them on demand using the API.

First, let’s learn how to connect to the Object Storage Endpoint:

from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver

auth_username = 'your_auth_username'
auth_password = 'your_auth_password'
auth_url = 'http://controller:5000'
project_name = 'your_project_name_or_id'
region_name = 'your_region_name'

provider = get_driver(Provider.OPENSTACK_SWIFT)
swift = provider(auth_username,
                 auth_password,
                 ex_force_auth_url=auth_url,
                 ex_force_auth_version='2.0_password',
                 ex_tenant_name=project_name,
                 ex_force_service_region=region_name)

To begin to store objects, we must first make a container. Call yours fractals:

container_name = 'fractals'
container = swift.create_container(container_name=container_name)
print(container)

You should see output such as:

<Container: name=fractals, provider=OpenStack Swift>

You should now be able to see this container appear in a listing of all containers in your account:

print(swift.list_containers())

You should see output such as:

[<Container: name=fractals, provider=OpenStack Swift>]

The next logical step is to upload an object. Find a photo of a goat online, name it goat.jpg and upload it to your container fractals:

file_path = 'goat.jpg'
object_name = 'an amazing goat'
container = swift.get_container(container_name=container_name)
object = container.upload_object(file_path=file_path, object_name=object_name)

List objects in your container fractals to see if the upload was successful, then download the file to verify the md5sum is the same:

objects = container.list_objects()
print(objects)

[<Object: name=an amazing goat, size=191874, hash=439884df9c1c15c59d2cf43008180048, provider=OpenStack Swift ...>]

object = swift.get_object(container_name, object_name)
print object

<Object: name=an amazing goat, size=954465, hash=7513986d3aeb22659079d1bf3dc2468b, provider=OpenStack Swift ...>

import hashlib
print(hashlib.md5(open('goat.jpg', 'rb').read()).hexdigest())

7513986d3aeb22659079d1bf3dc2468b

Finally, let’s clean up by deleting our test object:

swift.delete_object(object)

Now there should be no more objects be available in the container fractals.

objects = container.list_objects()
print(objects)

[]

container_name = 'fractals'
container = swift.get_container(container_name)

import base64
import cStringIO
import json

import requests

endpoint = 'http://IP_API_1'
params = { 'results_per_page': '-1' }
response = requests.get('%s/v1/fractal' % endpoint, params=params)
data = json.loads(response.text)
for fractal in data['objects']:
    response = requests.get('%s/fractal/%s' % (endpoint, fractal['uuid']), stream=True)
    container.upload_object_via_stream(response.iter_content(), object_name=fractal['uuid'])

for object in container.list_objects():
    print(object)

<Object: name=025fd8a0-6abe-4ffa-9686-bcbf853b71dc, size=61597, hash=b7a8a26e3c0ce9f80a1bf4f64792cd0c, provider=OpenStack Swift ...>
<Object: name=26ca9b38-25c8-4f1e-9e6a-a0132a7a2643, size=136298, hash=9f9b4cac16893854dd9e79dc682da0ff, provider=OpenStack Swift ...>
<Object: name=3f68c538-783e-42bc-8384-8396c8b0545d, size=27202, hash=e6ee0cd541578981c294cebc56bc4c35, provider=OpenStack Swift ...>

Extra Features¶

for object in container.list_objects():
    container.delete_object(object)
swift.delete_container(container)

file_path = 'goat.jpg'
object_name = 'backup_goat.jpg'
extra = {'meta_data': {'description': 'a funny goat', 'created': '2015-06-02'}}
with open('goat.jpg', 'rb') as iterator:
    object = swift.upload_object_via_stream(iterator=iterator,
                                            container=container,
                                            object_name=object_name,
                                            extra=extra)

swift.ex_multipart_upload_object(file_path, container, object_name,
                                 chunk_size=33554432)

Next Steps¶

You should now be fairly confident working with Object Storage. You can find more about the Object Storage SDK calls at:

https://libcloud.readthedocs.org/en/latest/storage/api.html

Or try a different step in the tutorial, including:

• Section Five: Block Storage - to migrate the database to block storage, or use the database-as-as-service component
• Section Six: Orchestration - to automatically orchestrate the application
• Section Seven: Networking - to learn about more complex networking
• Section Eight: Advice for Developers new to Operations - for advice for developers new to operations