Section One: Getting Started¶

Who should read this book¶

This book has been written for software developers who wish to deploy applications to OpenStack clouds.

We’ve assumed that you’re an experienced programmer, but that you haven’t necessarily created an application for cloud in general, or for OpenStack in particular.

If you’re already familiar with OpenStack, you’ll save time learning about the general concepts, and you’ll still find value in learning how to work programmatically with it’s components.

What you will learn¶

Deploying applications in a cloud environment can be very different from the traditional siloed approachyou see in traditional IT, so in addition to learning to deploy applications on OpenStack, you will also learn some best practices for cloud application development. Overall, this guide covers the following:

• Section One: Getting Started - The most basic cloud application – creating and destroying virtual resources
• Section Two: Introduction to the Fractals Application Architecture - The architecture of a sample cloud-based application
• Section Three: Scaling Out - The importance of message queues
• Section Four: Making it Durable - Scaling up and down in response to changes in application load
• Section Five: Block Storage - Using object or block storage to create persistance
• Section Six: Orchestration - Orchestrating your cloud for better control of the environment
• Section Seven: Networking - Networking choices and actions to help relieve potential congestion
• Section Eight: Advice for Developers new to Operations - Advice for developers who may not have been exposed to operations tasks before
• Section Nine: Going Crazy - Taking your application to the next level by spreading it across multiple regions or clouds

A general overview¶

This tutorial actually involves two applications; the first, a fractal generator, simply uses mathematical equations to generate images. We’ll provide that application to you in its entirety, because really, it’s just an excuse; the real application we will be showing you is the code that enables you to make use of OpenStack to run it. That application includes:

• Creating and destroying compute resources. (Those are the virtual machine instances on which the Fractals app runs.)
• Cloud-related architecture decisions, such as breaking individual functions out into microservices and modularizing them.
• Scaling up and down to customize the amount of available resources.
• Object and block storage for persistance of files and databases.
• Orchestration services to automatically adjust to the environment.
• Networking customization for better performance and segregation.
• A few other crazy things we think ordinary folks won’t want to do ;).

Choosing your OpenStack SDK¶

Future versions of this book will cover completing these tasks with various toolkits, such as the OpenStack SDK, and using various languages, such as Java or Ruby. For now, however, this initial incarnation focuses on using Python with Apache Libcloud. That said, if you’re not a master Python programmer, don’t despair; the code is fairly straightforward, and should be readable to anyone with a programming background.

If you’re a developer for an alternate toolkit and would like to see this book support it, great! Please feel free to submit alternate code snippets, or to contact any of the authors or members of the Documentation team to coordinate.

Although this guide (initially) covers only Libcloud, you actually have several choices when it comes to building an application for an OpenStack cloud. These choices include:

A list of all available SDKs is available on the OpenStack wiki.

What you need¶

We assume you already have access to an OpenStack cloud. You should have a project (tenant) with a quota of at least 6 instances. The Fractals application itself runs in Ubuntu, Debian, and Fedora-based and openSUSE-based distributions, so you’ll need to be creating instances using one of these operating systems.

Interact with the cloud itself, you will also need to have

libcloud 0.15.1 or better installed.

You will need the following 5 pieces of information, which you can obtain from your cloud provider:

• auth URL
• username
• password
• project id or name (Projects are also known as tenants.)
• cloud region

You can also get this information by downloading the OpenStack RC file from the OpenStack Dashboard. To download this file, log into the Horizon dashboard and click Project->Access & Security->API Access->Download OpenStack RC file. If you choose this route, be aware that the “auth URL” doesn’t include the path. In other words, if your openrc.sh file shows:

export OS_AUTH_URL=http://controller:5000/v2.0

the actual auth URL will be

http://controller:5000

How you’ll interact with OpenStack¶

Throughout this tutorial, you’ll be interacting with your OpenStack cloud through code, using one of the SDKs listed in section “Choosing your OpenStack SDK”. In this initial version, the code snippets assume that you’re using libcloud.

To try it out, add the following code to a Python script (or use an interactive Python shell) by calling python -i.

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

auth_username = 'your_auth_username'
auth_password = 'your_auth_password'
auth_url = 'http://controller:5000'
project_name = 'your_project_name_or_id'
region_name = 'your_region_name'

provider = get_driver(Provider.OPENSTACK)
conn = provider(auth_username,
                auth_password,
                ex_force_auth_url=auth_url,
                ex_force_auth_version='2.0_password',
                ex_tenant_name=project_name,
                ex_force_service_region=region_name)

Flavors and Images¶

In order to run your application, the first thing you’ll need to do is create a virtual machine, or launch an instance. This instance behaves (for all intents and purposes) as a normal server.

In order to launch an instance, you will need to choose a flavor and an image. The flavor is essentially the size of the instance, such as its number of CPUs, amount of RAM and disk. An image is a prepared OS instalation from which your instance is cloned. Keep in mind when booting instances that larger flavors can be more expensive (in terms of resources, and therefore monetary cost, if you’re working in a public cloud) than smaller ones.

You can easily find out the images available in your cloud by running some API calls:

images = conn.list_images()
for image in images:
    print(image)

You should see a result something like:

<NodeImage: id=2cccbea0-cea9-4f86-a3ed-065c652adda5, name=ubuntu-14.04, driver=OpenStack  ...>
<NodeImage: id=f2a8dadc-7c7b-498f-996a-b5272c715e55, name=cirros-0.3.3-x86_64, driver=OpenStack  ...>

You can also get information on the various flavors:

flavors = conn.list_sizes()
for flavor in flavors:
    print(flavor)

This code should produce output something like:

<OpenStackNodeSize: id=1, name=m1.tiny, ram=512, disk=1, bandwidth=None, price=0.0, driver=OpenStack, vcpus=1,  ...>
<OpenStackNodeSize: id=2, name=m1.small, ram=2048, disk=20, bandwidth=None, price=0.0, driver=OpenStack, vcpus=1,  ...>
<OpenStackNodeSize: id=3, name=m1.medium, ram=4096, disk=40, bandwidth=None, price=0.0, driver=OpenStack, vcpus=2,  ...>
<OpenStackNodeSize: id=4, name=m1.large, ram=8192, disk=80, bandwidth=None, price=0.0, driver=OpenStack, vcpus=4,  ...>
<OpenStackNodeSize: id=5, name=m1.xlarge, ram=16384, disk=160, bandwidth=None, price=0.0, driver=OpenStack, vcpus=8,  ...>

Your images and flavors will be different, of course.

Choose an image and flavor to use for your first instance. To start with, we only need about 1GB of RAM, 1 CPU and a GB of disk, so in this example, the m1.small flavor, which exceeds these requirements, in conjuction with the Ubuntu image, is a safe choice. The flavor and image you choose here will be used throughout this guide, so you will need to change the IDs in the following tutorial sections to correspond to your desired flavor and image.

If you don’t see the image you want available in your cloud, you can usually upload a new one - depending on your cloud’s policy settings. There is a guide on how to aquire images available here.

Set the image and size variables to appropriate values for your cloud. We’ll use these in later sections.

First tell the connection to retrieve a specific image, using the ID of the image you have chosen to work with in the previous section:

image_id = '2cccbea0-cea9-4f86-a3ed-065c652adda5'
image = conn.get_image(image_id)
print(image)

You should see output something like this:

<NodeImage: id=2cccbea0-cea9-4f86-a3ed-065c652adda5, name=ubuntu-14.04, driver=OpenStack  ...>

Next tell the script what flavor you want to use:

flavor_id = '3'
flavor = conn.ex_get_size(flavor_id)
print(flavor)

You should see output something like this:

<OpenStackNodeSize: id=3, name=m1.medium, ram=4096, disk=40, bandwidth=None, price=0.0, driver=OpenStack, vcpus=2,  ...>

Now you’re ready to actually launch the instance.

Booting an instance¶

Now that you have selected an image and flavor, use it to create an instance.

Start by creating the instance.

instance_name = 'testing'
testing_instance = conn.create_node(name=instance_name, image=image, size=flavor)
print(testing_instance)

You should see output something like:

<Node: uuid=1242d56cac5bcd4c110c60d57ccdbff086515133, name=testing, state=PENDING, public_ips=[], private_ips=[], provider=OpenStack ...>

If you then output a list of existing instances...

instances = conn.list_nodes()
for instance in instances:
    print(instance)

... you should see the new instance appear.

<Node: uuid=1242d56cac5bcd4c110c60d57ccdbff086515133, name=testing, state=RUNNING, public_ips=[], private_ips=[], provider=OpenStack ...>

Before we move on, there’s one more thing you need to do.

Destroying an instance¶

It is important to keep in mind that cloud resources (including running instances you are no longer using) can cost money. Learning to remove cloud resources will help you avoid any unexpected costs incurred by unnecessary cloud resources.

conn.destroy_node(testing_instance)

If you then list the instances again, you’ll see that the instance no longer appears.

Leave your shell open, as you will use it for another instance deployment in this section.

Deploy the application to a new instance¶

Now that you are familiar with how to create and destroy instances, it is time to deploy the sample application. The instance you create for the app will be similar to the first instance you created, but this time, we’ll briefly introduce a few extra concepts.

When you create an instance for the application, you’re going to want to give it a bit more information than the bare instance we created and destroyed a little while ago. We’ll go into more detail in later sections, but for now, simply create these resources so you can feed them to the instance:

• A key pair. In order to access your instance, you will need to import an SSH public key into OpenStack to create a key pair. This key pair will be installed on the new instance by OpenStack. Typically, your public key is written to .ssh/id_rsa.pub. If you do not have an SSH public key file, follow the instructions here first. We’ll cover this in depth in section 2.

In the following example, pub_key_file should be set to the location of your public SSH key file.

print('Checking for existing SSH key pair...')
keypair_name = 'demokey'
pub_key_file = '~/.ssh/id_rsa.pub'
keypair_exists = False
for keypair in conn.list_key_pairs():
    if keypair.name == keypair_name:
        keypair_exists = True

if keypair_exists:
    print('Keypair already exists.  Skipping import.')
else:
    print('adding keypair...')
    conn.import_key_pair_from_file(keypair_name, pub_key_file)

for keypair in conn.list_key_pairs():
    print(keypair)

<KeyPair name=demokey fingerprint=aa:bb:cc... driver=OpenStack>

• Network access. By default, OpenStack will filter all traffic. You’ll need to create a security group that will allow HTTP and SSH access and apply it to your instance. We’ll go into more detail in section 2.

security_group_exists = False
for security_group in conn.ex_list_security_groups():
    if security_group.name =='all-in-one':
        all_in_one_security_group = security_group
        security_group_exists = True

if security_group_exists:
    print('Security Group already exists.  Skipping creation.')
else:
    all_in_one_security_group = conn.ex_create_security_group('all-in-one', 'network access for all-in-one application.')
    conn.ex_create_security_group_rule(all_in_one_security_group, 'TCP', 80, 80)
    conn.ex_create_security_group_rule(all_in_one_security_group, 'TCP', 22, 22)

• Userdata. During instance creation, userdata may be provided to OpenStack in order to configure instances after they boot. The userdata is applied to an instance by the cloud-init service. This service should be pre-installed on the image you have chosen. We’ll go into more detail in section 2.

userdata = '''#!/usr/bin/env bash
curl -L -s https://git.openstack.org/cgit/stackforge/faafo/plain/contrib/install.sh | bash -s -- \
    -i faafo -i messaging -r api -r worker -r demo
'''

Now you’re ready to boot and configure the new instance.

instance_name = 'all-in-one'
testing_instance = conn.create_node(name=instance_name,
                                    image=image,
                                    size=flavor,
                                    ex_keyname=keypair_name,
                                    ex_userdata=userdata,
                                    ex_security_groups=[all_in_one_security_group])
conn.wait_until_running([testing_instance])

print('Checking for unused Floating IP...')
unused_floating_ip = None
for floating_ip in conn.ex_list_floating_ips():
    if floating_ip.node_id:
        unused_floating_ip = floating_ip
        break


if not unused_floating_ip:
    pool = conn.ex_list_floating_ip_pools()[0]
    print('Allocating new Floating IP from pool: {}'.format(pool))
    unused_floating_ip = pool.create_floating_ip()

Accessing the application¶

Deploying application data and configuration to the instance can take some time. Consider enjoying a cup of coffee while you wait. After the application has been deployed, you will be able to visit the awesome graphic interface at the link provided below using your preferred browser.

print('The Fractals app will be deployed to http://%s' % unused_floating_ip.ip_address)

Note

If you are not using floating IPs, substitute another IP address as appropriate

Next Steps¶

Don’t worry if you don’t understand every part of what just happened. As we move on to Section Two: Introduction to the Fractals Application Architecture, we’ll go into these concepts in more detail.

• Section Three: Scaling Out - to learn how to scale the application further
• Section Four: Making it Durable - to learn how to make your application more durable using Object Storage
• Section Five: Block Storage - to migrate the database to block storage, or use the database-as-as-service component
• Section Six: Orchestration - to automatically orchestrate the application
• Section Seven: Networking - to learn about more complex networking
• Section Eight: Advice for Developers new to Operations - for advice for developers new to operations
• Section Nine: Going Crazy - to see all the crazy things we think ordinary folks won’t want to do ;)

Full example code¶

Here’s every code snippet into a single file, in case you want to run it all in one, or you are so experienced you don’t need instruction ;) If you are going to use this, don’t forget to set your authentication information and the flavor and image ID.

# step-1
from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

auth_username = 'your_auth_username'
auth_password = 'your_auth_password'
auth_url = 'http://controller:5000'
project_name = 'your_project_name_or_id'
region_name = 'your_region_name'

provider = get_driver(Provider.OPENSTACK)
conn = provider(auth_username,
                auth_password,
                ex_force_auth_url=auth_url,
                ex_force_auth_version='2.0_password',
                ex_tenant_name=project_name,
                ex_force_service_region=region_name)

# step-2
images = conn.list_images()
for image in images:
    print(image)

# step-3
flavors = conn.list_sizes()
for flavor in flavors:
    print(flavor)

# step-4
image_id = '2cccbea0-cea9-4f86-a3ed-065c652adda5'
image = conn.get_image(image_id)
print(image)

# step-5
flavor_id = '3'
flavor = conn.ex_get_size(flavor_id)
print(flavor)

# step-6
instance_name = 'testing'
testing_instance = conn.create_node(name=instance_name, image=image, size=flavor)
print(testing_instance)

# step-7
instances = conn.list_nodes()
for instance in instances:
    print(instance)

# step-8
conn.destroy_node(testing_instance)

# step-9
print('Checking for existing SSH key pair...')
keypair_name = 'demokey'
pub_key_file = '~/.ssh/id_rsa.pub'
keypair_exists = False
for keypair in conn.list_key_pairs():
    if keypair.name == keypair_name:
        keypair_exists = True

if keypair_exists:
    print('Keypair already exists.  Skipping import.')
else:
    print('adding keypair...')
    conn.import_key_pair_from_file(keypair_name, pub_key_file)

for keypair in conn.list_key_pairs():
    print(keypair)


# step-10
security_group_exists = False
for security_group in conn.ex_list_security_groups():
    if security_group.name =='all-in-one':
        all_in_one_security_group = security_group
        security_group_exists = True

if security_group_exists:
    print('Security Group already exists.  Skipping creation.')
else:
    all_in_one_security_group = conn.ex_create_security_group('all-in-one', 'network access for all-in-one application.')
    conn.ex_create_security_group_rule(all_in_one_security_group, 'TCP', 80, 80)
    conn.ex_create_security_group_rule(all_in_one_security_group, 'TCP', 22, 22)

# step-11
userdata = '''#!/usr/bin/env bash
curl -L -s https://git.openstack.org/cgit/stackforge/faafo/plain/contrib/install.sh | bash -s -- \
    -i faafo -i messaging -r api -r worker -r demo
'''

# step-12
instance_name = 'all-in-one'
testing_instance = conn.create_node(name=instance_name,
                                    image=image,
                                    size=flavor,
                                    ex_keyname=keypair_name,
                                    ex_userdata=userdata,
                                    ex_security_groups=[all_in_one_security_group])
conn.wait_until_running([testing_instance])

# step-13
print('Checking for unused Floating IP...')
unused_floating_ip = None
for floating_ip in conn.ex_list_floating_ips():
    if floating_ip.node_id:
        unused_floating_ip = floating_ip
        break


if not unused_floating_ip:
    pool = conn.ex_list_floating_ip_pools()[0]
    print('Allocating new Floating IP from pool: {}'.format(pool))
    unused_floating_ip = pool.create_floating_ip()

# step-14
conn.ex_attach_floating_ip_to_node(testing_instance, unused_floating_ip)

# step-15
print('The Fractals app will be deployed to http://%s' % unused_floating_ip.ip_address)