Going further with CREATE Cloud

On this page we step into some more advanced use cases of CREATE Cloud. As ever we encourage you to get in touch if anything is unclear, we are always happy discuss your specific needs.

Volume types

As detailed in the getting started doc, for storing and processing research data we expect people mount our RDS or HPC scratch storage. This leaves OpenStack volume storage for use as VM disks or specific use cases (e.g. database storage) where larger "local" disks may be preferable to networked storage.

Volume storage is provided by our Ceph cluster and is either backed by HDD (slower) or SSD (faster) drives. Free tier projects will use HDD for their 40GB volume quota and paid projects can choose whether to opt for HDD or SSD, with the latter being highly advisable. Projects that purchase SSD volume quota will have their default volume type set to SSD so in general this distinction will not need to be considered during project configuration.

Tenant Networking

In our Getting Started doc to keep things simple we attach our instances directly to the King's network and use Security Groups to allow access via SSH. For more complex architectures that involve traffic between multiple VMs within an OpenStack project we recommend configuring an internal "tenant" network.

Navigating the "Network" section of the OpenStack UI yields the following sections.

Network Topology

Gives a graphical overview of the whole network in the project and provides shortcuts to most objects.

Networks

An internal network can be created for VMs to be associated with. Open the dialogue to create a network and:

1. Leave the default settings and add a unique network name.
2. Select 'Subnet', add a unique subnet name and network address. The hint suggests
   192.168.0.0/24 which will provide 254 usable addresses (although one will be taken for the
   gateway address).
3. Additional subnet details can be added although it is not necessary for most uses.

For VMs that need to be connected to from outside the project (e.g. via SSH, HTTP/S or VDI) a floating IP on the external network will also need to be associated in addition to any internal network interfaces.

Routers

To allow VMs with only internal network interfaces to communicate with external services one must configure a router.

1. Create a router with a unique name attached to the external network
2. Select the router by clicking on its name
3. Select the 'Interfaces' tab and add an interface
4. Choose the internal subnet you want the router to connect to

CREATE HPC access to Cloud database

Attention

This approach is only suitable for processing non-sensitive data. The security of the database relies on a single password authentication step from any other CREATE user. For processing sensitive data please contact us about the CREATE Trusted Research Environment.

In this section we talk about how to setup a simple MariaDB (MySQL fork) database that can be accessed from the CREATE HPC cluster. This is intended to be a basic "Hello, World!" type example for building such integrations between HPC and Cloud sub-systems of CREATE.This is not a MySQL setup, administration or API interaction guide, further readings on those topics should be sought in line with your design goals.

• Follow the Getting Started doc to create a new VM.
• On the VM
  • sudo apt update && sudo apt install -y mariadb-server
  • sudo systemctl start mariadb.service
  • sudo mysql_secure_installation
    • enter for none
    • n for set root password
    • Y for everything else
  • Setup your database, for example:

ubuntu@cloudpig:~$ sudo mysql
MariaDB [(none)]> CREATE DATABASE mysql_example;
MariaDB [(none)]> use mysql_example;
MariaDB [mysql_example]> CREATE TABLE stuff (
    -> id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
    -> thing VARCHAR(100) NOT NULL);
MariaDB [mysql_example]> INSERT INTO stuff (thing) VALUES ('foo'), ('bar'), ('baz');
MariaDB [mysql_example]> select * from stuff;
+----+-------+
| id | thing |
+----+-------+
|  1 | foo   |
|  2 | bar   |
|  3 | baz   |
+----+-------+
CREATE USER alice@'10.211.%' IDENTIFIED BY 'horsebatterystaple';
GRANT ALL PRIVILEGES ON mysql_example.* TO alice@'10.211.%';

• Back in the OpenStack UI:
  • 'Edit Security Groups' for your VM
  • Click the + next to 'er_mysql_from_hpc'
• On your VM instance:
  • sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf
  • Change the variable bind-address = 0.0.0.0
• On CREATE HPC:
  • module load py-mysql-connector-python
  • You can then use python code within your submissions to interact with your database, e.g.:

import mysql.connector

db = mysql.connector.connect(
    host="10.211.X.Y",
    user="alice",
    password="horsebatterystaple",
    database="mysql_example"
)

cursor = db.cursor()

cursor.execute("SELECT * FROM stuff")

results = cursor.fetchall()

for result in results:
    print(result)

Internet facing Apache server

Attention

Before embarking on an Internet facing OpenStack project one should ensure they rule out the use of simpler managed web hosting solutions beforehand.

Using OpenStack will allow you run whatever web stack you wish but with great flexibility comes great responsibility. The options to rule out (in order) are:

• King's website using CMS content types e.g. 'research project'
• CREATE Web for automated static (HTML/CSS/JS) content hosting from a Git repo or self-service WordPress provisioning

Once the above have been ruled out:

• Follow the Getting Started doc to create a new VM.
• On the VM:
  • sudo apt update && sudo apt install -y apache2
  • sudo a2enmod ssl
  • sudo systemctl restart apache2
  • sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt
  • Fill in the prompts appropriately, for Common Name enter .sites.er.kcl.ac.uk where is a descriptive name for your site
  • Create a file /etc/apache2/sites-available/<somename>.sites.er.kcl.ac.uk.conf with the content

<VirtualHost *:443>
   ServerName <somename>.sites.er.kcl.ac.uk
   DocumentRoot /var/www/<somename>.sites.er.kcl.ac.uk

   SSLEngine on
   SSLCertificateFile /etc/ssl/certs/apache-selfsigned.crt
   SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key
</VirtualHost>

• sudo mkdir /var/www/<somename>.sites.er.kcl.ac.uk
• echo '<h1>Hello, World!</h1>' | sudo tee /var/www/<somename>.sites.er.kcl.ac.uk/index.html
• sudo a2ensite <somename>.sites.er.kcl.ac.uk
• sudo systemctl reload apache2
• If everything went as planned you should be able to check things locally with curl:

ubuntu@web1:~$ curl https://localhost:443 --insecure
<h1>Hello, World!</h1>

• Moving to the e-Research Portal click the group you wish to work with under the 'Groups' section of the navigation panel
  • Click 'Add resource' and choose 'Standard website'
  • Enter a Site name and a description for your site
  • Enter Site name in the domain name box (note that names outside of .sites.er.kcl.ac.uk will require specific DNS changes, please get in touch to discuss)
  • Select External (Internal would limit access to devices on the King's network or users connected to the e-Research VPN)
  • Choose "Require KCL authentication" and choose the group this site is part of (this will
    bounce users via King's multi factor authentication and check they are group members before they
    access the site, this setting can be removed later if open access is required but is good to
    operate with while setting things up initially)
  • The site will now show as building, click 'View'
  • Initially the Locations Add/Actions buttons may show as locked, if so wait a minute and refresh the page
  • Choose 'Actions' > 'Edit' for the Location '/'
  • Click 'Proxy' and select the VM you have been working with in the Target field (this may take a second or two to populate)
  • Click 'Edit' to update the configuration
• Back in the OpenStack UI:
  • 'Edit Security Groups' for your VM
  • Click the + next to 'er_https_from_lbs'
• If all is well you should now be able to browse to https://.sites.er.kcl.ac.uk