1. Introduction#

1.1. Context#

The present documentation has been written as supporting material for the Engineering Team hackathon held in Berlin from January 22 until January 25, 2024.

1.2. Goal#

The goal is to guide the user through the setup of a fully functional AWS-backed Cloudspace, by manually setting up the underlying infrastructure (networking, backing services,…) and by using the ansible playbook to provision the different virtual machines.

Please note that this guide is not intended to provide a full understanding of the concepts, choices, and limitations of the current setup. Those notions are rather intended to be transmitted interactively during the walkthrough.

1.3. Structure#

This documentation is structured into six main parts, each building upon each other and progressively adding more functionality to the Cloudspace:

  1. Logging into your AWS environment

    Before we can start working on setting up the infrastructure, we need to gain access to an AWS account to use as playground. This section guides the reader through the process to operations and configuration necessary to access a sub-account via role assumption.

  2. Virtual Private Cloud

    The “container” for all resources. It’s a network namespace where all the components will run and that allows them to communicate between each other.

  3. SSH Bastion

    Our initial foothold into the new network we just created. It will act as an SSH Bastion host (or jumphost) allowing us to get access to the private network and, in the following phase, accessing other virtual machines in the Cloudspace.

  4. Application Controller

    The application controller is the heart of a Cloudspace. It is a custom-built orchestrator that knows about all applications running in a Cloudspace and makes sure that the containers are scheduled correctly and that the ingress tier knows where to route the traffic to.

  5. Application Runners and Builders

    Applications runners and builders are virtual machines used by the application controller to build application docker images during deployment and to run their containers as required.

  6. Ingress Tier

    Having application containers running is not very useful if they are not able to receive and respond to HTTP requests. The ingress tier is responsible to accept incoming requests and route them to the correct runner and container for the application responsible for their handling.

  7. Services API

    Finally, without backing services, the running applications would not have a way to persist their state. The services API is responsible for the provisioning, deprovisioning, backup, and restore of backing services such as databases, message queues, and object storage.

The content available in the appendices provides additional information for the interested reader but is not required for a successful setup.

1.4. Prerequisites#

To successfully follow this guide, you will need the following:

  • Deployment repo setup with 1Password integration

  • Access to the following 1Password items:

    • AWS Hackathon AWS

    • AWS Hackathon Elasticsearch

    • AWS Hackathon RabbitMQ

    • dnsimple.com

    • divio-cloud-deployment-vault/development

  • A OneLogin account with the developer role assigned

  • SSH keys added to the agent

1.5. Some Homework for You#

While going through this guide, please:

  • Think critically about the setup: why are things done this way? Are there better solutions? What could be improved? How is security, availability, and scalability affected by the design decisions?

  • Think about and write down very concrete improvements to be made both at the infrastructure level but also at the application level. Consider operational issues (e.g., can we cycle all hosts without manual intervention?).

  • If we would replace the all the manual work described in the present guide with an automation tool (e.g., Terraform or Ansible), what needs to be improved? What are the limitations?

  • Ask questions! This guide is meant as a way to know what to do to set up a Cloudspace. It is on purpose very light on contextual information and explanations, but the goal is that you get a full picture of the current situation, including design decisions, understanding of the various moving parts, and the ability to work on individual components in the future.