Bamboo 3.4 : Configuring Elastic Bamboo

Elastic Bamboo is a feature in Bamboo that allows you to utilise computing resources from the Amazon Elastic Compute Cloud (EC2) to run Job builds. Elastic Bamboo uses a remote agent AMI (Amazon Machine Image) to create instances of remote agents in the Amazon EC2. Job builds can be run on these 'elastic agents' in the same way that Job builds are run on local and remote agents.

Elastic Bamboo configuration settings include information that is required for Elastic Bamboo to work and parameters for optional features. These settings are divided into the following logical groups:

All of these settings are located on the Elastic Bamboo Configuration screen (see screenshot). The instructions below describe how to update each of the groups of settings separately, but you can update all of the settings at once, if you wish.

(warning) If you have disabled remote agent support, you must enable it before you can enable Elastic Bamboo. Refer to Disabling and Enabling Remote Agents Support for instructions on how to enable remote agent support.

Screenshots above: Elastic Bamboo Configuration

Amazon Web Services (AWS) Account Settings

 

Before you can use Elastic Bamboo in your Bamboo instance, you must enter your Amazon Web Services (AWS) account details into the Bamboo application. If you do not have an AWS account, you must register for one on the AWS registration page before you can enable Elastic Bamboo.

Before you begin:

  • Please note, Elastic Bamboo dynamically creates and runs remote agents in the Amazon Elastic Compute Cloud (EC2). Hence, if you choose to use Elastic Bamboo, you will be charged by Amazon for your EC2 compute usage (separate to your Bamboo license fee). These charges will be billed to the AWS account that you provide. Please read Elastic Bamboo Costs for more details.
  • Please note, if you change your AWS account details, Bamboo will stop all elastic agents that are currently running.

To configure your Amazon Web Services (AWS) account details:

  1. Click the 'Administration' link in the top navigation bar.
  2. Click the 'Configuration' link in the left navigation column under the 'Elastic Bamboo' sub-header.
  3. Click the 'Edit' button to edit your Elastic Bamboo configuration (see screenshot above).
  4. Enter or update your 'AWS Access Key ID'.
  5. Check the 'Change AWS Secret Access Key?' checkbox, if you wish to enter or update your AWS Secret Access Key.
  6. Enter or update 'AWS Secret Access Key' as desired (see Notes section below).
  7. Click the 'Save' button to update your Account Details. If the details you have provided are invalid, a validation message will display and you will be required to update your account details to correct the error.

Elastic Bamboo Global Settings

 

Elastic Bamboo provides you with a number of global configuration options to help you optimise EC2 usage for your Bamboo Job builds. These settings control how the Bamboo server operates and how it manages its elastic instances and agents.

To configure your Elastic Bamboo global settings:

  1. Click the 'Administration' link in the top navigation bar.
  2. Click the 'Configuration' link in the left navigation column under the 'Elastic Bamboo' sub-header.
  3. Click the 'Edit' button to edit your Elastic Bamboo configuration.
  4. In the 'Elastic Bamboo Global Settings' section of the 'Elastic Bamboo Configuration' screen, update these settings as desired:
    • 'Maximum Number of Elastic Instances' — this setting controls how many elastic instances can be running at any one time. For example, you may wish to change this setting to a small value if you are concerned about EC2 compute costs, and you have a large number of concurrent Job builds that cannot be supported by your non-elastic agents.
    • 'Automatically shut down elastic instance when elastic agent process ends' — this setting controls whether your elastic instances will automatically shut down after the elastic agent processes running on them terminate. If you have automated the starting and stopping of elastic agents (i.e. via the Bamboo Remote API), you can use this option to automatically shut down their instances as well.
      • 'Shutdown Delay' — this setting will only display if you have checked the 'Automatically shut down elastic instance when elastic agent process ends' option. It controls how long an elastic instance will wait before shutting down, after its elastic agent process terminates.
  5. Click the 'Save' button to update your elastic instance settings.

Configuring Elastic Bamboo to Use EC2 Spot Instances

 

Elastic Bamboo provides support for Amazon EC2 Spot Instances. Amazon spot instances allow you to bid on unused EC2 capacity and use it, as long as your bid exceeds the current "Spot price". You can configure Elastic Bamboo to bid for a spot instance of a particular type, and fall back to a regular instance after a set amount of time if no instances are available.

To configure Elastic Bamboo to use EC2 Spot Instances:

  1. Click the 'Administration' link in the top navigation bar.
  2. Click the 'Configuration' link in the left navigation column under the 'Elastic Bamboo' sub-header.
  3. Click the 'Edit' button to edit your Elastic Bamboo configuration.
  4. In the 'Spot instances configuration' section , update these settings as desired:
    • 'Enable support for spot instances' — Tick this checkbox to enable support for spot instances.
    • 'Fallback to a regular instance after' — Enter the time (in minutes) after which Elastic Bamboo will fall back to using a regular instance, if a spot instance has not become available.
    • 'Your current bid levels (per hour)' — Fill out this table with your bids. The bids are categorised by EC2 instance type and operating system.
  5. Click the 'Save' button.

Elastic Bamboo AWS Settings

 

These settings allow you to specify your AWS configuration settings in Bamboo so that Bamboo can operate elastic instances through your AWS account. This section includes settings that are used to configure elastic instances to work with the Amazon Elastic Block Store (EBS).

Using EBS with your elastic instances can significantly reduce the amount of data transfer required to run a Job build, compared with starting a clean elastic instance. To find out more about this feature and how to set it up in Elastic Bamboo, read Configuring Elastic Instances to use the EBS.

To configure your elastic instance settings:

  1. Click the 'Administration' link in the top navigation bar.
  2. Click the 'Configuration' link in the left navigation column under the 'Elastic Bamboo' sub-header.
  3. Click the 'Edit' button to edit your account details.
  4. In the 'Elastic Bamboo AWS Settings' section of the 'Elastic Bamboo Configuration' screen, update your elastic instance settings as desired:
    • Upload AWS account identifiers to new elastic instances (mandatory if EBS Snapshot ID specified) — Check this option to upload the AWS Account Private Key File and Account Certificate File to all new elastic instances started. This is mandatory if you wish to use EBS to store Job build information in a snapshot. However, you can also check this option if you are not using EBS (e.g. if you wish upload the AWS account identifiers in order to use Amazon's AWS command line tools).
      • Account Private Key File — this setting will only display if you have checked the 'Upload AWS account identifiers to new elastic instances (mandatory if EBS Snapshot ID specified)' option. You must specify the location of this file (on your Bamboo server) to use the Amazon EBS with Elastic Bamboo.
      • Account Certificate File — this setting will only display if you have checked the 'Upload AWS account identifiers to new elastic instances (mandatory if EBS Snapshot ID specified)' option. You must specify the location of this file (on your Bamboo server) to use the Amazon EBS with Elastic Bamboo.
        (info) If you haven't downloaded an AWS private key file or certificate file to your Bamboo server yet, please see Generating your AWS Private Key File and Certificate File for instructions.
  5. Click the 'Save' button to update your elastic instance settings.

Automatic Elastic Instance Management

The Automatic Elastic Instance Management feature allows Bamboo to start and shut down elastic instances automatically (based on build queue demands), so that you do not have to perform these action manually. This feature reduces Bamboo administration overhead and can help minimise your overall elastic instance usage costs.

If a Job's requirements cannot be met by any available online agents, this feature will start any elastic instance whose elastic agent has the capabilities to execute the Job, so that the Job's build can be generated. Regardless of how an elastic instance was started, all elastic instances will be shut down based on the settings specified below.

To configure Bamboo's elastic instance settings:

  1. Click the 'Administration' link in the top navigation bar.
  2. Click the 'Configuration' link in the left navigation column under the 'Elastic Bamboo' sub-header.
  3. Click the 'Edit' button to edit your account details.
  4. In the 'Automatic Elastic Instance Management' section of the 'Elastic Bamboo Configuration' screen, choose one of Bamboo's elastic instance management settings:
    • Preset Options — You can choose from the following elastic instance management presets. Each of these presets sets predefined values for the five criteria described in the 'Custom' user-defined options (below). (Bear in mind that both the 'Aggressive' and 'Passive' presets have trade-offs.)
      • 'Default' — Balances build queue clearance rates with elastic instance usage costs.
      • 'Aggressive' — Favours higher build queue clearance rates but with higher elastic instance usage costs.
      • 'Passive' — Favours lower instance usage costs but with lower build queue clearance rates.
    • User-defined Options — If you want more control over this feature than what's provided by the presets (above), or you want to disable elastic instance management, choose the appropriate option.
      • 'Custom' — Choose this option to individually specify the following five criteria, which are used to manage Bamboo's elastic images.
        • 'Idle Agent Shutdown Delay' — Specify the number of minutes that an elastic agent must be idle before Bamboo shuts down the elastic instance running that agent.
          (info) Elastic instances running in the Amazon EC2 compute cloud are charged in hourly blocks from the time they are started. Hence, to maximise usage of elastic instances in a cost-effective manner, Bamboo only performs these checks just prior to the expiry of each hourly block.
        • 'Maximum Number of Instances to Start at Once' — specify the maximum number of elastic instances that Bamboo can start in one go. Bamboo only starts this maximum number of elastic instances on a 'per minute' basis.
        • The remaining three criteria are thresholds for starting new elastic instances. New elastic instances will only be started when all of the following thresholds have been reached:
          • 'Maximum Number of Builds in Queue Threshold' — Specify the maximum threshold for the total number of Job builds in the queue.
          • 'Number of Elastic Builds in Queue Threshold' — Specify the threshold for the maximum number of Job builds in the queue that can be executed on elastic instances.
          • 'Average Queue Time Threshold' — Specify the maximum threshold for the average number of minutes that Job builds have been waiting in a queue.
      • 'Disabled' — Choose this option to completely disable Bamboo's elastic instance management feature.
  5. Click the 'Save' button to update your elastic instance settings.

Notes

  • AWS Access Key ID and AWS Secret Access Key — Your AWS Access Key ID and AWS Secret Access Key are used together to identify yourself when accessing Amazon EC2 services. If you are unsure what your AWS Account ID and AWS Secret Access Key are, please refer to the Amazon documentation on AWS access identifiers.
Related Topics

Generating your AWS Private Key File and Certificate File
Configuring Elastic Instances to use the EBS
Managing Elastic Bamboo
Disabling Elastic Bamboo