Description
This recipe is designed to give an example on how to configure a factory and frontend to submit user jobs on Amazon EC2.
| Requirement | Description |
| A functioning glideinwms factory | The factory should be completely configured and functioning for Grid submissions. The main reason for this is to be able to be assured that the factory is running and works before we do any configuration for cloud. |
| A functioning glideinwms frontend | The frontend should be completely configured and functioning for Grid submissions. The same reasoning for the factory applies here. |
| Specifically, the AccessKey and SecretKey are needed for submission. |
Example EC2 Factory Entry
<entry name="Amazon_Vandy"
auth_method="key_pair+vm_id+vm_type"
enabled="True"
gatekeeper="https://us-east-1.ec2.amazonaws.com"
gridtype="ec2"
schedd_name="cms-xen6.fnal.gov"
trust_domain="Cloud"
verbosity="std"
work_dir=".">
<config>
<max_jobs glideins="3" held="2" idle="1">
<max_job_frontends></max_job_frontends>
</max_jobs>
<release max_per_cycle="20" sleep="0.2"/>
<remove max_per_cycle="5" sleep="0.2"/>
<restrictions require_voms_proxy="False"/>
<submit cluster_size="10" max_per_cycle="100" sleep="0.2"/>
</config>
<allow_frontends></allow_frontends>
<attrs>
<attr name="CONDOR_ARCH" const="True" glidein_publish="False" job_publish="False"
parameter="True" publish="False" type="string" value="default"/>
<attr name="CONDOR_OS" const="True" glidein_publish="False" job_publish="False"
parameter="True" publish="False" type="string" value="default"/>
<attr name="GLEXEC_BIN" const="True" glidein_publish="False" job_publish="False"
parameter="True" publish="True" type="string" value="NONE"/>
<attr name="GLIDEIN_Site" const="True" glidein_publish="True" job_publish="True"
parameter="True" publish="True" type="string" value="Amazon_EC2"/>
<attr name="USE_CCB" const="True" glidein_publish="True" job_publish="False"
parameter="True" publish="True" type="string" value="True"/>
</attrs>
<files></files>
<infosys_refs></infosys_refs>
<monitorgroups></monitorgroups>
</entry>
The important pieces of the entry stanza listed above are listed below:
| Name | Type | Value | Description |
| auth_method |
The key pair in this case refers to the AccessKey and SecretKey that EC2-like cloud providers give for their REST interface. The vm_id and vm_type correspond to EC2's AMI_ID and AMI_TYPE descriptors. Each cloud implementation will have their own definitions for what these descriptors mean. In this example, the actual values will be configured by the frontend. See Factory Configuration for a complete description. |
||
| gatekeeper |
The gatekeeper attribute in the cloud case is similar enough to a grid gatekeeper that there is no function difference as far as the glideinWMS factory admin is concerned. EC2 has regional gatekeepers, so choose the gatekeeper for the region in which you would like to run in. In this example, the US-EAST region has bee selected. See Factory Configuration for a complete description. |
||
| gridtype | "ec2" |
To submit to EC2-like clouds, this attribute must be set to "ec2". See Factory Configuration for a complete description. |
|
| trust_domain | "Cloud" |
The trust domain can be any arbitrary value. The only caveat is that both the factory and the frontend must be configured to use the same value. In this example, "Cloud" is the arbitrary value. See Factory Configuration for a complete description. |
|
| work_dir | "." |
The working directory that the pilot starts up in must be "." for this example. The reason is that the VM that the example is pointing to makes specific use of the scratch space Amazon provides. This is in a non-standard location. For all intents and practical purposes, it will be the VOs responsibility to define the working directory on the VM and have the contextualization scripts handle the setup of where the pilot starts. See Factory Configuration for a complete description. |
|
| glideins | "3" |
This attribute is very important for cloud use. Even more so when real money is being used to pay for the computing cycles. This is a hard limit for the number of VMs that the factory will start. For testing purposes this example was restricted to 3 running VMs. See Factory Configuration for a complete description. |
|
| held | "1" |
This is a limit for the number of VM requests that can be in held state. If the number of held requests match this number, the factory will stop asking for more. For purposes of testing, this number was set extremely low. See Factory Configuration for a complete description. |
|
| idle | "1" |
This is a limit for the number of VM requests that can be in idle state. Ordinarily, this attribute is used to determine "pressure" at a grid site. However, the cloud use case is different considering that most cloud implementations do not operate on "allocations" or something similare, but are operated on a "pay-as-you-go" principle. Therefore, real money is exchanged for actual usage. By setting this value to "1", we basically turn off the "pressure" and ask for as many VMs as there are jobs up to the max set by the glideins attribute. See Factory Configuration for a complete description. |
Example EC2 Frontend Configuration
This only configuration for the frontend in this example is for the credential setup. The credential setup can be included in the group credential definition or in the global credential definition.
<credential absfname="/path/to/Cloud_AccessKey"
keyabsfname="/path/to/Cloud_SecretKey"
security_class="Security Class"
trust_domain="Cloud"
type="key_pair+vm_id+vm_type"
vm_id="ami-7bf43812"
vm_type="m1.large"
pilotabsfname="/path/to/pilot_proxy"/>
The important pieces of the credential stanza listed above are listed below:
| Name | Type | Value | Description |
| absfname | "/path/to/Cloud_AccessKey" |
This is the full path to the file containing the AccessKey for the account that will be used to submit the VM request See Frontend Configuration for a complete description. |
|
| keyabsfname | "/path/to/Cloud_SecretKey" |
This is the full path to the file containing the SectretKey for the account that will be used to submit
the VM request
See Frontend Configuration for a complete description. |
|
| security_class | "Security Class" |
This is the security class that is defined for the other credentials on this frontend See Frontend Configuration for a complete description. |
|
| trust_domain | "Cloud" |
The trust domain can be any arbitrary value. The only caveat is that both the factory and the frontend must be configured to use the same value. In this example, "Cloud" is the arbitrary value. See Frontend Configuration for a complete description. |
|
| type | "key_pair+vm_id+vm_type" |
The key pair in this case refers to the AccessKey and SecretKey that EC2-like cloud providers give for their REST interface. The vm_id and vm_type correspond to EC2's AMI_ID and AMI_TYPE descriptors. Each cloud implementation will have their own definitions for what these descriptors mean. In this example, the actual values will be configured by the frontend. This must match the value specified in the factory for the credentials to be matched properly See Frontend Configuration for a complete description. |
|
| vm_id | "ami-7bf43812" |
Since the <type> attribute contains vm_id, it must be specified here. See the specific cloud implementation for the correct vm_id value. In this example, a generic VM has been uploaded to Amazon EC2 and is ready for use. See Frontend Configuration for a complete description. |
|
| vm_type | "m1.large" |
Since the <type> attribute contains vm_type, it must be specified here. See the specific cloud implementation for the correct vm_type value. In this example, a generic VM has been uploaded to Amazon EC2 and is ready for use. See Frontend Configuration for a complete description. |
|
| pilotabsfname | "/path/to/pilot_proxy" |
A proxy for the pilot is required in all cases, even if proxies are not used to authenticate on the gatekeeper. This is because the proxy is used to establish secure communication between the pilot and the user collector. See Frontend Configuration for a complete description. |