Skip to content

updated ec2 instance types #6014

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

updated ec2 instance types #6014

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1657c26
updated ec2 instance types
May 8, 2015
85c6236
vertically aligned whitespace
May 8, 2015
80c2aa6
vertically aligned whitespace
May 8, 2015
fc2931f
updated ec2 instance types
May 8, 2015
6366c45
added back cc1.4xlarge
May 8, 2015
d7b4191
Merge branch 'ec2-instance-types-update' of github.com:brendancol/spa…
May 8, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 47 additions & 23 deletions ec2/spark_ec2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,46 +347,57 @@ def get_validate_spark_version(version, repo):


# Source: http://aws.amazon.com/amazon-linux-ami/instance-type-matrix/
# Last Updated: 2014-06-20
# Last Updated: 2015-05-08
# For easy maintainability, please keep this manually-inputted dictionary sorted by key.
EC2_INSTANCE_TYPES = {
"c1.medium": "pvm",
"c1.xlarge": "pvm",
"c3.large": "pvm",
"c3.xlarge": "pvm",
"c3.2xlarge": "pvm",
"c3.4xlarge": "pvm",
"c3.8xlarge": "pvm",
"c3.large": "pvm",
"c3.xlarge": "pvm",
"c4.large": "hvm",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as below

"c4.xlarge": "hvm",
"c4.2xlarge": "hvm",
"c4.4xlarge": "hvm",
"c4.8xlarge": "hvm",
"cc1.4xlarge": "hvm",
"cc2.8xlarge": "hvm",
"cg1.4xlarge": "hvm",
"cr1.8xlarge": "hvm",
"d2.xlarge": "hvm",
"d2.2xlarge": "hvm",
"d2.4xlarge": "hvm",
"d2.8xlarge": "hvm",
"g2.2xlarge": "hvm",
"g2.8xlarge": "hvm",
"hi1.4xlarge": "pvm",
"hs1.8xlarge": "pvm",
"i2.xlarge": "hvm",
"i2.2xlarge": "hvm",
"i2.4xlarge": "hvm",
"i2.8xlarge": "hvm",
"i2.xlarge": "hvm",
"m1.large": "pvm",
"m1.medium": "pvm",
"m1.small": "pvm",
"m1.medium": "pvm",
"m1.large": "pvm",
"m1.xlarge": "pvm",
"m2.xlarge": "pvm",
"m2.2xlarge": "pvm",
"m2.4xlarge": "pvm",
"m2.xlarge": "pvm",
"m3.2xlarge": "hvm",
"m3.large": "hvm",
"m3.medium": "hvm",
"m3.large": "hvm",
"m3.xlarge": "hvm",
"m3.2xlarge": "hvm",
"r3.large": "hvm",
"r3.xlarge": "hvm",
"r3.2xlarge": "hvm",
"r3.4xlarge": "hvm",
"r3.8xlarge": "hvm",
"r3.large": "hvm",
"r3.xlarge": "hvm",
"t1.micro": "pvm",
"t2.medium": "hvm",
"t2.micro": "hvm",
"t2.small": "hvm",
"t2.medium": "hvm",
}


Expand Down Expand Up @@ -878,44 +889,57 @@ def wait_for_cluster_state(conn, opts, cluster_instances, cluster_state):
# Get number of local disks available for a given EC2 instance type.
def get_num_disks(instance_type):
# Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like cc1.4xlarge has been removed - Is that intentional ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah so I was going off of:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html

Its absense may be an error, or intentional from AWS. I've seen:

EC2 is steering support away from cc1.4xlarge instances and toward
cc2.8xlarge instances. The cc2.8xlarge instance supports additional
features such as VPC and availability in all regions. When creating
instances in the RightScale Dashboard, use cc2.8xlarge instances instead of
cc1.4xlarge.

On Fri, May 8, 2015 at 3:36 PM, Shivaram Venkataraman <
[email protected]> wrote:

In ec2/spark_ec2.py
#6014 (comment):

     "c3.2xlarge":  2,
     "c3.4xlarge":  2,
     "c3.8xlarge":  2,

  •    "c3.large":    2,

  •    "c3.xlarge":   2,

  •    "cc1.4xlarge": 2,

Looks like cc1.4xlarge has been removed - Is that intentional ?

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/6014/files#r29969797.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm - I can still choose cc1.4xlarge as an instance type in the launch wizard in us-east-1. While its not recommended there might be some users using it -- Lets keep it around for some more time.

# Last Updated: 2014-06-20
# Last Updated: 2015-05-08
# For easy maintainability, please keep this manually-inputted dictionary sorted by key.
disks_by_instance = {
"c1.medium": 1,
"c1.xlarge": 4,
"c3.large": 2,
"c3.xlarge": 2,
"c3.2xlarge": 2,
"c3.4xlarge": 2,
"c3.8xlarge": 2,
"c3.large": 2,
"c3.xlarge": 2,
"c4.large": 0,
"c4.xlarge": 0,
"c4.2xlarge": 0,
"c4.4xlarge": 0,
"c4.8xlarge": 0,
"cc1.4xlarge": 2,
"cc2.8xlarge": 4,
"cg1.4xlarge": 2,
"cr1.8xlarge": 2,
"d2.xlarge": 3,
"d2.2xlarge": 6,
"d2.4xlarge": 12,
"d2.8xlarge": 24,
"g2.2xlarge": 1,
"g2.8xlarge": 2,
"hi1.4xlarge": 2,
"hs1.8xlarge": 24,
"i2.xlarge": 1,
"i2.2xlarge": 2,
"i2.4xlarge": 4,
"i2.8xlarge": 8,
"i2.xlarge": 1,
"m1.large": 2,
"m1.medium": 1,
"m1.small": 1,
"m1.medium": 1,
"m1.large": 2,
"m1.xlarge": 4,
"m2.xlarge": 1,
"m2.2xlarge": 1,
"m2.4xlarge": 2,
"m2.xlarge": 1,
"m3.2xlarge": 2,
"m3.large": 1,
"m3.medium": 1,
"m3.large": 1,
"m3.xlarge": 2,
"m3.2xlarge": 2,
"r3.large": 1,
"r3.xlarge": 1,
"r3.2xlarge": 1,
"r3.4xlarge": 1,
"r3.8xlarge": 2,
"r3.large": 1,
"r3.xlarge": 1,
"t1.micro": 0,
"t2.micro": 0,
"t2.small": 0,
"t2.medium": 0,
}
if instance_type in disks_by_instance:
return disks_by_instance[instance_type]
Expand Down