Sum volume size of all volumes attached to a subset of instances

I needed to get the total size used in attached volumes to a subset of ec2 instances which could be filtered out by tag name.

$ for i in `aws ec2 describe-instances --filters Name=tag:Name,Values=somepltf-* --query 'Reservations[].Instances[].BlockDeviceMappings[*].Ebs.VolumeId' --output text | xargs aws ec2 describe-volumes --query 'Volumes[].Size' --output text --volume-ids `; do echo $i; done | perl -ne ' $total += $_; END{print $total}'
15400

I’m iterating over the output as the list of sizes in output text doesn’t come with line breaks by default so it wasn’t that easy to break in perl.

Any improvement over this is welcome.

Happy CLI!

Let’s Tag!

After deploying a bunch of VMs we were asked to add additional tags. If you have ever tried to add them through the console it’s just a pain in the ass, so AWS CLI comes to the rescue.

We’ll first obtain a list of instance ids that we need to tag. Obviously this list needs to be taken out AWS.

aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].InstanceId' --filters Name=tag:Name,Values=xxxx-pre* | tr '\n' ' '

Note that we filter out some instances by tag name. Also we flatten the whole list by changing every newline character by a space. Now we have a flat space separated list of instance ids to feed the create-tags command.

aws ec2 create-tags --tags 'Key=Scope,Value="Linux Server Management"' --resources i-yyyyyyyy i-xxxxxxxx

Note that tags argument is all enclosed by single quotes in order to preserve spaces within tag value. Resources argument can contain any resource id that can be tagged.

Happy CLI!

Document EC2 instances on a table

I needed to get a list with instance information on it, formatted as a table as it’s handy to just pour it into a wiki as it’s wiki formatted.


aws ec2 describe-instances --output table --query 'Reservations[].Instances[].[Tags[?Key==`Name`]|[0].Value,Tags[?Key==`Environment`]|[0].Value,InstanceId,InstanceType,PublicIpAddress,PrivateIpAddress,Placement.AvailabilityZone]' --filters Name=tag:Environment,Values=Production

We get something similar to this table:

| DescribeInstances |
+-----------------------------+----------------+-------------+-------------+-----------------+----------------+---------------+
| xxxxxxx-pre-admin02 | Preproduction | i-xxxxxxxx | c4.large | 54.xxx.xx.xx | 10.xxx.x.xx | cn-north-1b |
| xxxxxxx-pre-httpfo01 | Preproduction | i-xxxxxxxx | c4.xlarge | 54.xxx.xxx.xxx | 10.xxx.x.xx | cn-north-1a |

On the query side of the aws command we get the value for some specific tags. Tags[?Key==\`Name\`]|[0].Value This basically filters out Tags for each instance whose Key is Name and gather the value field of that Tag. We are using the pipe expression of JMESPath for this.

Another important argument is the filters. We are filtering by the value of an specific tag.

Happy CLI!

virsh assigned memory per environment

In a KVM with a naming convention for guests of vm1.environment.foobar.foo we can get a report of memory assigned by environment:

virsh list --all | perl -MData::Dumper -ne '/[-\d]+\s+(\w+.(\w+).\w+.\w+)\s+/ and $sername = $1 and $environ = $2 and virsh dominfo $sername=~/Max memory:\s+(\d+)\s+KiB/ and $total{$environ}+=$1; END{print Dumper(\%total);}'

We get this output as a result:

$VAR1 = {
          'uat' => '29360128',
          'staging' => '29360128',
          'prod' => '29360128'
        };

Happy CLI!

Describing network interfaces

This post will cover some commands that I’ve found useful when working with network interfaces and for getting information of what we currently have.

List all private ips in use (actually you can change in-use in the filter to availability to get free allocated ips):

aws ec2 describe-network-interfaces \
--filters Name=status,Values=in-use \
--query 'NetworkInterfaces[*].PrivateIpAddress' \
--output table

List private ips in use within an AZ

aws ec2 describe-network-interfaces \
--filters Name=status,Values=in-use Name=availability-zone,Values=eu-west-1a \
--query 'NetworkInterfaces[*].PrivateIpAddress' \
--output table

I needed a list of ips with the security groups they have attached to it:

aws ec2 describe-network-interfaces \
--filters Name=status,Values=available Name=availability-zone,Values=eu-west-1b \
--query 'NetworkInterfaces[].[PrivateIpAddress,NetworkInterfaceId,Groups[]|[*].GroupName]'

As you see, we are projecting the groups attribute list just to have a the group name for each ip.

List all resources on a cloudformation stack

Getting a list on a table with the relevant information for all the resources in it.


aws cloudformation describe-stack-resources --stack-name STACKNAME --output table --query 'StackResources[*].[StackName,ResourceType,LogicalResourceId,PhysicalResourceId]'

Happy CLI!

S3 Bucket Usage

I needed to get the actual “disk size” of an s3 bucket.

aws s3 ls s3://my_bucket --recursive | \
perl -ne '/\d\d:\d\d\s+(\d+)\s/; $total += $1;
END{printf "Total size %.2fGB\n", $total/1024/1024/1024;}'

With that you get: Total size 2054.13GB

You can now multiply this to the aws costs to get what you’ll be paying for that.

Happy CLI

List pending snapshots

We needed to get a list of pending snapshots for an account. Use the following one-liner to get it into a table show snapshotId, volume size, progress and start time.

$ aws ec2 describe-snapshots --owner xxxxxxxxxxx --query 'Snapshots[?State==`pending`].[SnapshotId,VolumeSize,Progress,StartTime]' --output table
------------------------------------------------------------
|                     DescribeSnapshots                    |
+----------------+------+-----+----------------------------+
|  snap-52016xxx |  100 |  47 |  2014-11-11T14:44:52.000Z  |
|  snap-ec177xxx |  60  |  91 |  2014-11-11T14:51:41.000Z  |
+----------------+------+-----+----------------------------+

Remember to add –owner to describre-snapshots otherwise you’ll get a list of all snapshots available from shared AMI in the marketplace.

Happy CLI

Get a list of instance with id, name and type

So I needed to get a list of current instances in an account with it’s human readable name and instance type. Output table came really at hand to format it and make it more viewable.

$ aws --output table  ec2 describe-instances --query 'Reservations[].Instances[].[Tags[?Key==`Name`] | [0].Value,InstanceId,InstanceType]'
-----------------------------------------------
|              DescribeInstances              |
+----------------+--------------+-------------+
|  xxxxxxxxxx    |  i-a0169xxx  |  r3.large   |
|  yyyyyyyyyy    |  i-11a46xxx  |  m3.large   |
|  zzzzzzzzzzzzzz|  i-07c4axxx  |  t2.medium  |
+----------------+--------------+-------------+

The key was getting the readable name into that table as it is stored as a Tag. With –query you filter tags whose key is ‘Name’ and use the pipe operator which pipes the former command into a new JMESPath query where we get first result’s value attribute.

Read further on pipe expresions for jmespath.

Happy CLI

Schedule autoscaling up and down

Sometimes we want to pre scale the instances for a several amount of time and then enable the scale down some hours after that.

We can use the disable action on cloudwatch to enable/disable the scale down after we set the desired instances:

Disable scale down action on our cloudwatch using cron:

55 16 20 12 * aws cloudwatch disable-alarm-actions --alarm-names Test-app01-ScaleDown

Continue reading

EC2 Instance termination protection

Termination protection basically stops some fool tinkering around the console from accidentally terminate any of the instances that have it enabled. Here you are some onliner to get job done on that matter.

Getting a list of all your instances and their instance termination protection status:

for instance in `aws --profile custrb ec2 describe-instances \
  --query Reservations[*].Instances[*].InstanceId \
  | grep "i-" | cut -d\" -f 2`;
 do aws --profile custrb ec2 describe-instance-attribute \
  --instance-id $instance --attribute disableApiTermination \
  --query [InstanceId,DisableApiTermination]; 
done

On one hand you need to get the list of instances id, pretty ugly sorry for that, and iterate through them and issue a describe-instance-attribute command so we can check disableApiTermination. Upon common sense, all attributes are displayed although without any value except for the one requested.

To enable termination protection for one instance:

aws --profile custrb ec2 modify-instance-attribute \
  --instance-id i-42179402 --disable-api-termination

To enable termination protection for all instances in account:

for instance in `aws --profile custrb ec2 describe-instances \ 
 --query Reservations[*].Instances[*].InstanceId \
 | grep "i-" | cut -d\" -f 2`;
  do aws --profile custrb ec2 modify-instance-attribute \
   --instance-id $instance --disable-api-termination; 
done

To modify the attribue we just need to issue a modify-instance.attribute with –disable-api-termination, which sets disableApiTermination to true, and thus Termination protection is effectively activated.

Happy CLI!

Cloudformation Stacks

Cloudformation, what a joyful tool to get new platform up and ready in a breeze within AWS. So documentation on cloudformation is really extensive, so I’ll digest here mainly its use through the command line.

Here’s the full documentation for cloudformation and the cli reference.

Validation of a stack template

This is only a syntax validation of the cloudformation template. Semantic validation occurs when you actually issue a create stack command.

# validation of a local template file
aws cloudformation validate-template --template-body file:///path/to/cf-template.txt

# validation of a remote template file
aws cloudformation validate-template --template-url http://yourtempaltes/cf-template.txt

Stack operations

Basic actions to operate the creation, deletion and update of a cloudformation stack.

# creation of a stack
aws cloudformation create-stack --stack-name myStack --template-body file:///path/to/cf-template.txt

# deletion of a stack
aws cloudformation delete-stack --stack-name myStack

Stack update

A stack can be updated, that means you can issue and update-stack command with a modified version of the template for an specific stack. Cloudformation then will diff the new version with the current one and applies any change on the resources in it. Take into account that this can disrupt operation on some resources.

# Update of a stack
aws cloudformation update-stack --stack-name myStack --template-body file:///path/to/cf-template_v2.txt

Happy CLI

Loop on S3 bucket

There’s an script running on some hosts that upload a tar files of some folders within a bucket, each instance having its own folder. I needed to get a list of each instance’s folder content.
One liner in bash:

for folder in `aws s3 ls backups_bucket/ | awk '{print $2}' | cut -d '/' -f 1` ;
do
  echo "$folder";
  aws s3 ls backups_bucket/$folder/2014_10_12/;
done

Happy CLI

Blog purpose. Exploring every CLI

I’ve been always in love of any CLI available. Every time I need to perform a task over a new technology or product I look for it’s CLI.

You know, when you can just type and describe what you want that software to do, going through the point-and-click option is really cumbersome.

This blog will be a great repository for any snippet I come around or that I can think of. Any quick and dirty script that will serve for a purpose will have it’s place in here. To be honest, this will be oriented towards Amazon Webservices CLI, but surely any command can get its way in it.

Enjoy. Happy CLI