BogoToBogo
  • Home
  • About
  • Big Data
  • Machine Learning
  • AngularJS
  • Python
  • C++
  • go
  • DevOps
  • Kubernetes
  • Algorithms
  • More...
    • Qt 5
    • Linux
    • FFmpeg
    • Matlab
    • Django 1.8
    • Ruby On Rails
    • HTML5 & CSS

Ansible - AWS Auto Scaling via AMI

Ansible_Logo.png




Bookmark and Share





bogotobogo.com site search:






Ansible 2.0

  • What is Ansible?
  • Quick Preview - Setting up web servers with Nginx, configure environments, and deploy an App
  • SSH connection & running commands
  • Ansible: Playbook for Tomcat 9 on Ubuntu 18.04 systemd with AWS
  • Modules
  • Playbooks
  • Handlers
  • Roles
  • Playbook for LAMP HAProxy
  • Installing Nginx on a Docker container
  • AWS : Creating an ec2 instance & adding keys to authorized_keys
  • AWS : Auto Scaling via AMI
  • AWS : creating an ELB & registers an EC2 instance from the ELB
  • Deploying Wordpress micro-services with Docker containers on Vagrant box via Ansible
  • Setting up Apache web server
  • Deploying a Go app to Minikube
  • Ansible with Terraform


  • Note

    We're going to create an instance via AMI with Auto Scaling group. We'll use Ansible Dynamic Inventory.


    ASG-Tree.png



    Requirements

    Requirements (on host that executes module):

    1. python >= 2.6
    2. boto - Ansible's EC2 module uses python-boto library to call AWS API, and boto needs AWS credentials in order to function. So we need the boto installed on our local host. Though there are many ways to set our AWS credentials, in this tutorial, we'll create a file under our user home folder (~/.boto):
      [Credentials]
      AWS_ACCESS_KEY_ID=KID...CWU
      AWS_SECRET_ACCESS_EY=3qv...DSP
      

    We need to create:

    1. An IAM group with "AdministratorAccess" and "IAMFullAccess" privileges, we named it "PowerUsers".

    2. PowerUsersIAMFullAccess.png
    3. We need to create an IAM policy that gives zero access to any of our resources. The reason for this is that we'll be provisioning EC2 instances with an IAM policy attached, and if those instances get compromised, we don't want them to have permission to make any changes to our AWS account.
      In our case, we named the role as 'noaccess' which will be specified in "group_vars/all.yml".

      iam_profile: "noaccess"
      

      The policy should look like this:

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Action": "*",
                  "Effect": "Deny",
                  "Resource": "*"
              }
          ]
      }
      

      We can check it by clicking on the "Show Policy" action for the "IAMFullAccess" policy.


    4. Create a keypair, "autoscaling":
      CreateKeyPair.png
      The private key will now be downloaded by browser. Save the key in ~/.ssh folder:
      $ mv autoscaling.pem ~/.ssh
      $ chmod 600 ~/.ssh/autoscaling.pem
      

      Add the private key to ssh-agent by executing the following commands:
      $ eval `ssh-agent`
      $ ssh-add ~/.ssh/autoscaling.pem
      




    Ansible Dynamic Inventory

    Ansible uses inventory information about hosts and groups of hosts to connect to and manage client devices. By default, inventory information is stored in /etc/ansible/hosts. This inventory information can also be expanded to include the 'group_vars' and 'host_vars' directories.

    Now managing a large flat file can be unwieldy. If we are frequently adding and removing devices, then keeping the inventory current might be problematic and time consuming. Additionally, we might be duplicating information that is stored in an alternate system.

    Ansible has an alternate way of managing inventory called dynamic inventory. Basically, what we can do is pass in an alternate inventory source using the '-i' option.

    In order for this to work properly, we need to setup the EC2 external inventory script in our playbook.

    $ mkdir -p ~/bogo/inventory 
    $ cd ~/bogo/inventory
    

    To get started with dynamic inventory management, we'll need to grab the EC2.py script and the EC2.ini config file. The EC2.py script is written using the Boto EC2 library and will query AWS for our running Amazon EC2 instances. The EC2.ini file is the config file for EC2.py, and can be used to limit the scope of Ansible's reach. We can specify the regions, instance tags, or roles that the EC2.py script will find.

    Let's download the EC2 external inventory script from Ansible:

    $ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py
    $ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini
    
    $ chmod +x ec2.py
    

    Next, create the default Ansible configuration for this playbook by editing a file in the root of the playbook directory (in our example, ~/bogo), named ansible.cfg. This file should have the following text in it:

    [defaults]
    remote_user = ubuntu
    host_key_checking = False
    inventory = ./inventory/ec2.py
    [ssh_connection]
    control_path = %(directory)s/%%C
    

    To test and ensure that this script is working properly (and that your boto credentials are setup properly), execute the following command:

    $ ~/bogo/inventory/ec2.py --list
    

    We should see the following output:

    {
      "_meta": {
        "hostvars": {}
      }
    }
    



    group_vars/all.yml

    The following group variables will apply to any tasks configured in our playbook which allow us to specify a default set that will work for our provisioning:

    ---
    # group_vars/all.yml
     
    region: us-east-1
    zone: us-east-1a # zone that the master AMI will be configured in
    keypair: immutable
    security_groups: ['default']
    instance_type: t2.nano
    # specify group_name on the command line with -e group_name=devX
    group_name: test
    instances_min: 1 # minimum number of instances in the auto scaling group
    instances_max: 1 # maximum number of instances in the auto scaling group
    iam_profile: "noaccess"
    volumes:
      - device_name: /dev/sda1
        device_type: gp2
        volume_size: 8 # size of the root disk
        delete_on_termination: true
    




    roles/launch/tasks/main.yml

    Roles are used to automatically load certain variables and tasks into an Ansible playbook, and allow us to reuse our tasks in a modular way.

    The launch role launches a new EC2 compute instance, in the region and availability zone specified in our "group_vars" file:

    ---
    # roles/launch/tasks/main.yml
     
    - name: Search for the latest Ubuntu 14.04 AMI
      ec2_ami_find:
        region: "{{ region }}"
        name: "ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"
        owner: 099720109477
        sort: name
        sort_order: descending
        sort_end: 1
        no_result_action: fail
      register: ami_result
     
    - name: Launch new instance
      ec2:
        region: "{{ region }}"
        keypair: "{{ keypair }}"
        zone: "{{ zone }}"
        group: "{{ security_groups }}"
        image: "{{ ami_result.results[0].ami_id }}"
        instance_type: "{{ instance_type }}"
        instance_tags:
          Name: "{{ name }}"
        volumes: "{{ volumes }}"
        wait: yes
      register: ec2
     
    - name: Add new instances to host group
      add_host:
        name: "{{ item.public_dns_name }}"
        groups: "{{ name }}"
        ec2_id: "{{ item.id }}"
      with_items: ec2.instances
     
    - name: Wait for instance to boot
      wait_for:
        host: "{{ item.public_dns_name }}"
        port: 22
        delay: 30
        timeout: 300
        state: started
      with_items: ec2.instances
    




    roles/workstation/tasks/main.yml

    Now that we have a role that can launch a brand new t2.nano instance, our next role will allow us to configure this instance to be used as a workstation.

    roles/workstation/tasks/main.yml:

    ---
    # roles/workstation/tasks/main.yml
      - name: add timezone configuration
        command: bash -c 'echo US/Eastern > /etc/timezone'
     
      - name: add kernel parameters to /etc/sysctl.d/60-custom.conf
        blockinfile: |
          dest=/etc/sysctl.d/60-custom.conf
          create=yes
          content="# Auto-reboot linux 10 seconds after a kernel panic
            kernel.panic = 10
            kernel.panic_on_oops = 10
            kernel.unknown_nmi_panic = 10
            kernel.panic_on_unrecovered_nmi = 10
            kernel.panic_on_io_nmi = 10
            # Controls whether core dumps will append the PID to the core filename, useful for debugging multi-threaded applications
            kernel.core_uses_pid = 1
            # Turn on address space randomization - security is super important
            kernel.randomize_va_space = 2
            vm.swappiness = 0
            vm.dirty_ratio = 80
            vm.dirty_background_ratio = 5
            vm.dirty_expire_centisecs = 12000
            vm.overcommit_memory = 1
            # ------ VM ------
            fs.file-max = 204708
            #fs.epoll.max_user_instances = 4096
            fs.suid_dumpable = 0
            # ------ NETWORK SECURITY ------
            # Turn on protection for bad icmp error messages
            net.ipv4.icmp_ignore_bogus_error_responses = 1
            # Turn on syncookies for SYN flood attack protection
            net.ipv4.tcp_syncookies = 1
            net.ipv4.tcp_max_syn_backlog = 8096
            net.ipv4.tcp_synack_retries = 2
            net.ipv4.tcp_syn_retries = 2
            # Log suspicious packets, such as spoofed, source-routed, and redirect
            net.ipv4.conf.all.log_martians = 1
            net.ipv4.conf.default.log_martians = 1
            # Disables these ipv4 features, not very legitimate uses
            net.ipv4.conf.all.accept_source_route = 0
            net.ipv4.conf.default.accept_source_route = 0
            # ------ NETWORK PERFORMANCE ------
            # Netflix 2014 recommendations
            net.core.netdev_max_backlog = 5000
            net.core.rmem_max = 16777216
            net.core.wmem_max = 16777216
            net.ipv4.tcp_wmem = 4096 12582912 16777216
            net.ipv4.tcp_rmem = 4096 12582912 16777216
            # Allow reusing sockets in TIME_WAIT state for new connections
            net.ipv4.tcp_tw_reuse = 1
            # Socket max connections waiting to get accepted; the listen() backlog.
            # Default is 128.
            net.core.somaxconn = 4096
            # Decrease fin timeout. After telling the client we are closing, how long to wait for a FIN, ACK?
            # Default is 60.
            net.ipv4.tcp_fin_timeout = 10
            # Avoid falling back to slow start after a connection goes idle
            # keeps our cwnd large with the keep alive connections
            net.ipv4.tcp_slow_start_after_idle = 0"
     
      - name: reload sysctl kernel parameter settings
        command: bash -c 'sysctl -p /etc/sysctl.d/60-custom.conf'
     
      - name: copy swapfile init script
        copy:  src=aws-swap-init dest=/etc/init.d/aws-swap-init mode=0755
     
      - name: register swapfile init script
        command: bash -c 'update-rc.d aws-swap-init defaults'
     
      - name: add entries to sudoers
        lineinfile: |
          dest=/etc/sudoers.d/90-cloud-init-users
          line="luke ALL=(ALL) NOPASSWD:ALL"
     
      - name: add luke user account
        user: name=luke uid=1001 comment="Luke Youngblood" createhome=yes shell=/bin/bash
     
      - name: accept Oracle license part 1
        command: bash -c 'echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections'
     
      - name: accept Oracle license part 2
        command: bash -c 'echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections'
     
      - name: Update apt cache
        apt: update_cache=yes
     
      - name: Add apt key for ubuntu
        apt_key: keyserver=keyserver.ubuntu.com id=E56151BF
     
      - name: Determine Linux distribution distributor
        shell: lsb_release -is | tr ''[:upper:]'' ''[:lower:]''
        register: release_distributor
     
      - name: Determine Linux distribution codename
        command: lsb_release -cs
        register: release_codename
     
      - name: Add Mesosphere repository to sources list
        lineinfile: >
          dest=/etc/apt/sources.list.d/mesosphere.list
          line="deb http://repos.mesosphere.io/{{ release_distributor.stdout }} {{ release_codename.stdout }} main"
          mode=0644
          create=yes
     
      - name: Add java apt repository
        apt_repository: repo='ppa:webupd8team/java'
     
      - name: Add Google Chrome key
        shell: wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
     
      - name: Add Google Chrome repo
        copy: >
          content="deb http://dl.google.com/linux/chrome/deb/ stable main"
          dest="/etc/apt/sources.list.d/google-chrome.list"
          owner=root
          group=root
          mode=0644
     
      - name: Update apt cache
        apt: update_cache=yes
     
      - name: Install packages
        apt: pkg={{ item }} state=installed
        with_items: 
          - nfs-common
          - oracle-java8-installer
          - liblapack-dev
          - libblas-dev
          - gfortran
          - ntp
          - vnc4server 
          - xfce4 
          - xfce4-goodies 
          - emacs 
          - firefox 
          - git 
          - maven 
          - npm 
          - python-pip 
          - clusterssh
          - graphviz
          - google-chrome-stable
          - python-numpy
          - python-scipy
          - python-dev
          - python-nose
          - g++
          - libopenblas-dev
     
      - name: Install boto and run setup
        command: bash -c 'git clone git://github.com/boto/boto.git;cd /home/ubuntu/boto;python setup.py install'
        args:
          chdir: /home/ubuntu
     
      - name: Install bower
        command: bash -c 'npm install -g bower'
     
      - name: Update all packages to the latest version
        apt: upgrade=dist
     
      - name: Create nodejs symlink
        file: src=/usr/bin/nodejs dest=/usr/bin/node state=link
     
      - name: execute dpkg-reconfigure
        command: "dpkg-reconfigure -f noninteractive tzdata"
    

    Note that it takes a while doing all tasks. So, we may want to comment out majority of the task if we just want to know how we do Auto Scaling!


    roles/workstation/files/aws-swap-init:

    #!/bin/sh
    #
    # aws-swap-init
    #
    
    DEFAULTSIZE=4194304
    
    start () {
        # Check to see if there is swap mounted right now
        # If there is, we're done here
        ISSWAP=`/bin/cat /proc/meminfo | /bin/grep SwapTotal | /usr/bin/awk '{print $2}'`
        if [ $ISSWAP -ne 0 ]
        then
            exit 0
        fi
    
        # What OS are we running?
        if [ -f /etc/system-release -o -f /etc/redhat-release ]
        then
            OSTYPE="amzn"
        elif [ -f /etc/lsb-release ]
        then
            OSTYPE="ubuntu"
        fi
    
        # Set the target directory. If unsupported platform, use root
        case "$OSTYPE" in
            amzn)
                TARGET="/media/ephemeral0"
                ;;
            ubuntu)
                TARGET="/mnt"
                ;;
            *)
                TARGET="/"
                ;;
        esac
    
        # Does a swapfile already exist? If so, activate and be done
        if [ -f $TARGET/swapfile00 ]
        then
            /sbin/swapon $TARGET/swapfile00
            exit 0
        fi
     
        # OK, so there's no existing swapfile. Let's make one and activate it.
    
        # If we're on an unsupported OS, or ephemeral disk isn't mounted, use a safe
        # default size. Otherwise, use RAM size
        if [ $TARGET = "/" ]
        then
            SIZE=$DEFAULTSIZE
        else
            /bin/mount | grep -q " on $TARGET type "
            if [ $? -eq 0 ]
            then
                SIZE=`/bin/cat /proc/meminfo | /bin/grep "^MemTotal" | /usr/bin/awk '{print $2}'`
            else
                SIZE=$DEFAULTSIZE
                TARGET="/"
            fi
        fi
            
        # OK, time to get down to business.
        /bin/dd if=/dev/zero of=$TARGET/swapfile00 bs=1024 count=$SIZE
        /sbin/mkswap $TARGET/swapfile00
        /sbin/swapon $TARGET/swapfile00
    }
    
    stop () {
        exit 0
    }
    
    case "$1" in
        start)
            $1
            ;;
        stop)
            $1
            ;;
        *)
            echo $"Usage: $0 {start|stop}"
            exit 2
    esac
    exit $?
    




    Testing the playbook, site.yml

    Here is our playbook to deploy the workstation:

    site.yml:

    ---
    # deployworkstation.yml
      - hosts: localhost
        connection: local
        gather_facts: no
        roles:
          - role: launch
            name: "{{ group_name }}amibuild"
     
      - hosts: "{{ group_name }}amibuild"
        become: True
        become_user: root
        become_method: sudo
        roles:
          - workstation
    

    To test:

    $ ansible-playbook -vv -e group_name=test site.yml --private_key=/home/k/.ssh/einsteinish.pem
    

    We should see the following output at the end of the playbook run:

    PLAY RECAP *********************************************************************
    ec2-34-206-52-93.compute-1.amazonaws.com : ok=25   changed=22   unreachable=0    failed=0   
    localhost                  : ok=4    changed=2    unreachable=0    failed=0   
    

    To make it sure, let's ssh to the instance:

    $ ssh ubuntu@34.206.52.93 ... ubuntu@ip-172-31-70-155:~$



    roles/create-ami/tasks/main.yml

    Now, we're going to create an AMI (Amazon Machine Image) out of our workstation after it's configured, then create a launch configuration pointing at that AMI. Then, we'll create an autoscaling group that points at the launch configuration.

    After our workstation role has fully configured the instance, we are going to add a role to create an AMI by snapshotting our running instance.

    roles/create-ami/tasks/main.yml:

    ---
    # roles/create-ami/tasks/main.yml
    - name: Create AMI
      ec2_ami:
        region: "{{ region }}"
        instance_id: "{{ ec2.instance_ids[0] }}"
        name: "{{ asg_name }}-{{ ansible_date_time.iso8601 | regex_replace('[^a-zA-Z0-9]', '-') }}"
        wait: yes
        wait_timeout: 1200
        state: present
      register: ami
    




    roles/create-launch-config/tasks/main.yml

    The following create-launch-config role allow us to create a launch configuration that points at the AMI we just created. This is a necessary before we can create the autoscaling group.

    roles/create-launch-config/tasks/main.yml:

    ---
    # roles/create-launch-config/tasks/main.yml
     
    - name: Create Launch Configuration
      ec2_lc:
        region: "{{ region }}"
        name: "{{ asg_name }}-{{ ansible_date_time.iso8601 | regex_replace('[^a-zA-Z0-9]', '-') }}"
        image_id: "{{ ami.image_id }}"
        key_name: "{{ keypair }}"
        instance_type: "{{ instance_type }}"
        instance_profile_name: "{{ iam_profile }}"
        security_groups: "{{ security_groups }}"
        volumes: "{{ volumes }}"
        instance_monitoring: yes
      register: lc_result
     
    - debug: var=lc_result
    




    Autoscaling role : roles/auto-scaling/tasks/main.yml

    First, we'll create a launch configuration that points at the AMI we just created. This is a necessary before we can create the autoscaling group.

    Then, we're going to create is the role that actually creates the autoscaling group.

    This role first checks to see if an Auto Scaling group with the same name already exists, and if so, it just updates it.

    By updating the Auto Scaling group to point at the new launch configuration, with a new AMI, the Auto Scaling group will automatically do a rolling upgrade, where it starts a new instance, waits until the OS is loaded and healthy, then terminates an old instance. It repeats this process until all instances in the Auto Scaling group are running the new AMI.

    We can configure this by changing replace_batch_size, however, we've set a sensible default based on the size of the group divided by 4. For example, if you had an Auto Scaling group with 8 running instances, Auto Scaling would deploy 2 new instances at once, to speed up the rolling upgrade process.

    roles/auto-scaling/tasks/main.yml:

    ---
    # roles/auto-scaling/tasks/main.yml
    
    - name: Create Launch Configuration
      ec2_lc:
        region: "{{ region }}"
        name: "{{ asg_name }}-{{ ansible_date_time.iso8601 | regex_replace('[^a-zA-Z0-9]', '-') }}"
        image_id: "{{ ami.image_id }}"
        key_name: "{{ keypair }}"
        instance_type: "{{ instance_type }}"
        instance_profile_name: "{{ iam_profile }}"
        security_groups: "{{ security_groups }}"
        volumes: "{{ volumes }}"
        instance_monitoring: yes
      register: lc_result
    
    - name: Retrieve current Auto Scaling Group properties
      command: "aws --region {{ region }} autoscaling describe-auto-scaling-groups --auto-scaling-group-names {{ asg_name }}"
      register: asg_properties_result
     
    - name: Set asg_properties variable from JSON output if the Auto Scaling Group already exists
      set_fact:
        asg_properties: "{{ (asg_properties_result.stdout | from_json).AutoScalingGroups[0] }}"
      when: (asg_properties_result.stdout | from_json).AutoScalingGroups | count
     
    - name: Configure Auto Scaling Group and perform rolling deploy
      ec2_asg:
        region: "{{ region }}"
        name: "{{ asg_name }}"
        launch_config_name: "{{ lc_result.name }}"
        availability_zones: "{{ zone }}"
        health_check_type: EC2
        health_check_period: 300
    #    desired_capacity: "{{ asg_properties_result.DesiredCapacity | default(2) }}"
        desired_capacity: "{{ instances_min }}"
        replace_all_instances: yes
        replace_batch_size: "{{ (asg_properties_result.DesiredCapacity | default(2) / 4) | round(0, 'ceil') | int }}"
        min_size: "{{ instances_min }}"
        max_size: "{{ instances_max }}"
        tags:
         - Name: "{{ asg_name }}"
    #    load_balancers:
    #      - master
        state: present
      register: asg_result
     
    - debug: var=asg_result
     
    - name: Configure Scaling Policies
      ec2_scaling_policy:
        region: "{{ region }}"
        name: "{{ item.name }}"
        asg_name: "{{ asg_name }}"
        state: present
        adjustment_type: "{{ item.adjustment_type }}"
        min_adjustment_step: "{{ item.min_adjustment_step }}"
        scaling_adjustment: "{{ item.scaling_adjustment }}"
        cooldown: "{{ item.cooldown }}"
      with_items:
        - name: "Increase Group Size"
          adjustment_type: "ChangeInCapacity"
          scaling_adjustment: +1
          min_adjustment_step: 1
          cooldown: 180
        - name: "Decrease Group Size"
          adjustment_type: "ChangeInCapacity"
          scaling_adjustment: -1
          min_adjustment_step: 1
          cooldown: 300
      register: sp_result
     
    - name: Determine Metric Alarm configuration
      set_fact:
        metric_alarms:
          - name: "{{ asg_result.name }}-ScaleUp"
            comparison: ">="
            threshold: 50.0
            alarm_actions:
              - "{{ sp_result.results[0].arn }}"
          - name: "{{ asg_result.name }}-ScaleDown"
            comparison: "<="
            threshold: 20.0
            alarm_actions:
              - "{{ sp_result.results[1].arn }}"
     
    - name: Configure Metric Alarms and link to Scaling Policies
      ec2_metric_alarm:
        region: "{{ region }}"
        name: "{{ item.name }}"
        state: present
        metric: "CPUUtilization"
        namespace: "AWS/EC2"
        statistic: "Average"
        comparison: "{{ item.comparison }}"
        threshold: "{{ item.threshold }}"
        period: 60
        evaluation_periods: 5
        unit: "Percent"
        dimensions:
          AutoScalingGroupName: "{{ asg_result.name }}"
        alarm_actions: "{{ item.alarm_actions }}"
      with_items: metric_alarms
      when: "{{ asg_result.max_size }} > 1"
      register: ma_result
    

    If it's creating a new Auto Scaling group, it also sets some CloudWatch metric alarms based on CPU utilization, and links the metric alarms to the scaling policies. The way we set these alarms, if average CPU utilization is greater than 50% for 5 minutes, the group will scale up by adding another instance. If average CPU utilization is less than 20% for 5 minutes, the group will scale down by terminating an instance.

    There are also some cooldown times set so that this doesn't happen too often; scaling up can only happen every 3 minutes, and scaling down can only happen every 5 minutes.





    site-final.yml

    Here is our updated deploy play located in the root of our playbook folder. This new site_final.yml play should look like this:

    ---
    # site-final.yml
      - hosts: localhost
        connection: local
        gather_facts: no
        roles:
          - role: launch
            name: "{{ group_name }}amibuild"
    
      - hosts: "{{ group_name }}amibuild"
        become: True
        become_user: root
        become_method: sudo
        roles:
          - workstation
    
      - hosts: localhost
        connection: local
        gather_facts: no
        roles:
          - { role: create-ami, asg_name: "{{ group_name }}" }
          - { role: create-launch-config, asg_name: "{{ group_name }}" }
          - { role: auto-scaling, asg_name: "{{ group_name }}", instances_max: 1 }
    
      #- hosts: localhost
      #  connection: local
      #  gather_facts: no
      #  roles:
      #    - { role: delete-old-launch-configurations, asg_name: "{{ group_name }}" }
      #    - { role: delete-old-amis, asg_name: "{{ group_name }}" }
    
      #- hosts: "{{ group_name }}amibuild"
      #  connection: local
      #  gather_facts: no
      #  roles:
      #    - terminate
    




    Deleting old launch configurations role

    Here is the file (roles/delete-old-launch-configurations/tasks/main.yml):

    ---
    # roles/delete-old-launch-configurations/tasks/main.yml
     
    - lc_find:
        region: "{{ region }}"
        name_regex: "{{ asg_name }}-.*"
        sort: yes
        sort_end: -5
      register: old_lc_result
     
    - ec2_lc:
        region: "{{ region }}"
        name: "{{ item.name }}"
        state: absent
      with_items: old_lc_result.results
      ignore_errors: yes
    

    This role also requires a python script (roles/delete-old-launch-configurations/library/lc_find.py):

    #!/usr/bin/python
    
    # roles/delete-old-launch-configurations/library/lc_find.py
    
    import json
    import subprocess
    
    def main():
        argument_spec = ec2_argument_spec()
        argument_spec.update(dict(
                region = dict(required=True,
                    aliases = ['aws_region', 'ec2_region']),
                name_regex = dict(required=False),
                sort = dict(required=False, default=None, type='bool'),
                sort_order = dict(required=False, default='ascending',
                    choices=['ascending', 'descending']),
                sort_start = dict(required=False),
                sort_end = dict(required=False),
            )
        )
        module = AnsibleModule(
            argument_spec=argument_spec,
        )
        name_regex = module.params.get('name_regex')
        sort = module.params.get('sort')
        sort_order = module.params.get('sort_order')
        sort_start = module.params.get('sort_start')
        sort_end = module.params.get('sort_end')
        lc_cmd_result = subprocess.check_output(["aws", "autoscaling", "describe-launch-configurations", "--region",  module.params.get('region')])
        lc_result = json.loads(lc_cmd_result)
        results = []
        for lc in lc_result['LaunchConfigurations']:
            data = {
                'arn': lc["LaunchConfigurationARN"],
                'name': lc["LaunchConfigurationName"],
            }
            results.append(data)
        if name_regex:
            regex = re.compile(name_regex)
            results = [result for result in results if regex.match(result['name'])]
        if sort:
            results.sort(key=lambda e: e['name'], reverse=(sort_order=='descending'))
        try:
            if sort and sort_start and sort_end:
                results = results[int(sort_start):int(sort_end)]
            elif sort and sort_start:
                results = results[int(sort_start):]
            elif sort and sort_end:
                results = results[:int(sort_end)]
        except TypeError:
            module.fail_json(msg="Please supply numeric values for sort_start and/or sort_end")
        module.exit_json(results=results)
    
    from ansible.module_utils.basic import *
    from ansible.module_utils.ec2 import *
    
    if __name__ == '__main__':
        main()
    




    Deleting old AMIs role

    This role simply deletes any AMIs other than the 5 most recently created ones for the particular autoscaling group we are deploying.

    :

    ---
    # roles/delete-old-amis/tasks/main.yml
     
    - ec2_ami_find:
        region: "{{ region }}"
        owner: self
        name: "{{ asg_name }}-*"
        sort: name
        sort_end: -5
      register: old_ami_result
     
    - ec2_ami:
        region: "{{ region }}"
        image_id: "{{ item.ami_id }}"
        delete_snapshot: yes
        state: absent
      with_items: old_ami_result.results
      ignore_errors: yes
    




    Terminate role

    Now that we've captured the AMI snapshot of our fully configured system, created a launch config, and created an Auto Scaling group based on it, we no longer need our temporary "amibuild" system. This role will terminate it.

    roles/terminate/tasks/main.yml:

    ---
    # roles/terminate/tasks/main.yml
    - name: Terminate old instance(s)
      ec2:
        instance_ids: "{{ ec2_id }}"
        region: "{{ region }}"
        state: absent
        wait: yes
    




    Sample output

    Here is the output when we run the playbook (skipped majority of package installations in workstation role):

    k@laptop:~/bogo$ eval `ssh-agent`
    
    k@laptop:~/bogo$ ssh-add ~/.ssh/autoscaling.pem
    Identity added: /home/k/.ssh/autoscaling.pem (/home/k/.ssh/autoscaling.pem)
    
    k@laptop:~/bogo$ ansible-playbook -vv -e group_name=test site-final.yml
    Using /home/k/bogo/ansible.cfg as config file
     [WARNING]: provided hosts list is empty, only localhost is available
    
    5 plays in site-final.yml
    
    PLAY ***************************************************************************
    
    TASK [launch : Search for the latest Ubuntu 14.04 AMI] *************************
    ok: [localhost] => {"changed": false, "results": [{"ami_id": "ami-e5bf05f3", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "encrypted": false, "size": 8, "snapshot_id": "snap-052d62986f51e13b9", "volume_type": "gp2"}, "/dev/sdb": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}, "/dev/sdc": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}}, "creationDate": "2017-03-30T23:48:12.000Z", "description": null, "hypervisor": "xen", "is_public": true, "location": "099720109477/ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20170330", "name": "ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20170330", "owner_id": "099720109477", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": {}, "virtualization_type": "hvm"}]}
    
    TASK [launch : Launch new instance] ********************************************
    changed: [localhost] => {"changed": true, "instance_ids": ["i-0f55d61570fb6af45"], "instances": [{"ami_launch_index": "0", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "status": "attached", "volume_id": "vol-0ec625fad763c77d9"}}, "dns_name": "ec2-34-205-9-162.compute-1.amazonaws.com", "ebs_optimized": false, "groups": {"sg-22897246": "default"}, "hypervisor": "xen", "id": "i-0f55d61570fb6af45", "image_id": "ami-e5bf05f3", "instance_type": "t2.nano", "kernel": null, "key_name": "autoscaling", "launch_time": "2017-04-04T22:16:16.000Z", "placement": "us-east-1a", "private_dns_name": "ip-172-31-65-27.ec2.internal", "private_ip": "172.31.65.27", "public_dns_name": "ec2-34-205-9-162.compute-1.amazonaws.com", "public_ip": "34.205.9.162", "ramdisk": null, "region": "us-east-1", "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "running", "state_code": 16, "tags": {"Name": "testamibuild"}, "tenancy": "default", "virtualization_type": "hvm"}], "tagged_instances": []}
    
    TASK [launch : Add new instances to host group] ********************************
    creating host via 'add_host': hostname=ec2-34-205-9-162.compute-1.amazonaws.com
    changed: [localhost] => (item={u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-65-27.ec2.internal', u'public_ip': u'34.205.9.162', u'private_ip': u'172.31.65.27', u'id': u'i-0f55d61570fb6af45', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/sda1': {u'status': u'attached', u'delete_on_termination': True, u'volume_id': u'vol-0ec625fad763c77d9'}}, u'key_name': u'autoscaling', u'image_id': u'ami-e5bf05f3', u'tenancy': u'default', u'groups': {u'sg-22897246': u'default'}, u'public_dns_name': u'ec2-34-205-9-162.compute-1.amazonaws.com', u'state_code': 16, u'tags': {u'Name': u'testamibuild'}, u'placement': u'us-east-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-34-205-9-162.compute-1.amazonaws.com', u'region': u'us-east-1', u'launch_time': u'2017-04-04T22:16:16.000Z', u'instance_type': u't2.nano', u'root_device_name': u'/dev/sda1', u'hypervisor': u'xen'}) => {"add_host": {"groups": ["testamibuild"], "host_name": "ec2-34-205-9-162.compute-1.amazonaws.com", "host_vars": {"ec2_id": "i-0f55d61570fb6af45"}}, "changed": true, "item": {"ami_launch_index": "0", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "status": "attached", "volume_id": "vol-0ec625fad763c77d9"}}, "dns_name": "ec2-34-205-9-162.compute-1.amazonaws.com", "ebs_optimized": false, "groups": {"sg-22897246": "default"}, "hypervisor": "xen", "id": "i-0f55d61570fb6af45", "image_id": "ami-e5bf05f3", "instance_type": "t2.nano", "kernel": null, "key_name": "autoscaling", "launch_time": "2017-04-04T22:16:16.000Z", "placement": "us-east-1a", "private_dns_name": "ip-172-31-65-27.ec2.internal", "private_ip": "172.31.65.27", "public_dns_name": "ec2-34-205-9-162.compute-1.amazonaws.com", "public_ip": "34.205.9.162", "ramdisk": null, "region": "us-east-1", "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "running", "state_code": 16, "tags": {"Name": "testamibuild"}, "tenancy": "default", "virtualization_type": "hvm"}}
    
    TASK [launch : Wait for instance to boot] **************************************
    ok: [localhost] => (item={u'kernel': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-65-27.ec2.internal', u'public_ip': u'34.205.9.162', u'private_ip': u'172.31.65.27', u'id': u'i-0f55d61570fb6af45', u'ebs_optimized': False, u'state': u'running', u'virtualization_type': u'hvm', u'architecture': u'x86_64', u'ramdisk': None, u'block_device_mapping': {u'/dev/sda1': {u'status': u'attached', u'delete_on_termination': True, u'volume_id': u'vol-0ec625fad763c77d9'}}, u'key_name': u'autoscaling', u'image_id': u'ami-e5bf05f3', u'tenancy': u'default', u'groups': {u'sg-22897246': u'default'}, u'public_dns_name': u'ec2-34-205-9-162.compute-1.amazonaws.com', u'state_code': 16, u'tags': {u'Name': u'testamibuild'}, u'placement': u'us-east-1a', u'ami_launch_index': u'0', u'dns_name': u'ec2-34-205-9-162.compute-1.amazonaws.com', u'region': u'us-east-1', u'launch_time': u'2017-04-04T22:16:16.000Z', u'instance_type': u't2.nano', u'root_device_name': u'/dev/sda1', u'hypervisor': u'xen'}) => {"changed": false, "elapsed": 31, "item": {"ami_launch_index": "0", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "status": "attached", "volume_id": "vol-0ec625fad763c77d9"}}, "dns_name": "ec2-34-205-9-162.compute-1.amazonaws.com", "ebs_optimized": false, "groups": {"sg-22897246": "default"}, "hypervisor": "xen", "id": "i-0f55d61570fb6af45", "image_id": "ami-e5bf05f3", "instance_type": "t2.nano", "kernel": null, "key_name": "autoscaling", "launch_time": "2017-04-04T22:16:16.000Z", "placement": "us-east-1a", "private_dns_name": "ip-172-31-65-27.ec2.internal", "private_ip": "172.31.65.27", "public_dns_name": "ec2-34-205-9-162.compute-1.amazonaws.com", "public_ip": "34.205.9.162", "ramdisk": null, "region": "us-east-1", "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "running", "state_code": 16, "tags": {"Name": "testamibuild"}, "tenancy": "default", "virtualization_type": "hvm"}, "path": null, "port": 22, "search_regex": null, "state": "started"}
    
    PLAY ***************************************************************************
    
    TASK [setup] *******************************************************************
    ok: [ec2-34-205-9-162.compute-1.amazonaws.com]
    
    TASK [workstation : add timezone configuration] ********************************
    changed: [ec2-34-205-9-162.compute-1.amazonaws.com] => {"changed": true, "cmd": ["bash", "-c", "echo US/Eastern > /etc/timezone"], "delta": "0:00:00.001982", "end": "2017-04-04 22:17:25.651955", "rc": 0, "start": "2017-04-04 22:17:25.649973", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
    
    PLAY ***************************************************************************
    
    TASK [setup] *******************************************************************
    ok: [localhost]
    
    TASK [create-ami : Create AMI] *************************************************
    changed: [localhost] => {"changed": true, "image_id": "ami-8bdf5f9d", "msg": "AMI creation operation complete", "state": "available"}
    
    TASK [create-launch-config : Create Launch Configuration] **********************
    changed: [localhost] => {"arn": "arn:aws:autoscaling:us-east-1:526262051452:launchConfiguration:fa0ba6bc-fed8-4664-8374-d0ccdb14c561:launchConfigurationName/test-2017-04-04T22-17-28Z", "changed": true, "created_time": "2017-04-04 22:18:57.768000", "image_id": "ami-8bdf5f9d", "instance_type": "t2.nano", "name": "test-2017-04-04T22-17-28Z", "result": {"associate_public_ip_address": null, "block_device_mappings": [{"device_name": "/dev/sda1", "ebs": {"snapshot_id": null, "volume_size": "8"}, "virtual_name": null}], "classic_link_vpc_id": null, "classic_link_vpc_security_groups": [], "created_time": "2017-04-04 22:18:57.768000", "delete_on_termination": true, "ebs_optimized": false, "image_id": "ami-8bdf5f9d", "instance_monitoring": true, "instance_profile_name": "noaccess", "instance_type": "t2.nano", "iops": null, "kernel_id": "", "key_name": "autoscaling", "launch_configuration_arn": "arn:aws:autoscaling:us-east-1:526262051452:launchConfiguration:fa0ba6bc-fed8-4664-8374-d0ccdb14c561:launchConfigurationName/test-2017-04-04T22-17-28Z", "member": "\n      ", "name": "test-2017-04-04T22-17-28Z", "ramdisk_id": "", "security_groups": ["default"], "spot_price": null, "use_block_device_types": false, "user_data": "", "volume_type": null}, "security_groups": ["default"]}
    
    TASK [create-launch-config : debug] ********************************************
    ok: [localhost] => {
        "lc_result": {
            "arn": "arn:aws:autoscaling:us-east-1:526262051452:launchConfiguration:fa0ba6bc-fed8-4664-8374-d0ccdb14c561:launchConfigurationName/test-2017-04-04T22-17-28Z", 
            "changed": true, 
            "created_time": "2017-04-04 22:18:57.768000", 
            "image_id": "ami-8bdf5f9d", 
            "instance_type": "t2.nano", 
            "name": "test-2017-04-04T22-17-28Z", 
            "result": {
                "associate_public_ip_address": null, 
                "block_device_mappings": [
                    {
                        "device_name": "/dev/sda1", 
                        "ebs": {
                            "snapshot_id": null, 
                            "volume_size": "8"
                        }, 
                        "virtual_name": null
                    }
                ], 
                "classic_link_vpc_id": null, 
                "classic_link_vpc_security_groups": [], 
                "created_time": "2017-04-04 22:18:57.768000", 
                "delete_on_termination": true, 
                "ebs_optimized": false, 
                "image_id": "ami-8bdf5f9d", 
                "instance_monitoring": true, 
                "instance_profile_name": "noaccess", 
                "instance_type": "t2.nano", 
                "iops": null, 
                "kernel_id": "", 
                "key_name": "autoscaling", 
                "launch_configuration_arn": "arn:aws:autoscaling:us-east-1:526262051452:launchConfiguration:fa0ba6bc-fed8-4664-8374-d0ccdb14c561:launchConfigurationName/test-2017-04-04T22-17-28Z", 
                "member": "\n      ", 
                "name": "test-2017-04-04T22-17-28Z", 
                "ramdisk_id": "", 
                "security_groups": [
                    "default"
                ], 
                "spot_price": null, 
                "use_block_device_types": false, 
                "user_data": "", 
                "volume_type": null
            }, 
            "security_groups": [
                "default"
            ]
        }
    }
    
    TASK [auto-scaling : Retrieve current Auto Scaling Group properties] ***********
    changed: [localhost] => {"changed": true, "cmd": ["aws", "--region", "us-east-1", "autoscaling", "describe-auto-scaling-groups", "--auto-scaling-group-names", "test"], "delta": "0:00:04.689517", "end": "2017-04-04 15:19:03.266688", "rc": 0, "start": "2017-04-04 15:18:58.577171", "stderr": "", "stdout": "{\n    \"AutoScalingGroups\": []\n}", "stdout_lines": ["{", "    \"AutoScalingGroups\": []", "}"], "warnings": []}
    
    TASK [auto-scaling : Set asg_properties variable from JSON output if the Auto Scaling Group already exists] ***
    skipping: [localhost] => {"changed": false, "skip_reason": "Conditional check failed", "skipped": true}
    
    TASK [auto-scaling : Configure Auto Scaling Group and perform rolling deploy] **
    changed: [localhost] => {"availability_zones": ["us-east-1a"], "changed": true, "default_cooldown": 300, "desired_capacity": 1, "health_check_period": 300, "health_check_type": "EC2", "healthy_instances": 1, "in_service_instances": 1, "instance_facts": {"i-0cc64e3642c26f75d": {"health_status": "Healthy", "launch_config_name": "test-2017-04-04T22-17-28Z", "lifecycle_state": "InService"}}, "instances": ["i-0cc64e3642c26f75d"], "launch_config_name": "test-2017-04-04T22-17-28Z", "load_balancers": [], "max_size": 1, "min_size": 1, "name": "test", "pending_instances": 0, "placement_group": null, "tags": {"Name": "test"}, "terminating_instances": 0, "termination_policies": ["Default"], "unhealthy_instances": 0, "viable_instances": 1, "vpc_zone_identifier": ""}
    
    TASK [auto-scaling : debug] ****************************************************
    ok: [localhost] => {
        "asg_result": {
            "availability_zones": [
                "us-east-1a"
            ], 
            "changed": true, 
            "default_cooldown": 300, 
            "desired_capacity": 1, 
            "health_check_period": 300, 
            "health_check_type": "EC2", 
            "healthy_instances": 1, 
            "in_service_instances": 1, 
            "instance_facts": {
                "i-0cc64e3642c26f75d": {
                    "health_status": "Healthy", 
                    "launch_config_name": "test-2017-04-04T22-17-28Z", 
                    "lifecycle_state": "InService"
                }
            }, 
            "instances": [
                "i-0cc64e3642c26f75d"
            ], 
            "launch_config_name": "test-2017-04-04T22-17-28Z", 
            "load_balancers": [], 
            "max_size": 1, 
            "min_size": 1, 
            "name": "test", 
            "pending_instances": 0, 
            "placement_group": null, 
            "tags": {
                "Name": "test"
            }, 
            "terminating_instances": 0, 
            "termination_policies": [
                "Default"
            ], 
            "unhealthy_instances": 0, 
            "viable_instances": 1, 
            "vpc_zone_identifier": ""
        }
    }
    
    TASK [auto-scaling : Configure Scaling Policies] *******************************
    changed: [localhost] => (item={u'cooldown': 180, u'min_adjustment_step': 1, u'name': u'Increase Group Size', u'adjustment_type': u'ChangeInCapacity', u'scaling_adjustment': 1}) => {"adjustment_type": "ChangeInCapacity", "arn": "arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:f79610a9-f6f0-458f-8135-40c8f059baf1:autoScalingGroupName/test:policyName/Increase Group Size", "as_name": "test", "changed": true, "cooldown": 180, "item": {"adjustment_type": "ChangeInCapacity", "cooldown": 180, "min_adjustment_step": 1, "name": "Increase Group Size", "scaling_adjustment": 1}, "min_adjustment_step": null, "name": "Increase Group Size", "scaling_adjustment": 1}
    changed: [localhost] => (item={u'cooldown': 300, u'min_adjustment_step': 1, u'name': u'Decrease Group Size', u'adjustment_type': u'ChangeInCapacity', u'scaling_adjustment': -1}) => {"adjustment_type": "ChangeInCapacity", "arn": "arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:be2141aa-dc6e-4d84-8a07-61942d0c0d26:autoScalingGroupName/test:policyName/Decrease Group Size", "as_name": "test", "changed": true, "cooldown": 300, "item": {"adjustment_type": "ChangeInCapacity", "cooldown": 300, "min_adjustment_step": 1, "name": "Decrease Group Size", "scaling_adjustment": -1}, "min_adjustment_step": null, "name": "Decrease Group Size", "scaling_adjustment": -1}
    
    TASK [auto-scaling : Determine Metric Alarm configuration] *********************
    ok: [localhost] => {"ansible_facts": {"metric_alarms": [{"alarm_actions": ["arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:f79610a9-f6f0-458f-8135-40c8f059baf1:autoScalingGroupName/test:policyName/Increase Group Size"], "comparison": ">=", "name": "test-ScaleUp", "threshold": 50.0}, {"alarm_actions": ["arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:be2141aa-dc6e-4d84-8a07-61942d0c0d26:autoScalingGroupName/test:policyName/Decrease Group Size"], "comparison": "<=", "name": "test-ScaleDown", "threshold": 20.0}]}, "changed": false}
    
    TASK [auto-scaling : Configure Metric Alarms and link to Scaling Policies] *****
    skipping: [localhost] => (item={u'threshold': 50.0, u'comparison': u'>=', u'alarm_actions': [u'arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:f79610a9-f6f0-458f-8135-40c8f059baf1:autoScalingGroupName/test:policyName/Increase Group Size'], u'name': u'test-ScaleUp'})  => {"changed": false, "item": {"alarm_actions": ["arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:f79610a9-f6f0-458f-8135-40c8f059baf1:autoScalingGroupName/test:policyName/Increase Group Size"], "comparison": ">=", "name": "test-ScaleUp", "threshold": 50.0}, "skip_reason": "Conditional check failed", "skipped": true}
    skipping: [localhost] => (item={u'threshold': 20.0, u'comparison': u'<=', u'alarm_actions': [u'arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:be2141aa-dc6e-4d84-8a07-61942d0c0d26:autoScalingGroupName/test:policyName/Decrease Group Size'], u'name': u'test-ScaleDown'})  => {"changed": false, "item": {"alarm_actions": ["arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:be2141aa-dc6e-4d84-8a07-61942d0c0d26:autoScalingGroupName/test:policyName/Decrease Group Size"], "comparison": "<=", "name": "test-ScaleDown", "threshold": 20.0}, "skip_reason": "Conditional check failed", "skipped": true}
    
    PLAY ***************************************************************************
    
    TASK [delete-old-launch-configurations : lc_find] ******************************
    ok: [localhost] => {"changed": false, "results": []}
    
    TASK [delete-old-launch-configurations : ec2_lc] *******************************
    
    TASK [delete-old-amis : ec2_ami_find] ******************************************
    ok: [localhost] => {"changed": false, "results": [{"ami_id": "ami-ae33b2b8", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "encrypted": false, "size": 8, "snapshot_id": "snap-6d9288d9", "volume_type": "gp2"}, "/dev/sdb": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}, "/dev/sdc": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}}, "creationDate": "2017-04-04T18:08:36.000Z", "description": null, "hypervisor": "xen", "is_public": false, "location": "526262051452/test-2017-04-04T18-08-34Z", "name": "test-2017-04-04T18-08-34Z", "owner_id": "526262051452", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": {}, "virtualization_type": "hvm"}, {"ami_id": "ami-ac37b6ba", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "encrypted": false, "size": 8, "snapshot_id": "snap-62d950ea", "volume_type": "gp2"}, "/dev/sdb": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}, "/dev/sdc": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}}, "creationDate": "2017-04-04T18:41:05.000Z", "description": null, "hypervisor": "xen", "is_public": false, "location": "526262051452/test-2017-04-04T18-41-04Z", "name": "test-2017-04-04T18-41-04Z", "owner_id": "526262051452", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": {}, "virtualization_type": "hvm"}, {"ami_id": "ami-f5c646e3", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "encrypted": false, "size": 8, "snapshot_id": "snap-64c81cd7", "volume_type": "gp2"}, "/dev/sdb": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}, "/dev/sdc": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}}, "creationDate": "2017-04-04T19:51:07.000Z", "description": null, "hypervisor": "xen", "is_public": false, "location": "526262051452/test-2017-04-04T19-51-06Z", "name": "test-2017-04-04T19-51-06Z", "owner_id": "526262051452", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": {}, "virtualization_type": "hvm"}]}
    
    TASK [delete-old-amis : ec2_ami] ***********************************************
    changed: [localhost] => (item={u'ami_id': u'ami-ae33b2b8', u'root_device_type': u'ebs', u'description': None, u'tags': {}, u'hypervisor': u'xen', u'block_device_mapping': {u'/dev/sdb': {u'encrypted': None, u'snapshot_id': None, u'delete_on_termination': False, u'volume_type': None, u'size': None}, u'/dev/sda1': {u'encrypted': False, u'snapshot_id': u'snap-6d9288d9', u'delete_on_termination': True, u'volume_type': u'gp2', u'size': 8}, u'/dev/sdc': {u'encrypted': None, u'snapshot_id': None, u'delete_on_termination': False, u'volume_type': None, u'size': None}}, u'architecture': u'x86_64', u'owner_id': u'526262051452', u'platform': None, u'state': u'available', u'location': u'526262051452/test-2017-04-04T18-08-34Z', u'is_public': False, u'creationDate': u'2017-04-04T18:08:36.000Z', u'root_device_name': u'/dev/sda1', u'virtualization_type': u'hvm', u'name': u'test-2017-04-04T18-08-34Z'}) => {"changed": true, "item": {"ami_id": "ami-ae33b2b8", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "encrypted": false, "size": 8, "snapshot_id": "snap-6d9288d9", "volume_type": "gp2"}, "/dev/sdb": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}, "/dev/sdc": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}}, "creationDate": "2017-04-04T18:08:36.000Z", "description": null, "hypervisor": "xen", "is_public": false, "location": "526262051452/test-2017-04-04T18-08-34Z", "name": "test-2017-04-04T18-08-34Z", "owner_id": "526262051452", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": {}, "virtualization_type": "hvm"}, "msg": "AMI deregister/delete operation complete"}
    changed: [localhost] => (item={u'ami_id': u'ami-ac37b6ba', u'root_device_type': u'ebs', u'description': None, u'tags': {}, u'hypervisor': u'xen', u'block_device_mapping': {u'/dev/sdb': {u'encrypted': None, u'snapshot_id': None, u'delete_on_termination': False, u'volume_type': None, u'size': None}, u'/dev/sda1': {u'encrypted': False, u'snapshot_id': u'snap-62d950ea', u'delete_on_termination': True, u'volume_type': u'gp2', u'size': 8}, u'/dev/sdc': {u'encrypted': None, u'snapshot_id': None, u'delete_on_termination': False, u'volume_type': None, u'size': None}}, u'architecture': u'x86_64', u'owner_id': u'526262051452', u'platform': None, u'state': u'available', u'location': u'526262051452/test-2017-04-04T18-41-04Z', u'is_public': False, u'creationDate': u'2017-04-04T18:41:05.000Z', u'root_device_name': u'/dev/sda1', u'virtualization_type': u'hvm', u'name': u'test-2017-04-04T18-41-04Z'}) => {"changed": true, "item": {"ami_id": "ami-ac37b6ba", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "encrypted": false, "size": 8, "snapshot_id": "snap-62d950ea", "volume_type": "gp2"}, "/dev/sdb": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}, "/dev/sdc": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}}, "creationDate": "2017-04-04T18:41:05.000Z", "description": null, "hypervisor": "xen", "is_public": false, "location": "526262051452/test-2017-04-04T18-41-04Z", "name": "test-2017-04-04T18-41-04Z", "owner_id": "526262051452", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": {}, "virtualization_type": "hvm"}, "msg": "AMI deregister/delete operation complete"}
    changed: [localhost] => (item={u'ami_id': u'ami-f5c646e3', u'root_device_type': u'ebs', u'description': None, u'tags': {}, u'hypervisor': u'xen', u'block_device_mapping': {u'/dev/sdb': {u'encrypted': None, u'snapshot_id': None, u'delete_on_termination': False, u'volume_type': None, u'size': None}, u'/dev/sda1': {u'encrypted': False, u'snapshot_id': u'snap-64c81cd7', u'delete_on_termination': True, u'volume_type': u'gp2', u'size': 8}, u'/dev/sdc': {u'encrypted': None, u'snapshot_id': None, u'delete_on_termination': False, u'volume_type': None, u'size': None}}, u'architecture': u'x86_64', u'owner_id': u'526262051452', u'platform': None, u'state': u'available', u'location': u'526262051452/test-2017-04-04T19-51-06Z', u'is_public': False, u'creationDate': u'2017-04-04T19:51:07.000Z', u'root_device_name': u'/dev/sda1', u'virtualization_type': u'hvm', u'name': u'test-2017-04-04T19-51-06Z'}) => {"changed": true, "item": {"ami_id": "ami-f5c646e3", "architecture": "x86_64", "block_device_mapping": {"/dev/sda1": {"delete_on_termination": true, "encrypted": false, "size": 8, "snapshot_id": "snap-64c81cd7", "volume_type": "gp2"}, "/dev/sdb": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}, "/dev/sdc": {"delete_on_termination": false, "encrypted": null, "size": null, "snapshot_id": null, "volume_type": null}}, "creationDate": "2017-04-04T19:51:07.000Z", "description": null, "hypervisor": "xen", "is_public": false, "location": "526262051452/test-2017-04-04T19-51-06Z", "name": "test-2017-04-04T19-51-06Z", "owner_id": "526262051452", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": {}, "virtualization_type": "hvm"}, "msg": "AMI deregister/delete operation complete"}
    
    PLAY ***************************************************************************
    
    TASK [terminate : Terminate old instance(s)] ***********************************
    changed: [ec2-34-205-9-162.compute-1.amazonaws.com] => {"changed": true, "instance_ids": ["i-0f55d61570fb6af45"], "instances": [{"ami_launch_index": "0", "architecture": "x86_64", "block_device_mapping": {}, "dns_name": "", "ebs_optimized": false, "groups": {}, "hypervisor": "xen", "id": "i-0f55d61570fb6af45", "image_id": "ami-e5bf05f3", "instance_type": "t2.nano", "kernel": null, "key_name": "autoscaling", "launch_time": "2017-04-04T22:16:16.000Z", "placement": "us-east-1a", "private_dns_name": "", "private_ip": null, "public_dns_name": "", "public_ip": null, "ramdisk": null, "region": "us-east-1", "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "terminated", "state_code": 48, "tags": {"Name": "testamibuild"}, "tenancy": "default", "virtualization_type": "hvm"}], "tagged_instances": []}
    
    PLAY RECAP *********************************************************************
    ec2-34-205-9-162.compute-1.amazonaws.com : ok=3    changed=2    unreachable=0    failed=0   
    localhost                  : ok=16   changed=8    unreachable=0    failed=0   
    
    k@laptop:~/bogo$ 
    



    Screen shots

    A newly created Auto Scaling group:

    AutoScalingGroup.png

    Launch configuration:

    LaunchConfiguration.png

    We can see we have a new instance (test) and a deleted amibuild (testamibuild) instance:

    CreatedInstanceByASG.png




    References

    Immutable Infrastructure with AWS and Ansible - Part 3 - Autoscaling





    Ansible 2.0

  • What is Ansible?
  • Quick Preview - Setting up web servers with Nginx, configure environments, and deploy an App
  • SSH connection & running commands
  • Ansible: Playbook for Tomcat 9 on Ubuntu 18.04 systemd with AWS
  • Modules
  • Playbooks
  • Handlers
  • Roles
  • Playbook for LAMP HAProxy
  • Installing Nginx on a Docker container
  • AWS : Creating an ec2 instance & adding keys to authorized_keys
  • AWS : Auto Scaling via AMI
  • AWS : creating an ELB & registers an EC2 instance from the ELB
  • Deploying Wordpress micro-services with Docker containers on Vagrant box via Ansible
  • Setting up Apache web server
  • Deploying a Go app to Minikube
  • Ansible with Terraform




  • Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization

    YouTubeMy YouTube channel

    Sponsor Open Source development activities and free contents for everyone.

    Thank you.

    - K Hong







    Ansible 2.0



    What is Ansible?

    Quick Preview - Setting up web servers with Nginx, configure environments, and deploy an App

    SSH connection & running commands

    Ansible: Playbook for Tomcat 9 on Ubuntu 18.04 systemd with AWS

    Modules

    Playbooks

    Handlers

    Roles

    Playbook for LAMP HAProxy

    Installing Nginx on a Docker container

    AWS : Creating an ec2 instance & adding keys to authorized_keys

    AWS : Auto Scaling via AMI

    AWS : creating an ELB & registers an EC2 instance from the ELB

    Deploying Wordpress micro-services with Docker containers on Vagrant box via Ansible

    Setting up Apache web server

    Deploying a Go app to Minikube

    Ansible with Terraform




    Sponsor Open Source development activities and free contents for everyone.

    Thank you.

    - K Hong







    DevOps



    Phases of Continuous Integration

    Software development methodology

    Introduction to DevOps

    Samples of Continuous Integration (CI) / Continuous Delivery (CD) - Use cases

    Artifact repository and repository management

    Linux - General, shell programming, processes & signals ...

    RabbitMQ...

    MariaDB

    New Relic APM with NodeJS : simple agent setup on AWS instance

    Nagios on CentOS 7 with Nagios Remote Plugin Executor (NRPE)

    Nagios - The industry standard in IT infrastructure monitoring on Ubuntu

    Zabbix 3 install on Ubuntu 14.04 & adding hosts / items / graphs

    Datadog - Monitoring with PagerDuty/HipChat and APM

    Install and Configure Mesos Cluster

    Cassandra on a Single-Node Cluster

    Container Orchestration : Docker Swarm vs Kubernetes vs Apache Mesos

    OpenStack install on Ubuntu 16.04 server - DevStack

    AWS EC2 Container Service (ECS) & EC2 Container Registry (ECR) | Docker Registry

    CI/CD with CircleCI - Heroku deploy

    Introduction to Terraform with AWS elb & nginx

    Docker & Kubernetes

    Kubernetes I - Running Kubernetes Locally via Minikube

    Kubernetes II - kops on AWS

    Kubernetes III - kubeadm on AWS

    AWS : EKS (Elastic Container Service for Kubernetes)

    CI/CD Github actions

    CI/CD Gitlab



    DevOps / Sys Admin Q & A



    (1A) - Linux Commands

    (1B) - Linux Commands

    (2) - Networks

    (2B) - Networks

    (3) - Linux Systems

    (4) - Scripting (Ruby/Shell)

    (5) - Configuration Management

    (6) - AWS VPC setup (public/private subnets with NAT)

    (6B) - AWS VPC Peering

    (7) - Web server

    (8) - Database

    (9) - Linux System / Application Monitoring, Performance Tuning, Profiling Methods & Tools

    (10) - Trouble Shooting: Load, Throughput, Response time and Leaks

    (11) - SSH key pairs, SSL Certificate, and SSL Handshake

    (12) - Why is the database slow?

    (13) - Is my web site down?

    (14) - Is my server down?

    (15) - Why is the server sluggish?

    (16A) - Serving multiple domains using Virtual Hosts - Apache

    (16B) - Serving multiple domains using server block - Nginx

    (16C) - Reverse proxy servers and load balancers - Nginx

    (17) - Linux startup process

    (18) - phpMyAdmin with Nginx virtual host as a subdomain

    (19) - How to SSH login without password?

    (20) - Log Rotation

    (21) - Monitoring Metrics

    (22) - lsof

    (23) - Wireshark introduction

    (24) - User account management

    (25) - Domain Name System (DNS)

    (26) - NGINX SSL/TLS, Caching, and Session

    (27) - Troubleshooting 5xx server errors

    (28) - Linux Systemd: journalctl

    (29) - Linux Systemd: FirewallD

    (30) - Linux: SELinux

    (31) - Linux: Samba

    (0) - Linux Sys Admin's Day to Day tasks






    Terraform



    Introduction to Terraform with AWS elb & nginx

    Terraform Tutorial - terraform format(tf) and interpolation(variables)

    Terraform Tutorial - user_data

    Terraform Tutorial - variables

    Terraform 12 Tutorial - Loops with count, for_each, and for

    Terraform Tutorial - creating multiple instances (count, list type and element() function)

    Terraform Tutorial - State (terraform.tfstate) & terraform import

    Terraform Tutorial - Output variables

    Terraform Tutorial - Destroy

    Terraform Tutorial - Modules

    Terraform Tutorial - Creating AWS S3 bucket / SQS queue resources and notifying bucket event to queue

    Terraform Tutorial - AWS ASG and Modules

    Terraform Tutorial - VPC, Subnets, RouteTable, ELB, Security Group, and Apache server I

    Terraform Tutorial - VPC, Subnets, RouteTable, ELB, Security Group, and Apache server II

    Terraform Tutorial - Docker nginx container with ALB and dynamic autoscaling

    Terraform Tutorial - AWS ECS using Fargate : Part I

    Hashicorp Vault

    HashiCorp Vault Agent

    HashiCorp Vault and Consul on AWS with Terraform

    Ansible with Terraform

    AWS IAM user, group, role, and policies - part 1

    AWS IAM user, group, role, and policies - part 2

    Delegate Access Across AWS Accounts Using IAM Roles

    AWS KMS

    terraform import & terraformer import

    Terraform commands cheat sheet

    Terraform Cloud

    Terraform 14

    Creating Private TLS Certs





    Jenkins



    Install

    Configuration - Manage Jenkins - security setup

    Adding job and build

    Scheduling jobs

    Managing_plugins

    Git/GitHub plugins, SSH keys configuration, and Fork/Clone

    JDK & Maven setup

    Build configuration for GitHub Java application with Maven

    Build Action for GitHub Java application with Maven - Console Output, Updating Maven

    Commit to changes to GitHub & new test results - Build Failure

    Commit to changes to GitHub & new test results - Successful Build

    Adding code coverage and metrics

    Jenkins on EC2 - creating an EC2 account, ssh to EC2, and install Apache server

    Jenkins on EC2 - setting up Jenkins account, plugins, and Configure System (JAVA_HOME, MAVEN_HOME, notification email)

    Jenkins on EC2 - Creating a Maven project

    Jenkins on EC2 - Configuring GitHub Hook and Notification service to Jenkins server for any changes to the repository

    Jenkins on EC2 - Line Coverage with JaCoCo plugin

    Setting up Master and Slave nodes

    Jenkins Build Pipeline & Dependency Graph Plugins

    Jenkins Build Flow Plugin

    Pipeline Jenkinsfile with Classic / Blue Ocean

    Jenkins Setting up Slave nodes on AWS

    Jenkins Q & A





    Puppet



    Puppet with Amazon AWS I - Puppet accounts

    Puppet with Amazon AWS II (ssh & puppetmaster/puppet install)

    Puppet with Amazon AWS III - Puppet running Hello World

    Puppet Code Basics - Terminology

    Puppet with Amazon AWS on CentOS 7 (I) - Master setup on EC2

    Puppet with Amazon AWS on CentOS 7 (II) - Configuring a Puppet Master Server with Passenger and Apache

    Puppet master /agent ubuntu 14.04 install on EC2 nodes

    Puppet master post install tasks - master's names and certificates setup,

    Puppet agent post install tasks - configure agent, hostnames, and sign request

    EC2 Puppet master/agent basic tasks - main manifest with a file resource/module and immediate execution on an agent node

    Setting up puppet master and agent with simple scripts on EC2 / remote install from desktop

    EC2 Puppet - Install lamp with a manifest ('puppet apply')

    EC2 Puppet - Install lamp with a module

    Puppet variable scope

    Puppet packages, services, and files

    Puppet packages, services, and files II with nginx Puppet templates

    Puppet creating and managing user accounts with SSH access

    Puppet Locking user accounts & deploying sudoers file

    Puppet exec resource

    Puppet classes and modules

    Puppet Forge modules

    Puppet Express

    Puppet Express 2

    Puppet 4 : Changes

    Puppet --configprint

    Puppet with Docker

    Puppet 6.0.2 install on Ubuntu 18.04





    Chef



    What is Chef?

    Chef install on Ubuntu 14.04 - Local Workstation via omnibus installer

    Setting up Hosted Chef server

    VirtualBox via Vagrant with Chef client provision

    Creating and using cookbooks on a VirtualBox node

    Chef server install on Ubuntu 14.04

    Chef workstation setup on EC2 Ubuntu 14.04

    Chef Client Node - Knife Bootstrapping a node on EC2 ubuntu 14.04





    Docker & K8s



    Docker install on Amazon Linux AMI

    Docker install on EC2 Ubuntu 14.04

    Docker container vs Virtual Machine

    Docker install on Ubuntu 14.04

    Docker Hello World Application

    Nginx image - share/copy files, Dockerfile

    Working with Docker images : brief introduction

    Docker image and container via docker commands (search, pull, run, ps, restart, attach, and rm)

    More on docker run command (docker run -it, docker run --rm, etc.)

    Docker Networks - Bridge Driver Network

    Docker Persistent Storage

    File sharing between host and container (docker run -d -p -v)

    Linking containers and volume for datastore

    Dockerfile - Build Docker images automatically I - FROM, MAINTAINER, and build context

    Dockerfile - Build Docker images automatically II - revisiting FROM, MAINTAINER, build context, and caching

    Dockerfile - Build Docker images automatically III - RUN

    Dockerfile - Build Docker images automatically IV - CMD

    Dockerfile - Build Docker images automatically V - WORKDIR, ENV, ADD, and ENTRYPOINT

    Docker - Apache Tomcat

    Docker - NodeJS

    Docker - NodeJS with hostname

    Docker Compose - NodeJS with MongoDB

    Docker - Prometheus and Grafana with Docker-compose

    Docker - StatsD/Graphite/Grafana

    Docker - Deploying a Java EE JBoss/WildFly Application on AWS Elastic Beanstalk Using Docker Containers

    Docker : NodeJS with GCP Kubernetes Engine

    Docker : Jenkins Multibranch Pipeline with Jenkinsfile and Github

    Docker : Jenkins Master and Slave

    Docker - ELK : ElasticSearch, Logstash, and Kibana

    Docker - ELK 7.6 : Elasticsearch on Centos 7 Docker - ELK 7.6 : Filebeat on Centos 7

    Docker - ELK 7.6 : Logstash on Centos 7

    Docker - ELK 7.6 : Kibana on Centos 7 Part 1

    Docker - ELK 7.6 : Kibana on Centos 7 Part 2

    Docker - ELK 7.6 : Elastic Stack with Docker Compose

    Docker - Deploy Elastic Cloud on Kubernetes (ECK) via Elasticsearch operator on minikube

    Docker - Deploy Elastic Stack via Helm on minikube

    Docker Compose - A gentle introduction with WordPress

    Docker Compose - MySQL

    MEAN Stack app on Docker containers : micro services

    Docker Compose - Hashicorp's Vault and Consul Part A (install vault, unsealing, static secrets, and policies)

    Docker Compose - Hashicorp's Vault and Consul Part B (EaaS, dynamic secrets, leases, and revocation)

    Docker Compose - Hashicorp's Vault and Consul Part C (Consul)

    Docker Compose with two containers - Flask REST API service container and an Apache server container

    Docker compose : Nginx reverse proxy with multiple containers

    Docker compose : Nginx reverse proxy with multiple containers

    Docker & Kubernetes : Envoy - Getting started

    Docker & Kubernetes : Envoy - Front Proxy

    Docker & Kubernetes : Ambassador - Envoy API Gateway on Kubernetes

    Docker Packer

    Docker Cheat Sheet

    Docker Q & A

    Kubernetes Q & A - Part I

    Kubernetes Q & A - Part II

    Docker - Run a React app in a docker

    Docker - Run a React app in a docker II (snapshot app with nginx)

    Docker - NodeJS and MySQL app with React in a docker

    Docker - Step by Step NodeJS and MySQL app with React - I

    Installing LAMP via puppet on Docker

    Docker install via Puppet

    Nginx Docker install via Ansible

    Apache Hadoop CDH 5.8 Install with QuickStarts Docker

    Docker - Deploying Flask app to ECS

    Docker Compose - Deploying WordPress to AWS

    Docker - WordPress Deploy to ECS with Docker-Compose (ECS-CLI EC2 type)

    Docker - ECS Fargate

    Docker - AWS ECS service discovery with Flask and Redis

    Docker & Kubernetes: minikube version: v1.31.2, 2023

    Docker & Kubernetes 1 : minikube

    Docker & Kubernetes 2 : minikube Django with Postgres - persistent volume

    Docker & Kubernetes 3 : minikube Django with Redis and Celery

    Docker & Kubernetes 4 : Django with RDS via AWS Kops

    Docker & Kubernetes : Kops on AWS

    Docker & Kubernetes : Ingress controller on AWS with Kops

    Docker & Kubernetes : HashiCorp's Vault and Consul on minikube

    Docker & Kubernetes : HashiCorp's Vault and Consul - Auto-unseal using Transit Secrets Engine

    Docker & Kubernetes : Persistent Volumes & Persistent Volumes Claims - hostPath and annotations

    Docker & Kubernetes : Persistent Volumes - Dynamic volume provisioning

    Docker & Kubernetes : DaemonSet

    Docker & Kubernetes : Secrets

    Docker & Kubernetes : kubectl command

    Docker & Kubernetes : Assign a Kubernetes Pod to a particular node in a Kubernetes cluster

    Docker & Kubernetes : Configure a Pod to Use a ConfigMap

    AWS : EKS (Elastic Container Service for Kubernetes)

    Docker & Kubernetes : Run a React app in a minikube

    Docker & Kubernetes : Minikube install on AWS EC2

    Docker & Kubernetes : Cassandra with a StatefulSet

    Docker & Kubernetes : Terraform and AWS EKS

    Docker & Kubernetes : Pods and Service definitions

    Docker & Kubernetes : Headless service and discovering pods

    Docker & Kubernetes : Service IP and the Service Type

    Docker & Kubernetes : Kubernetes DNS with Pods and Services

    Docker & Kubernetes - Scaling and Updating application

    Docker & Kubernetes : Horizontal pod autoscaler on minikubes

    Docker & Kubernetes : NodePort vs LoadBalancer vs Ingress

    Docker & Kubernetes : Load Testing with Locust on GCP Kubernetes

    Docker & Kubernetes : From a monolithic app to micro services on GCP Kubernetes

    Docker & Kubernetes : Rolling updates

    Docker & Kubernetes : Deployments to GKE (Rolling update, Canary and Blue-green deployments)

    Docker & Kubernetes : Slack Chat Bot with NodeJS on GCP Kubernetes

    Docker & Kubernetes : Continuous Delivery with Jenkins Multibranch Pipeline for Dev, Canary, and Production Environments on GCP Kubernetes

    Docker & Kubernetes - MongoDB with StatefulSets on GCP Kubernetes Engine

    Docker & Kubernetes : Nginx Ingress Controller on minikube

    Docker & Kubernetes : Setting up Ingress with NGINX Controller on Minikube (Mac)

    Docker & Kubernetes : Nginx Ingress Controller for Dashboard service on Minikube

    Docker & Kubernetes : Nginx Ingress Controller on GCP Kubernetes

    Docker & Kubernetes : Kubernetes Ingress with AWS ALB Ingress Controller in EKS

    Docker & Kubernetes : MongoDB / MongoExpress on Minikube

    Docker & Kubernetes : Setting up a private cluster on GCP Kubernetes

    Docker & Kubernetes : Kubernetes Namespaces (default, kube-public, kube-system) and switching namespaces (kubens)

    Docker & Kubernetes : StatefulSets on minikube

    Docker & Kubernetes : StatefulSets on minikube

    Docker & Kubernetes : RBAC

    Docker & Kubernetes Service Account, RBAC, and IAM

    Docker & Kubernetes - Kubernetes Service Account, RBAC, IAM with EKS ALB, Part 1

    Docker & Kubernetes : Helm Chart

    Docker & Kubernetes : My first Helm deploy

    Docker & Kubernetes : Readiness and Liveness Probes

    Docker & Kubernetes : Helm chart repository with Github pages

    Docker & Kubernetes : Deploying WordPress and MariaDB with Ingress to Minikube using Helm Chart

    Docker & Kubernetes : Deploying WordPress and MariaDB to AWS using Helm 2 Chart

    Docker & Kubernetes : Deploying WordPress and MariaDB to AWS using Helm 3 Chart

    Docker & Kubernetes : Helm Chart for Node/Express and MySQL with Ingress

    Docker & Kubernetes : Docker_Helm_Chart_Node_Expess_MySQL_Ingress.php

    Docker & Kubernetes: Deploy Prometheus and Grafana using Helm and Prometheus Operator - Monitoring Kubernetes node resources out of the box

    Docker & Kubernetes : Deploy Prometheus and Grafana using kube-prometheus-stack Helm Chart

    Docker & Kubernetes : Istio (service mesh) sidecar proxy on GCP Kubernetes

    Docker & Kubernetes : Istio on EKS

    Docker & Kubernetes : Istio on Minikube with AWS EC2 for Bookinfo Application

    Docker & Kubernetes : Deploying .NET Core app to Kubernetes Engine and configuring its traffic managed by Istio (Part I)

    Docker & Kubernetes : Deploying .NET Core app to Kubernetes Engine and configuring its traffic managed by Istio (Part II - Prometheus, Grafana, pin a service, split traffic, and inject faults)

    Docker & Kubernetes : Helm Package Manager with MySQL on GCP Kubernetes Engine

    Docker & Kubernetes : Deploying Memcached on Kubernetes Engine

    Docker & Kubernetes : EKS Control Plane (API server) Metrics with Prometheus

    Docker & Kubernetes : Spinnaker on EKS with Halyard

    Docker & Kubernetes : Continuous Delivery Pipelines with Spinnaker and Kubernetes Engine

    Docker & Kubernetes: Multi-node Local Kubernetes cluster - Kubeadm-dind(docker-in-docker)

    Docker & Kubernetes: Multi-node Local Kubernetes cluster - Kubeadm-kind(k8s-in-docker)

    Docker & Kubernetes : nodeSelector, nodeAffinity, taints/tolerations, pod affinity and anti-affinity - Assigning Pods to Nodes

    Docker & Kubernetes : Jenkins-X on EKS

    Docker & Kubernetes : ArgoCD App of Apps with Heml on Kubernetes

    Docker & Kubernetes : ArgoCD on Kubernetes cluster

    Docker & Kubernetes : GitOps with ArgoCD for Continuous Delivery to Kubernetes clusters (minikube) - guestbook





    Elasticsearch search engine, Logstash, and Kibana



    Elasticsearch, search engine

    Logstash with Elasticsearch

    Logstash, Elasticsearch, and Kibana 4

    Elasticsearch with Redis broker and Logstash Shipper and Indexer

    Samples of ELK architecture

    Elasticsearch indexing performance



    Vagrant



    VirtualBox & Vagrant install on Ubuntu 14.04

    Creating a VirtualBox using Vagrant

    Provisioning

    Networking - Port Forwarding

    Vagrant Share

    Vagrant Rebuild & Teardown

    Vagrant & Ansible





    Big Data & Hadoop Tutorials



    Hadoop 2.6 - Installing on Ubuntu 14.04 (Single-Node Cluster)

    Hadoop 2.6.5 - Installing on Ubuntu 16.04 (Single-Node Cluster)

    Hadoop - Running MapReduce Job

    Hadoop - Ecosystem

    CDH5.3 Install on four EC2 instances (1 Name node and 3 Datanodes) using Cloudera Manager 5

    CDH5 APIs

    QuickStart VMs for CDH 5.3

    QuickStart VMs for CDH 5.3 II - Testing with wordcount

    QuickStart VMs for CDH 5.3 II - Hive DB query

    Scheduled start and stop CDH services

    CDH 5.8 Install with QuickStarts Docker

    Zookeeper & Kafka Install

    Zookeeper & Kafka - single node single broker

    Zookeeper & Kafka - Single node and multiple brokers

    OLTP vs OLAP

    Apache Hadoop Tutorial I with CDH - Overview

    Apache Hadoop Tutorial II with CDH - MapReduce Word Count

    Apache Hadoop Tutorial III with CDH - MapReduce Word Count 2

    Apache Hadoop (CDH 5) Hive Introduction

    CDH5 - Hive Upgrade to 1.3 to from 1.2

    Apache Hive 2.1.0 install on Ubuntu 16.04

    Apache HBase in Pseudo-Distributed mode

    Creating HBase table with HBase shell and HUE

    Apache Hadoop : Hue 3.11 install on Ubuntu 16.04

    Creating HBase table with Java API

    HBase - Map, Persistent, Sparse, Sorted, Distributed and Multidimensional

    Flume with CDH5: a single-node Flume deployment (telnet example)

    Apache Hadoop (CDH 5) Flume with VirtualBox : syslog example via NettyAvroRpcClient

    List of Apache Hadoop hdfs commands

    Apache Hadoop : Creating Wordcount Java Project with Eclipse Part 1

    Apache Hadoop : Creating Wordcount Java Project with Eclipse Part 2

    Apache Hadoop : Creating Card Java Project with Eclipse using Cloudera VM UnoExample for CDH5 - local run

    Apache Hadoop : Creating Wordcount Maven Project with Eclipse

    Wordcount MapReduce with Oozie workflow with Hue browser - CDH 5.3 Hadoop cluster using VirtualBox and QuickStart VM

    Spark 1.2 using VirtualBox and QuickStart VM - wordcount

    Spark Programming Model : Resilient Distributed Dataset (RDD) with CDH

    Apache Spark 2.0.2 with PySpark (Spark Python API) Shell

    Apache Spark 2.0.2 tutorial with PySpark : RDD

    Apache Spark 2.0.0 tutorial with PySpark : Analyzing Neuroimaging Data with Thunder

    Apache Spark Streaming with Kafka and Cassandra

    Apache Spark 1.2 with PySpark (Spark Python API) Wordcount using CDH5

    Apache Spark 1.2 Streaming

    Apache Drill with ZooKeeper install on Ubuntu 16.04 - Embedded & Distributed

    Apache Drill - Query File System, JSON, and Parquet

    Apache Drill - HBase query

    Apache Drill - Hive query

    Apache Drill - MongoDB query





    Redis In-Memory Database



    Redis vs Memcached

    Redis 3.0.1 Install

    Setting up multiple server instances on a Linux host

    Redis with Python

    ELK : Elasticsearch with Redis broker and Logstash Shipper and Indexer



    GCP (Google Cloud Platform)



    GCP: Creating an Instance

    GCP: gcloud compute command-line tool

    GCP: Deploying Containers

    GCP: Kubernetes Quickstart

    GCP: Deploying a containerized web application via Kubernetes

    GCP: Django Deploy via Kubernetes I (local)

    GCP: Django Deploy via Kubernetes II (GKE)





    AWS (Amazon Web Services)



    AWS : EKS (Elastic Container Service for Kubernetes)

    AWS : Creating a snapshot (cloning an image)

    AWS : Attaching Amazon EBS volume to an instance

    AWS : Adding swap space to an attached volume via mkswap and swapon

    AWS : Creating an EC2 instance and attaching Amazon EBS volume to the instance using Python boto module with User data

    AWS : Creating an instance to a new region by copying an AMI

    AWS : S3 (Simple Storage Service) 1

    AWS : S3 (Simple Storage Service) 2 - Creating and Deleting a Bucket

    AWS : S3 (Simple Storage Service) 3 - Bucket Versioning

    AWS : S3 (Simple Storage Service) 4 - Uploading a large file

    AWS : S3 (Simple Storage Service) 5 - Uploading folders/files recursively

    AWS : S3 (Simple Storage Service) 6 - Bucket Policy for File/Folder View/Download

    AWS : S3 (Simple Storage Service) 7 - How to Copy or Move Objects from one region to another

    AWS : S3 (Simple Storage Service) 8 - Archiving S3 Data to Glacier

    AWS : Creating a CloudFront distribution with an Amazon S3 origin

    AWS : Creating VPC with CloudFormation

    WAF (Web Application Firewall) with preconfigured CloudFormation template and Web ACL for CloudFront distribution

    AWS : CloudWatch & Logs with Lambda Function / S3

    AWS : Lambda Serverless Computing with EC2, CloudWatch Alarm, SNS

    AWS : Lambda and SNS - cross account

    AWS : CLI (Command Line Interface)

    AWS : CLI (ECS with ALB & autoscaling)

    AWS : ECS with cloudformation and json task definition

    AWS : AWS Application Load Balancer (ALB) and ECS with Flask app

    AWS : Load Balancing with HAProxy (High Availability Proxy)

    AWS : VirtualBox on EC2

    AWS : NTP setup on EC2

    AWS: jq with AWS

    AWS : AWS & OpenSSL : Creating / Installing a Server SSL Certificate

    AWS : OpenVPN Access Server 2 Install

    AWS : VPC (Virtual Private Cloud) 1 - netmask, subnets, default gateway, and CIDR

    AWS : VPC (Virtual Private Cloud) 2 - VPC Wizard

    AWS : VPC (Virtual Private Cloud) 3 - VPC Wizard with NAT

    AWS : DevOps / Sys Admin Q & A (VI) - AWS VPC setup (public/private subnets with NAT)

    AWS : OpenVPN Protocols : PPTP, L2TP/IPsec, and OpenVPN

    AWS : Autoscaling group (ASG)

    AWS : Setting up Autoscaling Alarms and Notifications via CLI and Cloudformation

    AWS : Adding a SSH User Account on Linux Instance

    AWS : Windows Servers - Remote Desktop Connections using RDP

    AWS : Scheduled stopping and starting an instance - python & cron

    AWS : Detecting stopped instance and sending an alert email using Mandrill smtp

    AWS : Elastic Beanstalk with NodeJS

    AWS : Elastic Beanstalk Inplace/Rolling Blue/Green Deploy

    AWS : Identity and Access Management (IAM) Roles for Amazon EC2

    AWS : Identity and Access Management (IAM) Policies, sts AssumeRole, and delegate access across AWS accounts

    AWS : Identity and Access Management (IAM) sts assume role via aws cli2

    AWS : Creating IAM Roles and associating them with EC2 Instances in CloudFormation

    AWS Identity and Access Management (IAM) Roles, SSO(Single Sign On), SAML(Security Assertion Markup Language), IdP(identity provider), STS(Security Token Service), and ADFS(Active Directory Federation Services)

    AWS : Amazon Route 53

    AWS : Amazon Route 53 - DNS (Domain Name Server) setup

    AWS : Amazon Route 53 - subdomain setup and virtual host on Nginx

    AWS Amazon Route 53 : Private Hosted Zone

    AWS : SNS (Simple Notification Service) example with ELB and CloudWatch

    AWS : Lambda with AWS CloudTrail

    AWS : SQS (Simple Queue Service) with NodeJS and AWS SDK

    AWS : Redshift data warehouse

    AWS : CloudFormation - templates, change sets, and CLI

    AWS : CloudFormation Bootstrap UserData/Metadata

    AWS : CloudFormation - Creating an ASG with rolling update

    AWS : Cloudformation Cross-stack reference

    AWS : OpsWorks

    AWS : Network Load Balancer (NLB) with Autoscaling group (ASG)

    AWS CodeDeploy : Deploy an Application from GitHub

    AWS EC2 Container Service (ECS)

    AWS EC2 Container Service (ECS) II

    AWS Hello World Lambda Function

    AWS Lambda Function Q & A

    AWS Node.js Lambda Function & API Gateway

    AWS API Gateway endpoint invoking Lambda function

    AWS API Gateway invoking Lambda function with Terraform

    AWS API Gateway invoking Lambda function with Terraform - Lambda Container

    Amazon Kinesis Streams

    Kinesis Data Firehose with Lambda and ElasticSearch

    Amazon DynamoDB

    Amazon DynamoDB with Lambda and CloudWatch

    Loading DynamoDB stream to AWS Elasticsearch service with Lambda

    Amazon ML (Machine Learning)

    Simple Systems Manager (SSM)

    AWS : RDS Connecting to a DB Instance Running the SQL Server Database Engine

    AWS : RDS Importing and Exporting SQL Server Data

    AWS : RDS PostgreSQL & pgAdmin III

    AWS : RDS PostgreSQL 2 - Creating/Deleting a Table

    AWS : MySQL Replication : Master-slave

    AWS : MySQL backup & restore

    AWS RDS : Cross-Region Read Replicas for MySQL and Snapshots for PostgreSQL

    AWS : Restoring Postgres on EC2 instance from S3 backup

    AWS : Q & A

    AWS : Security

    AWS : Security groups vs. network ACLs

    AWS : Scaling-Up

    AWS : Networking

    AWS : Single Sign-on (SSO) with Okta

    AWS : JIT (Just-in-Time) with Okta





    Powershell 4 Tutorial



    Powersehll : Introduction

    Powersehll : Help System

    Powersehll : Running commands

    Powersehll : Providers

    Powersehll : Pipeline

    Powersehll : Objects

    Powershell : Remote Control

    Windows Management Instrumentation (WMI)

    How to Enable Multiple RDP Sessions in Windows 2012 Server

    How to install and configure FTP server on IIS 8 in Windows 2012 Server

    How to Run Exe as a Service on Windows 2012 Server

    SQL Inner, Left, Right, and Outer Joins





    Git/GitHub Tutorial



    One page express tutorial for GIT and GitHub

    Installation

    add/status/log

    commit and diff

    git commit --amend

    Deleting and Renaming files

    Undoing Things : File Checkout & Unstaging

    Reverting commit

    Soft Reset - (git reset --soft <SHA key>)

    Mixed Reset - Default

    Hard Reset - (git reset --hard <SHA key>)

    Creating & switching Branches

    Fast-forward merge

    Rebase & Three-way merge

    Merge conflicts with a simple example

    GitHub Account and SSH

    Uploading to GitHub

    GUI

    Branching & Merging

    Merging conflicts

    GIT on Ubuntu and OS X - Focused on Branching

    Setting up a remote repository / pushing local project and cloning the remote repo

    Fork vs Clone, Origin vs Upstream

    Git/GitHub Terminologies

    Git/GitHub via SourceTree II : Branching & Merging

    Git/GitHub via SourceTree III : Git Work Flow

    Git/GitHub via SourceTree IV : Git Reset

    Git wiki - quick command reference






    Subversion

    Subversion Install On Ubuntu 14.04

    Subversion creating and accessing I

    Subversion creating and accessing II








    Contact

    BogoToBogo
    contactus@bogotobogo.com

    Follow Bogotobogo

    About Us

    contactus@bogotobogo.com

    YouTubeMy YouTube channel
    Pacific Ave, San Francisco, CA 94115

    Pacific Ave, San Francisco, CA 94115

    Copyright © 2024, bogotobogo
    Design: Web Master