Legacy Saltstack

This is a COMPLETE public release of my personal salt states.

The states presented here were used for running this network.

This repository includes a lot of neat tricks I’ve learned and shows off many salt best practices (bad practices are bugs). It also includes some components that will be eventually be described on my personal blog [1], such as IDI (Inventory Defined Infrastructure) [2] and automated highstates [3].

At this point, the infrastructure described within has been retired and rebuilt from scratch from an absolute minimalist approach. The approach shared within works exceptionally well for small, medium, and multi-national corporations. It also worked well for my home lab/network and gave me the chance to explore some amazing concepts. Ultimately, as I hosted more and more open source projects (enabled by automation), my $hobby and $home[‘life’] devolved further and further into $work; I stopped having fun. … hence, the entire public export of the FINAL and COMPLETE incantation of my home network. It is complete, and it is retired. So long, and thanks for all the fish.

Cloud Nodes

Visualization of the pillar data generated by my salt setup.

salt-cloud:

  defaults:
    digitalocean:
      image: "8.7 x64"
      location: "New York 3"
      size: 512MB
    proxint:
      image: slowdisk:vztmpl/debian-8.0_Lustfield.tar.gz
      disk: "slow;8"
      type: lxc 
      host: prox1
      onboot: 1
      swap: 0
      cpu: 2
      mem: 256 

  digitalocean-nodes:

    oob-c1.domain.tld:
      id: oob-c1

    pbin.ngx.cc:
      id: pbin

    irc.domain.tld:
      id: irc 

    pubweb00.domain.tld:
      id: pubweb00

    pubweb02.domain.tld:
      id: pubweb02
      location: "Singapore 1"

  proxint-nodes:

    forge.domain.tld:
      id: "210;forge"
      type: qemu
      mem: 256-8192
      cpu: 16
      disk: "slow;64"
      iface:
        net0:
          name: eth0
          vlan: 2

    ## 
    # PROD
    ##

    boot.domain.tld:
      id: "101;boot"
      mem: 512 
      cpu: 1
      disk: "fast;8"
      iface:
        net0:
          name: eth0
          vlan: 57
          ipv4_addr: 10.41.57.10/24
          ipv4_gw: 10.41.57.1

    salt.domain.tld:
      id: "111;salt"
      mem: 2048
      cpu: 8
      disk: "crypt;16"
      iface:
        net0:
          name: eth0
          vlan: 50
          ipv4_addr: 10.41.50.11/24
          ipv4_gw: 10.41.50.1
          ipv6_addr: 2001:dead:b33f:50::11/64
          ipv6_gw: 2001:dead:b33f:50::1

    git.domain.tld:
      id: "112;git"
      mem: 2048
      cpu: 4
      disk: "crypt;32"
      iface:
        net0:
          name: eth0
          vlan: 50
          ipv4_addr: 10.41.50.12/24
          ipv4_gw: 10.41.50.1
          ipv6_addr: 2001:dead:b33f:50::12/64
          ipv6_gw: 2001:dead:b33f:50::1

    log.domain.tld:
      id: "113;log"
      mem: 512
      disk: "crypt;64"
      iface:
        net0:
          name: eth0
          vlan: 50
          ipv4_addr: 10.41.50.13/24
          ipv4_gw: 10.41.50.1
          ipv6_addr: 2001:dead:b33f:50::13/64
          ipv6_gw: 2001:dead:b33f:50::1

    apt.domain.tld:
      id: "114;apt"
      mem: 1024
      disk: "fast;16"
      iface:
        net0:
          name: eth0
          vlan: 50
          ipv4_addr: 10.41.50.14/24
          ipv4_gw: 10.41.50.1
          ipv6_addr: 2001:dead:b33f:50::14/64
          ipv6_gw: 2001:dead:b33f:50::1

    #[...]

Global Highstate

Server-side git hook that sends notifications to salt server:

root@prd-git-01:/var/lib/gitolite3/repositories/salt/states.git# cat hooks/post-receive.h00-salt-global-highstate 
#!/bin/sh
sudo /usr/local/sbin/saltevent 'global' 'highstate'

Expose a cache database to salt master:

localsdb:
  driver: sqlite3
  database: /var/cache/salt/local.db
  table: sdb_local
  create_table: True

Use salt reactor and sdb to orchestrate the delayed execution of a highstate when a git trigger is received:

{% if 'action' in data['data'] %}
{% set action  = data['data']['action'] %}{% endif %}

{% if 'host' in data['data'] %}
{% set host    = data['data']['host'] %}{% endif %}

{% if 'env' in data['data'] %}
{% set env     = data['data']['env'] %}{% endif %}

{% set tick    = salt['sdb.get']('sdb://localsdb/hst_tick') %}

{% set src     = data['id'] %}


##
# git.lustfield.net
##
{% if src == 'git.lustfield.net' and action == 'highstate' %}
update_fileserver:
  runner.fileserver.update:
    - tgt: salt.lustfield.net

  {% if 'env' in data['data'] %}
highstate_run:
  local.state.highstate:
    - tgt: "G@id:{{ host }}* and G@environment:{{ env }}"
    - expr_form: compound
    - require:
      - runner: update_fileserver

  {% elif host == 'global' %}
    {% if not tick or (tick > 10 or tick == 0) %}
{##
  # This is my magic highstate scheduling!
  # When a salt event comes in from the git server that requests
  # a highstate be pushed across the environment, check the
  # current tick value. If a global highstate was not recently
  # executed, then set the tick to 15.
  #
  # A separate process on the salt master is used to decrease
  # this tick value by one every minute and triggers a global
  # highstate runner when the value reaches zero.
  #
  # This results in a global highstate orchestration run being
  # scheduled with a five minute delay when certain git repositories
  # are pushed to. It also creates a delay before the next global
  # higshtate can be triggered, meaning time for this process to
  # complete.
  #
  # Message me if this deserves a separate blog post!
  ##}
highstate_countdown:
  local.sdb.set:
    - tgt: salt.lustfield.net
    - arg:
      - sdb://local/hst_tick
      - '15'
    {% endif %}

  {% else %}
highstate_run:
  local.state.highstate:
    - tgt: "{{ host }}*"
    - require:
      - runner: update_fileserver

{% endif %}

Script that handles tick and highstate execution logic:

#!/usr/bin/env python
'''
Author:      Michael Lustfield (MTecknology)
License:     CC BY-NC-ND 4.0
             https://creativecommons.org/licenses/by-nc-nd/4.0/
Description: This script is meant to be triggered via the salt scheduler. It
             reads a counter stored in sdb and drops the value by one. If the
             counter reaches a specified value, a global highstate is triggered
             via the orchestrate runner. Setting the value and the conditions
             in which it's set are handled by separate logic.
'''
import salt.config
import subprocess
import time
from salt.utils.sdb import sdb_get, sdb_set

opts = salt.config.minion_config('/etc/salt/minion')
tick = int(sdb_get('sdb://localsdb/hst_tick', opts))

if tick:
    if tick > 0:
        sdb_set('sdb://localsdb/hst_tick', tick - 1, opts)
    if tick == 10:
        p = subprocess.Popen(
                ['/usr/bin/salt-run', 'state.orch', '_orchestrate.global_highstate'],
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE)
        stdout, stderr = p.communicate()
        if stdout or stderr:
            with open('/var/log/salt/global_highstate.log', 'a') as fh:
                fh.write('{}\n'.format(time.strftime("%c")))
                if stdout:
                    fh.write('{}\n'.format(stdout))
                if stderr:
                    fh.write('{}\n'.format(stderr))
                fh.close()