{Digital Safe}

Operating in a Cloud Environment

# ABOUT THE INSTRUCTOR
  • Former Forward Communications

 

  • Software Engineer

 

  • Linux Enthusiast
# AGENDA

1

- Introduction

- Procurement & Attribution

- Basic Setup

- Version Control

- Scripting and Automation

2

- Building Locally

- Containers & Namespaces

- Custom Images

- Recipe Tools

3

- Dynamic Monitoring

- Accessing Remote Services

- Cloud Hardening

- Firewalls

4

- Automated Hardening and Scanning

- Q & A

- Capstone

Training Days
# Our TECH STACK
  • Fedora (35)

 

  • KDE Plasma Shell

 

  • ZSH

 

  • Some Extra Applications and Tweaks along the way
# OUR TECH STACK
# TAKING NOTES ONLINE

Taking Notes

  • www.github.com
  • Open Source Code Repositories
  • Create a Repository for "ClassNotes"
  • Use Markdown in ".md" files

https://obsidian.md

# TAKING NOTES
$ sudo dnf install flatpak
$ flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
$ flatpak install flathub md.obsidian.Obsidian

Markdown in 30 Seconds

# TAKING NOTES
# Title

## Sub Title

### etc. etc. etc.

Comments about `commands`

List:
 
- **bold**
- *italics*
- ++underline++
- ~~strikethrough~~
- ==Highlight==

- [ ] Incomplete Checkbox
- [x] Complete Checkbox

Even [hyper-links](www.someurl.com)
```python
# Code Here
def say_hi(name="Kevin"):
  print(f'Hello {name}')
  
say_hi("Tom")
```

{Day 01}

From Zero to ________

{Procurement}

How and Where do we get a "Cloud Environment"

 Define Requirements

 

  • Resources
  • POPs
  • Attribution
  • Timelines of use

1.

2.

Research Providers

 

  • Big names vs small names
  • Capabilities
  • Payment Model

3.

Rotate Responsibly

 

  • Data exfil costs
  • Rotating within a provider
  • Rotating resources
# PROCUREMENT PROCESS

Most providers are trying to prevent crypto mining, scams, and illegal activity

https://prv.to

{Attribution}

Who am I? Who are We?

  • Assume Compromise
  • Spiking Interest AKA Being too Sneaky
  • Historical Backstopping
  • Should You Use Third Parties?
    • Trust but verify

Attribution Pillars

# ATTRIBUTION
  • Network / Application (Packets, Ports, Protocols, Timestamps)
  • Logistical
  • Usage (Timezone, etc.)
  • Internal State (Language, Configuration, etc.)
# TYPES OF ATTRIBUTION
  • Consider the Threat Model
    • What can they see?
    • Where are they looking from?
    • What don't I want them to see?
    • What do I want them to see?
    • Who can they collaborate with?
  • Consider trade-offs in usability, security, and noise floor
# THREAT MODEL

{Linux}

Catching Up

# What is Linux?
  • Technically an Open Source kernel
  • Commonly known as a Open Source Operating System
    • An Operating System is a kernel and tools together
    • Also referred to as GNU/Linux to credit the tools + the kernel
  • Created in 1991 by Linus Torvalds based on Unix
  • Heavily used in servers, embedded, and development environments
  • Reputation for being advanced or difficult
# Open Source?
  • Software where the source code used to build the software is open to the public
  • Usually released under special licenses
    • Different licenses have different restrictions
    • Licenses have been the topic of holy wars
  • 'Free' as in Speech, sometimes 'Free' as in Beer
# Key Linux Concepts
  • Everything is a file (including directories)
    • Absolute paths start with a leading '/'
    • Relative paths start with anything else
  • 'Hidden' files start with a leading '.'
  • Many configurations and settings are simply text files
  • Many GUI programs simply 'wrap' command line programs
  • Commands can be 'linked' (or 'chained') with one another
# LINUX COMPONENTS

CPU

Screen

HDD

RAM

...

Linux Kernel

GNU Core Utilities

System Config

PKG MGMT

User Applications

User Applications

Desktop Environment

# COMMON DISTROS
  • Debian Based
    • Includes Ubuntu, Mint, Elementary
    • DEB Packages
  • Redhat Based
    • Includes CentOS (and Clones), Fedora
    • RPM Packages
  • Other Notable
    • Arch Based (Manjaro, Garuda)
    • OpenSUSE
# COMMON Desktop Envrionments

Gnome 3

KDE Plasma Shell

Specialized (i3, etc.)

# File System
# Permissions
  • Three sections of three "types" of permissions
    • r  = read
    • w = write
    • x  = execute
    • -   = None
  • Sections are: Owner, Group, and Other (or World)
  • A fourth prefix section includes
    • - = file
    • d = directory
    • l = link
$  touch file

$  ls -l file
-rw-rw-r-- 1 student student 0 Jan 16 13:58 file

$  chmod o+x file

$  ls -l file
-rw-rw-r-x 1 student student 0 Jan 16 13:58 file
# Permissions
  • Commands
    • chown
    • chmod

{Shell Setup}

Becoming a Nerd

Combining Commands (Pipes)

  • Commands can be combined by using pipes
  • Pipes take the output of one command, and make it the input for another command
  • Pipes use the | character
  • Purpose is to change or modify the output in some way
  • This is extremely useful!
$  command1 | command2 | command3
# Pipes

Redirecting

  • Redirecting is similar to piping, except that either the input, output, or error output can be "sent somewhere else"
  • Useful for:
    • Sending the output of a command to a file
    • Using the contents of a file as input to a command
  • Three Operators
    • > Output - Start from the beginning
    • < Input
    • >> Append
$  echo "Send me to a file" > file.txt

$  echo < file.txt
Send me to a file
# Redirecting

Special Redirecting

  • All files are accessible via "special numbers" (known as file descriptors)
    • stdin    = "0"
    • stdout = "1"
    • stderr  = "2"
  • Useful to send errors to places such as to log errors, or ignore
$  cat some_file.txt 2> err_file.txt
  # Redirect stderr to 'err_file.txt'

$  cat some_file.txt 2>&1
  # Redirect stderr to stdout (stdout still goes to screen)

$  cat some_file.txt 2>&1 > out.txt
  # Redirect stderr to stdout, then stdout to 'out.txt' file

$  cat some_file.txt &> some_file.txt
  # Redirect stderr AND stdout to a file
# Redirecting
  • Nobody remembers all the commands and options!
  • Google!
  • Help Docs exist for most commands
    • $ command --help
    • $ command -h
  • Manpages are good if there is no internet
    • $ man command
# Finding Help
  • quickref.me
  • curl cheat.sh/CMD
  • tldr CMD
  • Completions
  • Suggestions
  • Themes
  • Human Readable Scripting
  • However...
# FISH SHELL
  • Completions
  • Corrections
  • Plugins

Let's do it!

$ sudo dnf install zsh util-linux-user git

$ sudo chsh -s $(which zsh) $USER
# in ~/.zshrc

ENABLE_CORRECTION="true"
COMPLETION_WAITING_DOTS="true"

export PATH=/usr/local/bin:$HOME/.local/bin:$PATH
export SSH_KEY_PATH="~/.ssh/rsa_id"
export TERM="xterm-256color"
fpath+=~/.zfunc
compinit -U
# ZSH SHELL

Oh My ZSH

$ git clone \
  https://github.com/robbyrussell/oh-my-zsh \
  ~/.oh-my-zsh
# in the TOP of ~/.zshrc

export ZSH=$HOME/.oh-my-zsh
ZSH_THEME="robbyrussell"
plugins=()
source $ZSH/oh-my-zsh.sh

But mind how many plugins you use!

# ZSH PLUGINS

Auto Suggestions

$ git clone \
  https://github.com/zsh-users/zsh-autosuggestions \
  ~/.oh-my-zsh/custom/plugins/zsh-autosuggestions
# in ~/.zshrc

plugins=(
  zsh-autosuggestions
)
# ZSH PLUGINS

Syntax Highlighting

$ git clone \
  https://github.com/zsh-users/zsh-syntax-highlighting.git \
  ~/.oh-my-zsh/custom/plugins/zsh-syntax-highlighting
# in ~/.zshrc

plugins=(
  zsh-autosuggestions 
  zsh-syntax-highlighting
)
# ZSH PLUGINS

Abbrev Alias

$ git clone \
  https://github.com/momo-lab/zsh-abbrev-alias \
  ~/.config/zsh-abbrev-alias/
# in ~/.zshrc

export USE_ABBREV_ALIAS=1
source $HOME/.config/zsh-abbrev-alias/abbrev-alias.plugin.zsh

abbrev-alias gl='git log --graph --all --oneline --decorate'

When to use Regular Aliases?

# ZSH PLUGINS

Built In Plugins

# in ~/.zshrc

plugins=(
  zsh-autosuggestions 
  zsh-syntax-highlighting
  sudo
  dnf
  git
)
# ZSH PLUGINS

Custom Prompts - Starship

$ curl -fsSL https://starship.rs/install.sh | bash
# In ~/.zshrc

# Remove ZSH_THEME...

eval "$(starship init zsh)"
# STARSHIP

╭─kevin@shenron in repo: toast on  main [$] is 📦 v0.45.2 via  v1.60.0-nightly took 28ms
╰─➤

FZF - The Fuzzy Finder

$ git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf

$ ~/.fzf/install
# In ~/.zshrc

[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh

Commands

  • CTRL + r
  • CTRL + t
# FZF

Navi - Personal Cheat Sheet

$ curl -sL https://raw.githubusercontent.com/denisidoro/navi/master/scripts/install | sh
# In ~/.zshrc

eval "$(navi widget zsh)"

Commands

  • CTRL + g
# NAVI

Navi Cheats Example

% dnf

# List files included in an RPM
sudo dnf repoquery --installed -l <PKG>

# List files included in an installed RPM
sudo dnf repoquery --installed -l <EXISTING_PKG>

# Disable repository
sudo dnf config-manager --set-disabled <EXISTING_REPO>

# Add repository
sudo dnf config-manager --add-repo <REPOSITORY>

$ EXISTING_REPO: dnf repolist | awk '{print $1}'
$ EXISTING_PKG: rpm -qa | grep -oP '^[^.]*(?=-[0-9])'
# NAVI

{Pre-Scripting}

Advanced Command Line

# SHELL-FU

Variables

  • We can store information in variables
  • These variables are like "buckets", we may not know exactly what is in the bucket, but we can refer to the contents at use
  • Syntax is a dollar sign ($) followed by a word or underscore
    • Convention uses UPPER_SNAKE_CASE
  • Your environment stores many variables
# SHELL-FU

Variable Continued

  • To see current variables in your environment, use the command 'env'
  • Some important variables are:
    • USER   - your current username
    • SHELL - Which shell program you are using
    • PWD    - Your current directory
    • EDITOR - Your text editor
    • PATH    - Where to look for binary executables
# SHELL-FU

Creating and Using Variables

  • Create a variable using the variable name, followed by an equals sign, followed by the "value"
  • Then get the "value" out of the variable by using a dollar sign ($) followed by the variable name
$  ANIMAL=cat

$  echo $ANIMAL
cat

{Substitutions and Expansion}

AKA Magic

# EXPANSIONS
  • Saves typing
  • Reduce errors
  • Two types of expansions
    • globs
    • parameter expansions
$ mv /some/long/path/file.old /some/long/path/file.new

# vs

$ mv /some/long/path/file.{old,new}
# GLOBS

Matches zero or more characters

Matches exactly one character

Matches given characters only

Matches any except given chars

$ ls -l
total 264896
-rw-r--r--  1 kevin kevin 15400455 Jan 17 10:33 09b25a9a5c720.tar.gz
-rw-rw-r--  1 kevin kevin 43487904 Dec  7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r--  1 kevin kevin 27765264 Dec  7 13:01 encryptr_2.0.0-1_amd64.deb
-rw-rw-r--  1 kevin kevin  1407463 Jan 10 14:14 FEL-r22.pdf
-rw-rw-r--  1 kevin kevin   397750 Jan 10 14:14 FEL.tar.gz
-rw-rw-r--  1 kevin kevin 42004318 Dec  7 13:01 haroopad-v0.13.1-x64.deb
-rw-rw-r--  1 kevin kevin 79941824 Dec  7 13:04 keybase_amd64.deb

$ ls -l *.deb
total 264896
-rw-rw-r--  1 kevin kevin 43487904 Dec  7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r--  1 kevin kevin 27765264 Dec  7 13:01 encryptr_2.0.0-1_amd64.deb
-rw-rw-r--  1 kevin kevin 42004318 Dec  7 13:01 haroopad-v0.13.1-x64.deb
-rw-rw-r--  1 kevin kevin 79941824 Dec  7 13:04 keybase_amd64.deb

*

[abc]

In short...wild cards

Special Groups

[[:named:]]

[!abc]

alpha    digit    alnum

lower    upper

punct    space

?

# PARAMETER EXPANSIONS
  • AKA "curly brace expansion"
  • Expands to "lists"
  • Can be combined with globs
$ echo {one,two,three}
one two three

$ ls -l {*.deb,0*}
-rw-r--r-- 1 kevin kevin 15400455 Jan 17 10:33 09b2af7a5c720.tar.gz
-rw-rw-r-- 1 kevin kevin 43487904 Dec  7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r-- 1 kevin kevin 27765264 Dec  7 13:01 encryptr_2.0.0-1_amd64.deb
-rw-rw-r-- 1 kevin kevin 49035482 Dec 19 15:23 google-chrome-stable_current_amd64.deb
-rw-rw-r-- 1 kevin kevin 42004318 Dec  7 13:01 haroopad-v0.13.1-x64.deb
-rw-rw-r-- 1 kevin kevin 79941824 Dec  7 13:04 keybase_amd64.deb
  • Supports basic sequences (similar to 'seq' command)
$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
# COMMAND SUBSTITUTION
  • Allows using the output of a command as the actual input for another command
    • In other words, the output is used inline as if it was directly typed on the command line
$ ls $(echo "-l")
total 264896
-rw-r--r--  1 student student 15400455 Jan 17 10:33 09b2afa5c720.tar.gz
-rw-rw-r--  1 student student 43487904 Dec  7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r--  1 student student 27765264 Dec  7 13:01 encryptr_2.0.0-1_amd64.deb

$ ls -l
total 264896
-rw-r--r--  1 student student 15400455 Jan 17 10:33 09b2afa5c720.tar.gz
-rw-rw-r--  1 student student 43487904 Dec  7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r--  1 student student 27765264 Dec  7 13:01 encryptr_2.0.0-1_amd64.deb
  • Like parameter expansion, but uses a dollar sign (  ) and expands variables
$ ANIMAL=cat

$ echo ${ANIMAL}
cat

$ echo ${ANIMAL}s
cats

$ echo $ANIMALs
# PARAMETER SUBSTITUTION

$

  • Supports internal substitution via forward slashes (   )
  • Supports Default values via 
$ ANIMAL=cat

$ echo ${ANIMAL/at/ow}
cow

$ echo ${ANIMALs:-dog}
dog
$ CAT=tabby

$ ANIMAL=CAT

$ echo ${ANIMAL}
CAT

$ echo ${#ANIMAL}
3

$ echo ${!ANIMAL}
tabby
# PARAMETER SUBSTITUTION

/

:-

#

!

Variable Length

Variable Indirection

Other operators available

  • Using the 'diff' command
$ diff <(ls -l) <(ls -al)    
1c1,3
< total 264896
---
> total 264904
> drwxr-xr-x  3 student student     4096 Jan 17 11:11 .
> drwxr-xr-x 35 student student     4096 Jan 17 12:34 ..

$ diff $(ls -l) $(ls -al)    
diff: invalid option -- '-'
diff: Try 'diff --help' for more information.
  • Very close to command substitution, but subtly different!
  • Process substitution uses <(command) whereas command substitution uses $(command)
  • Allows using the output of a command, but stored in a "temporary file" (/dev/fd/<#>) then that file is passed to the next command
# PROCESS SUBSTITUTION

{Version Control}

Taming the chaos

  • Version control software that tracks changes to files
  • Allows you (or multiple people) to contribute to a single project
  • Allows multiple 'branches' to be worked on at the same time and merged together
  • git is SUPER complex, we will only scratch the surface
# GIT

Basics

  • Enable git tracking on a folder (i.e. project)
$ git init
  • By default you are working on the 'main' branch
  • By default git tracks all files in the folder (if you add new files later use the command git add <files>)
  • Work on files as normal
  • Once you are ready to "save" your code issue the command
$ git commit -a 
  • This "saves" the files and asks you to add a message about what you changed
  • Use detailed messages!
  • To add the message inline use 'git commit -am "message"'
# GIT

Branching

  • Lets say you have a working script, but want to add a new feature or work on a bug, but you don't want to break existing code
  • You can add a new branch
  • This in effect copies your project, and allows you work on both the working code and the 'copy' code at the same time
  • To branch, use
$ git switch my-branch-name
# GIT

Branches (cont.)

  • You switch back and forth between branches at will
$ git switch awesomefeature 
  • Once you have a working feature, or bug fix you can 'merge' the two branches back into one
$ git switch main

$ git merge awesomefeature 
  • When you have no need for a branch anymore you can delete it
$ git branch -D awesomefeature
# GIT
# GIT

github.com (and alternatives)

  • Online git repositories
  • Allows you to save projects and work with others easily
  • Free service requires public repositories (i.e. projects)
  • Can be used for more than just scripting/coding

 

There are self hosted alternatives!

# GIT
# GIT

{Scripting}

Just do this for me...

# SCRIPTING
  • Great for lazy people!
  • Also repetitive tests
  • At it's most basic form, it's a file containing a list of commands to run

Making a Script

  • Sh-Bang
  • Commands
#!/bin/bash

echo "Hello world!"
  • Save the file to ~/.local/bin/script_01.sh
# SCRIPTING

Running a Script

  • We can run a script with      notation
  • Alternatively if the script is located in a directory listed in our                variable, we can just use the script name
$ ./.local/bin/script_01.sh
zsh: permission denied: ./.local/bin/script_01.sh
# SCRIPTING

$PATH

./

Making Scripts Executable

  • If we look at our script, we'll see it's not executable
$ ls -l .local/bin
total 4
-rw-rw-r-- 1 student student 33 Jan 29 10:46 script_01.sh
$ chmod u+x .local/bin/script_01.sh

$ ls -l .local/bin
-rwxrw-r-- 1 student student 33 Jan 29 10:46 script_01.sh

$ ./.local/bin/script_01.sh
Hello world!
# SCRIPTING

Modifying $PATH

  • ./ syntax is fine for one-off scripts, but if this is something we'll want to run many times, it's better to have it inside a director in $PATH
$ export PATH=$PATH:$HOME/.local/bin

$ script_01.sh
Hello world!
  • We can take this a step further by modifying               in our shell startup configuration file
# SCRIPTING

$PATH

Loops

  • Sometimes we want to do a task repeatedly
  • We can use a          loop to do something a set number of times, or over a set of data
  • We'll start by using a for loop on the command line
  • Then move it into a script
  • The basic syntax is
    • for <VAR> in <SET>; do <CMD>; done
$ for NUM in $(seq 1 10); do echo "Hello World! x$NUM"; done
Hello World! x1
Hello World! x2
Hello World! x3
Hello World! x4
Hello World! x5
Hello World! x6
Hello World! x7
Hello World! x8
Hello World! x9
Hello World! x10
# SCRIPTING

for

Loops in Scripts

  • Although we can do complex things in loops on the command line, it quickly becomes hard to read
  • If we move the loop to a script, readability improves
  • The basic syntax becomes
for <VAR> in <SET>;
do
    <CMDS>
done
for NUM in $(seq 1 10);
do
    echo "Hello $NUM times:"
    for _ in $(seq 1 $NUM);
    do
        echo -e "\tHello"
    done
done

We can even nest loops

# SCRIPTING

Making Decisions

  • So far we can only do a set number of tasks in a script
    • It's flexible...but it could be better
  • Now we'll learn how to do a specific task only in some circumstances, and a different task otherwise
  • These types of statements are called conditionals
  • The basic syntax is
if [ <CONDITION> ];
then
    <command if true>
else
    <command if NOT true>
fi
  • The 'else' could even be omitted if we only want to run a command in some circumstances, but nothing in others
# SCRIPTING

Caveat to Conditionals

  • '[' vs '[['
  • [ <condition> ]
    • older
    • more portable
    • supported everywhere
    • less flexible
  • [[ <condition> ]]
    • Newer
    • not as well supported
    • allows for much easier testing
  • My personal opinion, default to `[[ <condition> ]]` unless you have a reason not to
# SCRIPTING

Failing

  • Another form of conditionals is boolean logic
  • Instead of if...else...fi blocks, we can use boolean operators
    •        - means and (i.e. success)
    •        - means or (i.e. failure)
  • The basic syntax is
    • <command1> && <command2>
      • Run command1
      • Run command2 only if command1 succeeds
    • <command1> || <command2>
      • Run command1
      • Run command2 only if command1 fails
  • Can be nested with parenthesis
  • Works great on the command line
  • Very useful to exit scripts early
    • <command that can fail> || (echo "reason" && false)
# SCRIPTING

&&

||

{Python}

Getting friendly with the snake

What is Python?

  • A scripting language (specifically an interpreter)
  • Uses text files (typically ending with        ) to perform specific tasks
  • Each text file is read and executed from top to bottom
    • NOT compiled  
    • ...Buuuuut JIT compiling does happen (.pyc)
  • Must have interpreter installed to run python scripts
# PYTHON

.py

Why use Python?

  • Tasks that are: Repeatable, Mundane, Complex, Precise, Time Consuming, etc.
  • Bridge the end-user technology gap
  • Rapid prototyping of larger projects
  • Brevity
  • Short development time
  • HUGE repository of libraries and modules (i.e. VERY popular)
# PYTHON

Hello World

# in hello.py
print('Hello World!')

Run via

$ python hello.py
Hello world!
# PYTHON

Variables

  • Must start with a letter
  • Typically snake_case_names
  • Use descriptive names when possible
  • Cannot be a keyword
  • Assigned and re-assigned using '='
name = 'something'
age = 29
more_text = a_var
# PYTHON

Types

  • Dynamic Types
  • Common Types
    • strings
    • numbers (int, float, etc.)
    • characters
    • bools
    • lists, dicts, sets, and tuples
    • objects
    • None
num1 = 2
num2 = 2
both = num1 + num2
print('Adding: ', both)
both = num1 - num2
print('Subtracting: ', both)
both = num1 / num2
print('Dividing: ', both)
both = num1 * num2
print('Multiplying: ', both)
both = num1 % num2
print('Modding: ', both)
text1 = '2'
text2 = '2'
both = text1 + text2
print('String and String: ', both)
fl1 = 2.2
int1 = 2
both = fl1 + int1
print('Float and Int: ', both)
fl2 = 2.2
both = fl1 + fl2
print('Float and Float: ', both)
# PYTHON

Strings

  • In Python 3.x all strings are a sequence of valid UTF-8 Unicode characters
  • ASCII vs UTF-8
    • 1 byte vs maybe several
    • ASCII byte strings in Python 3 (str type, or b'literal')
  • Hidden characters in strings
    • Escape character is \ (i.e. 'string with\'quotes\'')
  • Common hidden characters
    • \n
    • \t
    • \c\r
  • Strings are simply lists of characters and can be sliced (more on that later...)
# PYTHON

Lists

  • An unordered sequence of values
  • Values do not need to be same type
  • Formed using the [ and ] characters, separating values by , character
  • Can be indexed via an integer
    • Starts at 0!!
    • A negative integer means, "From the end"
my_list = ['Kevin', 'Knapp', 29]
print(my_list[0])
print(my_list[1])
print(my_list[2])
# PYTHON

List Slicing

  • Lists can be sliced to refer to a specific section of the list using the [:] characters and 0, 1, or 2 numbers
  • Using 0 numbers refers the to whole list
my_list[:]
  • The number before the : is the start index
  • The number after the : is the end index
  • Omitting either says, "Everything till the end, or from the beginning" respectivly
from_start_to_index_5[:5]
from_index_2_to_end[2:]
from_index_1_to_4[1:4]
  • A negative number for either start or end, simply reverses the index (i.e. from start or from end)
  • Start must ALWAYS be below End
# PYTHON

dict

  • A.k.a. hash-map or a key-value list
  • Keys do not need to be same type as value
  • Literal uses {, :, and }
  • Keys MUST not conflict
  • Values accesses via [ and ] with the key
person = {'first': 'Kevin',
          'last': 'Knapp',
          'age': 29 }
my_age = person['age']
# PYTHON

Set

  • An unordered list of UNIQUE items
  • Literals use { and }
  • Cannot be access via [ and ], but can be iterated
  • A good use is to remove duplicates from other collections
my_set = set()
my_set.add('kevin')
# PYTHON

Conditionals

  • Sometimes you want to take different actions based on certain conditions
  • Python provides if/else statements
  • Conditions use Boolean logic to determine which path to take
if some_expression_or_variable:
    # Actions take if TRUE
else:
    # Actions to take if FALSE
  • Python also provides multi-condition statements by adding elif
if some_expression_or_variable:
    # Actions take if TRUE
elif some_new_test:
    # Actions to take if TRUE
else:
    # Actions to take if all tests are FALSE
# PYTHON

Loops

  • Sometimes you need to do a specific operation multiple times
  • Python provides loops for these occasions
  • There are two types of loops in Python
    • for
    • while
# PYTHON

for Loop

  • Used when you need to do something a specific number of times, or for an entire collection of values
  • The syntax is
for value in some_collection:
    # do something
  • For example, to print every item in a list
my_list = ['item 1', 'item 2', 'item 3', 'item 4']
for item in my_list:
    print(item)
# PYTHON

for Loop Cont.

  • If you want to do something a specific number of times you can use the range() function (Functions will be covered later)
# say 'Hello' 10 times
for _ in range(10):
    print('Hello')
  • Notice we used the '_' character, that's because we didn't need the 'item' variable (in this case it would be a number). Alternatively if we wanted to use it to count we could do
# Count to 10
for num in range(1,11):
    print(num)
    print(',')

Note the start and end values passed to range

# PYTHON

Functions

  • Functions start with the def keyword, then the name, then parenthesis containing any optional arguments
  • Arguments are positional and mandatory (with one exception, more on this later...)
  • If functions wish to send the results back, they use the return keyword followed by the results variable or literal
# A function that takes 2 arguments and doesn't
# return a result
def my_function(some_arg, other_arg):
    # do some work here...

# A function that takes 0 arguments and
# returns a result
def my_other_function():
    # do some more work...
    return True
# PYTHON

Keyword Arguments

  • Keyword Arguments (a.k.a. kwargs/kargs) allow you to have optional arguments to a function or method (along with mixing the order)
def cool_function(name='kevin', age=29):
    # do stuff here
    return (name, age)

# use the function
n, a = cool_function(age=50)
print('name={}, age={}'.format(n,a))
  • kwargs can be mixed with positional args too
# PYTHON

Testing with a REPL

  • Read Evaluate Print Loop
  • Python's REPL is called IDLE
  • Accessed via 'python' command
    • To stop, type 'quit()'
  • Most python modules are available, plus some extras (namely dir() )
  • Sessions non-persistent
  • Great for testing (and short term memory loss, aka 'getting old')
# PYTHON

{Ansible}

Controlling a Fleet

Ansible

  • Made by Red Hat
  • Automation via "scripting light"
  • Playbooks made up of Plays
---
- name: apply non-kernel updates
  hosts: "{{ HOSTS | default('web') }}"
  become: true
  gather_facts: false

  tasks:
  - name: upgrade all packages except kernel
    yum:
      name: '*'
      state: latest
      exclude: kernel*
    tags: all

  - name: upgrade all packages security related except kernel
    yum:
      name: '*'
      state: latest
      security: true
      exclude: kernel*
    tags: security
  • Written in YAML
  • Built-in Modules
  • Inventories!
  • Under the hood its a lot of Python
  • Can make your own modules in Python
# ANSIBLE

Inventories

---
all:
  children:
    controller:
      hosts:
        localhost:
          ansible_connection: local
    user_nodes:
      hosts:
        user01:
          ansible_host: 192.168.1.20
        user02: 
          ansible_host: 192.168.1.21
        user03: 
          ansible_host: 192.168.1.22
    database_server:
      hosts:
        db01:
        db02:
        db03:
    projext_x:
      children:
        user_nodes:
        database_server:
  • Written in YAML
  • Can group nodes
  • Groups can have children
  • Can use IPs or Hostnames
# ANSIBLE

Running Playbooks

$ ansible-playbook -i inventory_file playbook.yml
# ANSIBLE

{Day 02}

Containers, Containers, Containers

{Containers}

Buzzword Bingo

# What is a Container
  • Software bundled with all dependencies
    • Sort of like a .ZIP with with special permissions
  • Has it's own "view" of the world (aka Namespace)
    • Process Tree
    • Network Stack
    • Filesystem / mount points
    • Resource Limits (CPU, RAM, etc)
    • UID / GID
  • "Image" vs "Container"
  • "Container" vs "VM"
# What is a Container

Hardware

Linux Kernel

GNU Core Utilities

System Config

PKG MGMT

App

Container

VM

GNU Core Utilities

System Config

PKG MGMT

App

Linux Kernel

GNU Core Utilities

System Config

PKG MGMT

App

Virtual

Hardware

# What is a Container

Hardware

Linux Kernel

GNU Core Utilities

System Config

PKG MGMT

App

Container

VM

App

Linux Kernel

GNU Core Utilities

System Config

PKG MGMT

App

Virtual

Hardware

# Container Software
  • Kernel Built In
    • LXC
    • Namespaces
    • CGroups

 

  • User Facing
    • Docker
    • Podman

 

  • Orchestration Layer
    • Kubernetes
    • Nomad
    • Rancher
# Container Software

Install Docker

$ curl -LSs https://get.docker.com | sh
$ sudo usermod -aG docker $USER
$ sudo systemctl enable --now docker

Install Podman

$ sudo dnf install podman
# Container Software

Testing It Out

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

$ podman run hello-world
 
 [...]
# Container Software

Fedora Toolbox

$ sudo dnf install toolbox

$ toolbox enter

Creates an overlay system with a shared home directory

 

Great for testing and development!

# docker-compose
  • Allows cooperative containers
  • Helps sharing resources such as mounts or networks
  • Declarative YAML format file
  • Will eventually be integrated into the actual docker command line tool
version: "2"

services:
  bookstack:
    image: lscr.io/linuxserver/bookstack
    container_name: bookstack
    environment:
      - APP_URL=http://localhost/
      - DB_HOST=bookstack_db
      - DB_USER=bookstackuser
      - DB_PASS=bookstackpass
      - DB_DATABASE=bookstackapp
      - PUID=996
      - PGID=994
    ports:
      - 6875:80
    depends_on:
      - bookstack_db
    networks:
      - bookstack-tier
    restart: unless-stopped
    volumes:
      - ./data:/config


  bookstack_db:
    image: lscr.io/linuxserver/mariadb
    container_name: bookstack_db
    networks:
      - bookstack-tier
    environment:
      - MYSQL_ROOT_PASSWORD=bookstackdbrootpass
      - TZ=America/New_York
      - MYSQL_DATABASE=bookstackapp
      - MYSQL_USER=bookstackuser
      - MYSQL_PASSWORD=bookstackpass
      - PUID=996
      - PGID=994
    restart: unless-stopped
    volumes:
      - ./data:/config

networks:
  bookstack-tier:
    driver: bridge
$ sudo dnf install docker-compose
$ sudo dnf install podman-compose

{Container Alternatives}

VMs, and alternative tools

# VAGRANT
$ sudo dnf install vagrant
$ vagrant init hashicorp/bionic64
$ vagrant up
  Bringing machine 'default' up with 'virtualbox' provider...
  ==> default: Importing base box 'hashicorp/bionic64'...
  ==> default: Forwarding ports...
  default: 22 (guest)
  => 2222 (host) (adapter 1)
  ==> default: Waiting for machine to boot...
 
$ vagrant ssh
  vagrant@bionic64:~$ _
  • Allows for creating VMs from Code
  • Under the hood uses Virtual Box or other hypervisors
  • Each VM is configured with Ruby scripts
  • Can then provision VMs using inline shell, or dedicated shell scripts
  • EXCELLENT for test environments!
# VAGRANTFILE
Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"
  config.vm.synced_folder ".", "/vagrant", disabled: true

  # StandAlone Server Only
  config.vm.define "server2" do |server2|
    server2.vm.network "private_network", ip: "192.168.42.20",
      virtualbox__intnet: "thnet"

    server2.vm.provider "virtualbox" do |vb|
      vb.name = "server2"
      vb.memory = "1024"
    end

    server2.vm.provision :shell, path: "./bootstrap_server.sh"
  end

  # Sensor to Server/Sensor
  config.vm.define "sensor1" do |sensor1|
    sensor1.vm.network "private_network", ip: "192.168.42.11",
      virtualbox__intnet: "thnet"

    sensor1.vm.provider "virtualbox" do |vb|
      vb.name = "sensor1"
      vb.memory = "1024"
    end

    sensor1.vm.provision :shell, path: "./bootstrap_sensor.sh"
  end

  # APT Repo
  config.vm.define "repo" do |repo|
    repo.vm.network "private_network", ip: "192.168.42.2",
      virtualbox__intnet: "thnet"

    repo.vm.provider "virtualbox" do |vb|
      vb.name = "repo"
      vb.memory = "1024"
    end

    repo.vm.provision :file, source: '../pkgs/deps.tgz', destination: "/home/vagrant/"
    repo.vm.provision :shell, path: "../bootstrap_repo.sh"
  end
end
# LXD
  • Made by Canonical
  • Orchestrate containers
    • Can also do VMs with --vm
  • Easy snapshot and restore
$ sudo snap install lxd
$ sudo lxd init
$ lxc launch ubuntu:20.04 [--vm]
# MULTIPASS
$ sudo snap install multipass
$ multipass launch
  • Made by Canonical
  • Easy VMs from the command line

{Custom Images}

Have it your way

# DOCKERFILE
# Dockerfile
FROM alpine:latest
RUN apk add python3
ENTRYPOINT ["python3", "-m", "http.server"]
$ docker build -t pyserv .
 
$ docker image ls
REPOSITORY    TAG           IMAGE ID       CREATED         SIZE
pyserv        latest        fec6935a627e   3 minutes ago   54MB
  • Declarative document for building container images
  • Should be checked in to version control

Each directive creates a "layer" which is built upon (and can lead to huge sizes!)

# DOCKERFILE LINTS
$ docker run --rm -i hadolint/hadolint < Dockerfile

Unable to find image 'hadolint/hadolint:latest' locally
latest: Pulling from hadolint/hadolint
d0727728368a: Pull complete
Digest: sha256:1b47f3192d2c69c622863b84d5cccbfa0c1fac9022bfc297aa931e6d44e51608
Status: Downloaded newer image for hadolint/hadolint:latest
-:1 DL3007 warning: Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag
-:2 DL3018 warning: Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`
-:2 DL3019 info: Use the `--no-cache` switch to avoid the need to use `--update` and remove `/var/cache/apk/*` when done installing packages

Lint your Dockerfiles!

https://github.com/hadolint/hadolint

# Container Registries
  • Repositories (public and private) for container images
  • Allows easily sharing and pulling images
  • Public Registries
    • Docker Hub
    • Quay.io
    • AWS
    • Others...
  • GitLab contains per-repository Registries
# Container Registries
  • Can also export containers, or save images to TAR files
  • These can then be loaded on another system
$ docker save pyserv -o pyserv.tar

$ ls -l
   inode Permissions Links Size Blocks User  Group Date Modified Name
62391693 .rw-r--r--@     1   83      8 kevin kevin 15 Jan 14:48  Dockerfile
62391694 .rw-------@     1 54Mi 110264 kevin kevin 15 Jan 14:57  pyserv.tar

$ tar tf pyserv.tar
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/VERSION
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/json
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/layer.tar
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/VERSION
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/json
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/layer.tar
fec6935a627eb73a17b0269f95aa5cda4b1c5d70c032c87e333d7827ce1d4154.json
manifest.json
repositories
# Working with Images

Tips for working with Container Images and Layers

$ curl -OL \
  https://github.com/wagoodman/dive/releases/download/v0.9.2/dive_0.9.2_linux_amd64.rpm
$ sudo dnf install ./dive*.rpm

https://github.com/wagoodman/dive

{Recipes}

Scripting...lite

github.com/casey/just

# JUST
$ curl \
  --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | \
  bash -s -- --to ~/.local/bin/

github.com/stepchowfun/toast

# TOAST
image: ubuntu
tasks:
  install_figlet:
    command: |
      apt-get update
      apt-get install --yes figlet

  greet:
    dependencies:
      - install_figlet 
    command: figlet 'Hello, World!'
$ curl https://raw.githubusercontent.com/stepchowfun/toast/main/install.sh -LSfs | sh

{Day 03}

Monitoring, Remote Admin, and Hardening

{Monitoring}

Watching everything

# MONITORING
  • Known Requirements
  • Ephemeral Data
  • Useful in Troubleshooting
  • May not be useful long term

Live

Historical

  • Long term storage
  • Can "go back in time" to visualize events
  • May be raw/unprocessed data

Actioned

  • Actions taken on events
  • AKA Alerting
  • May spawn more logging
  • Consider attribution!

Built In-(ish) Monitoring

# MONITORING
  • The tops (top, memtop, ptop, sntop, vtop, htop)
    • + Modern Alternatives
  • ps, free, du (and family), df, lsblk, watch
  • systemd
  • network monitoring utilities
    • Basic tools can be used to view the state of the system (ip, ifconfig, netstat, lsof, iptables, tcpdump, ping, nslookup, arp, traceroute/mtr)
    • nethogs

Consider the difference between local node monitoring, and "entire system" monitoring

# CONTAINER MONITORING
$ docker stats
CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT    MEM %     NET I/O           BLOCK I/O         PIDS
036b8ad6c7c2   tak_takserver_1           1.82%     5.313GiB / 31.4GiB   16.92%    444MB / 2.08GB    0B / 1.13MB       368
a3c79a328776   tak_pg_1                  0.11%     259.1MiB / 31.4GiB   0.81%     927MB / 183MB     16.4kB / 2.96GB   10
3a36813ff86c   tak_openfire_1            0.12%     678.7MiB / 31.4GiB   2.11%     9.01MB / 12.2MB   0B / 852kB        64
b360b0eee623   metrics_node-exporter_1   0.00%     21.22MiB / 31.4GiB   0.07%     0B / 0B           17.9MB / 0B       19
73be46dd33fa   metrics_cadvisor_1        6.09%     202MiB / 31.4GiB     0.63%     155MB / 8.29GB    31.4MB / 0B       42

docker stats

https://github.com/jesseduffield/lazydocker

# PROMETHEUS
  • Time Series Database
  • Primarily a "Pull/Scrape" Methodology
    • Does support a "Push Gateway"
  • Metrics are provided by the logged application via "Exporters"
  • Uses a Query Language called PromQL
  • Great for ad-hoc queries and basic visualization
# Graphana
  • The visualization tool of choice
  • Works amazing with Prometheus (and many other databases too)
  • Has a decent learning curve
  • Can download/use pre-made dashboards
  • Many Prometheus Exporters come with Graphana dashboards
  • Key Dashboards / Exporters
    • node_exporter
    • cAdvisor
# Graphana - NODE_EXPORTER
# Graphana - CADVISOR

{Private Services}

Remote Services...but private

# CONNECTIVITY
  • Internal vs. External Connectivity
  • Common
    • SSH (Derivatives; Mosh, Auto-SSH, sshuttle etc.)
    • wstunnel, stunnel, OBFSProxy
    • Layer 3 VPNs
      • IPSec
      • OpenVPN
      • Tinc
      • Wireguard (Derivatives; Nebula, Tailscale, Innernet, etc.)
    •  Layer 2 VPNs
      • ZeroTier
# WIREGUARD
  • In Kernel (userspace implementations and kmods exist too)
  • Noise Protocol
  • Simple (vs. easy)
  • ChaCha20Poly1305
    • Performance
    • Efficiency (i.e. battery life)
  • NOT Configurable
  • Low Level Primitive
  • Possibly Quantum Resistant with Optional PSK
# WIREGUARD
$ sudo dnf install wireguard-tools
$ sudo mkdir /etc/wireguard
$ cd /etc/wireguard
$ sudo wg genkey | tee hub.private | wg pubkey > hub.public
$ sudo wg genkey | tee client.private | wg pubkey > client.public
# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = 2DiO+Ox5cURGbFLWniVmx/N1laqYGIZpv/IPDuiYkVg=
Address = 10.10.10.1/24
ListenPort = 51820
DNS = 1.1.1.1

# Client
[Peer]
PublicKey = hrbwZZxxjVuP7QvlWm9YDyg+40ftgQo5mKRrWmekpFw=
AllowedIPs = 10.10.10.2/32
PersistentKeepalive = 25
$ sudo systemctl enable --now wg-quick@wg0
# SSH CONFIG FILES
Host myserver
  HostName 51.127.128.14
  User kevink
  IdentityFile ~/.ssh/roshi
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentitiesOnly yes
  LogLevel FATAL
  Compression yes
  Port 22
  • Use them!
  • Makes SSH (and SCP) a breeze to use
  • Stored in ~/.ssh/config
  • Supports Globs and ! syntax
  • Can be overridden with

-F FILE

-o "option=value"

# SSH TUNNELS
  • Can be used to forward *local* traffic to a remote server (extremely useful for remote dashboards)
  • Can be used to forward local through *through* a remote server (i.e. a poor man's VPN)
$ ssh user@host -L 8080:127.0.0.1:3000 -N
  •      can be specified multiple times
$ pip3 install --upgrade --user sshuttle
$ sshuttle -r user@host 0.0.0.0/0
  •      can be used to exclude subnets

-x

-L

{Easy Mode Remote Admin}

Enough of this terminal...

# COCKPIT
  • A Web GUI for your servers
  • Made by Redhat
  • Works on Ubuntu based distros as well
  • Additional plugins control things like Containers, VMs, etc.
  • Can be used for local machine too!
$ sudo dnf install cockpit
$ sudo systemctl enable --now cockpit cockpit.socket
$ sudo firewall-cmd --permanent --zone=public --add-service=cockpit
$ sudo firewall-cmd --reload

{Hardening}

Make 'em work for it

# HARDENING
  • Shotgun Blast vs Targeted
  • Whats the goal: Meeting a policy?
  • Consider a Zero Trust Model
  • Types of hardening, and from whom:
    • Network
    • System
    • Application
  • Schedule Red Team Assessments (even if only internal)

The Most Low Hanging Fruit Will Be the Exploit!

The defense must be correct 100% of the time, but the offense only needs to be correct once!

Many issues boil down to permissions

# SSH HARDENING

If you do nothing else - Do this

PermitRootLogin prohibit-password # or 'no' even better
PasswordAuthentication yes # Normally this should be 'no'
PrintLastLog yes
PermitEmptyPasswords no
MaxAuthTries 3
ClientAliveInterval 200
ClientAliveCountMax 3
MaxSessions 3
HostbasedAuthentication no

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr

Macs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
$ sudo chown root:root /etc/ssh/sshd_config
$ sudo chmod 0600 /etc/ssh/sshd_config
$ sudo systemctl restart sshd
# RUST-MOTD

github.com/rust-motd/rust-motd

  • Configure via: $HOME/.config/rust-motd/config.toml

[service_status]
docker = "docker"
gitlab-runner = "gitlab-runner"

[uptime]
prefix = "Up"

[filesystems]
root = "/"

[fail_2_ban]
jails = ["sshd", "anotherjail"]

[last_login]
kevin = 2
root = 3

[last_run]
# SYSTEMD SERVICE HARDENING

We can harden on a per-service basis

This should be the first thing you reach for!

$ sudo systemd-analyze security
UNIT             EXPOSURE PREDICATE
atd.service           9.6 UNSAFE
crond.service         9.6 UNSAFE
httpd.service         9.2 UNSAFE
mariadb.service       8.8 EXPOSED
sshd.service          9.6 UNSAFE

$ sudo systemd-analyze security httpd.service
  NAME                   DESCRIPTION                                          EXPOSURE
✗ PrivateDevices         Service potentially has access to hardware devices        0.2
✓ PrivateTmp             Service has no access to other software's temporary files
✗ ProtectControlGroups   Service may modify to the control group file system       0.2
✗ ProtectHome            Service has full access to home directories               0.2
✗ ProtectKernelTunables  Service may alter kernel tunables                         0.2
✗ ProtectSystem          Service has full access to the OS file hierarchy          0.2
✗ RestrictSUIDSGID       Service may create SUID/SGID files                        0.2
✗ RootDirectory          Service runs within the host's root directory             0.1
…
→ Overall exposure level for httpd.service: 9.2 UNSAFE
# FAIL2BAN

Ban IPs based on failed login attempts

There are times fail2ban isn't as useful as you'd think

$ sudo dnf install fail2ban

$ sudo cp /etc/fail2ban/fail2ban.{conf,local}
$ sudo cp /etc/fail2ban/jail.{conf,local}

$ sudo systemctl enable --now fail2ban

$ sudo tail -f /var/log/fail2ban.log

To see "jailed" IPs

$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	0
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	0
   |- Total banned:	0
   `- Banned IP list:	

# unban an IP
$ sudo fail2ban-client set sshd unbanip 1.2.3.4
# CROWDSEC

github.com/crowdsecurity/crowdsec

"CrowdSec is a free, modern & collaborative behavior detection engine, coupled with a global IP reputation network"

  • Similar to a distributed fail2ban
  • Requires a detection engine + a "bouncer"
# HARDENING

CentOS Hardening will be covered in Day 04 via OpenSCAP

# UBUNTU-HARDENING

github.com/konstruktoid/hardening

$ git clone https://github.com/konstruktoid/hardening
$ cd hardening
$ sed -i "s/CHANGEME=''/CHANGEME=foo" ubuntu.cfg
$ sudo bash ubuntu.sh

Primarily only good for Ubuntu based systems, however some of the internal scripts are applicable across many distros

{SELinux}

Oh no...

# SELINUX
  • One of two popular Linux Security Module (LSM) (a third is being built)
    • AppArmor (Ubuntu) and Sandbox are the other two
  • Originally developed by NSA
  • Now maintained by Red Hat and the default LSM on RHEL based distros
  • A reputation for being extremely complex
  • Most people unfortunately just disable it

Three Primary Modes:

  • Enforcing
  • Permissive
  • Disabled
# SELINUX

Configuration File: /etc/selinux/config

# Get the current mode
$ sudo getenforce
$ sudo sestatus

# Temporarily Change Modes
$ sudo setenforce [enforcing|permissive|1|0]

Common Commands:

  • getenforce | setenforce
  • semanage [boolean|port|fcontext] [-l]
  • ausearch
  • chcon | restorecon | matchpathcon
  • semodule
# SELINUX CONTEXT LABELS
  • Uses a MAC pattern vs traditional Unix DAC pattern
  • Fundamentally all files contain "labels" which consist of contexts in the form of
    • USER : ROLE : TYPE : LEVEL
    • suffixed _u, _r, _t (and prefix s)
  • Adding -Z to a command will display SELinux information, such as with ls or ps
$ ps -eZ
LABEL                               PID TTY          TIME CMD
system_u:system_r:init_t:s0           1 ?        00:00:14 systemd
system_u:system_r:kernel_t:s0         2 ?        00:00:00 kthreadd
system_u:system_r:httpd_t:s0      25134 ?        00:00:00 httpd

restorecon PATH will set the file context to whatever the system configuration specifies (view or change with semanage fcontext [--list])

# SELINUX FCONTEXT

restorecon PATH will set the file context to whatever the system configuration specifies (view or change with semanage fcontext [--list])

$ sudo semanage fcontext -l
SELinux fcontext                    type               Context

/home                               directory          system_u:object_r:home_root_t:s0
/home                               symbolic link      system_u:object_r:home_root_t:s0
/home/(.*/)?\.snapshots(/.*)?       all files     

$ sudo semanage fcontext -a -t unconfined_u:object_r:user_home_t:s0 /home
$ sudo restorecon -v -r /home

Caution

files retain the context they had previously
files *may not* retain the context they had previously
files inherit the context of the parent directory

MOVED

COPIED

CREATED

# SELINUX BOOLEANS

Booleans are on/off switches

  • List with semanage boolean --list
$ sudo semanage boolean -l
SELinux boolean                State  Default Description
  ...
xend_run_blktap                (on   ,   on)  Allow xend to run blktap
xend_run_qemu                  (on   ,   on)  Allow xend to run qemu
xguest_connect_network         (on   ,   on)  Allow xguest to connect network
xguest_exec_content            (on   ,   on)  Allow xguest to exec content
xguest_mount_media             (on   ,   on)  Allow xguest to mount media
  ...
  
$ setsbool [-P] xguest_connect_network off

Means permanent

-P

# SELINUX PORTS

The Firewall **AND** SELinux Control Network Access

  • List with semanage port --list
$ sudo semanage port -l
SELinux Port Type              Proto    Port Number

afs3_callback_port_t           tcp      7001
afs_fs_port_t                  udp      7000, 7005
afs_ka_port_t                  udp      7004
amanda_port_t                  tcp      10080-10083
amqp_port_t                    tcp      15672, 5671-5672
amqp_port_t                    udp      5671-5672
asterisk_port_t                udp      2427, 2727, 4569
  
$ sudo semanage port -a -p udp -t asterisk_port_t 8989

Means ADD

-a

Means DELETE

-d

Means MODIFY

-m

# SELINUX TROUBLESHOOTING
  • ausearch is most helpful in displaying errors
$ sudo ausearch -m avc -ts recent
----
time->Tue Dec 21 10:15:53 2021
type=AVC msg=audit(1640099753.877:245): avc:  denied  { write } for  pid=2368 comm="flatpak" name="system_bus_socket" dev="tmpfs" ino=2513 scontext=system_u:system_r:fedoratp_t:s0-s0:c0.c1023 tcontext=system_u:object_r:system_dbusd_var_run_t:s0 tclass=sock_file permissive=0
----
time->Tue Dec 21 10:15:53 2021
type=AVC msg=audit(1640099753.893:246): avc:  denied  { write } for  pid=2370 comm="flatpak" name="system_bus_socket" dev="tmpfs" ino=2513 scontext=system_u:system_r:fedoratp_t:s0-s0:c0.c1023 tcontext=system_u:object_r:system_dbusd_var_run_t:s0 tclass=sock_file permissive=0
----
  • Narrow down to a specific component (comm= above) with -c
  • Other valid time values: this-year, this-month, week-ago, yesterday, today, boot, now
  • audit2allow can be used to create a custom policy (and loaded via semodule)
# SELINUX

Instead of setting the entire system to permissive, consider changing just the problem domain

$ ps -eZ
LABEL                               PID TTY          TIME CMD
system_u:system_r:init_t:s0           1 ?        00:00:14 systemd
system_u:system_r:kernel_t:s0         2 ?        00:00:00 kthreadd
system_u:system_r:httpd_t:s0      25134 ?        00:00:00 httpd

$ sudo semanage permissive -a httpd_t

Means ADD

-a

Means DELETE

-d

Means MODIFY

-m

{Firewalls}

Not in my house

# IPTABLES
  • Rule based firewalls
  • Rules are designated for different "chains" which are part of different "tables"
  • Each packet is filtered through these chains of rules, one after the other
  • At each rule, the kernel decides whether to perform an action (such as drop) on the packet if it matches the rule, or let it continue processing
    • The action may be something simple like, "log and continue processing" or "allow"
  • The packet is sent through these chains of rules and through the different tables until either an action is performed on the packet that drops it, or ultimately the default rule is applied (i.e. it makes it all the way through all rules of all chains of all tables)
  • Packets can match multiple rules, but are processed in order
  • Let's visualize this...
# IPTABLES

Chains

  • In it's simplest form (the default table known as the FILTER table), there are three chains in order of processing
    • INPUT: Packets coming in
    • FORWARD: Packets being forwarded
    • OUTPUT: Packets going out
  • When we talk about tables, different tables may have different chains
# IPTABLES

 Tables

  • Although you'll primarily use the default table, there is one other table you should know
    • nat
  • The nat table includes the following four chains in order of processing:
    • PREROUTING
    • INPUT
    • OUTPUT
    • POSTROUTING
# IPTABLES

Rule Syntax

  • The first thing you should know: iptables -L
    • Lists the chains for the default table
    • The table can be changed with -t <TABLE>
  • One can "Flush" or reset the rules of a table with -F
    • iptables -F
    • iptables -t nat -F
  • Before we spoke about the default policy, which is if the rule makes it to the end of a chain, what should happen.
    • By default the policy is ALLOW for all chains
    • You can change the policy of a particular chain with -P
      • iptables -P INPUT DROP
# IPTABLES

Policy

  • When changing the default policy, ESPECIALLY of a remote box, be sure not to cut your current connection
  • By default iptables rules are non-persistent and will be reset on reboot
  • We can save and restore rulesets with
    • iptables-save > /tmp/iptables-v4
    • iptables-restore < /tmp/iptables-v4
  • There are packages which will do this for us automatically at boot
    • iptables-persistent
      • To save (which will auto restore at next boot):
        • dpkg-reconfigure iptables-persistent
# IPTABLES

Modifying Chains

  • Rules can be modified (appended, inserted, or deleted) on chains with these various commands
Command Function
iptables -A <CHAIN> "Append" a rule at the bottom of <CHAIN>
tcpdump -I <CHAIN> <NUM> Insert a rule on <CHAIN> and rule number <NUM>
iptables -D <CHAIN> <NUM> Delete rule number <NUM> from <CHAIN>
  • TIP: it can be helpful to use the `--line-numbers` option which will display the line numbers of the rules for easier inserting/deleting
# IPTABLES

Rules

  • Now that we know how to view tables and chains, and list or modify the current rules, we actually learn about the rules themselves
  • Most rules are similar to using tcpdump, you can specify "filters" to match packets
  • There are various flags and switches which modify or add these filters to the rule
  • Some flags/switches enable other flags and switches
    • These are known as "match extensions"
# IPTABLES

Rules

Option / Flag Function
-j <ACTION> What to do with the packet. Normally:
  * ACCEPT
  * LOG
  * DROP
  * MASQUERADE
-p [!] <PROTO> Match a particular protocol like 'tcp' or 'udp'
-i <IFACE> Match only on input interface <IFACE>. If '!' is included it reverses the rule
-d [!] <IP>[/cidr] Destination IP/network. If '!' is included it reverses the rule
-s [!] <IP> Source IP/network. If '!' is included it reverses the rule
-o <IFACE> Only match output interface <IFACE>
-m <NAME> Performs a "match" matches can be thought of as "modules" and enable other "matching options/flags"
More on this later...
--dport <NUM> Destination port
--sport <NUM> Source port

Common Rule options/switches

# IPTABLES

Matches

Option / Flag Function
-m conntrack Adds a --ctstate <STATE> flag where you can match connection states
  * ESTABLISHED
  * RELATED
  * NEW
  * INVALID
Can be comma separated like ESTABLISHED,RELATED
Also adds --ctproto
-m comment Allows adding comments with --comment "some comment"
-m multiport Allows matching on more than one port with --dports <port1,port2>
-m iprange Allows --src-range and --dst-range for a range of IPs
-m log Adds --log-level <#> to only log at specified logging levels
-m udp OR -m tcp Adds --sport and --dport that will match only TCP or UDP ports
  • Common match names (modules)

List of all extensions: http://ipset.netfilter.org/iptables-extensions.man.html

# IPTABLES

Tips

  • When working with iptables open two terminals, or a split screen
  • In one screen run the iptables commands
  • In the other run:
    • watch -n 1 iptables -L --line-numbers
# IPTABLES
# IPTABLES
*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT
-N port-scanning
-N syn_flood
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp --syn -j syn_flood
-A INPUT -p icmp -m limit --limit 1/sec --limit-burst 1 -j ACCEPT
-A INPUT -p icmp -m limit --limit 1/sec --limit-burst 1 -j LOG --log-prefix PING-DROP
-A INPUT -p icmp -j DROP
-A INPUT -f -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A INPUT -p tcp -m connlimit --connlimit-above 80 --connlimit-mask 32 --connlimit-saddr -j REJECT --reject-with tcp-reset
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -p icmp -j ACCEPT
-A port-scanning -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -m limit --limit 1/sec --limit-burst 2 -j RETURN
-A port-scanning -j DROP
-A syn_flood -m limit --limit 1/sec --limit-burst 3 -j RETURN
-A syn_flood -j DROP
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
COMMIT
*mangle
:PREROUTING ACCEPT
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT
:POSTROUTING ACCEPT
-A PREROUTING -m conntrack --ctstate INVALID -j DROP
-A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,RST FIN,RST -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,ACK FIN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags ACK,URG URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,ACK FIN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags PSH,ACK PSH -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,PSH,URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,ACK,URG -j DROP
COMMIT
# IPTABLES
[Unit]
Description=Packet Filtering Framework
After=network-online.target
Requires=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables-restore /etc/ipv4.rules
ExecReload=/sbin/iptables-restore /etc/ipv4.rules
ExecStop=/sbin/iptables -P INPUT ACCEPT
ExecStop=/sbin/iptables -P FORWARD ACCEPT
ExecStop=/sbin/iptables -F

[Install]
WantedBy=multi-user.target
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now iptables.service

Enable IP Forwarding

  • In order to use the FORWARD chains or any nat'ing you'll have to enable IP forwarding in the kernel
  • To check if forwarding is already enabled
    • $ cat /proc/sys/net/ipv4/ip_forward
    • if you see a 0 fowarding is NOT enabled
    • if you see a 1 forwarding is enabled
  • To enable forwarding use
    • # echo 1 > /proc/sys/net/ipv4/ip_forward
# IPTABLES

The Uncomplicated FireWall

  • Default in Ubuntu based distros
  • Uses iptables under the hood
  • Sometimes deceptively "simple"
# UFW
$ sudo ufw status
ufw: active

$ sudo ufw allow ssh/tcp
$ sudo ufw deny 8888/udp
$ sudo ufw limit 80/tcp
$ sudo ufw deny from 192.168.1.100/24 on eth0 proto any
  • Default in RHEL based distros
  • Uses nftables under the hood by default (can be switched to iptables)
  • MUCH easier (but larger!) CLI
  • Still a complex system due to trying to paper over a complex system
  • Focuses on "zones"
  • Can be controlled via XML files on the system...(blech)
  • Ephemeral!
    • Don't forget --reload or --permanent 
  • Supports raw iptables syntax too
# FIREWALLD
$ sudo firewall-cmd --state
$ sudo firewall-cmd --get-*
$ sudo firewall-cmd --zone=internal --add-port=25/tcp
# IPTABLES
  • How does this compare to nftables?
  • Where does firewalld and firewall-cmd fit in?
  • What about ufw?

Outstanding Questions

{Day 04}

Scanning, Recap and Capstone

{Scanning}

Tell me what's wrong

# LYNIS

Automated heath and security scanning for Linux

# LinPEAS

Searches for Privilege Escalation Paths

# OpenSCAP
  • Very comprehensive scan against "policies"
  • Generates a nice interactive HTML report
  • Can generate "remediation" fixes in either shell scripts or Ansible playbooks based on findings
  • Used to meet "enterprise" standards
  • Some distros lack the needed "policies" (Ubuntu)
  • Some distros require workarounds (CentOS)

Running an OpenSCAP Scan

# OpenSCAP
$ sudo dnf install -y openscap-scanner scap-security-guide

$ sudo oscap xccdf eval \
	--profile xccdf_org.ssgproject.content_profile_stig \
	--report $(hostname)-scap-report.$(date +%Y%m%d).html \
	--results $(hostname)-scap-xccdf.$(date +%Y%m%d).xml \
	--cpe /usr/share/xml/scap/ssg/content/ssg-rhel8-cpe-dictionary.xml \
	--fetch-remote-resources \
	/usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml
    
# first scan gives `notapplicable` for everything
# The workaround...

$ sudo cp /usr/share/openscap/cpe/openscap-cpe-dict.xml{,.dist}
$ sudo cp /usr/share/openscap/cpe/openscap-cpe-oval.xml{,.dist}
$ sudo curl -L \
  https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.3/cpe/openscap-cpe-dict.xml \
  -o /usr/share/openscap/cpe/openscap-cpe-dict.xml
$ sudo curl -L \
  https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.3/cpe/openscap-cpe-oval.xml \
  -o /usr/share/openscap/cpe/openscap-cpe-oval.xml
$ sudo sed -i \
-e 's|idref="cpe:/o:redhat:enterprise_linux|idref="cpe:/o:centos:centos|g' \
-e 's|ref_id="cpe:/o:redhat:enterprise_linux|ref_id="cpe:/o:centos:centos|g' \
/usr/share/xml/scap/ssg/content/ssg-rhel*.xml

OpenSCAP Workbench

# OpenSCAP

{Q&A}

Questions?

{Capstone}

Now do it.

Student Control Back Proxy Front Proxy
Student01 137.184.98.24
10.136.0.3
143.198.167.106
10.136.0.2
104.248.115.67
10.136.0.22
Student02 159.223.122.166
10.136.0.9
137.184.158.238
10.136.0.19
198.211.99.131
10.136.0.31
Student03 159.223.114.187
10.136.0.7
159.223.96.252
10.136.0.21
104.248.115.77
10.136.0.23
Student04 137.184.219.176
10.136.0.5
147.182.186.155
10.136.0.13
104.248.115.87
10.136.0.25
Student05 137.184.195.184
10.136.0.4
137.184.99.29
10.136.0.17
104.248.115.102
10.136.0.28
Student06 159.223.118.134
10.136.0.12
147.182.218.170
10.136.0.14
104.248.115.84
10.136.0.24
Student07 159.223.122.250
10.136.0.10
137.184.63.109
10.136.0.15
104.248.115.99
10.136.0.27
Student08 159.223.122.135
10.136.0.8
137.184.213.156
10.136.0.20
104.248.115.90
10.136.0.26
Student09 159.223.114.42
10.136.0.6
137.184.135.121
10.136.0.18
104.248.115.120
10.136.0.29
Student10 159.223.118.112
10.136.0.11
137.184.105.180
10.136.0.16
104.248.115.132
10.136.0.30
# IP SPACE
# DIAGRAM
Made with Slides.com