Operating in a Cloud Environment
# ABOUT THE INSTRUCTOR
Former Forward Communications
# AGENDA
- Introduction
- Procurement & Attribution
- Basic Setup
- Version Control
- Scripting and Automation
- Building Locally
- Containers & Namespaces
- Custom Images
- Recipe Tools
- Dynamic Monitoring
- Accessing Remote Services
- Cloud Hardening
- Firewalls
- Automated Hardening and Scanning
- Q & A
- Capstone
Training Days
# Our TECH STACK
# OUR TECH STACK
# TAKING NOTES ONLINE
https://obsidian.md
# TAKING NOTES
$ sudo dnf install flatpak
$ flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
$ flatpak install flathub md.obsidian.Obsidian
Markdown in 30 Seconds
# TAKING NOTES
# Title
## Sub Title
### etc. etc. etc.
Comments about `commands`
List:
- **bold**
- *italics*
- ++underline++
- ~~strikethrough~~
- ==Highlight==
- [ ] Incomplete Checkbox
- [x] Complete Checkbox
Even [hyper-links](www.someurl.com)
```python
# Code Here
def say_hi(name="Kevin"):
print(f'Hello {name}')
say_hi("Tom")
```
From Zero to ________
How and Where do we get a "Cloud Environment"
Define Requirements
Research Providers
Rotate Responsibly
# PROCUREMENT PROCESS
Most providers are trying to prevent crypto mining, scams, and illegal activity
https://prv.to
Who am I? Who are We?
# ATTRIBUTION
# TYPES OF ATTRIBUTION
# THREAT MODEL
Catching Up
# What is Linux?
# Open Source?
# Key Linux Concepts
# LINUX COMPONENTS
CPU
Screen
HDD
RAM
...
Linux Kernel
GNU Core Utilities
System Config
PKG MGMT
User Applications
User Applications
Desktop Environment
# COMMON DISTROS
# COMMON Desktop Envrionments
Gnome 3
KDE Plasma Shell
Specialized (i3, etc.)
# File System
# Permissions
$ touch file
$ ls -l file
-rw-rw-r-- 1 student student 0 Jan 16 13:58 file
$ chmod o+x file
$ ls -l file
-rw-rw-r-x 1 student student 0 Jan 16 13:58 file
# Permissions
Becoming a Nerd
$ command1 | command2 | command3
# Pipes
$ echo "Send me to a file" > file.txt
$ echo < file.txt
Send me to a file
# Redirecting
$ cat some_file.txt 2> err_file.txt
# Redirect stderr to 'err_file.txt'
$ cat some_file.txt 2>&1
# Redirect stderr to stdout (stdout still goes to screen)
$ cat some_file.txt 2>&1 > out.txt
# Redirect stderr to stdout, then stdout to 'out.txt' file
$ cat some_file.txt &> some_file.txt
# Redirect stderr AND stdout to a file
# Redirecting
# Finding Help
# FISH SHELL
Let's do it!
$ sudo dnf install zsh util-linux-user git
$ sudo chsh -s $(which zsh) $USER
# in ~/.zshrc
ENABLE_CORRECTION="true"
COMPLETION_WAITING_DOTS="true"
export PATH=/usr/local/bin:$HOME/.local/bin:$PATH
export SSH_KEY_PATH="~/.ssh/rsa_id"
export TERM="xterm-256color"
fpath+=~/.zfunc
compinit -U
# ZSH SHELL
$ git clone \
https://github.com/robbyrussell/oh-my-zsh \
~/.oh-my-zsh
# in the TOP of ~/.zshrc
export ZSH=$HOME/.oh-my-zsh
ZSH_THEME="robbyrussell"
plugins=()
source $ZSH/oh-my-zsh.sh
But mind how many plugins you use!
# ZSH PLUGINS
$ git clone \
https://github.com/zsh-users/zsh-autosuggestions \
~/.oh-my-zsh/custom/plugins/zsh-autosuggestions
# in ~/.zshrc
plugins=(
zsh-autosuggestions
)
# ZSH PLUGINS
$ git clone \
https://github.com/zsh-users/zsh-syntax-highlighting.git \
~/.oh-my-zsh/custom/plugins/zsh-syntax-highlighting
# in ~/.zshrc
plugins=(
zsh-autosuggestions
zsh-syntax-highlighting
)
# ZSH PLUGINS
$ git clone \
https://github.com/momo-lab/zsh-abbrev-alias \
~/.config/zsh-abbrev-alias/
# in ~/.zshrc
export USE_ABBREV_ALIAS=1
source $HOME/.config/zsh-abbrev-alias/abbrev-alias.plugin.zsh
abbrev-alias gl='git log --graph --all --oneline --decorate'
When to use Regular Aliases?
# ZSH PLUGINS
# in ~/.zshrc
plugins=(
zsh-autosuggestions
zsh-syntax-highlighting
sudo
dnf
git
)
Full List of Plugins
# ZSH PLUGINS
$ curl -fsSL https://starship.rs/install.sh | bash
# In ~/.zshrc
# Remove ZSH_THEME...
eval "$(starship init zsh)"
# STARSHIP
╭─kevin@shenron in repo: toast on main [$] is 📦 v0.45.2 via v1.60.0-nightly took 28ms
╰─➤
$ git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf
$ ~/.fzf/install
# In ~/.zshrc
[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh
Commands
# FZF
$ curl -sL https://raw.githubusercontent.com/denisidoro/navi/master/scripts/install | sh
# In ~/.zshrc
eval "$(navi widget zsh)"
Commands
# NAVI
% dnf
# List files included in an RPM
sudo dnf repoquery --installed -l <PKG>
# List files included in an installed RPM
sudo dnf repoquery --installed -l <EXISTING_PKG>
# Disable repository
sudo dnf config-manager --set-disabled <EXISTING_REPO>
# Add repository
sudo dnf config-manager --add-repo <REPOSITORY>
$ EXISTING_REPO: dnf repolist | awk '{print $1}'
$ EXISTING_PKG: rpm -qa | grep -oP '^[^.]*(?=-[0-9])'
# NAVI
Advanced Command Line
# SHELL-FU
# SHELL-FU
# SHELL-FU
$ ANIMAL=cat
$ echo $ANIMAL
cat
AKA Magic
# EXPANSIONS
$ mv /some/long/path/file.old /some/long/path/file.new
# vs
$ mv /some/long/path/file.{old,new}
# GLOBS
Matches zero or more characters
Matches exactly one character
Matches given characters only
Matches any except given chars
$ ls -l
total 264896
-rw-r--r-- 1 kevin kevin 15400455 Jan 17 10:33 09b25a9a5c720.tar.gz
-rw-rw-r-- 1 kevin kevin 43487904 Dec 7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r-- 1 kevin kevin 27765264 Dec 7 13:01 encryptr_2.0.0-1_amd64.deb
-rw-rw-r-- 1 kevin kevin 1407463 Jan 10 14:14 FEL-r22.pdf
-rw-rw-r-- 1 kevin kevin 397750 Jan 10 14:14 FEL.tar.gz
-rw-rw-r-- 1 kevin kevin 42004318 Dec 7 13:01 haroopad-v0.13.1-x64.deb
-rw-rw-r-- 1 kevin kevin 79941824 Dec 7 13:04 keybase_amd64.deb
$ ls -l *.deb
total 264896
-rw-rw-r-- 1 kevin kevin 43487904 Dec 7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r-- 1 kevin kevin 27765264 Dec 7 13:01 encryptr_2.0.0-1_amd64.deb
-rw-rw-r-- 1 kevin kevin 42004318 Dec 7 13:01 haroopad-v0.13.1-x64.deb
-rw-rw-r-- 1 kevin kevin 79941824 Dec 7 13:04 keybase_amd64.deb
*
[abc]
In short...wild cards
Special Groups
[[:named:]]
[!abc]
alpha digit alnum
lower upper
punct space
?
# PARAMETER EXPANSIONS
$ echo {one,two,three}
one two three
$ ls -l {*.deb,0*}
-rw-r--r-- 1 kevin kevin 15400455 Jan 17 10:33 09b2af7a5c720.tar.gz
-rw-rw-r-- 1 kevin kevin 43487904 Dec 7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r-- 1 kevin kevin 27765264 Dec 7 13:01 encryptr_2.0.0-1_amd64.deb
-rw-rw-r-- 1 kevin kevin 49035482 Dec 19 15:23 google-chrome-stable_current_amd64.deb
-rw-rw-r-- 1 kevin kevin 42004318 Dec 7 13:01 haroopad-v0.13.1-x64.deb
-rw-rw-r-- 1 kevin kevin 79941824 Dec 7 13:04 keybase_amd64.deb
$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
# COMMAND SUBSTITUTION
$ ls $(echo "-l")
total 264896
-rw-r--r-- 1 student student 15400455 Jan 17 10:33 09b2afa5c720.tar.gz
-rw-rw-r-- 1 student student 43487904 Dec 7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r-- 1 student student 27765264 Dec 7 13:01 encryptr_2.0.0-1_amd64.deb
$ ls -l
total 264896
-rw-r--r-- 1 student student 15400455 Jan 17 10:33 09b2afa5c720.tar.gz
-rw-rw-r-- 1 student student 43487904 Dec 7 13:00 code_1.18.1-1510857349_amd64.deb
-rw-rw-r-- 1 student student 27765264 Dec 7 13:01 encryptr_2.0.0-1_amd64.deb
$ ANIMAL=cat
$ echo ${ANIMAL}
cat
$ echo ${ANIMAL}s
cats
$ echo $ANIMALs
# PARAMETER SUBSTITUTION
$
$ ANIMAL=cat
$ echo ${ANIMAL/at/ow}
cow
$ echo ${ANIMALs:-dog}
dog
$ CAT=tabby
$ ANIMAL=CAT
$ echo ${ANIMAL}
CAT
$ echo ${#ANIMAL}
3
$ echo ${!ANIMAL}
tabby
# PARAMETER SUBSTITUTION
/
:-
#
!
Variable Length
Variable Indirection
Other operators available
$ diff <(ls -l) <(ls -al)
1c1,3
< total 264896
---
> total 264904
> drwxr-xr-x 3 student student 4096 Jan 17 11:11 .
> drwxr-xr-x 35 student student 4096 Jan 17 12:34 ..
$ diff $(ls -l) $(ls -al)
diff: invalid option -- '-'
diff: Try 'diff --help' for more information.
# PROCESS SUBSTITUTION
Taming the chaos
# GIT
$ git init
$ git commit -a
# GIT
$ git switch my-branch-name
# GIT
$ git switch awesomefeature
$ git switch main
$ git merge awesomefeature
$ git branch -D awesomefeature
# GIT
# GIT
There are self hosted alternatives!
# GIT
# GIT
Just do this for me...
# SCRIPTING
#!/bin/bash
echo "Hello world!"
# SCRIPTING
$ ./.local/bin/script_01.sh
zsh: permission denied: ./.local/bin/script_01.sh
# SCRIPTING
$PATH
./
$ ls -l .local/bin
total 4
-rw-rw-r-- 1 student student 33 Jan 29 10:46 script_01.sh
$ chmod u+x .local/bin/script_01.sh
$ ls -l .local/bin
-rwxrw-r-- 1 student student 33 Jan 29 10:46 script_01.sh
$ ./.local/bin/script_01.sh
Hello world!
# SCRIPTING
$ export PATH=$PATH:$HOME/.local/bin
$ script_01.sh
Hello world!
# SCRIPTING
$PATH
$ for NUM in $(seq 1 10); do echo "Hello World! x$NUM"; done
Hello World! x1
Hello World! x2
Hello World! x3
Hello World! x4
Hello World! x5
Hello World! x6
Hello World! x7
Hello World! x8
Hello World! x9
Hello World! x10
# SCRIPTING
for
for <VAR> in <SET>;
do
<CMDS>
done
for NUM in $(seq 1 10);
do
echo "Hello $NUM times:"
for _ in $(seq 1 $NUM);
do
echo -e "\tHello"
done
done
We can even nest loops
# SCRIPTING
if [ <CONDITION> ];
then
<command if true>
else
<command if NOT true>
fi
# SCRIPTING
# SCRIPTING
# SCRIPTING
&&
||
Getting friendly with the snake
# PYTHON
.py
# PYTHON
Hello World
# in hello.py
print('Hello World!')
Run via
$ python hello.py
Hello world!
# PYTHON
name = 'something'
age = 29
more_text = a_var
# PYTHON
num1 = 2
num2 = 2
both = num1 + num2
print('Adding: ', both)
both = num1 - num2
print('Subtracting: ', both)
both = num1 / num2
print('Dividing: ', both)
both = num1 * num2
print('Multiplying: ', both)
both = num1 % num2
print('Modding: ', both)
text1 = '2'
text2 = '2'
both = text1 + text2
print('String and String: ', both)
fl1 = 2.2
int1 = 2
both = fl1 + int1
print('Float and Int: ', both)
fl2 = 2.2
both = fl1 + fl2
print('Float and Float: ', both)
# PYTHON
# PYTHON
my_list = ['Kevin', 'Knapp', 29]
print(my_list[0])
print(my_list[1])
print(my_list[2])
# PYTHON
my_list[:]
from_start_to_index_5[:5]
from_index_2_to_end[2:]
from_index_1_to_4[1:4]
# PYTHON
person = {'first': 'Kevin',
'last': 'Knapp',
'age': 29 }
my_age = person['age']
# PYTHON
my_set = set()
my_set.add('kevin')
# PYTHON
if some_expression_or_variable:
# Actions take if TRUE
else:
# Actions to take if FALSE
if some_expression_or_variable:
# Actions take if TRUE
elif some_new_test:
# Actions to take if TRUE
else:
# Actions to take if all tests are FALSE
# PYTHON
# PYTHON
for value in some_collection:
# do something
my_list = ['item 1', 'item 2', 'item 3', 'item 4']
for item in my_list:
print(item)
# PYTHON
# say 'Hello' 10 times
for _ in range(10):
print('Hello')
# Count to 10
for num in range(1,11):
print(num)
print(',')
Note the start and end values passed to range
# PYTHON
# A function that takes 2 arguments and doesn't
# return a result
def my_function(some_arg, other_arg):
# do some work here...
# A function that takes 0 arguments and
# returns a result
def my_other_function():
# do some more work...
return True
# PYTHON
def cool_function(name='kevin', age=29):
# do stuff here
return (name, age)
# use the function
n, a = cool_function(age=50)
print('name={}, age={}'.format(n,a))
# PYTHON
# PYTHON
Controlling a Fleet
---
- name: apply non-kernel updates
hosts: "{{ HOSTS | default('web') }}"
become: true
gather_facts: false
tasks:
- name: upgrade all packages except kernel
yum:
name: '*'
state: latest
exclude: kernel*
tags: all
- name: upgrade all packages security related except kernel
yum:
name: '*'
state: latest
security: true
exclude: kernel*
tags: security
# ANSIBLE
---
all:
children:
controller:
hosts:
localhost:
ansible_connection: local
user_nodes:
hosts:
user01:
ansible_host: 192.168.1.20
user02:
ansible_host: 192.168.1.21
user03:
ansible_host: 192.168.1.22
database_server:
hosts:
db01:
db02:
db03:
projext_x:
children:
user_nodes:
database_server:
# ANSIBLE
$ ansible-playbook -i inventory_file playbook.yml
# ANSIBLE
Containers, Containers, Containers
Buzzword Bingo
# What is a Container
# What is a Container
Hardware
Linux Kernel
GNU Core Utilities
System Config
PKG MGMT
App
Container
VM
GNU Core Utilities
System Config
PKG MGMT
App
Linux Kernel
GNU Core Utilities
System Config
PKG MGMT
App
Virtual
Hardware
# What is a Container
Hardware
Linux Kernel
GNU Core Utilities
System Config
PKG MGMT
App
Container
VM
App
Linux Kernel
GNU Core Utilities
System Config
PKG MGMT
App
Virtual
Hardware
# Container Software
# Container Software
Install Docker
$ curl -LSs https://get.docker.com | sh
$ sudo usermod -aG docker $USER
$ sudo systemctl enable --now docker
Install Podman
$ sudo dnf install podman
# Container Software
Testing It Out
$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
$ podman run hello-world
[...]
# Container Software
Fedora Toolbox
$ sudo dnf install toolbox
$ toolbox enter
Creates an overlay system with a shared home directory
Great for testing and development!
# docker-compose
version: "2"
services:
bookstack:
image: lscr.io/linuxserver/bookstack
container_name: bookstack
environment:
- APP_URL=http://localhost/
- DB_HOST=bookstack_db
- DB_USER=bookstackuser
- DB_PASS=bookstackpass
- DB_DATABASE=bookstackapp
- PUID=996
- PGID=994
ports:
- 6875:80
depends_on:
- bookstack_db
networks:
- bookstack-tier
restart: unless-stopped
volumes:
- ./data:/config
bookstack_db:
image: lscr.io/linuxserver/mariadb
container_name: bookstack_db
networks:
- bookstack-tier
environment:
- MYSQL_ROOT_PASSWORD=bookstackdbrootpass
- TZ=America/New_York
- MYSQL_DATABASE=bookstackapp
- MYSQL_USER=bookstackuser
- MYSQL_PASSWORD=bookstackpass
- PUID=996
- PGID=994
restart: unless-stopped
volumes:
- ./data:/config
networks:
bookstack-tier:
driver: bridge
$ sudo dnf install docker-compose
$ sudo dnf install podman-compose
VMs, and alternative tools
# VAGRANT
$ sudo dnf install vagrant
$ vagrant init hashicorp/bionic64
$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'hashicorp/bionic64'...
==> default: Forwarding ports...
default: 22 (guest)
=> 2222 (host) (adapter 1)
==> default: Waiting for machine to boot...
$ vagrant ssh
vagrant@bionic64:~$ _
# VAGRANTFILE
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.synced_folder ".", "/vagrant", disabled: true
# StandAlone Server Only
config.vm.define "server2" do |server2|
server2.vm.network "private_network", ip: "192.168.42.20",
virtualbox__intnet: "thnet"
server2.vm.provider "virtualbox" do |vb|
vb.name = "server2"
vb.memory = "1024"
end
server2.vm.provision :shell, path: "./bootstrap_server.sh"
end
# Sensor to Server/Sensor
config.vm.define "sensor1" do |sensor1|
sensor1.vm.network "private_network", ip: "192.168.42.11",
virtualbox__intnet: "thnet"
sensor1.vm.provider "virtualbox" do |vb|
vb.name = "sensor1"
vb.memory = "1024"
end
sensor1.vm.provision :shell, path: "./bootstrap_sensor.sh"
end
# APT Repo
config.vm.define "repo" do |repo|
repo.vm.network "private_network", ip: "192.168.42.2",
virtualbox__intnet: "thnet"
repo.vm.provider "virtualbox" do |vb|
vb.name = "repo"
vb.memory = "1024"
end
repo.vm.provision :file, source: '../pkgs/deps.tgz', destination: "/home/vagrant/"
repo.vm.provision :shell, path: "../bootstrap_repo.sh"
end
end
# LXD
$ sudo snap install lxd
$ sudo lxd init
$ lxc launch ubuntu:20.04 [--vm]
# MULTIPASS
$ sudo snap install multipass
$ multipass launch
Have it your way
# DOCKERFILE
# Dockerfile
FROM alpine:latest
RUN apk add python3
ENTRYPOINT ["python3", "-m", "http.server"]
$ docker build -t pyserv .
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
pyserv latest fec6935a627e 3 minutes ago 54MB
Each directive creates a "layer" which is built upon (and can lead to huge sizes!)
# DOCKERFILE LINTS
$ docker run --rm -i hadolint/hadolint < Dockerfile
Unable to find image 'hadolint/hadolint:latest' locally
latest: Pulling from hadolint/hadolint
d0727728368a: Pull complete
Digest: sha256:1b47f3192d2c69c622863b84d5cccbfa0c1fac9022bfc297aa931e6d44e51608
Status: Downloaded newer image for hadolint/hadolint:latest
-:1 DL3007 warning: Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag
-:2 DL3018 warning: Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`
-:2 DL3019 info: Use the `--no-cache` switch to avoid the need to use `--update` and remove `/var/cache/apk/*` when done installing packages
Lint your Dockerfiles!
https://github.com/hadolint/hadolint
# Container Registries
# Container Registries
$ docker save pyserv -o pyserv.tar
$ ls -l
inode Permissions Links Size Blocks User Group Date Modified Name
62391693 .rw-r--r--@ 1 83 8 kevin kevin 15 Jan 14:48 Dockerfile
62391694 .rw-------@ 1 54Mi 110264 kevin kevin 15 Jan 14:57 pyserv.tar
$ tar tf pyserv.tar
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/VERSION
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/json
9ba2039e49404b3ae8654f99e9d22564cfbeaf6588ec1c06ac048a5357853341/layer.tar
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/VERSION
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/json
fc9537bd0694747ea59e49ff8617bf343626b0fb3a4657c0cb8f02c859c4f909/layer.tar
fec6935a627eb73a17b0269f95aa5cda4b1c5d70c032c87e333d7827ce1d4154.json
manifest.json
repositories
# Working with Images
Tips for working with Container Images and Layers
$ curl -OL \
https://github.com/wagoodman/dive/releases/download/v0.9.2/dive_0.9.2_linux_amd64.rpm
$ sudo dnf install ./dive*.rpm
https://github.com/wagoodman/dive
Scripting...lite
github.com/casey/just
# JUST
$ curl \
--proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | \
bash -s -- --to ~/.local/bin/
# TOAST
image: ubuntu
tasks:
install_figlet:
command: |
apt-get update
apt-get install --yes figlet
greet:
dependencies:
- install_figlet
command: figlet 'Hello, World!'
$ curl https://raw.githubusercontent.com/stepchowfun/toast/main/install.sh -LSfs | sh
Monitoring, Remote Admin, and Hardening
Watching everything
# MONITORING
# MONITORING
Consider the difference between local node monitoring, and "entire system" monitoring
# CONTAINER MONITORING
$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
036b8ad6c7c2 tak_takserver_1 1.82% 5.313GiB / 31.4GiB 16.92% 444MB / 2.08GB 0B / 1.13MB 368
a3c79a328776 tak_pg_1 0.11% 259.1MiB / 31.4GiB 0.81% 927MB / 183MB 16.4kB / 2.96GB 10
3a36813ff86c tak_openfire_1 0.12% 678.7MiB / 31.4GiB 2.11% 9.01MB / 12.2MB 0B / 852kB 64
b360b0eee623 metrics_node-exporter_1 0.00% 21.22MiB / 31.4GiB 0.07% 0B / 0B 17.9MB / 0B 19
73be46dd33fa metrics_cadvisor_1 6.09% 202MiB / 31.4GiB 0.63% 155MB / 8.29GB 31.4MB / 0B 42
docker stats
https://github.com/jesseduffield/lazydocker
# PROMETHEUS
# Graphana
# Graphana - NODE_EXPORTER
# Graphana - CADVISOR
Remote Services...but private
# CONNECTIVITY
# WIREGUARD
# WIREGUARD
$ sudo dnf install wireguard-tools
$ sudo mkdir /etc/wireguard
$ cd /etc/wireguard
$ sudo wg genkey | tee hub.private | wg pubkey > hub.public
$ sudo wg genkey | tee client.private | wg pubkey > client.public
# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = 2DiO+Ox5cURGbFLWniVmx/N1laqYGIZpv/IPDuiYkVg=
Address = 10.10.10.1/24
ListenPort = 51820
DNS = 1.1.1.1
# Client
[Peer]
PublicKey = hrbwZZxxjVuP7QvlWm9YDyg+40ftgQo5mKRrWmekpFw=
AllowedIPs = 10.10.10.2/32
PersistentKeepalive = 25
$ sudo systemctl enable --now wg-quick@wg0
# SSH CONFIG FILES
Host myserver
HostName 51.127.128.14
User kevink
IdentityFile ~/.ssh/roshi
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentitiesOnly yes
LogLevel FATAL
Compression yes
Port 22
-F FILE
-o "option=value"
# SSH TUNNELS
$ ssh user@host -L 8080:127.0.0.1:3000 -N
$ pip3 install --upgrade --user sshuttle
$ sshuttle -r user@host 0.0.0.0/0
-x
-L
Enough of this terminal...
# COCKPIT
$ sudo dnf install cockpit
$ sudo systemctl enable --now cockpit cockpit.socket
$ sudo firewall-cmd --permanent --zone=public --add-service=cockpit
$ sudo firewall-cmd --reload
Make 'em work for it
# HARDENING
The Most Low Hanging Fruit Will Be the Exploit!
The defense must be correct 100% of the time, but the offense only needs to be correct once!
Many issues boil down to permissions
# SSH HARDENING
If you do nothing else - Do this
PermitRootLogin prohibit-password # or 'no' even better
PasswordAuthentication yes # Normally this should be 'no'
PrintLastLog yes
PermitEmptyPasswords no
MaxAuthTries 3
ClientAliveInterval 200
ClientAliveCountMax 3
MaxSessions 3
HostbasedAuthentication no
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr
Macs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
$ sudo chown root:root /etc/ssh/sshd_config
$ sudo chmod 0600 /etc/ssh/sshd_config
$ sudo systemctl restart sshd
# RUST-MOTD
github.com/rust-motd/rust-motd
Configure via: $HOME/.config/rust-motd/config.toml
[service_status]
docker = "docker"
gitlab-runner = "gitlab-runner"
[uptime]
prefix = "Up"
[filesystems]
root = "/"
[fail_2_ban]
jails = ["sshd", "anotherjail"]
[last_login]
kevin = 2
root = 3
[last_run]
# SYSTEMD SERVICE HARDENING
We can harden on a per-service basis
This should be the first thing you reach for!
$ sudo systemd-analyze security
UNIT EXPOSURE PREDICATE
atd.service 9.6 UNSAFE
crond.service 9.6 UNSAFE
httpd.service 9.2 UNSAFE
mariadb.service 8.8 EXPOSED
sshd.service 9.6 UNSAFE
$ sudo systemd-analyze security httpd.service
NAME DESCRIPTION EXPOSURE
✗ PrivateDevices Service potentially has access to hardware devices 0.2
✓ PrivateTmp Service has no access to other software's temporary files
✗ ProtectControlGroups Service may modify to the control group file system 0.2
✗ ProtectHome Service has full access to home directories 0.2
✗ ProtectKernelTunables Service may alter kernel tunables 0.2
✗ ProtectSystem Service has full access to the OS file hierarchy 0.2
✗ RestrictSUIDSGID Service may create SUID/SGID files 0.2
✗ RootDirectory Service runs within the host's root directory 0.1
…
→ Overall exposure level for httpd.service: 9.2 UNSAFE
# FAIL2BAN
Ban IPs based on failed login attempts
There are times fail2ban isn't as useful as you'd think
$ sudo dnf install fail2ban
$ sudo cp /etc/fail2ban/fail2ban.{conf,local}
$ sudo cp /etc/fail2ban/jail.{conf,local}
$ sudo systemctl enable --now fail2ban
$ sudo tail -f /var/log/fail2ban.log
To see "jailed" IPs
$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/auth.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
# unban an IP
$ sudo fail2ban-client set sshd unbanip 1.2.3.4
# CROWDSEC
github.com/crowdsecurity/crowdsec
"CrowdSec is a free, modern & collaborative behavior detection engine, coupled with a global IP reputation network"
# HARDENING
CentOS Hardening will be covered in Day 04 via OpenSCAP
# UBUNTU-HARDENING
github.com/konstruktoid/hardening
$ git clone https://github.com/konstruktoid/hardening
$ cd hardening
$ sed -i "s/CHANGEME=''/CHANGEME=foo" ubuntu.cfg
$ sudo bash ubuntu.sh
Primarily only good for Ubuntu based systems, however some of the internal scripts are applicable across many distros
Oh no...
# SELINUX
Three Primary Modes:
# SELINUX
Configuration File: /etc/selinux/config
# Get the current mode
$ sudo getenforce
$ sudo sestatus
# Temporarily Change Modes
$ sudo setenforce [enforcing|permissive|1|0]
Common Commands:
# SELINUX CONTEXT LABELS
$ ps -eZ
LABEL PID TTY TIME CMD
system_u:system_r:init_t:s0 1 ? 00:00:14 systemd
system_u:system_r:kernel_t:s0 2 ? 00:00:00 kthreadd
system_u:system_r:httpd_t:s0 25134 ? 00:00:00 httpd
restorecon PATH will set the file context to whatever the system configuration specifies (view or change with semanage fcontext [--list])
# SELINUX FCONTEXT
restorecon PATH will set the file context to whatever the system configuration specifies (view or change with semanage fcontext [--list])
$ sudo semanage fcontext -l
SELinux fcontext type Context
/home directory system_u:object_r:home_root_t:s0
/home symbolic link system_u:object_r:home_root_t:s0
/home/(.*/)?\.snapshots(/.*)? all files
$ sudo semanage fcontext -a -t unconfined_u:object_r:user_home_t:s0 /home
$ sudo restorecon -v -r /home
Caution
files retain the context they had previously
files *may not* retain the context they had previously
files inherit the context of the parent directory
MOVED
COPIED
CREATED
# SELINUX BOOLEANS
Booleans are on/off switches
$ sudo semanage boolean -l
SELinux boolean State Default Description
...
xend_run_blktap (on , on) Allow xend to run blktap
xend_run_qemu (on , on) Allow xend to run qemu
xguest_connect_network (on , on) Allow xguest to connect network
xguest_exec_content (on , on) Allow xguest to exec content
xguest_mount_media (on , on) Allow xguest to mount media
...
$ setsbool [-P] xguest_connect_network off
Means permanent
-P
# SELINUX PORTS
The Firewall **AND** SELinux Control Network Access
$ sudo semanage port -l
SELinux Port Type Proto Port Number
afs3_callback_port_t tcp 7001
afs_fs_port_t udp 7000, 7005
afs_ka_port_t udp 7004
amanda_port_t tcp 10080-10083
amqp_port_t tcp 15672, 5671-5672
amqp_port_t udp 5671-5672
asterisk_port_t udp 2427, 2727, 4569
$ sudo semanage port -a -p udp -t asterisk_port_t 8989
Means ADD
-a
Means DELETE
-d
Means MODIFY
-m
# SELINUX TROUBLESHOOTING
$ sudo ausearch -m avc -ts recent
----
time->Tue Dec 21 10:15:53 2021
type=AVC msg=audit(1640099753.877:245): avc: denied { write } for pid=2368 comm="flatpak" name="system_bus_socket" dev="tmpfs" ino=2513 scontext=system_u:system_r:fedoratp_t:s0-s0:c0.c1023 tcontext=system_u:object_r:system_dbusd_var_run_t:s0 tclass=sock_file permissive=0
----
time->Tue Dec 21 10:15:53 2021
type=AVC msg=audit(1640099753.893:246): avc: denied { write } for pid=2370 comm="flatpak" name="system_bus_socket" dev="tmpfs" ino=2513 scontext=system_u:system_r:fedoratp_t:s0-s0:c0.c1023 tcontext=system_u:object_r:system_dbusd_var_run_t:s0 tclass=sock_file permissive=0
----
# SELINUX
Instead of setting the entire system to permissive, consider changing just the problem domain
$ ps -eZ
LABEL PID TTY TIME CMD
system_u:system_r:init_t:s0 1 ? 00:00:14 systemd
system_u:system_r:kernel_t:s0 2 ? 00:00:00 kthreadd
system_u:system_r:httpd_t:s0 25134 ? 00:00:00 httpd
$ sudo semanage permissive -a httpd_t
Means ADD
-a
Means DELETE
-d
Means MODIFY
-m
Not in my house
# IPTABLES
# IPTABLES
# IPTABLES
# IPTABLES
# IPTABLES
# IPTABLES
Command | Function |
---|---|
iptables -A <CHAIN> | "Append" a rule at the bottom of <CHAIN> |
tcpdump -I <CHAIN> <NUM> | Insert a rule on <CHAIN> and rule number <NUM> |
iptables -D <CHAIN> <NUM> | Delete rule number <NUM> from <CHAIN> |
# IPTABLES
# IPTABLES
Option / Flag | Function |
---|---|
-j <ACTION> | What to do with the packet. Normally: * ACCEPT * LOG * DROP * MASQUERADE |
-p [!] <PROTO> | Match a particular protocol like 'tcp' or 'udp' |
-i <IFACE> | Match only on input interface <IFACE>. If '!' is included it reverses the rule |
-d [!] <IP>[/cidr] | Destination IP/network. If '!' is included it reverses the rule |
-s [!] <IP> | Source IP/network. If '!' is included it reverses the rule |
-o <IFACE> | Only match output interface <IFACE> |
-m <NAME> | Performs a "match" matches can be thought of as "modules" and enable other "matching options/flags" More on this later... |
--dport <NUM> | Destination port |
--sport <NUM> | Source port |
Common Rule options/switches
# IPTABLES
Option / Flag | Function |
---|---|
-m conntrack | Adds a --ctstate <STATE> flag where you can match connection states * ESTABLISHED * RELATED * NEW * INVALID Can be comma separated like ESTABLISHED,RELATED Also adds --ctproto |
-m comment | Allows adding comments with --comment "some comment" |
-m multiport | Allows matching on more than one port with --dports <port1,port2> |
-m iprange | Allows --src-range and --dst-range for a range of IPs |
-m log | Adds --log-level <#> to only log at specified logging levels |
-m udp OR -m tcp | Adds --sport and --dport that will match only TCP or UDP ports |
List of all extensions: http://ipset.netfilter.org/iptables-extensions.man.html
# IPTABLES
# IPTABLES
# IPTABLES
*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT
-N port-scanning
-N syn_flood
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp --syn -j syn_flood
-A INPUT -p icmp -m limit --limit 1/sec --limit-burst 1 -j ACCEPT
-A INPUT -p icmp -m limit --limit 1/sec --limit-burst 1 -j LOG --log-prefix PING-DROP
-A INPUT -p icmp -j DROP
-A INPUT -f -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A INPUT -p tcp -m connlimit --connlimit-above 80 --connlimit-mask 32 --connlimit-saddr -j REJECT --reject-with tcp-reset
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -p icmp -j ACCEPT
-A port-scanning -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -m limit --limit 1/sec --limit-burst 2 -j RETURN
-A port-scanning -j DROP
-A syn_flood -m limit --limit 1/sec --limit-burst 3 -j RETURN
-A syn_flood -j DROP
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
COMMIT
*mangle
:PREROUTING ACCEPT
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT
:POSTROUTING ACCEPT
-A PREROUTING -m conntrack --ctstate INVALID -j DROP
-A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,RST FIN,RST -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,ACK FIN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags ACK,URG URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,ACK FIN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags PSH,ACK PSH -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,PSH,URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,ACK,URG -j DROP
COMMIT
# IPTABLES
[Unit]
Description=Packet Filtering Framework
After=network-online.target
Requires=network.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables-restore /etc/ipv4.rules
ExecReload=/sbin/iptables-restore /etc/ipv4.rules
ExecStop=/sbin/iptables -P INPUT ACCEPT
ExecStop=/sbin/iptables -P FORWARD ACCEPT
ExecStop=/sbin/iptables -F
[Install]
WantedBy=multi-user.target
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now iptables.service
# IPTABLES
# UFW
$ sudo ufw status
ufw: active
$ sudo ufw allow ssh/tcp
$ sudo ufw deny 8888/udp
$ sudo ufw limit 80/tcp
$ sudo ufw deny from 192.168.1.100/24 on eth0 proto any
# FIREWALLD
$ sudo firewall-cmd --state
$ sudo firewall-cmd --get-*
$ sudo firewall-cmd --zone=internal --add-port=25/tcp
# IPTABLES
Scanning, Recap and Capstone
Tell me what's wrong
# LYNIS
Automated heath and security scanning for Linux
# LinPEAS
Searches for Privilege Escalation Paths
# OpenSCAP
# OpenSCAP
$ sudo dnf install -y openscap-scanner scap-security-guide
$ sudo oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_stig \
--report $(hostname)-scap-report.$(date +%Y%m%d).html \
--results $(hostname)-scap-xccdf.$(date +%Y%m%d).xml \
--cpe /usr/share/xml/scap/ssg/content/ssg-rhel8-cpe-dictionary.xml \
--fetch-remote-resources \
/usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml
# first scan gives `notapplicable` for everything
# The workaround...
$ sudo cp /usr/share/openscap/cpe/openscap-cpe-dict.xml{,.dist}
$ sudo cp /usr/share/openscap/cpe/openscap-cpe-oval.xml{,.dist}
$ sudo curl -L \
https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.3/cpe/openscap-cpe-dict.xml \
-o /usr/share/openscap/cpe/openscap-cpe-dict.xml
$ sudo curl -L \
https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.3/cpe/openscap-cpe-oval.xml \
-o /usr/share/openscap/cpe/openscap-cpe-oval.xml
$ sudo sed -i \
-e 's|idref="cpe:/o:redhat:enterprise_linux|idref="cpe:/o:centos:centos|g' \
-e 's|ref_id="cpe:/o:redhat:enterprise_linux|ref_id="cpe:/o:centos:centos|g' \
/usr/share/xml/scap/ssg/content/ssg-rhel*.xml
# OpenSCAP
Questions?
Now do it.
Student | Control | Back Proxy | Front Proxy |
---|---|---|---|
Student01 | 137.184.98.24 10.136.0.3 |
143.198.167.106 10.136.0.2 |
104.248.115.67 10.136.0.22 |
Student02 | 159.223.122.166 10.136.0.9 |
137.184.158.238 10.136.0.19 |
198.211.99.131 10.136.0.31 |
Student03 | 159.223.114.187 10.136.0.7 |
159.223.96.252 10.136.0.21 |
104.248.115.77 10.136.0.23 |
Student04 | 137.184.219.176 10.136.0.5 |
147.182.186.155 10.136.0.13 |
104.248.115.87 10.136.0.25 |
Student05 | 137.184.195.184 10.136.0.4 |
137.184.99.29 10.136.0.17 |
104.248.115.102 10.136.0.28 |
Student06 | 159.223.118.134 10.136.0.12 |
147.182.218.170 10.136.0.14 |
104.248.115.84 10.136.0.24 |
Student07 | 159.223.122.250 10.136.0.10 |
137.184.63.109 10.136.0.15 |
104.248.115.99 10.136.0.27 |
Student08 | 159.223.122.135 10.136.0.8 |
137.184.213.156 10.136.0.20 |
104.248.115.90 10.136.0.26 |
Student09 | 159.223.114.42 10.136.0.6 |
137.184.135.121 10.136.0.18 |
104.248.115.120 10.136.0.29 |
Student10 | 159.223.118.112 10.136.0.11 |
137.184.105.180 10.136.0.16 |
104.248.115.132 10.136.0.30 |
# IP SPACE
# DIAGRAM