How to train your

Marcin Stożek "Perk"

# figure out the absolute path to the script being run a bit
# non-obvious, the ${0%/*} pulls the path out of $0, cd's into the
# specified directory, then uses $PWD to figure out where that
# directory lives - and all this in a subshell, so we don't affect
# $PWD

STEAMROOT="$(cd "${0%/*}" && echo $PWD)"

https://github.com/ValveSoftware/steam-for-linux/issues/3671

rm -rf "$STEAMROOT/"*

https://github.com/ValveSoftware/steam-for-linux/issues/3671

What is Bash?

Shell

(command processor)

Command language

(scripting language subtype)

It's a DSL for job control

It's Turing complete

https://en.wikipedia.org/wiki/Bash_(Unix_shell)

$ whatis bash

 $ whatis bash
 bash (1) - GNU Bourne-Again SHell

 $ whatis javascript
 javascript: nothing appropriate.

 $ whatis
 whatis what?

How does a usual Bash script look like at the beginning?

How does it look like after some time?

Let's see how to
tame the beast

Myth: variable has a type

Bash variables are untyped

Bash variables are character strings

Bash is stringly typed

* it allows for arithmetic operations if
a variable contains digits only;
also there are arrays

https://www.tldp.org/LDP/abs/html/untyped.html

$1, $2, $3, $4... - positional parameters
$# - number of parameters
$@ - expansion of positional parameters - "$1" "$2" ...
$* - expansion of positional parameters by $IFS

$0 - name of the script (or shell)
$$ - PID of the current shell

$! - PID of the last command
$? - last command's exit status

$IFS - internal field separator, used for word splitting

Dollar sign variables

Don't

Do

Rule: always use quotes

echo $1
echo one two *

echo "$1"
echo "one two *"

$ VAR="one two *"

$ echo "${VAR}"
one two *

$ echo ${VAR}
one two bin boot cdrom dev etc home...

Why?

Don't

Do

Rule: no space around =

Why?

LANGUAGE= "w"
LANGUAGE =w
LANGUAGE = w

LANGUAGE="w"

$ FOO = bar
FOO: command not found

$ env | grep bar

$ FOO=bar env | grep bar
FOO=bar

Don't

Do

Rule: assignment quotes

OWNERSHIP=Ala has a cat

OWNERSHIP="Ala has a cat"

Why?

$ OWNERSHIP=Ala has a cat
has: command not found

$ OWNERSHIP="Ala has a cat"

$ echo "${OWNERSHIP}"
Ala has a cat

Don't

Do

Rule: immutable globals

RULES

Keep them to minimum
USE_UPPER_CASE_FORMAT
make them readonly

#!/bin/bash

my_var="Some value"

#!/bin/bash

readonly MY_VAR="Some value"

Don't

Do

Rule: name the parameters

with the help of ${1:-"default"}

function list_files() {
  find . -iname "*.$1"
}

function list_files() {
  local type=${1:-"tmp"}
  find . -iname "*.${type}"
}

Don't

Do

Rule: know your ARGS

#!/bin/bash

for f in "$@"; do
  echo "$1"
  shift
done

#!/bin/bash

declare -ra ARGS=("$@")

main "${ARGS[@]}"

Why use array?

Rule: know your ARGS

$ fun() { ARGS="$@"; for f in $ARGS; do echo "$f"; done; }
$ fun 1 "2 3"
1
2
3

$ fun() { ARGS="$@"; for f in "$ARGS"; do echo "$f"; done; }
$ fun 1 "2 3"
1 2 3

$ fun() { declare -a ARGS=("$@"); for f in "${ARGS[@]}"; do echo "$f"; done; }
$ fun 1 "2 3"
1
2 3

Don't

Do

Rule: use explicit paths ./

If you need recursion:

use find

for file in *; do
  something "$file"
done

for file in ./*; do
  something "$file"
done

Don't

Do

Rule: use $() instead of ``

Why

Backticks `` are legacy and harder to read
$() can be nested easily

var="`command`"

var="$(command)"

Don't

Do

Rule: use subshells ()

echo "$IFS"
(IFS="\n"; echo "$IFS")
echo "$IFS"

var_backup="$var"
var="whatev"
do_something_with "$var"
var="$var_backup"

Why?

$ VAR="one"
$ echo "${VAR}"
one
$ (VAR="two"; echo "${VAR}")
two
$ echo "${VAR}"
one

Don't

Do

Rule: split pipelines with \

command1 \
| command2 \
| command3 \
| command4 \
...

command1 | command2 | command3 | command4 | command5...

command1 \
&& command2 \
&& command3 \
|| command4 \
...

Rule: use conditional expressions

pure bash instead of spawning external processes

[[ -z $string ]] \
  && echo 'True if the length of string is zero.'

[[ -n $string ]] \
  && echo 'True if the length of string is non-zero.'

[[ $string1 == $string2 ]] \
  && echo "True if the strings are equal"

[[ -e $file ]] && echo 'True if file exists.'

[[ -x $file ]] \
  && echo 'True if file exists and is executable.'

[[ -L $file ]] \
  && echo 'True if file exists and is a symbolic link.'

https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html
https://github.com/dylanaraps/pure-bash-bible

Don't

Do

Rule: use [[ rather than [

[[ "$text" = 'some text' ]] && echo 'it works'

[[ $text = 'some text' ]] && echo 'this works to!'

[[ 1 > 2 ]] \
  && echo true \
  || echo false # finally some sane result

[ $file = "bar" ] # wrong quotes
[ 1 > 2 ] # wrong - it's not what you think it is!

Don't

Do

Rule: use functions

function delete_temp_files() {
  local path="$1"
  local current_pid="$$"
  find "${path}" -iname "*.${current_pid}.tmp" -delete
}

delete_temp_files "${path}"

cd $path
find . -iname "*.$$.tmp" -delete

Rule: use functions

Variables can be local

Code is easier to debug

Code is self documenting

Easier to understand code flow

Don't

Do

Rule: use local

$ echo "'$MEH'"
''
$ echo "'$BLAH'"
''

$ fun() { BLAH="nasty global"; local MEH="nice local"; }
$ fun

$ echo "'$MEH'"
''
$ echo "'$BLAH'"
'nasty global'

function fun() {
  path="$1"
}

function fun() {
  local path="$1"
}

Why?

What about the nested functions calls?

It sort of works for nested calls...

$ ./test.sh
some value

$ cat test.sh
#!/bin/bash

function one() {
  local local_variable="some value"
  two
}

function two() {
  echo $local_variable
}

one

Rule: use main function

Most of the variables can be local (sort of)
We can easily locate the script's entrypoint
If script is really short - don't bother

#!/bin/bash

function main() {
  for i in "$@"; do
    echo "$i"
  done
}

main "$@"

 _________________________________________
/ You take the red pill — you stay in     \
| Wonderland, and I show you how deep the |
\ rabbit hole goes.                       /
 -----------------------------------------
        \   ^__^
         \  (@@)\_______
            (__)\       )\/\
                ||----w |
                ||     ||




$ cowsay -p "blah blah blah..."

Trivia: functional Bash

# Higher order functions

function stalosie() { echo "zgasla mi fajka mi zgasla"; }
function stao() { echo "lazienka jest zamknieta"; }

function cosie() {
  local co="$($1)"
  echo ">> ${co} <<"
}

$ cosie stalosie
>> zgasla mi fajka mi zgasla <<

$ cosie stao
>> lazienka jest zamknieta <<

Trivia: Bash monad

| is not a pipe any more

It's an >>= operator equivalent

It's a part of a monad

Rule: use unofficial strict mode

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

http://redsymbol.net/articles/unofficial-bash-strict-mode/

set -e - exit on all non-zero exit statuses
set -u - do not allow usage of undefined variables
set -o pipefail - fail pipe if any non-zero exit codes
IFS=$'\n\t' - split words on newlines and tabs only

Rule: clean after yourself

function cleanup() {
  # don't do this
  # rm -rf --no-preserve-root /

  for file in /my/cache/*; do
    rm -v "${file}"
  done
}
trap cleanup EXIT

http://redsymbol.net/articles/bash-exit-traps/

clean when you exit (because AWS is not cheap)
very useful with set -e

Rule: use debug mode -x

Prints out every command before executing - expanded.

#!/bin/bash
set -x

$ bash -x /path/to/script.sh

#!/bin/bash

set -x
some_buggy_function
set +x

#!/bin/bash -x

$ cat test.sh 
#!/bin/bash

function main() {
  for i in "$@"; do
    echo "$i"
  done
}

main "$@"



$ bash -x test.sh one "2 three"
+ main one '2 three'
+ for i in "$@"
+ echo one
one
+ for i in "$@"
+ echo '2 three'
2 three

Rule: log your output

readonly LOG_FILE="/tmp/$(basename "$0").log"
info()    { echo "[INFO]    $*" | tee -a "$LOG_FILE" >&2 ; }
warning() { echo "[WARNING] $*" | tee -a "$LOG_FILE" >&2 ; }
error()   { echo "[ERROR]   $*" | tee -a "$LOG_FILE" >&2 ; }
fatal()   { echo "[FATAL]   $*" | tee -a "$LOG_FILE" >&2 ; exit 1 ; }

https://dev.to/thiht/shell-scripts-matter

Rule: use shellcheck

https://www.shellcheck.net/

Shell script static analysis tool

Gives warnings and suggestions

Checks for incorrect command use

Can make suggestions to improve style

$ cat skrypcik.sh 
#!/bin/bash

function main() {
  local args=$@
  local date=`date`

  for $i in args; do
    echo $date
  done
}

main $*

$ shellcheck skrypcik.sh 

In skrypcik.sh line 4:
  local args=$@
        ^-- SC2034: args appears unused. Verify it or export it.
             ^-- SC2124: Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.


In skrypcik.sh line 5:
  local date=`date`
        ^-- SC2155: Declare and assign separately to avoid masking return values.
             ^-- SC2006: Use $(..) instead of legacy `..`.


In skrypcik.sh line 7:
  for $i in args; do
  ^-- SC2034: i appears unused. Verify it or export it.
      ^-- SC1086: Don't use $ on the iterator name in for loops.
            ^-- SC2043: This loop will only ever run once for a constant value. Did you perhaps mean to loop over dir/*, $var or $(cmd)?


In skrypcik.sh line 8:
    echo $date
         ^-- SC2086: Double quote to prevent globbing and word splitting.


In skrypcik.sh line 12:
main $*
     ^-- SC2048: Use "$@" (with quotes) to prevent whitespace problems.
     ^-- SC2086: Double quote to prevent globbing and word splitting.

rm -rf "${STEAMROOT:?}/"*

rm -rf "$STEAMROOT/"*

Don't

Do

Shellcheck rule SC2115

Rule: use Shell Style Guide

https://google.github.io/styleguide/shell.xml

It's battle tested

World is a better place if we have style conventions

Promotes good practises

Rule: unit test with shUnit2

https://github.com/kward/shunit2

shUnit2 is a unit test framework for Bourne shell scripts

#!/bin/bash
# file: equality_test.sh

testEquality() {
  assertEquals 1 1
}

# Load shUnit2.
. ./shunit2

$ ./equality_test.sh
testEquality

Ran 1 test.

OK

Rule: use dos2unix

The dos2unix converts line endings like a pro.

/bin/bash^M: bad interpreter: No such file or directory

If you feel like

$ [ whereis my brain?
bash: [: missing `]'

there is one more rule...

Rule: don't use Bash...

... if your script is > 100 lines long.

Use Python instead.

Some more #bash

$ echo 'Thank you!'

Marcin Stożek "Perk"

@marcinstozek / perk.pl

How to train your Bash

By Marcin Stożek

How to train your Bash

https://youtu.be/ccM0-QCcZ9A

1,754

How to train your

What is Bash?

$ whatis bash

How does a usual Bash script look like at the beginning?

How does it look like after some time?

Let's see how to tame the beast

Myth: variable has a type

Dollar sign variables

Don't

Do

Rule: always use quotes

Why?

Don't

Do

Rule: no space around =

Why?

Don't

Do

Rule: assignment quotes

Why?

Don't

Do

Rule: immutable globals

RULES

Don't

Do

Rule: name the parameters

Don't

Do

Rule: know your ARGS

Why use array?

Rule: know your ARGS

Don't

Do

Rule: use explicit paths ./

More

Don't

Do

Rule: use $() instead of ``

Why

Don't

Do

Rule: use subshells ()

Why?

Don't

Do

Rule: split pipelines with \

Rule: use conditional expressions

Don't

Do

Rule: use [[ rather than [

Don't

Do

Rule: use functions

Rule: use functions

Don't

Do

Rule: use local

Why?

What about the nested functions calls?

It sort of works for nested calls...

Rule: use main function

Trivia: functional Bash

Trivia: Bash monad

| is not a pipe any more

It's an >>= operator equivalent

It's a part of a monad

Rule: use unofficial strict mode

Rule: clean after yourself

Rule: use debug mode -x

Rule: log your output

Rule: use shellcheck

Don't

Do

Shellcheck rule SC2115

Rule: use Shell Style Guide

Rule: unit test with shUnit2

Rule: use dos2unix

If you feel like

there is one more rule...

Let's see how to
tame the beast