How to train your

Marcin Stożek "Perk"

# figure out the absolute path to the script being run a bit
# non-obvious, the ${0%/*} pulls the path out of $0, cd's into the
# specified directory, then uses $PWD to figure out where that
# directory lives - and all this in a subshell, so we don't affect
# $PWD

STEAMROOT="$(cd "${0%/*}" && echo $PWD)"

https://github.com/ValveSoftware/steam-for-linux/issues/3671

rm -rf "$STEAMROOT/"*

https://github.com/ValveSoftware/steam-for-linux/issues/3671

What is Bash?

Shell

(command processor)

 

Command language

(scripting language subtype)

 

It's a DSL for job control

 

It's Turing complete

https://en.wikipedia.org/wiki/Bash_(Unix_shell)

$ whatis bash

 $ whatis bash
 bash (1) - GNU Bourne-Again SHell
 $ whatis javascript
 javascript: nothing appropriate.
 $ whatis
 whatis what?

How does  a usual Bash script look like at the beginning?

How does it look like after some time?

Let's see how to
tame the beast

Myth: variable has a type

Bash variables are untyped

Bash variables are character strings

 

Bash is stringly typed

 

* it allows for arithmetic operations if
a variable contains digits only;
also there are arrays

https://www.tldp.org/LDP/abs/html/untyped.html

  • $1, $2, $3, $4... - positional parameters
  • $# - number of parameters
  • $@ - expansion of positional parameters - "$1" "$2" ...
  • $* - expansion of positional parameters by $IFS

 

  • $0 - name of the script (or shell)
  • $$ - PID of the current shell

 

  • $! - PID of the last command
  • $? - last command's exit status

 

  • $IFS - internal field separator, used for word splitting

Dollar sign variables

Don't

Do

Rule: always use quotes

echo $1
echo one two *
echo "$1"
echo "one two *"
$ VAR="one two *"

$ echo "${VAR}"
one two *

$ echo ${VAR}
one two bin boot cdrom dev etc home...

Why?

Don't

Do

Rule: no space around =

Why?

LANGUAGE= "w"
LANGUAGE =w
LANGUAGE = w
LANGUAGE="w"
$ FOO = bar
FOO: command not found

$ env | grep bar

$ FOO=bar env | grep bar
FOO=bar

Don't

Do

Rule: assignment quotes

OWNERSHIP=Ala has a cat
OWNERSHIP="Ala has a cat"

Why?

$ OWNERSHIP=Ala has a cat
has: command not found

$ OWNERSHIP="Ala has a cat"

$ echo "${OWNERSHIP}"
Ala has a cat

Don't

Do

Rule: immutable globals

RULES

  • Keep them to minimum
  • USE_UPPER_CASE_FORMAT
  • make them readonly 
#!/bin/bash

my_var="Some value"
#!/bin/bash

readonly MY_VAR="Some value"

Don't

Do

Rule: name the parameters

with the help of ${1:-"default"}

function list_files() {
  find . -iname "*.$1"
}
function list_files() {
  local type=${1:-"tmp"}
  find . -iname "*.${type}"
}

Don't

Do

Rule: know your ARGS

#!/bin/bash

for f in "$@"; do
  echo "$1"
  shift
done
#!/bin/bash

declare -ra ARGS=("$@")

main "${ARGS[@]}"

Why use array?

Rule: know your ARGS

$ fun() { ARGS="$@"; for f in $ARGS; do echo "$f"; done; }
$ fun 1 "2 3"
1
2
3

$ fun() { ARGS="$@"; for f in "$ARGS"; do echo "$f"; done; }
$ fun 1 "2 3"
1 2 3

$ fun() { declare -a ARGS=("$@"); for f in "${ARGS[@]}"; do echo "$f"; done; }
$ fun 1 "2 3"
1
2 3

Don't

Do

Rule: use explicit paths ./

More

If you need recursion:

use find

for file in *; do
  something "$file"
done
for file in ./*; do
  something "$file"
done

Don't

Do

Rule: use $() instead of ``

Why

  • Backticks `` are legacy and harder to read
  • $() can be nested easily
var="`command`"
var="$(command)"

Don't

Do

Rule: use subshells ()

echo "$IFS"
(IFS="\n"; echo "$IFS")
echo "$IFS"
var_backup="$var"
var="whatev"
do_something_with "$var"
var="$var_backup"

Why?

$ VAR="one"
$ echo "${VAR}"
one
$ (VAR="two"; echo "${VAR}")
two
$ echo "${VAR}"
one

Don't

Do

Rule: split pipelines with \ 

command1 \
| command2 \
| command3 \
| command4 \
...
command1 | command2 | command3 | command4 | command5...
command1 \
&& command2 \
&& command3 \
|| command4 \
...

Rule: use conditional expressions

pure bash instead of spawning external processes

[[ -z $string ]] \
  && echo 'True if the length of string is zero.'

[[ -n $string ]] \
  && echo 'True if the length of string is non-zero.'

[[ $string1 == $string2 ]] \
  && echo "True if the strings are equal"

[[ -e $file ]] && echo 'True if file exists.'

[[ -x $file ]] \
  && echo 'True if file exists and is executable.'

[[ -L $file ]] \
  && echo 'True if file exists and is a symbolic link.'

https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html
https://github.com/dylanaraps/pure-bash-bible

Don't

Do

Rule: use [[ rather than [ 

[[ "$text" = 'some text' ]] && echo 'it works'

[[ $text = 'some text' ]] && echo 'this works to!'

[[ 1 > 2 ]] \
  && echo true \
  || echo false # finally some sane result
[ $file = "bar" ] # wrong quotes
[ 1 > 2 ] # wrong - this is not what you think it is!

Don't

Do

Rule: use functions

function delete_temp_files() {
  local path="$1"
  local current_pid="$$"
  find "${path}" -iname "*.${current_pid}.tmp" -delete
}

delete_temp_files "${path}"
cd $path
find . -iname "*.$$.tmp" -delete

Rule: use functions

Variables can be local​

Code is easier to debug

Code is self documenting

Easier to understand code flow

Don't

Do

Rule: use local 

$ echo "'$MEH'"
''
$ echo "'$BLAH'"
''

$ fun() { BLAH="nasty global"; local MEH="nice local"; }
$ fun

$ echo "'$MEH'"
''
$ echo "'$BLAH'"
'nasty global'
function fun() {
  path="$1"
}
function fun() {
  local path="$1"
}

Why?

Rule: use main function

  • Most of the variables can be local​
  • We can easily locate the script's entrypoint
  • If script is really short - don't bother
#!/bin/bash

function main() {
  for i in "$@"; do
    echo "$i"
  done
}

main "$@"
 _________________________________________
/ You take the red pill — you stay in     \
| Wonderland, and I show you how deep the |
\ rabbit hole goes.                       /
 -----------------------------------------
        \   ^__^
         \  (@@)\_______
            (__)\       )\/\
                ||----w |
                ||     ||




$ cowsay -p "blah blah blah..."

Trivia: functional Bash

# Higher order functions

function stalosie() { echo "zgasla mi fajka mi zgasla"; }
function stao() { echo "lazienka jest zamknieta"; }

function cosie() {
  local co="$($1)"
  echo ">> ${co} <<"
}

$ cosie stalosie
>> zgasla mi fajka mi zgasla <<

$ cosie stao
>> lazienka jest zamknieta <<

Trivia: Bash monad

| is not a pipe any more

It's an >>= operator equivalent

It's a part of a monad

Rule: use unofficial strict mode 

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

http://redsymbol.net/articles/unofficial-bash-strict-mode/

  • set -e - exit on all non-zero exit statuses
  • set -u - do not allow usage of undefined variables
  • set -o pipefail - fail pipe if any non-zero exit codes
  • IFS=$'\n\t' - split words on newlines and tabs only

Rule: clean after yourself

function cleanup() {
  # don't do this
  # rm -rf --no-preserve-root /

  for file in /my/cache/*; do
    rm -v "${file}"
  done
}
trap cleanup EXIT

http://redsymbol.net/articles/bash-exit-traps/

  • clean when you exit (because AWS is not cheap)
  • very useful with set -e

Rule: use debug mode -x

Prints out every command before executing - expanded.​

#!/bin/bash
set -x
$ bash -x /path/to/script.sh
#!/bin/bash

set -x
some_buggy_function
set +x
#!/bin/bash -x
$ cat test.sh 
#!/bin/bash

function main() {
  for i in "$@"; do
    echo "$i"
  done
}

main "$@"



$ bash -x test.sh one "2 three"
+ main one '2 three'
+ for i in "$@"
+ echo one
one
+ for i in "$@"
+ echo '2 three'
2 three

Rule: log your output

readonly LOG_FILE="/tmp/$(basename "$0").log"
info()    { echo "[INFO]    $*" | tee -a "$LOG_FILE" >&2 ; }
warning() { echo "[WARNING] $*" | tee -a "$LOG_FILE" >&2 ; }
error()   { echo "[ERROR]   $*" | tee -a "$LOG_FILE" >&2 ; }
fatal()   { echo "[FATAL]   $*" | tee -a "$LOG_FILE" >&2 ; exit 1 ; }

https://dev.to/thiht/shell-scripts-matter

Rule: use shellcheck

https://www.shellcheck.net/

 

Shell script static analysis tool

 

Gives warnings and suggestions

 

Checks for incorrect command use

 

Can make suggestions to improve style

$ cat skrypcik.sh 
#!/bin/bash

function main() {
  local args=$@
  local date=`date`

  for $i in args; do
    echo $date
  done
}

main $*
$ shellcheck skrypcik.sh 

In skrypcik.sh line 4:
  local args=$@
        ^-- SC2034: args appears unused. Verify it or export it.
             ^-- SC2124: Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.


In skrypcik.sh line 5:
  local date=`date`
        ^-- SC2155: Declare and assign separately to avoid masking return values.
             ^-- SC2006: Use $(..) instead of legacy `..`.


In skrypcik.sh line 7:
  for $i in args; do
  ^-- SC2034: i appears unused. Verify it or export it.
      ^-- SC1086: Don't use $ on the iterator name in for loops.
            ^-- SC2043: This loop will only ever run once for a constant value. Did you perhaps mean to loop over dir/*, $var or $(cmd)?


In skrypcik.sh line 8:
    echo $date
         ^-- SC2086: Double quote to prevent globbing and word splitting.


In skrypcik.sh line 12:
main $*
     ^-- SC2048: Use "$@" (with quotes) to prevent whitespace problems.
     ^-- SC2086: Double quote to prevent globbing and word splitting.
rm -rf "${STEAMROOT:?}/"*
rm -rf "$STEAMROOT/"*

Don't

Do

Shellcheck rule SC2115

Rule: use Shell Style Guide

https://google.github.io/styleguide/shell.xml

 

It's battle tested

 

World is a better place if we have style conventions

 

Promotes good practises

Rule: unit test with shUnit2

https://github.com/kward/shunit2

 

shUnit2 is a unit test framework for Bourne shell scripts

#!/bin/bash
# file: equality_test.sh

testEquality() {
  assertEquals 1 1
}

# Load shUnit2.
. ./shunit2
$ ./equality_test.sh
testEquality

Ran 1 test.

OK

Rule: use dos2unix

The dos2unix converts line endings like a pro.

/bin/bash^M: bad interpreter: No such file or directory

If you feel like

$ [ whereis my brain?
bash: [: missing `]'

there is one more rule...

Rule: don't use Bash...

... if your script is > 100 lines long.

 

Use Python instead.

Some more #bash

$ echo 'Thank you!'

Marcin Stożek "Perk"

@marcinstozek / perk.pl

Made with Slides.com