👻Spook👻
🐚In The Shell🐚

AKA

Spit: A Postmodern DevOps tool, written in Perl 6

https://github.com/spitsh/spitsh

https://slides.com/lloydfournier/spook-in-the-shell

 

.WHO

Lloyd Fournier

DevOps Engineer 😂

Financial Hacker

(Happily Unemployed)

lloyd.fourn@gmail.com

llfourn #perl6 on irc.freenode.net

@LLFOURN

Australian

#👻
say "hello world";
#🐚
BEGIN(){
  exec 3>&1
  say(){ printf '%s\n' "$1" >&3; }
}
MAIN(){ say 'hello world'; }
BEGIN && MAIN

.WHAT

It's a shell script compiler!

Spook

Kompile-time 

Only 

Operations

Perl

Six

Spook in the shell

llfourn$ spit hello-world.sp
BEGIN(){
  exec 3>&1
  say(){ printf '%s\n' "$1" >&3; }
}
MAIN(){ say 'hello world'; }
BEGIN && MAIN

It's the compiler

It's called spit.

It spits shell.

Like a 🐪 .

It's uge

10,000+ loc

1,400+ spec tests

.WHY

Shell scripts are terrible

DevOps tools are terrible

Maybe if I make devops tool out of shell scripts it will be OK.

CAASS

Configuration as a Shell Script

(A post-modern approach to configuration)

🐚⟹🐪⟹🦋

⟸👻

Can the operations you want to perform on your systems be reduced to a list of shell commands?

...Then lets produce that list of shell commands.

What you can do with shell commands:

  • Define/Create Docker images
  • Create virtual machines
  • Consume JSON APIs
  • Add DNS entries
  • Install/uninstall packages
  • Inspect machine state
  • SSH
  • Modify configuration files
  • Restart things
  • Add/delete users
  • Create X509 certificates
  • accidentally rm -rf things

What you can't do with shell commands:

  • Create an iOS app
  • Create a website
  • Create a VR game

What was wrong with shell?

  • Many gotchas
  • difficult to handle failure well
  • difficult to make libraries
  • No compile time checking
  • Not portable

Instead of writing shell let's describe our procedures in a Perl 6 like language and compile it out 🎉

Let's Have:    

  • Modules
  • Type checking
  • Function signatures
  • Classes and methods
  • Regexes
  • Testing frameworks
  • Emojis

Let's have typescript for /bin/sh!

.HOW

Write a compiler in rakudo

.HOW

  1. Parse with Perl 6 grammars
  2. Build an AST
  3. Compile the AST into shell

* Idnnowtfitalkinbout

$ ./spit eval 'say 1 + 1' --target=parse
「say 1 + 1」
 statementlist => 「say 1 + 1」
  statement => 「say 1 + 1」
   statement => 「say 1 + 1」
    EXPR-and-mod => 「say 1 + 1」
     EXPR => 「say 1 + 1」
      termish => 「say 1 + 1」
       term => 「say 1 + 1」
        name => 「say」
        call-args => 「 1 + 1」
         args => 「1 + 1」
          pos => 「1 + 1」
           termish => 「1」
            term => 「1」
             int => 「1」
           infix => 「+」
            eq-infix => 「+」
             sym => 「+」
            sym => 「+」
             sym => 「+」
           termish => 「1」
            term => 「1」
             int => 「1」
$ spit eval 'say 1 + 1' --target=stage1
CompUnit
  - Block --> Any
    - SubCall(say)
      - IntExpr(+)
        - IVal(1)
        - IVal(1)
$ spit eval 'say 1 + "foo"' --target=stage1
CompUnit
  - Block --> Any
    - SubCall(say) 
      - IntExpr(+) #Str context
        - IVal(1)   #Int context
        - SVal(foo) #Int context

$ spit eval 'say 1 + 1' --target=stage3
CompUnit
  - Block --> Any
    - SubCall(say)
      - IVal+{Force(Str,$)}(2)
depends:
Block --> FD
  - Cmd
    - SVal(exec)
    - IVal+{Force(FD,$)}(3)
    - IVal+{Force(FD,$)}(1)
  - ConstantDecl+{SAST::Option}($:OUT) = 
    - IVal+{Force(FD,$)}(3)
SubDeclare(say)
  - Signature(Str $str)
  - Block --> Bool
    - Return
      - Cmd
        - SVal(printf)
        - SVal(%s\n)
        - Var($str)
        - IVal+{Force(FD,$)}(1)
        - IVal+{Force(FD,$)}(3)
$ spit eval 'say 1 + 1'
BEGIN(){
  exec 3>&1
  say(){ printf '%s\n' "$1" >&3; }
}
MAIN(){ say 2; }
BEGIN && MAIN

.WHY....WHY

Don't have to be bound to ssh/agents. You can introduce procedures anywhere that allows you to run shell scripts.

  • Debian preseed
  • curl | sh
  • cloud-init
  • container images
  • ssh

Brief introduction to Spook 👻

Typed Strings

  • Everything is a string
  • Some strings we can attach more meaning to

 

 

say File</etc/hosts>.owner;

say GitHub<stedolan/jq>.latest-release-url;

for File</bin /usr/bin> {
    say "$_ was last modified at {.mtime}"
}
#👻
say File</etc/hosts>.owner;

say GitHub<stedolan/jq>.latest-release-url;

for File</bin /usr/bin> {
    say "$_ was last modified at {.mtime}"
}
#🐚
MAIN(){
  say "$(stat -c %U /etc/hosts)"
  say "$(latest_release_url stedolan/jq)"
  for _1 in $(list /bin /usr/bin); do
    say "$_1 was last modified at $(mtime "$_1")"
  done
}
#🐚 spit typed-strings.sp
BEGIN(){
  e(){ printf %s "$1"; }
  list(){ test "$*" && printf '%s\n' "$*"; }
  IFS='
'
  exec 3>&1
  say(){ printf '%s\n' "$1" >&3; }
  release_url(){ e "https://github.com/$1/releases/$2"; }
  exec 4>/dev/null
  exists(){ command -v "$1" >&4; }
  Ljoin(){ s="$1" awk '{if(NR != 1) printf "%s", ENVIRON["s"]; printf "%s", $0}'; }
  note(){ printf '%s\n' "$1" >&2; }
  Ffind(){ find "$1" ${name:+$(list -name "$name")}; }
  at_pos(){ sed -n "$(($1+1))p"|tr -d '\n'; }
  ctime(){ stat -c %z "$1" 2>&4|sed -r 's/ /T/;s/0{6}$//'; }
  last_updated(){ ctime "$(name='*.gz' Ffind /var/cache/apk/|at_pos 0)"; }
  now(){ date -u "+%FT$(nmeter -d0 %3t|head -n1)"; }
  posix(){ date -u -D %Y-%m-%dT%T -d "$1" +%s; }
  update_pkglist(){ apk update >&4 2>&4; }
  check_update(){
    last_updated="$(last_updated)"
    if ! test "$last_updated" || [ $(($(posix "$(now)")-$(posix "$last_updated"))) -gt 86400 ]; then
      note "Updating package list because it $(test "$last_updated" && e "was last updated at $last_updated" || e "doesn't exist")" >&4
      update_pkglist
    else
      false
    fi
  }
  Linstall(){
    note "Installing pacakges: $(e "$1"|Ljoin ', ')" >&4
    check_update
    apk add $1 --no-progress >&4 2>&2
  }
  ensure_install(){
    if ! exists "$1"; then
      Linstall "$1"
    fi
    e "$1"
  }
  curl="$(ensure_install curl)"
  redirect_url(){ "$curl" -Isw '%{redirect_url}' -o /dev/null "$1"; }
  latest_release_url(){ redirect_url "$(release_url "$1" latest)"; }
  mtime(){ stat -c %y "$1" 2>&4|sed -r 's/ /T/;s/0{6}$//'; }
}
MAIN(){
  say "$(stat -c %U /etc/hosts)"
  say "$(latest_release_url stedolan/jq)"
  for _1 in /bin /usr/bin; do
    say "$_1 was last modified at $(mtime "$_1")"
  done
}
BEGIN && MAIN
#🐚 spit typed-strings.sp --os centos
BEGIN(){
  e(){ printf %s "$1"; }
  exec 3>&1
  say(){ printf '%s\n' "$1" >&3; }
  release_url(){ e "https://github.com/$1/releases/$2"; }
  redirect_url(){ curl -Isw '%{redirect_url}' -o /dev/null "$1"; }
  latest_release_url(){ redirect_url "$(release_url "$1" latest)"; }
  exec 4>/dev/null
  mtime(){ date -u -r "$1" +%FT%T.%3N 2>&4; }
}
MAIN(){
  say "$(stat -c %U /etc/hosts)"
  say "$(latest_release_url stedolan/jq)"
  for _1 in /bin /usr/bin; do
    say "$_1 was last modified at $(mtime "$_1")"
  done
}
BEGIN && MAIN
spit compile type-demo.spt

👻DEMO🐚

Shell commands

#👻
${ echo "hello world" };
my $cmd = "echo";
${ $cmd "hello world" };
File</etc/hosts>.slurp.${ grep 'localhost' };
${ cat '/etc/hosts' | grep 'localhost' };
${ grep 'localhost' < '/etc/hosts' };
#🐚
echo 'hello world'
cmd=echo
"$cmd" 'hello world'
slurp /etc/hosts|grep localhost
cat /etc/hosts|grep localhost
grep localhost </etc/hosts

Object Orientated Shell

(yes really)

#👻
class User {
    method create( :$home-dir ) on {
        Alpine ${ adduser -D  ("-h$_" if $home-dir) $self }
        GNU    ${ useradd ("-d$_" if $home-dir) -N $self  }
    }

    method delete? on {
        RHEL ${ userdel $self *>X}
        Any  ${ deluser $self *>X}
    }

    method exists? ${ id -u $self *>X }
}

my $user = User<llfourn>;
$user.create;
say "$user exists: {$user.exists.gist}"
BEGIN(){
  IFS='
'
  e(){ printf %s "$1"; }
  create(){ adduser -D ${home_dir:+"-h$home_dir"} "$1"; }
  exec 4>/dev/null
  gist(){ test "$1" && e True || e False; }
  exec 3>&1
  say(){ printf '%s\n' "$1" >&3; }
  delete(){ deluser "$1" >&4 2>&4; }
}
MAIN(){
  user=llfourn
  create "$user"
  say "$user exists: $(gist "$(id -u "$user" >&4 2>&4 && e 1)")"
  delete "$user"
  say "$user exists: $(gist "$(id -u "$user" >&4 2>&4 && e 1)")"
}
BEGIN && MAIN

Type checking

  • Everything is a Str (almost)
  • Is only done at compile time (for now)

  • Only "primitive" type checking is implemented

Type checking

#👻
# Int is superfluous
my Int $a = 1;
$a = File</etc/hosts>; # not ok
$a = PID<42>; #ok
$ spit eval 'say Int.PRIMITIVE' -d
Int
$ spit eval 'say File.PRIMITIVE' -d
Str
$ spit eval 'say PID.PRIMITIVE' -d
Int

Variables

#👻
my $a = "foo";
{
    my $a = "looks it's me again";
}
#🐚
a=foo

a_1="looks it's me again"


  



myoption='the defualt value'
say "$option"


file="$(tmp)"
say File
#🐚
say 'a constant value'
#👻
constant $const = "a constant value";
say $const;

Options

#👻
constant $:foo = "bar";
say $:foo;
#🐚
#spit options.sp
MAIN(){ say bar; }
#spit options.sp -o foo=baz
MAIN(){ say baz; }

Namespaced

#👻
class MyClass {
    constant $:foo = "bar";
    
    method do-something { say $:foo }
}

say $MyClass:foo;

Required

#👻
class MyClass {
    constant $:foo is required;
    
    method do-something { say $:foo }
}

say $MyClass:foo;

Routines

#👻
sub foo($a, $b?, $c = "see", :$d, :$e = "eee") {
    say "$a $b $c $d $e";
}

foo("one", "two", :d<three> );
#🐚
MAIN(){
  foo(){ say "$1 $2 $3 $d $e"; }
  e=eee d=three foo one two see
}
  • named parameters
  • defaults
  • optionals
  • Mandatory return type

Slurpies

#👻
sub bar($a, $b, Int *@c) {
    say "$a $b @c";
}

bar "one","two", 1..5, 42, 1337;
#🐚
MAIN(){
  bar(){
    a=$1 b=$2 shift 2
    say "$a $b $*"
  }
  bar one two $(seq 1 5) 42 1337
}

Return types

#👻
sub string~ { "foo" }
sub forty-two+ { 42 }
sub question? { True }
sub dabien -->DockerImg {
    Docker.create('debian')
          .commit(name => "dabien");
}

eval

  • creates an embedded shell script
  • can have different options
  • can interpolate runtime values into the embedded shell script
  • It's pretty spicy
#🐚
MAIN(){
  script="$(cat <<-'🐈'
        {
          exec 3>&1
          say(){ printf '%s\n' "$1" >&3; }
          say 'hello world'
        }
        🐈
    )"
}
#👻
my $script = eval{ say "hello world" };
#👻
say Docker.create('debian')
          .cleanup
          .exec( eval(os => Debian){ Pkg<curl>.installed } ).gist;

say Docker.create('centos')
          .cleanup
          .exec( eval(os => CentOS){ Pkg<curl>.installed } ).gist;
#🐚
MAIN(){
  say "$(gist "$(Dexec "$(cleanup "$(create debian)")" "$(cat <<-'🐈'
        {
          exec 4>/dev/null
          dpkg -s curl >&4 2>&4
        }
        🐈
    )" && e 1)")"
  say "$(gist "$(Dexec "$(cleanup "$(create centos)")" "$(cat <<-'😺'
        {
          exec 4>/dev/null
          installed(){ yum list installed "$1" >&4 2>&4; }
          installed curl
        }
        😺
    )" && e 1)")"
}
END(){
  remove "$(slurp "$docker_cleanup")"
  rm -rf "$docker_cleanup"
}
trap END EXIT;trap 'trap : TERM; exit 1' TERM
BEGIN && MAIN

spit eval-docker.sp -h

#👻
my $one = "1";
my $two = "2";
my $three = "3";

eval{ say "$one $two $three" }.${sh};
#🐚
MAIN(){
  one=1
  two=2
  three=3
  cat <<-'🐈' | 
        {
          one='💐'
          two='🌼'
          three='💞'
          exec 3>&1
          say(){ printf '%s\n' "$1" >&3; }
          say "$one $two $three"
        }
        🐈
    subst_eval 💐 "$one"|subst_eval 💞 "$three"|subst_eval 🌼 "$two"|\
    sh
}
BEGIN && MAIN

Future

  • multi-dispatch on signatures
  • macros
  • better module support
  • s/regex/replace/
  • Async/Promises

spit.sh

curl https://spit.sh/minecraft?version=1.12.1 | sh

Rough Edges

  • Not ready for production
  • subshell problem

Demo: DigitalOcean/Twilio

class Twilio {
    constant $:auth-token is required;
    constant $:account-sid is required;
    constant $:api = "api.twilio.com/2010-04-01";
    constant $:from is required;


    static method send-sms(:$from = $:from, :$to, :$message) {
        HTTP("https://{$:account-sid}:{$:auth-token}@" ~
             "$:api/Accounts/$:account-sid/Messages")

        .request(
            'POST',
            form => (
                From => $from,
                To => $to,
                Body => $message,
            ),
        ).ok;
    }
}
constant $:to = '+310655512458';

my $droplet = DO.create-ssh-seeded(
    :name<twilio-test>,
    :region<ams2>
);

$droplet.ssh-seed: eval(os => Debian){
    Twilio.send-sms(
        :$:to,
        message => File</etc/debian_version>.slurp
    );
};

END $droplet.delete;
spit DO-twilio.sp -h -o log
#👻
sub uppercase($str)~ { 
    say "Uppercasing $str"; 
    $str.uc;
}

my $a = uppercase "foo";
say ">>>$a<<<";
#🐚
BEGIN(){
  e(){ printf %s "$1"; }
  exec 3>&1
  say(){ printf '%s\n' "$1" >&3; }
  uc(){ tr [:lower:] [:upper:]; }
}
MAIN(){
  uppercase(){
    say "Uppercasing $1"
    e "$1"|uc
  }
  a="$(uppercase foo)"
  say ">>>$a<<<"
}
BEGIN && MAIN

Spook in the Shell

By Lloyd Fournier