👻Spook👻
🐚In The Shell🐚
AKA
Spit: A Postmodern DevOps tool, written in Perl 6
https://github.com/spitsh/spitsh
https://slides.com/lloydfournier/spook-in-the-shell
.WHO
Lloyd Fournier
DevOps Engineer 😂
Financial Hacker
(Happily Unemployed)
lloyd.fourn@gmail.com
llfourn #perl6 on irc.freenode.net
@LLFOURN
Australian
#👻
say "hello world";
#🐚
BEGIN(){
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
}
MAIN(){ say 'hello world'; }
BEGIN && MAIN
.WHAT
It's a shell script compiler!
Spook
Kompile-time
Only
Operations
Perl
Six
Spook in the shell
llfourn$ spit hello-world.sp
BEGIN(){
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
}
MAIN(){ say 'hello world'; }
BEGIN && MAIN
It's the compiler
It's called spit.
It spits shell.
Like a 🐪 .
It's uge
10,000+ loc
1,400+ spec tests
.WHY
Shell scripts are terrible
DevOps tools are terrible
Maybe if I make devops tool out of shell scripts it will be OK.
CAASS
Configuration as a Shell Script
(A post-modern approach to configuration)
🐚⟹🐪⟹🦋
⟸👻
Can the operations you want to perform on your systems be reduced to a list of shell commands?
...Then lets produce that list of shell commands.
What you can do with shell commands:
- Define/Create Docker images
- Create virtual machines
- Consume JSON APIs
- Add DNS entries
- Install/uninstall packages
- Inspect machine state
- SSH
- Modify configuration files
- Restart things
- Add/delete users
- Create X509 certificates
- accidentally rm -rf things
What you can't do with shell commands:
- Create an iOS app
- Create a website
- Create a VR game
What was wrong with shell?
- Many gotchas
- difficult to handle failure well
- difficult to make libraries
- No compile time checking
- Not portable
Instead of writing shell let's describe our procedures in a Perl 6 like language and compile it out 🎉
Let's Have:
- Modules
- Type checking
- Function signatures
- Classes and methods
- Regexes
- Testing frameworks
- Emojis
Let's have typescript for /bin/sh!
.HOW
Write a compiler in rakudo
.HOW
- Parse with Perl 6 grammars
- Build an AST
- Compile the AST into shell
* Idnnowtfitalkinbout
$ ./spit eval 'say 1 + 1' --target=parse
「say 1 + 1」
statementlist => 「say 1 + 1」
statement => 「say 1 + 1」
statement => 「say 1 + 1」
EXPR-and-mod => 「say 1 + 1」
EXPR => 「say 1 + 1」
termish => 「say 1 + 1」
term => 「say 1 + 1」
name => 「say」
call-args => 「 1 + 1」
args => 「1 + 1」
pos => 「1 + 1」
termish => 「1」
term => 「1」
int => 「1」
infix => 「+」
eq-infix => 「+」
sym => 「+」
sym => 「+」
sym => 「+」
termish => 「1」
term => 「1」
int => 「1」
$ spit eval 'say 1 + 1' --target=stage1
CompUnit
- Block --> Any
- SubCall(say)
- IntExpr(+)
- IVal(1)
- IVal(1)
$ spit eval 'say 1 + "foo"' --target=stage1
CompUnit
- Block --> Any
- SubCall(say)
- IntExpr(+) #Str context
- IVal(1) #Int context
- SVal(foo) #Int context
$ spit eval 'say 1 + 1' --target=stage3
CompUnit
- Block --> Any
- SubCall(say)
- IVal+{Force(Str,$)}(2)
depends:
Block --> FD
- Cmd
- SVal(exec)
- IVal+{Force(FD,$)}(3)
- IVal+{Force(FD,$)}(1)
- ConstantDecl+{SAST::Option}($:OUT) =
- IVal+{Force(FD,$)}(3)
SubDeclare(say)
- Signature(Str $str)
- Block --> Bool
- Return
- Cmd
- SVal(printf)
- SVal(%s\n)
- Var($str)
- IVal+{Force(FD,$)}(1)
- IVal+{Force(FD,$)}(3)
$ spit eval 'say 1 + 1'
BEGIN(){
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
}
MAIN(){ say 2; }
BEGIN && MAIN
.WHY....WHY
Don't have to be bound to ssh/agents. You can introduce procedures anywhere that allows you to run shell scripts.
- Debian preseed
- curl | sh
- cloud-init
- container images
- ssh
Brief introduction to Spook 👻
Typed Strings
- Everything is a string
- Some strings we can attach more meaning to
say File</etc/hosts>.owner;
say GitHub<stedolan/jq>.latest-release-url;
for File</bin /usr/bin> {
say "$_ was last modified at {.mtime}"
}
#👻
say File</etc/hosts>.owner;
say GitHub<stedolan/jq>.latest-release-url;
for File</bin /usr/bin> {
say "$_ was last modified at {.mtime}"
}
#🐚
MAIN(){
say "$(stat -c %U /etc/hosts)"
say "$(latest_release_url stedolan/jq)"
for _1 in $(list /bin /usr/bin); do
say "$_1 was last modified at $(mtime "$_1")"
done
}
#🐚 spit typed-strings.sp
BEGIN(){
e(){ printf %s "$1"; }
list(){ test "$*" && printf '%s\n' "$*"; }
IFS='
'
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
release_url(){ e "https://github.com/$1/releases/$2"; }
exec 4>/dev/null
exists(){ command -v "$1" >&4; }
Ljoin(){ s="$1" awk '{if(NR != 1) printf "%s", ENVIRON["s"]; printf "%s", $0}'; }
note(){ printf '%s\n' "$1" >&2; }
Ffind(){ find "$1" ${name:+$(list -name "$name")}; }
at_pos(){ sed -n "$(($1+1))p"|tr -d '\n'; }
ctime(){ stat -c %z "$1" 2>&4|sed -r 's/ /T/;s/0{6}$//'; }
last_updated(){ ctime "$(name='*.gz' Ffind /var/cache/apk/|at_pos 0)"; }
now(){ date -u "+%FT$(nmeter -d0 %3t|head -n1)"; }
posix(){ date -u -D %Y-%m-%dT%T -d "$1" +%s; }
update_pkglist(){ apk update >&4 2>&4; }
check_update(){
last_updated="$(last_updated)"
if ! test "$last_updated" || [ $(($(posix "$(now)")-$(posix "$last_updated"))) -gt 86400 ]; then
note "Updating package list because it $(test "$last_updated" && e "was last updated at $last_updated" || e "doesn't exist")" >&4
update_pkglist
else
false
fi
}
Linstall(){
note "Installing pacakges: $(e "$1"|Ljoin ', ')" >&4
check_update
apk add $1 --no-progress >&4 2>&2
}
ensure_install(){
if ! exists "$1"; then
Linstall "$1"
fi
e "$1"
}
curl="$(ensure_install curl)"
redirect_url(){ "$curl" -Isw '%{redirect_url}' -o /dev/null "$1"; }
latest_release_url(){ redirect_url "$(release_url "$1" latest)"; }
mtime(){ stat -c %y "$1" 2>&4|sed -r 's/ /T/;s/0{6}$//'; }
}
MAIN(){
say "$(stat -c %U /etc/hosts)"
say "$(latest_release_url stedolan/jq)"
for _1 in /bin /usr/bin; do
say "$_1 was last modified at $(mtime "$_1")"
done
}
BEGIN && MAIN
#🐚 spit typed-strings.sp --os centos
BEGIN(){
e(){ printf %s "$1"; }
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
release_url(){ e "https://github.com/$1/releases/$2"; }
redirect_url(){ curl -Isw '%{redirect_url}' -o /dev/null "$1"; }
latest_release_url(){ redirect_url "$(release_url "$1" latest)"; }
exec 4>/dev/null
mtime(){ date -u -r "$1" +%FT%T.%3N 2>&4; }
}
MAIN(){
say "$(stat -c %U /etc/hosts)"
say "$(latest_release_url stedolan/jq)"
for _1 in /bin /usr/bin; do
say "$_1 was last modified at $(mtime "$_1")"
done
}
BEGIN && MAIN
spit compile type-demo.spt
👻DEMO🐚
Shell commands
#👻
${ echo "hello world" };
my $cmd = "echo";
${ $cmd "hello world" };
File</etc/hosts>.slurp.${ grep 'localhost' };
${ cat '/etc/hosts' | grep 'localhost' };
${ grep 'localhost' < '/etc/hosts' };
#🐚
echo 'hello world'
cmd=echo
"$cmd" 'hello world'
slurp /etc/hosts|grep localhost
cat /etc/hosts|grep localhost
grep localhost </etc/hosts
Object Orientated Shell
(yes really)
#👻
class User {
method create( :$home-dir ) on {
Alpine ${ adduser -D ("-h$_" if $home-dir) $self }
GNU ${ useradd ("-d$_" if $home-dir) -N $self }
}
method delete? on {
RHEL ${ userdel $self *>X}
Any ${ deluser $self *>X}
}
method exists? ${ id -u $self *>X }
}
my $user = User<llfourn>;
$user.create;
say "$user exists: {$user.exists.gist}"
BEGIN(){
IFS='
'
e(){ printf %s "$1"; }
create(){ adduser -D ${home_dir:+"-h$home_dir"} "$1"; }
exec 4>/dev/null
gist(){ test "$1" && e True || e False; }
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
delete(){ deluser "$1" >&4 2>&4; }
}
MAIN(){
user=llfourn
create "$user"
say "$user exists: $(gist "$(id -u "$user" >&4 2>&4 && e 1)")"
delete "$user"
say "$user exists: $(gist "$(id -u "$user" >&4 2>&4 && e 1)")"
}
BEGIN && MAIN
Type checking
- Everything is a Str (almost)
-
Is only done at compile time (for now)
-
Only "primitive" type checking is implemented
Type checking
#👻
# Int is superfluous
my Int $a = 1;
$a = File</etc/hosts>; # not ok
$a = PID<42>; #ok
$ spit eval 'say Int.PRIMITIVE' -d
Int
$ spit eval 'say File.PRIMITIVE' -d
Str
$ spit eval 'say PID.PRIMITIVE' -d
Int
Variables
#👻
my $a = "foo";
{
my $a = "looks it's me again";
}
#🐚
a=foo
a_1="looks it's me again"
myoption='the defualt value'
say "$option"
file="$(tmp)"
say File
#🐚
say 'a constant value'
#👻
constant $const = "a constant value";
say $const;
Options
#👻
constant $:foo = "bar";
say $:foo;
#🐚
#spit options.sp
MAIN(){ say bar; }
#spit options.sp -o foo=baz
MAIN(){ say baz; }
Namespaced
#👻
class MyClass {
constant $:foo = "bar";
method do-something { say $:foo }
}
say $MyClass:foo;
Required
#👻
class MyClass {
constant $:foo is required;
method do-something { say $:foo }
}
say $MyClass:foo;
Routines
#👻
sub foo($a, $b?, $c = "see", :$d, :$e = "eee") {
say "$a $b $c $d $e";
}
foo("one", "two", :d<three> );
#🐚
MAIN(){
foo(){ say "$1 $2 $3 $d $e"; }
e=eee d=three foo one two see
}
- named parameters
- defaults
- optionals
- Mandatory return type
Slurpies
#👻
sub bar($a, $b, Int *@c) {
say "$a $b @c";
}
bar "one","two", 1..5, 42, 1337;
#🐚
MAIN(){
bar(){
a=$1 b=$2 shift 2
say "$a $b $*"
}
bar one two $(seq 1 5) 42 1337
}
Return types
#👻
sub string~ { "foo" }
sub forty-two+ { 42 }
sub question? { True }
sub dabien -->DockerImg {
Docker.create('debian')
.commit(name => "dabien");
}
eval
- creates an embedded shell script
- can have different options
- can interpolate runtime values into the embedded shell script
- It's pretty spicy
#🐚
MAIN(){
script="$(cat <<-'🐈'
{
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
say 'hello world'
}
🐈
)"
}
#👻
my $script = eval{ say "hello world" };
#👻
say Docker.create('debian')
.cleanup
.exec( eval(os => Debian){ Pkg<curl>.installed } ).gist;
say Docker.create('centos')
.cleanup
.exec( eval(os => CentOS){ Pkg<curl>.installed } ).gist;
#🐚
MAIN(){
say "$(gist "$(Dexec "$(cleanup "$(create debian)")" "$(cat <<-'🐈'
{
exec 4>/dev/null
dpkg -s curl >&4 2>&4
}
🐈
)" && e 1)")"
say "$(gist "$(Dexec "$(cleanup "$(create centos)")" "$(cat <<-'😺'
{
exec 4>/dev/null
installed(){ yum list installed "$1" >&4 2>&4; }
installed curl
}
😺
)" && e 1)")"
}
END(){
remove "$(slurp "$docker_cleanup")"
rm -rf "$docker_cleanup"
}
trap END EXIT;trap 'trap : TERM; exit 1' TERM
BEGIN && MAIN
spit eval-docker.sp -h
#👻
my $one = "1";
my $two = "2";
my $three = "3";
eval{ say "$one $two $three" }.${sh};
#🐚
MAIN(){
one=1
two=2
three=3
cat <<-'🐈' |
{
one='💐'
two='🌼'
three='💞'
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
say "$one $two $three"
}
🐈
subst_eval 💐 "$one"|subst_eval 💞 "$three"|subst_eval 🌼 "$two"|\
sh
}
BEGIN && MAIN
Future
- multi-dispatch on signatures
- macros
- better module support
- s/regex/replace/
- Async/Promises
spit.sh
curl https://spit.sh/minecraft?version=1.12.1 | sh
Rough Edges
- Not ready for production
- subshell problem
Demo: DigitalOcean/Twilio
class Twilio {
constant $:auth-token is required;
constant $:account-sid is required;
constant $:api = "api.twilio.com/2010-04-01";
constant $:from is required;
static method send-sms(:$from = $:from, :$to, :$message) {
HTTP("https://{$:account-sid}:{$:auth-token}@" ~
"$:api/Accounts/$:account-sid/Messages")
.request(
'POST',
form => (
From => $from,
To => $to,
Body => $message,
),
).ok;
}
}
constant $:to = '+310655512458';
my $droplet = DO.create-ssh-seeded(
:name<twilio-test>,
:region<ams2>
);
$droplet.ssh-seed: eval(os => Debian){
Twilio.send-sms(
:$:to,
message => File</etc/debian_version>.slurp
);
};
END $droplet.delete;
spit DO-twilio.sp -h -o log
#👻
sub uppercase($str)~ {
say "Uppercasing $str";
$str.uc;
}
my $a = uppercase "foo";
say ">>>$a<<<";
#🐚
BEGIN(){
e(){ printf %s "$1"; }
exec 3>&1
say(){ printf '%s\n' "$1" >&3; }
uc(){ tr [:lower:] [:upper:]; }
}
MAIN(){
uppercase(){
say "Uppercasing $1"
e "$1"|uc
}
a="$(uppercase foo)"
say ">>>$a<<<"
}
BEGIN && MAIN
Spook in the Shell
By Lloyd Fournier
Spook in the Shell
- 2,156