#UnixTricks

Commands You Can't Live Without

​March 30th 2016 – viscaweb.com/meetings

#UnixTricks

First part:

Why is it so powerful?

#UnixTricks

Statistics...

#UnixTricks

90% of the world’s most powerful supercomputers are using GNU/Linux.

Top ten of supercomputers use Linux.

Last year, 75% of Linux code was developed by programmers working for corporations. GOOGLE has contributed about 1.1% of the code in the current Linux kernel.

33.8% of the world runs on Linux servers compared to 7.3% running Microsoft Windows operating system. 

Who's relying on Unix?

#UnixTricks

#UnixTricks

Second part:

The shell...

#UnixTricks

Terminal vs. Shell vs. Console

#UnixTricks

The shell processes commands and returns output
(bash, ksh, tcsh, zsh)

The terminal (aka console) is working on top of it
(Terminal, iTerm, GNOME Terminal , etc..)

Basic commands

#UnixTricks

Listing
ls

Change dir.
cd

Copying
cp

Displaying
cat

Moving
mv

Processes
top / htop

Searching
grep

Copying
cp

Advanced commands

#UnixTricks

Agreeing
yes / no

Drive status
df

Calculating space
du

Mesuring time
time

Last lines
tail

First lines
head

Paginate output
more

Extract parts
awk

Basic commands

#UnixTricks 

grep

# Insensitive case, recursively, and display line numbers
grep -nir .

# Print only what's matching
grep -o 'POST .........'

# Looking by regex
grep -E 'POST|GET'

# Looking only for PHP files
grep --include='*.php'

Aliases

#UnixTricks 

# Project alises
alias my_project1='cd /Volumes/HDD/Git/Project1'
alias my_project2='cd /Volumes/HDD/Git/Project2'
alias my_project1='cd /Volumes/HDD/Git/Project1'

# Moving around the drive
alias ..='cd ..'
alias ...='cd ../..';

# Symfony
alias c='app/console'
alias cc='app/console cache:clear'

# Others
alias alpha='pbpaste | sort | pbcopy'
alias composer7='php7 /usr/local/bin/composer'
alias grep='grep --color=auto'
alias grepphp='grep -nir --include="*.php"'

alias grpe='grep'
alias gerp='grep'
alias gepr='grep'
# Using functions
launchCommandVerbose(){
    php app/console $1 -vvv
}

alias commandVerbose=launchCommandVerbose

---

raul@viscaweb $ commandVerbose debug:container

Redirections

#UnixTricks

There are 3 standard ways for a program to communicate:

stdin

stdout (1)

stderr (2)

raul@viscaweb $ script.sh < input.txt
raul@viscaweb $ script.sh > output.txt
raul@viscaweb $ script.sh 1> output.txt
raul@viscaweb $ script.sh 2> errors.txt


Find out more examples on GitHub:
https://github.com/Viscaweb/ViscaMeeting-Examples
 

<?php
// Standard output
file_put_contents('php://stdout', "Standard output line (1) \n");
print "Standard output line (2) \n";

// Error output
file_put_contents('php://stderr', "Error output line\n");

Combinations

#UnixTricks 

# Count the numbers of files and directories
find . | wc -l

# List all files and their MD5 values
find . -type f | xargs md5

# Copying to the clip board
echo 'copy this' | pbcopy

A pipeline is a set of processes chained by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) to the next one

Operators

#UnixTricks 

# Say something if the command succeed
true && say 'done';
bash my-script.sh && say 'done';

# Say something if any cases
false; say 'done';
bash my-script.sh; say 'done';

# Run a script, or another if it fails
bash my-script.sh || bash script-alternative.sh;

A logic to your commands...

Long processes

#UnixTricks

Some processes can take hours (if not, days, or weeks) to run...

How to ensure your script will end-up properly even if you close your computers?

Long processes

#UnixTricks

Screen

A "screen" allow you to create independents and not-attached sessions.

raul@viscaweb $ bash my-script-taking-ages.sh

Running this script on your server will break...

when closing the terminal by mistake

loosing
Internet

closing your computer to take a beer

Long processes

#UnixTricks

Screen

To avoid this...

raul@viscaweb $ ssh ...
> connected to SSH

raul@viscaweb $ screen
> running inside the screen

raul@viscaweb $ bash archiving-files.sh
> script running...
> archiving file 1 on 100 (0%)

Then, detach the screen:

+

Leave the computer...

Later, just "reattach" the screen

raul@viscaweb $ ssh ...
> connected to SSH

raul@viscaweb $ screen -r
> archiving file 45 on 100 (45%)
> archiving file 46 on 100 (46%)
> archiving file 47 on 100 (47%)
> running...

Long processes

#UnixTricks

Nohup and &

Using nohup

raul@visacweb $ ssh ...
> connected to SSH

raul@viscaweb $ nohup archiving-files.sh &
> appending output to nohup.out

raul@viscaweb $ exit

Long processes

#UnixTricks

fg, bg

Run your script...

raul@viscaweb $ ssh ...
> connected to SSH
raul@viscaweb $ bash archiving-files.sh
> script running...
> archiving file 1 on 100 (0%)

Then, suspend the process:

+

Move this jobs to foreground

raul@visacweb $ fg
> reattached
# Relaunch this process in background
raul@viscaweb $ bg

# List active jobs
raul@viscaweb $ jobs
[1]+  Running  bash archiving-files.sh &

Long processes

#UnixTricks

Which one should I use?

nohup:

✓ Installed by default
✓ Can't be killed using Ctrl+C or Ctrl+D
✖ No more user interaction
✖ Not possible to get it back

screen:

✓ Ability to name each processes
✓ Possible to get it back
✓ Ability to follow a process easily during weeks
✖ Not installed by default

fg / bg:

✓ Installed by default
✓/✖ Possible to get it back through the current session only

#UnixTricks

Third part:

And even more...

#UnixTricks

tmate

#UnixTricks

TeamViewer for the Terminal - tmate.io

raul@viscaweb $ brew install tmate

raul@viscaweb $ tmate
> [tmate] ssh session : ssh xxx@tmate.io
felix@viscaweb $ ssh xxx@tmate.io
> Connecting...
raul@viscaweb $ ls
raul@viscaweb $ ls

1

2

ngrok

#UnixTricks

Be connected

raul@viscaweb $ brew install ngrok
raul@viscaweb $ ngrok 80

Tunnel Status                 online
Forwarding                    http://7a022fb2.ngrok.com -> 127.0.0.1:80
Forwarding                    https://7a022fb2.ngrok.com -> 127.0.0.1:80
Web Interface                 127.0.0.1:4040

watch

#UnixTricks

Execute a program periodically

raul@viscaweb $ watch -n 2 'ls -l'

Every 2.0s: ls -l                    Mon Mar 07 17:00:00 2016
-rw-r--r--   1 hj    wheel     0 Mar 28 17:00 downloading.file

Every 2.0s: ls -l                    Mon Mar 07 17:00:02 2016
-rw-r--r--   1 hj    wheel   450 Mar 28 17:00 downloading.file

Every 2.0s: ls -l                    Mon Mar 07 17:00:04 2016
-rw-r--r--   1 hj    wheel   910 Mar 28 17:00 downloading.file

#UnixTricks 

# Read headers only
http --headers example.org

# Include all responses that lead to the final one:
http --follow --all --max-redirects=5 httpbin.org/redirect/3

# View all intermediary requests
http --all --follow httpbin.org/redirect/3

# Force a specific request type
http PUT example.org name=John email=john@example.org

tree

#UnixTricks

List contents of directories in a tree-like format 

raul@viscaweb $ tree
.
├── CleanCode
│   └── Interfaces
│       ├── example_in_java.java
│       ├── example_in_php5.php
│       ├── example_in_php6.php
│       └── example_in_php7.php
├── README.md
├── UnixTricks
│   ├── ngrok
│   │   ├── run-server.sh
│   │   └── welcome.php
│   └── redirections
│       ├── catch_output.sh
│       └── output.php
└── logo.png

ncdu

#UnixTricks

Recursive `du`

raul@viscaweb $ brew install ncdu

raul@viscaweb $ ncdu /

.  15.4 GiB [##########] /var
.   4.0 GiB [##        ] /usr
.   1.6 GiB [#         ] /tmp
.  26.5 MiB [          ] /boot
   18.0 MiB [          ] /sbin
.  13.4 MiB [          ] /etc
   13.2 MiB [          ] /lib64

SCP + iTerm

#UnixTricks

SCP (Secure copy) is a means of securely transferring computer files between a local host and a remote host or between two remote hosts.

raul@viscaweb $ scp user@myhost.com:/home/dir1/file.txt user@myhost.com:/home/dir2

Links (1/4)

Interesting facts about Unix
http://www.tecmint.com/lesser-known-facts-about-gnu-linux/
http://www.techpraveen.com/2012/04/15-most-interesting-linux-facts.html
 

Mac OS based on Unix
http://askubuntu.com/questions/11392/what-are-the-differences-between-mac-os-and-linux


Differences between Terminal, Console, and Shell
http://superuser.com/questions/144666/what-is-the-difference-between-shell-console-and-terminal
 

Basic Unix commands (PDF)
https://ubuntudanmark.dk/filer/fwunixref.pdf

#UnixTricks

Links (2/4)

Create an alias

http://www.hostingadvice.com/how-to/set-command-aliases-linuxubuntudebian/
 

Examples used in these slides
https://github.com/Viscaweb/ViscaMeeting-Examples


Play with redirections
http://www.tutorialspoint.com/unix/unix-io-redirections.htm


Combine commands using pipeline
http://www.tutorialspoint.com/unix/unix-pipes-filters.htm
 

All operators
http://www.tutorialspoint.com/unix/unix-c-shell-operators.htm

#UnixTricks

Links (3/4)

Screen and nohup comparative
http://stackoverflow.com/questions/20766300/nohup-vs-screen-which-is-better-for-long-running-process
 

Play with screens
https://www.mattcutts.com/blog/a-quick-tutorial-on-screen/

Move running process to a screen
http://serverfault.com/questions/55880/moving-an-already-running-process-to-screen

Tmate (share your screen)
https://tmate.io/


Ngrok (create a tunnel)
https://ngrok.com/

#UnixTricks

Links (4/4)

Watch (periodical command)
http://www.linfo.org/watch.html
 

Http (alternative to cURL)
http://httpie.org


Tree
http://www.cyberciti.biz/faq/linux-show-directory-structure-command-line


Ncdu (recursive du)
https://dev.yorhel.nl/ncdu/scr


Scp + iTerm
https://www.iterm2.com/shell_integration.html

#UnixTricks

#UnixTricks

Discovering command and tools helping day to day

​March 30th 2016 – viscaweb.com/meetings

by

Made with Slides.com