Command-Line Kung Fu

Know just enough, to not be dangers!

Who am I?

Software Developer, Product Designer, Consultant & Entrepreneur

me@usmanbashir.com

@ubax

usmanbashir.com

Why learn all this?

Really? Why not?

Why is the Command-Line useful?

  • Because, if you want to code, you simply have to learn this. For example, a programming language is a really advanced way of controlling your computer. In comparison, the command-line is more like a little brother. But, it's a powerful tool in its own right.
  • At first, the command-line may seem complex and scary. But actually, it's quite simple once you understand what's going on under the hood.
  • Unix gives you a set of building blocks and lets you put them together. It allows you to build things to your exact needs. With some creativity and logical thinking, mixed in with an understanding of the blocks. You can build tools to do virtually anything you want.
  • Though, the CLI is not a replacement for the GUI. Each has different strengths and weakness. In the end, you will be using both.

Let's Bash

The Bourne-Again Shell

What is this Shell you speak off?

  • A shell is just a program, that lets you talk to your OS.
  • It can be a graphical interface, like Finder, Explorer & Unity for Mac, Windows & Ubuntu.
  • Or, it can just be a command-line interface, like Bash or ZSH.

How does Bash work?

  • You type commands into a Terminal.
  • Bash figures out those commands and makes things happen.
  • Easy Peasy! Right?

What can Bash do?

  • You should really be asking what it can't do!
  • You can use it to copy files, run programs, search and filter data streams and just a whole a lot more.
  • It even keeps a history of all the commands you've used. In case, you ever want to go back and check what you've been up to.

How do I use Bash?

  • It's pretty simple really. Just type your commands.
  • A shell command has two parts.
  • The first being the program that you're running.
  • And, the second part is all the arguments you're passing to the program.

That's all there is to Bash!

  • Most commands are pretty simple and easy.
  • But, what really makes Bash a badass. Is, it can be as complicated or simple as you need it to be.

The File Whisperer

Would you rather have me call it, file management? How boring!

Working with Directories

       ls   - Lists the contents of a directory.
  tree   - List the contents of directories in a tree-like format.
  pwd   - Feel lost and want to get your bearings?
Prints the name and path of the current directory.
     cd   - Move around and go places by changing directories. 
cd ./path   # Go to a sub directory from the current parent directory.
cd ~        # Go to your home directory.
cd /        # Go to the root/top of your file system.
cd ..       # Go one level above the current directory.
cd ../..    # Go two levels above the current directory
mkdir  - Create a directory if it doesn't already exists.

Working with Files

     cp   - Copy source files and directories to a provided destination.
    mv   - Move or rename files and directories.
    rm   - Delete files and directories.



What ever you do! Never, ever run "rm -rf /" as root.
touch  -  Need a new empty file? Just use touch! 
rm ./file.ext            # Delete the given file.

rm -rf ./directoryname   # Delete the given directory along with all of it's 
                         # files and sub-directories. Use, with caution.

Streams & Pipes
The Hard Way!

No easy way in, let's just push through!

What is this stream, you speak of precious?

  • Think of a stream as a river. And, like a river, a stream also flows in a direction.
  • When you type commands and data, that's a stream that goes in and it's called STDIN.
  • When a program is done and wants to share the results. It sends them out to you on the outgoing stream called STDOUT.
  • Though, just like life. Sometimes things don't go well for a program. And, it wants to tell you all about it. But, being considerate, it'll do it in a separate stream called STDERR.

Plumbing 101: Pipelines

  • A pipeline is something that helps you chain together the standard streams.
  • You can take the output stream (stdout) of a program and feed it directly as the input stream (stdin) to another program.
  • This way, you can keep piping data from one program to another, again and again.
  • Filter programs are often used along with pipes to change data in some way.

Using Pipes & Redirection

  • The pipe symbol, or simply called the pipe as in "|", is perhaps one of the most commonly used characters.
  • Let's say you've a long directory listing of the /bin directory. When you type ls -l and press Enter, the names flash by too fast to read for anyone. And, when the display stops, all you see are the last few entries.
  • Now, if you pipe the output of ls -l /bin to the program more. It would let you scan through the list a screenful at a time.
ls -l /dev | more

Redirecting Output

  • Normally, when a program is run, we get its output on the screen, which is what we want most of the time.
  • Though, sometimes we may need to save a result into a file to keep as a record, feed it into another system, or just to send it to someone else.
  • You can do this, by using the ">" operator. It tells the command-line that you want to save the output of the program or whatever that is being sent to the (stdout) to a file instead of printing it to the screen.

DEMO

# Insert the given content into a file by overwriting anything in the file, if it already exists.

echo "The quick brown fox jumps over the lazy dog" > readme.txt


# Append and not overwrite the given content to the end of a file, if it already exists.

echo "A phrase that has every letter in the alphabet in it." >> readme.txt

Analyzing
Text Files & Streams

This sounds boring! But wait, it is!

Using Cat & Tac

  • cat and tac, read data from both files and (stdout). And, they can concatenate whatever you send to them via (stdin).
  • cat is also used to display text files, copy text files into a new file, append the contents of one text file at the end of another, combining them.
  • cat is a simple but useful tool for working with data in your text files or from the streams.
  • tac is basically cat but it reads data from bottom to top instead of top to bottom.

DEMO

# Display the contents of a file(s).

cat filename.txt

cat filename.txt filename2.txt


# Copy a files content to another file.

cat oldtextfile.txt > newfile.txt


# Similarly, you can concatenate several files into a destination file.

cat filename.txt filename2.txt > newfilename.txt


# Append a files contents to another file.

cat filename.txt >> another-text-filename.txt

cat filename.txt filename2.txt >> another-text-filename.txt



# Add numbers to the output of a file or stream.

ls -la /bin | cat -n

Reading: Head & Tail

  • head - Takes a files or (stdout) as input and outputs the first 10 lines by default.
  • tail - Also takes files or (stdout) as input and outputs the last 10 lines of it by default.
  • Though, where tail really shines is in keeping a watch over an updating log file or stream.
  • You can use it to monitor log files and it will pipe the last 10 lines and any new lines added to the file or stream, to the screen.

tail -f access.log

Paging Data: More or Less?

  • Have a lot of data to browse through? Then use more for paging through the text one screenful at a time.
  • Though, on the other hand, less is similar to more. But, it allows backward and forward movement, instead of just showing you a screenful.
  • And, less is faster to start up especially with large files. Because, it does not reads the entire input file before starting.

The Data Fruit Ninja a.k.a. Cut

  • What does it do? It simply cuts out sections of each line from a file or stream. And, hands them over to you. It's a great tool for filtering data vertically.
  • To use it, all you need to tell it is the delimiter and which fields you want cut.

DEMO

# Extract the second IP address from the list.

echo "192.168.1.123 54.192.77.176 192.168.1.111 216.58.212.206" | cut -d' ' -f2

# This will return:
# 54.192.77.176


# Only cut, the first-through-second and fifth-through-sixth field of each line.

# filename.txt
# one two three four five six
# alpha beta gamma delta epsilon zeta

cat filename.txt | cut -d' ' -f 1-2,5-6


# Just give me the fourth field and every filed after that.

cat filename.txt | cut -d' ' -f 4-


# List all usernames on the system.

cut -d: -f1 /etc/passwd


# List all usernames and their home directories with a space in-between them.

cut -d: --output-delimiter=' ' -f1,6 /etc/passwd

Get your output organized!

  sort   - A simple but useful command to rearrange the contents of a text file or stream, line by line. So that, they are sorted, numerically and alphabetically.
  uniq   - It reports or filters out repeated lines in a file or stream. Though, it doesn't detect repeated lines unless they are adjacent. For that, you should first use sort.

DEMO

Hunting through Streams

In the end, everything is a stream that flows through the universe.

Grep This!

  • Have a huge Markdown file with a lot of links all over the place? But, you only need links for certain domains or topics?
  • Or maybe, every time you find a bug or need to do something in your code. You leave behind a comment with a prefix like TODO:, FIX: or BUG: to remind yourself to take care of it later?
  • Then again, maybe your app crashed dumping hundreds or thousands of lines of log data in your lap. How do you find the relevant information out of all that mess?
  • That's what grep is for, and a lot more. It takes a regular expression and processes text, line by line and prints any lines which match the specified pattern.
  • It is indeed a very powerful tool, in your toolbox.

DEMO

# Find all StackOverflow links from a Markdown file.

grep stackoverflow Reading-List.md


# Find all To Do items in all files under each directory, recursively.
# Along with a line number for each To Do task.

# code.rb
# ---------------------------
# def foo
#   # TODO: Test this code.
#   1 + 2
# end
# 
# def bar
#   # FIX: Divided By 0 Error
#   1 / 0
# end
# 
# def baz
#   # BUG: Need to fix this.
#   raise "SomeBugName"
# end

grep -rni --color 'TODO:' ./

# This performs a case-insensitive grep search and returns:
# ./code.rb:2:  # TODO: Test this code.

Grep: Leveling Up

  • What really makes grep so powerful is its ability to process data based on regular expressions.
  • Regular expressions is a huge topic on its own. And, they can be used across many different places. Both, in the command-line and in your everyday programming code.
  • But, regular expressions are beyond the scope of this talk.
  • Though, for those interested. Checkout, the last slide.

Hello Internet!

Long time, no see!

The Curly Curl!

  • cURL is a program for performing requests using a variety of protocols, but mainly HTTP. It's often used by developers to test 3rd-party APIs or their own API. As it supports low level HTTP functionality required to interact with the APIs.
  • You can use cURL to make all sorts of HTTP requests, like GET, POST, PUT, PATCH, HEAD and DELETE against a RESTful API, just from your command-line.
  • It also has other user cases. Though, they are far too numerous to mention in this talk.

DEMO

# Get user data from the JSON, User API endpoint of GitHub.

curl https://api.github.com/users/usmanbashir


# Get a list of all followers for the user.

curl https://api.github.com/users/usmanbashir/followers


# Authenticate with the Twilio API using GET HTTP requests.
# Their REST API is protected with HTTP Basic authentication.

curl -G https://api.twilio.com/2010-04-01/Accounts -u '[YOUR ACCOUNT SID]:[YOUR AUTH TOKEN]'


# Send an SMS to a friend.

curl -X POST \
'https://api.twilio.com/2010-04-01/Accounts/AC5ef872f6da5a21de157d80997a64bd33/Messages.json' \
--data-urlencode 'To=+16518675309'  \
--data-urlencode 'From=+14158141829'  \
--data-urlencode 'Body=Hey dude! Good luck with the exam!'  \
-u AC5ef872f6da5a21de157d80997a64bd33:[AuthToken]

Downloading the Interwebs!

  • You're in the middle of some work, but then you immediately need a file. It could be a Text file, Image, Mp3, Zip or even an ISO, it doesn't matter. Who are you gonna call?
  • Definitely, not the Ghostbusters!
  • What you need is Wget. It's a downloader alright, that can download anything from simple files to entire websites for archiving.
# Download a single file from the net.

wget http://example.com/file.zip

# Download a file but save it locally under a different name.

wget ‐‐output-document=filename.zip http://example.com/file.zip

# Resume an interrupted download previously started by Wget itself.

wget ‐‐continue example.com/some_big_file.zip

Behold the dark power you possess now!

Though, don't let it, get to your head.

# 1. Retrieve information about the eth1 network interface.

# 2. Filter it down only to the IPv4 line.

# 3. Cut out almost everything else and only leave the bits about the IP address.

# 4. Get the first field, which contains the IP address.

# 5. Get the IP bit for the computer and not the gateway part of it.


ifconfig eth1 | grep "inet addr" | cut -d: -f2 | awk "{print \$1}" | cut -d. -f4

Need Help?

For the love of God, yes please!

Let Me Help You

  • No body, is expected to memorize all this stuff. But, you're expected to know what you need and where to find it.
  • The first place you should look for any help would be the man pages. And, if you need help with man pages, then just run man man. It will answer any questions you might have about itself.
  • By the way, if you ever become stranded alone on Mars like Mark Watney from the excellent science fiction novel The Martian by Andy Weir. And, you need a HEX to ASCII table so you can talk with NASA to save your life.


    You are welcome! — Cue applause!!! 
man ascii

Just TL;DR it for me Doc!

  • Fine!
  • If you need something simple and to the point. Then try the npm  tldr package.
  • You need to have both Node.JS and NPM installed to use this.

I didn't think it would end this way.

End? No, the journey doesn't end here.

For the journey ahead of you!

Made with Slides.com