Module 5-1

Technical Environment & Scripting

ATTENTION PLEASE

When you see slides with a white background:

  • listen to your instructor
  • takes note if you want
  • ask questions ;)

🚀 HANDS-ON

When you see slides with a colored background:

  • do the exercices
  • talk to your instructor
  • ask questions ;)

AGENDA

FEW DAYS WITH 

  • Command line
  • Shell expansion
  • Pipes and commands
  • vi
  • Scripting
  • Local user management
  • File security

Back to basics

Computer ?

What's inside ?

Processor ?

Computer ?

  • Personnal Computer
    • Workstation, Laptop
    • Mac
    • ...
  • Terminal
  • Service

Server ?

Client-Server ?

Operating System

  • Human / Machine interface:
    • Textual
    • Graphical
  • Management role:
    • Resources (files, memory, devices)
    • Users data & applications
    • Conflicts btw users (multi-user architecture)
  • Ex: Windows, Linux, UNIX, MacOS, AS400
  • Commande language: shell/DOS, powershell ...

Back to basics #2
Computer Networks

Networking

Computer Network
 

 A collection of computing devices that are connected in various ways in order to communicate and share resources.

  Usually, the connection between computer in a network are made using physical wires or cables.

  However, some connections are wireless, using radio waves or infrared signals.

Networking

  • The generic term node or host refers to any device on a network.
  • Data transfer rate - The speed with which data is moved from one place on a network to another.
  • Data transfer rate is a key issue in computer networks.

Networking

  • Computer networks have opened up an entire frontier in the world of computing called the client/server model

Networking

  • File server - A computer that stores and manages files for multiple users on a network.
  • Web server - A computer dedicated to responding to requests (from the browser client) for web pages.

Types of Networks

  • Local-area network (LAN) - A network that connects a relatively small number of machines in a relatively close geographical area.

Types of Networks

  • Various configurations, called topologies, have been used to administer LANs
    • Ring topology - A configuration that connects all nodes in a closed loop on which messages travel in one direction.
    • Star topology - A configuration that centers around one node to which all others are connected and through which all messages are sent
    • Bus topology - All node are connected to a single communication line that carries messages in both directions

Types of Networks

Types of Networks

  • Wide-area network (WAN) - A network that connects two or more local-area networks over a potentially large geographic distance.

Often one particular node on a LAN is set up to server as a gateway to handle all communication going between that LAN and other networks.

Communication between networks is called internetworking. The Internet, as we know it today, is essentially the ultimate wide-are network, spanning the entire globe.

Types of Networks

  • Metropolitan-area network (MAN) - The communication infrastructures that have been developed in and around large cities.

So, who owns the Internet?

Well, nobody does. No single person or company owns the Internet or even controls it entirely. As a wide-are network, it is made up of many smaller networks. These smaller networks are often owned and managed by a person or organization.

The Internet, then, is really defined by how connections can be made between these networks.

Internet Connections

  • Internet backbone - A set of high-speed networks that carry Internet traffic. These network are provided by companies such as AT&T, GTE, IBM, ...
  • Internet service provider (ISP) - A company that provides other companies or individuals with access to the Internet.

Internet Connections

  • There are various technologies available that you can use to connect a home computer to the Internet :
    • A phone modem - converts computer data into an analog audio signal for transfer over a telephone line, and then a model at the destination converts it back again into data.
    • A digital subscriber line (DSL) uses regular copper phone lines to transfer digital data to and from the phone company's central office
    • A cable model uses the same line that you cable TV signals come in on to transfer the data back and forth.

Internet Connections

  • Broadband - A connection in which transfer speeds are faster than 128 bits per second
    • DSL connections and cable modems are broadband connections
    • The speed for downloads (getting data from the Internet to your home computer) may not be the same as uploads (sending data from your home computer to the Internet).

Packet Switching

  • To improve the efficiency of transferring information over a shared communication line, messages are divided into fixed-sized, numbered packets.
  • Network devices called routers are used to direct packets between networks.

Open Systems

  • Proprietary system - A system that uses technologies kept private by a particular commercial vendor.
  • Interoperability - The ability of software and hardware on multiple machines and from multiple commercial vendors to communicate.
  • Open systems - Systems based on a common model of network architecture and a suite of protocols used in its implementation.

Network Protocols

  • Network protocols are layered such that each one relies on the protocols that underlie it
  • Sometimes referred to as a protocol stack

TCP/IP

  • TCP stands for Transmission Control Protocol
    • TCP software breaks messages into packets, hand them off to the IP software for delivery, and the orders and reassembles the packets at their destination.
  • IP stands for Internet Protocol
    • IP software deals with the routing of packets through the maze of interconnected networks to their final destination.

TCP/IP (cont.)

  • UDP stands for User Datagram Protocol
    • It is an alternative to TCP
    • The main difference is that TCP is highly reliable, at the cost of decreased performance, while UDP is less reliable, but generally faster.

High-Level Protocols

  • Other protocols build on the foundation established by the TCP/IP protocol suite
    • Simple Mail Transfer Protocol (SMTP)
    • File Transfer Protocol (FTP)
    • Telnet
    • Hyper Text Transfer Protocol (HTTP)

MIME Types

Firewalls

  • Firewall - A machine and its software that serve as a special gateway to a network, protecting it from inappropriate access
    • Filters the network traffic that comes in, checking the validity of the messages as much as possible and perhaps denying some messages altogether
    • Enforces an organization's access control policy

Network Addresses

  • Hostname - A unique identification that specifies a particular computer on the Internet
    • For example :
      • ipme07.northeurope.cloudapp.azure.com

Network Addresses

  • Network software translates a hostname into its corresponding IP address
    • For example
      • ipme07.northeurope.cloudapp.azure.com
                  ==

        40.112.67.50

Network Addresses

  • An IP address can be split into :
    • Network address, which specifies a specific network
    • Host number, which specifies a particular machine in that network

Domain Name System

  • A hostname consists of the computer name followed by the domain name
  • A domain name is separated into two or more sections that specify the organization, and possibly a subset of an organization, of which the computer is a part
  • Two organizations can have a computer named the same thing because the domain name makes it clear which one is being referred to.

Domain Name System

  • The very last section of the domain is called its top-level domain (TLD) name

Domain Name System

  • Organizations based in countries other than the United States use a top-level domain that corresponds to their two-letter country codes

Domain Name System

  • The domain name system (DNS) is chiefly used to translate hostnames into numeric IP addresses
    • DNS is an example of distributed database
    • If that server can resolve the hostname, it does so
    • If not, that server asks another domain name server

History

Birth of free software

1983 - Richard Stallman, GNU project and free software concept

==> Beginning of the development off gcc, gdb, glibc and important other tools

1991 - Linux Torvalds, Linux kernel project, an Unix-like operating system kernel

==> Together with GNU software and many other open-source components: a completely free operating system : GNU/Linux

1995 - Linux on server systems

2000 - Linux on embedded systems

2008 - Linux on mobile devices

2010 - Linux on phones

Free software ?

A program is considered free when its license offers to all its users the following 4 freedoms :

  1. ​Freedom to run the software for any purpose
  2. Freedom to study the software and to change it
  3. Freedom to redistribute copies
  4. Freedom to distribute copies of modifies versions

 

These freedoms are granted for both commercial and non-commercial use.

They imply the availability of source code, software can be modified and distributed to customers.

Linux philosophy

There are nine major tenets to the Linux philosophy:

  1. Small is beautiful
  2. Each program does one thing well
  3. Build a prototype as soon as possible
  4. Choose portability over efficiency
  5. Store data in flat text files
  6. Use software leverage to your advantage
  7. Use shell scripts to increase leverage and portability
  8. Avoid captive user interface
  9. Make every program a filter

Linux distributions

Distribution = Kernel + Application + Branding

Linux

  • Powerful
  • Runs on multiple hardware platform
  • Users like its speed and stability
  • No requirement for the latest hardware
  • It's "free"
  • Licensed under GPL
  • Vendors are distributors who packages Linux

Linux

Multi-user

 A multi-user OS allows for multiple users to use the same computer at the same time and/or different time

Multi-processing

  An operating system capable of supporting and utilizing more than one computer processor

Multi-tasking

  An operating system capable of allowing multiple software processes to run at the same time

Multi-threading

  OS that allows different parts of a software program to run concurrently

Linux

COMMAND LINE

COMMAND LINE

MAN PAGES

man

# man $command
$ man whois
$ man man

# man $configfile
$ man syslog.conf

# man deamon
$ man syslogd

Press q to quit

whatis

To see just the description of a manual page, use whatis followed by a string

$ whatis route
route (8)            - affiche / manipule la table de routage IP

whereis

whereis locates source/binary and manual sections for specified files.

$ whereis httpd
httpd: /usr/sbin/httpd /usr/lib64/httpd /etc/httpd /usr/share/httpd

COMMAND LINE

WORKING WITH DIRECTORIES

pwd

The tool displays your current directory.

$ pwd
/root

cd

Change your current directory.

$ pwd
/root
$ cd ..
$ pwd
/
$ cd /var/log/
$ pwd
/var/log/
$ cd /etc
$ pwd
/etc
$ cd -
$ pwd
/var/log
$ cd ~
$ pwd
/root

ls

Display directory content.

$ ls
hello.txt
$ ls -l
total 0
-rw-r--r-- 1 root root 0  7 janv. 21:41 hello.txt
$ ls -al
total 8
drwxr-xr-x  2 root root 4096  7 janv. 21:42 .
dr-xr-x---. 8 root root 4096  7 janv. 21:42 ..
-rw-r--r--  1 root root    0  7 janv. 21:42 hello.txt
-rw-r--r--  1 root root    0  7 janv. 21:42 .hidded_file

mkdir

Create your own directories with mkdir.

$ mkdir a
$ cd a
$ ls -al
total 8
drwxr-xr-x 2 root root 4096  7 janv. 21:44 .
drwxr-xr-x 4 root root 4096  7 janv. 21:44 ..
$ mkdir b
$ mkdir c
$ ls -l
total 16
drwxr-xr-x 4 root root 4096  7 janv. 21:44 .
drwxr-xr-x 4 root root 4096  7 janv. 21:44 ..
drwxr-xr-x 2 root root 4096  7 janv. 21:44 b
drwxr-xr-x 2 root root 4096  7 janv. 21:44 c
$ mkdir -p d/e/f
$ cd d
$ ls -l
total 4
drwxr-xr-x 3 root root 4096  7 janv. 21:45 e

rmdir

Use rmdir to remove a directory.

$ ls -l
total 12
drwxr-xr-x 2 root root 4096  7 janv. 21:44 b
drwxr-xr-x 2 root root 4096  7 janv. 21:44 c
drwxr-xr-x 3 root root 4096  7 janv. 21:45 d
$ rmdir c/
$ rmdir -p d/e/f

🚀 HANDS-ON

  1. Display your current directory
  2. Change to /tmp
  3. List the content of /tmp directory
  4. Create the useless folder useless/
  5. Remove the directory useless/

COMMAND LINE

WORKING WITH FILES

FACTS

  1. All files are case sensitive
  2. Everything is a file ! 
$ ls
winter.txt Winter.txt

file

file utility determines the file type.

$ file winter.txt
winter.txt: ASCII text
$ file winter.tar.gz
winter.tar.gz: gzip compressed data, from Unix, last modified: Sun Jan  7 21:53:10 2018

touch

One easy way to create an empty file is with touch.

$ ls -l
total 0
$ touch a.txt
$ touch b.txt
$ ls -l
total 0
-rw-r--r-- 1 root root 0  7 janv. 21:57 a.txt
-rw-r--r-- 1 root root 0  7 janv. 21:57 b.txt

rm

Use rm to remove a file.

$ ls
a.txt b.txt c.txt
$ rm a.txt
$ ls
b.txt c.txt
$ rm -i b.txt
rm : supprimer fichier vide « b.txt » ? y
$ ls 
c.txt
$ rm -i c.txt
rm : supprimer fichier vide « b.txt » ? n
$ ls
c.txt
$ mkdir folder1/ folder2/
$ rm -r folder1/
rm : supprimer répertoire « folder1 » ? y
$ rm -rf folder2/

cp

Use cp to copy a file/directory.

$ ls
a.txt b.txt c.txt
$ cp a.txt a_old.txt
$ ls
a_old.txt a.txt b.txt c.txt
$ mkdir folder1/
$ cp -r folder1/ folder1_backup/

mv

Use mv to rename a file/directory.

$ ls
a.txt b.txt c.txt
$ mv a.txt d.txt
$ ls
b.txt c.txt d.txt
$ mkdir folder1
$ mv folder1/ folder2/
$ ls
b.txt c.txt d.txt folder2

🚀 HANDS-ON

  1. List files of /bin directory
  2. Display the file type of /bin/cat, /etc/passwd, /etc/shadow
  3. Create a directory ~/pets and enter it.
  4. Create files wolf.txt and cat.txt
  5. Rename cat.txt to unicorn.txt
  6. Copy unicorn.txt in dragon.txt
  7. Remove all *.txt files in ~/pets folder

COMMAND LINE

WORKING WITH FILES CONTENT

head

Use head to display the first ten lines of a file.

$ head /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
$ head -2 /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin

tail

Use tail to display the last ten lines of a file.

$ tail /etc/services
3gpp-cbsp       48049/tcp               # 3GPP Cell Broadcast Service Protocol
isnetserv       48128/tcp               # Image Systems Network Services
isnetserv       48128/udp               # Image Systems Network Services
blp5            48129/tcp               # Bloomberg locator
blp5            48129/udp               # Bloomberg locator
com-bardac-dw   48556/tcp               # com-bardac-dw
com-bardac-dw   48556/udp               # com-bardac-dw
iqobject        48619/tcp               # iqobject
iqobject        48619/udp               # iqobject
matahari        49000/tcp               # Matahari Broker

cat

Use cat to display a file on the screen.

$ cat /etc/resolv.conf
# Generated by NetworkManager
search openstacklocal ovh.net
nameserver 213.186.33.99

# Use cat to concatenate
$ echo one > part1
$ echo two > part2
$ echo three > part3
$ cat part1
one
$ cat part2
two
$ cat part1 part2 part3 > all
$ cat all
one
two
three

tac

Use tac to display a file (backwards) on the screen.

$ cat count
one
two
three
four
$ tac count
four
three
two
one

🚀 HANDS-ON

  1. Display the first 12 lines of /etc/services.
  2. Display the last line of /etc/passwd.
  3. Use more to display /etc/services.
  4. Use ls to find the biggest file in /etc.
  5. Display /var/log/ files based on last modified time (in reverse order)

COMMAND LINE

THE LINUX FILE TREE

THE ROOT DIRECTORY /

All Linux systems have a directory structure that starts at the root directory. The root directory is represented by a forward slash, like this: /.

$ cd /
$ ls
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  
proc  root  run  sbin  srv  sys  tmp  usr  var

BINARY DIRECTORIES

Binaries are file that contains compiled code (or machine code). Binaries can be executed on the computer. Sometimes binaries are called executables.

# The /bin directory contains binaries for use by all users
$ ls /bin
[                     gpg-error              nl-addr-list              setpriv
a2p                   gpgparsemail           nl-class-add              setsid
ab                    gpgsplit               nl-class-delete           setterm
...
# /sbin contains binaries to configure the operating system
accessdb              e2image                    intel-microcode2ucode  nl-qdisc-list
addgnupghome          e2label                    ip                     nologin
...
# Binaries found in /bin and /sbin ofter use shared libraries located in /lib
audit                    libcrack.so.2                 libnss_files.so.2
binfmt.d                 libcrack.so.2.9.0             libnss_hesiod-2.17.so
debug                    libcrypt-2.17.so              libnss_hesiod.so.2 

/opt

The purpose of /opt is to store optional software. In many cases this is software from outside the distribution repository. You may find an empty /opt directory on many systems.

CONFIGURATION DIRECTORIES

The /boot directory contains all files needed to boot the computer. These files don't change very often.

/etc: contains (almost) all of the machine-specific configuration files.

$ ls /etc/*.conf
/etc/asound.conf   /etc/GeoIP.conf  /etc/libaudit.conf   /etc/mke2fs.conf
/etc/chrony.conf   /etc/host.conf   /etc/libuser.conf    /etc/nsswitch.conf
/etc/dnsmasq.conf  /etc/kdump.conf  /etc/locale.conf     /etc/resolv.conf
/etc/dracut.conf   /etc/krb5.conf   /etc/logrotate.conf  /etc/rsyncd.conf
/etc/e2fsck.conf   /etc/ld.so.conf  /etc/man_db.conf     /etc/rsyslog.conf

DATA DIRECTORIES

# /home : store personal or project data
$ ls /home
pierre paul jacques david alexis

# /root : default location for personal data and profile of the root user

# /srv : contains site-specific data which is served by this system.

# /media : directory serves as mount point for removable media devices
$ ls /media
cdrom cdrom0 usbdisk

# /mnt : used for temporary mount points

# /tmp : Applications and users should use /tmp to store temporary data when needed.

IN-MEMORY DIRECTORIES

# /dev : populated with files as the kernel recognize hardware. 
         Not actually located on hard disk.

# /proc : special directory, not taking up disk space. View of 
          the kernel & what kernel manages.

# /sys : contains kernel information about hardware.

/usr UNIX SYSTEM RESOURCES

The /usr hierarchy should contain shareable, read only data.

# /usr/bin: contains a lot of commands
$ ls /usr/bin | wc -l
1395

# /usr/include: contains general use include files for C.
aalib.h        expat_config.h      features.h
...

# /usr/lib: contains libraries that are not executed directly by users of scripts.

# /usr/local: can be used by administrator to install software locally

# /usr/src: directory recommended location for kernel source file  

/var VARIABLE DATA

Files that are unpredictable in size, such as log, cache shoud be located in /var.

# /var/log: directory serves as a central point to contain all log files
$ ls /var/log
anaconda           btmp                   dmesg               maillog-20171224
audit              btmp-20180101          dmesg.old           maillog-20171231
boot.log           chrony                 firewalld           messages
...

# /var/log/messages: contains information on what just happened to the system

# /var/cache: contains cache data for several application

# /var/spool: contains spool directories for mail, cron ...

# /var/lib: contains application states

# /var: also contains Process ID files in /var/run and temporary files that survive 
# after a reboot

🚀 HANDS-ON

  1. Does the file /bin/cat exist ? What about /bin/dd and /bin/echo ? What is the type of these files ?
  2. Use cat to display /etc/hosts and /etc/resolv.conf. What is your idea about the purpose of these files ?
  3. Are there any files in /etc/skel ? Check also hidden files
  4. Display /proc/cpuinfo. On what architecture is your Linux running ?
  5. Is /var/log a file or a directory ? What about /var/spool ?

SHELL EXPANSION

SHELL EXPANSION

COMMANDS AND ARGUMENTS

Commands and arguments

The command line interface or shell used on most Linux systems is called bash, which stands for Bourne again shell.

 

The bash shell incorporates features from sh (the original Bourne shell), csh (the C shell) and ksh (the Korn shell).

echo

Display each argument it receives from the shell

$ echo Hello World
Hello World
$ echo Hello       World
Hello World
$ echo          Hello           World
Hello World
$     echo   Hello    World
Hello World
$ echo 'A line with    single quotes'
A line with    single quotes
$ echo "A line with     double     quotes"
A line with     double     quotes
$ echo -e "A line with \na newline"
A line with
a newline
$ echo -e 'A line with \ta tab'
A line with    a tab

type

To find out whether a command given to the shell will be executed as an external command or as a builtin command.

$ type cd
cd is a shell builtin
$ type cat
cat is /bin/cat
$ type ls
ls is aliased to `ls --color=auto`
$ type echo
echo is a shell builtin
echo is /bin/echo
$ /bin/echo Running the external echo command...
Running the external echo command...

which

To search for binaries in the $PATH environment variable.

$ which cp ls mkdir pwd
/bin/cp
/bin/ls
/bin/mkdir
/bin/pwd

alias

Alias are often used to create an easier to remember name for an existing command.

$ cat count.txt
one
two
three
$ alias dog=tac
$ dog count.txt
three
two
one
$ alias ll='ls -lh --color=auto'
$ alias c='clear'
$ alias c ll
alias c='clear'
alias ll='ls -lh --color=auto'
$ unalias c

🚀 HANDS-ON

  1. Is tac a shell builtin command ?
  2. Is there an existing alias for rm ?
  3. Execute: alias rm='rm -i'. Test you alias with a test file. Does this work as expected ?
  4. Read the man page of rm, make sure you understand the -i option of rm. Create and remove a file to test the -i option.
  5. List all current aliases.

SHELL EXPANSION

CONTROL OPERATORS

; semicolon

Put two or more commands on the same line separated by ;

$ echo Hello
Hello
$ echo World
World
$ echo Hello ; echo World
Hello
World

& ampersand

When a line ends with an ampersand &, the shell will not wait for the command to finish.

$ sleep 20 &
[1] 4512
$ 
...wait 20 seconds
$ 
[1]+  Done            sleep 20

&& double ampersand

Shell interprets && as a logical AND

$ echo first && echo second
first
second
$ zecho first && echo second
-bash: zecho: command not found
$ cd /tmp && ls
file1  file3  file5
file2  file4  file6

|| double vertical bar

|| represents a logical OR.

$ echo first || echo second ; echo third
first
third
$ zecho first || echo second ; echo third
second
third

combining && and ||

$ rm file1 && echo It worked! || echo It failed!
It worked!
$ rm file1 && echo It worked! || echo It failed!
rm: cannot remove `file1`: No such file or directory
It failed! 
$ 

MISCELLANEOUS

# Pound sign (#) is ignored by the shell
$ mkdir test # we create a directory

# \ escaping special characters
$ echo hello \; world
hello ; world
$ echo hello \ \ \ world
hello    world
$ echo escaping \\\ \#\ \&\ \"\ \'
escaping \ # & " '
$ echo escaping \\\?\*\"\'
escaping \?*"'

# End of line backslash
$ echo This command line \
> is split in three \
> parts
This command line is split in three parts
$ 

🚀 HANDS-ON

  1. When you type passwd, which file is executed ? What kind of file is that ?
  2. Execute ls after cd /etc, but only if cd /etc did not error.
  3. Execute cd /etc after cd etc, but only if cd etc fails.
  4. Echo it worked when touch test42 works, and echo it failed when the touch failed. All on one command line. Test it in your home directory and in /bin/.
  5. Execute sleep 6, what is this command doing ?

SHELL EXPANSION

SHELL VARIABLES

$ dollar sign

The shell will look for an environment variable named like the string following the dollar sign and replace it with the value.

$ echo This is the $SHELL shell
This is the /bin/bash shell
$ echo This is $SHELL on computer $HOSTNAME
This is /bin/bash on computer RHELv4u3.localdomain
$ echo The userid of $USER is $UID
The userid of paul is 500
$ echo My homedir is $HOME
My homedir is /home/paul

creating variables

$ MyVar=555
$ echo $MyVAR
555

# Case sensitive !
$ echo Hello $USER
paul
$ echo Hello $user
Hello 

# quotes
$ echo "$MyVar"
555
$ echo '$MyVar'
$MyVar

set / unset

$ MyVar=8472
$ echo $MyVar
8472
$ unset $MyVar
$ echo $MyVar

$ 

Use the set command to display a list of environment variables.

Use the unset command to remove a variable from your shell environment.

$PATH

$ echo $PATH
/usr/kerberos/bin:/usr/local/bin:/bin/:/user/bin:

# Difference btw su and su -
[paul@RHEL3 ~]$ su
Password:
[root@RHEL3 ~]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin
[root@RHEL3 ~]$ exit
[paul@RHEL3 ~]$ su -
Password:
[root@RHEL3 ~]$ echo $PATH
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:

The $PATH variable is determines where the shell is looking for commands to execute (unless the command is builtin or aliased). This variable contains a list of directories, separated by colons.

env

$ bash -c 'echo $SHELL $HOME $USER'
/bin/bash /home/paul paul
$ env -i bash -c 'echo $SHELL $HOME $USER'
/bin/bash

env commands without options will display a list of exported variables.

set lists all variables, including those not exported to child shells.

env can also be used to start a clean shell. 

export

$ var3=three
$ var4=four
$ export var4
$ echo $var3 $var4
three four
$ bash
$ echo $var3 $var4
four
# But will not export to the parent shell
$ export var5=five
$ echo $var3 $var4 $var5
four five
$ exit
exit
$ echo $var3 $var4 $var5
three four

You can export shell variables to other shells with the export command.

🚀 HANDS-ON

  1. Use echo to display Hello followed by your username.
  2. Create a variable answer with a value of 42.
  3. Copy the value of $LANG to $MyLANG.
  4. List all current shell variables.
  5. List all exported shell variables.
  6. Do the env and set commands displays your variable ?
  7. Destroy your answer variable.
  8. Create two variables, and export one of them.
  9. Display the exported variable in an interactive child shell.

SHELL EXPANSION

SHELL EMBEDDING AND OPTIONS

shell embedding

$ echo $var1

$ echo $(var1=5; echo $var1)
5
$ echo $var1

$

backticks

$ echo `cd /etc; ls -d * | grep pass`
passwd passwd- passwd.OLD
$ 

backticks or single quotes

$ echo `var1=5;echo $var1`
5
$ echo 'var1=5;echo $var1'
var1=5;echo $var1
$ 

shell options

$ echo $var123

$ set -u
$ echo $var123
bash-: var123: unbound variable
$ set +u
$ echo $var123

$ 

🚀 HANDS-ON

  1. Find the list of shell options in the man page of bash. What is the difference btw set -u and set -o nounset ?
  2. Activate nounset in your shell. Test that it shows an error message when using non-existing variables.
  3. Desactive nounset.
  4. Execute cd /var and ls in an embedded shell
  5. Create the variable embvar in an embedded shell and echo it. Does the variable exists in your current shell now ?
  6. Explain what "set -x" does. Can this be useful ?

SHELL EXPANSION

SHELL HISTORY

repeating the XX command(s)

$ echo this will be repeated > file42.txt
$ !!
this will be repeated > file42.txt
$ touch file42
$ cat file42
$ !to
touch file42
$ 

To repeat the last command in bash, type !!.

history

$ history 10
38 mkdir test
39 cd test
40 touch file1
41 echo hello > file2
42 echo It is very cold day > winter.txt
43 ls
44 ls -l
45 cp winter.txt summer.txt
46 ls -l
47 history 10

To see older commands, use history.

$HIST...

$ echo $HISTSIZE
500
# Can be changed :
$ HISTSIZE=15000
$ echo $HISTSIZE
15000
$ echo $HISTFILE
/home/paul/.bash_history
$ echo $HISTFILESIZE
15000

$HISTSIZE: number of commands that will be remembered

$HISTFILE: file that contains your history.

$HISTFILESIZE: number of commands kept in your history file

🚀 HANDS-ON

  1. Issue the command echo The answer to the meaning of life, the universe and everything is 42.
  2. Repeat the previous command using two characters (there are two solutions!).
  3. Display at least 5 commands you typed.
  4. How many commands can be kept in memory for your current shell session ?

SHELL EXPANSION

FILE GLOBBING

* asterisk

$ ls
file1  file2  file3  File4  File55  FileA  fileab  Fileab  FileAB  fileabc
$ ls File*
File4  File55  FileA  Fileab  FileAB
$ ls file*
file1  file2  file3  fileab  fileabc
$ ls *ile55
File55
$ ls F*ile55
File55
$ ls F*55
File55
$

? question mark

$ ls
file1  file2  file3  File4  File55  FileA  fileab  Fileab  FileAB  fileabc
$ ls File?
File4  FileA
$ ls Fil?4
File4
$ ls Fil??
File4  FileA
$ ls File??
File55  Fileab  FileAB
$

[] square brackets

$ ls
file1  file2  file3  File4  File55  FileA  fileab  Fileab  FileAB  fileabc
$ ls File[5A]
FileA
$ ls File[A5]
FileA
$ ls File[A5][5b]
File55
$ ls File[a5][5b]
File55  Fileab
$ ls File[a5][5b][abcdefghijklm]
ls: File[a5][5b][abcdefghijklm]: No such file or directory
$ ls file[a5][5b][abcdefghijklm]
fileabc
$

[ is interpreted by the shell as a sign to generate filenames, matching any of the characters between [ and the first subsequent ].

a-z and 0-9 ranges

$ ls
file1  file3  File55  fileab  FileAB   fileabc
file2  File4  FileA   Fileab  fileab2
$ ls file[a-z]*
fileab  fileab2  fileabc
$ ls file[0-9]
file1  file2  file3
$ ls file[a-z][a-z][0-9]*
fileab2
$

The bash shell will also understand ranges of characters between brackets.

preventing file globbing

$ mkdir test42
$ cd test42
$ echo *
*
$ touch file42 file33
$ echo *
file33 file42

🚀 HANDS-ON

  1. Create a test directory and enter in it.
  2. Create the following files : file1, file10, file11, file2, File2, File3, file33, fileAB, filea, fileA, fileAAA, file(, file 2.
  3. List (with ls) all files starting with :
    • file
    • File
    • file and ending in a number
    • File and having a digit as fifth character
  4. List all files that have exactly five characters.
  5. List all files that do not start with the letter F.

PIPES AND COMMANDS

PIPES AND COMMANDS

I/O redirection

stdin, stdout and stderr

output redirection

# > stdout
$ echo It is cold today! > winter.txt
$ cat winter.txt
It is cold today !
# Output file is erased !
$ zcho It is cold today! > winter.txt
-bash: zcho: command not found
$ cat winter.txt
$
# >> append 
$ echo It is cold today! > winter.txt
$ echo Where is the summer ? >> winter.txt
$ cat winter.txt
It is cold today!
Where is the summer ?
$

error redirection

# Redirecting stderr is done with 2>
$ find / > allfiles.txt 2> /dev/null # Redirect stdout to file and stderr to /dev/null

# 2>&1 Redirect both stdout and stderr to the same file
$ find / > allfiles_and_errors.txt 2>&1

# &> construction will put both stdout and stderr in one stream
$ rm file42 &> out_and_err
$ cat ou_and_err
rm: cannot remove `file42`: No such file or directory
$ echo file42 &> out_and_err
$ cat out_and_err
file42
$ 

input redirection

# < stdin
$ cat < text.txt
one two

# << here document
$ cat << EOF
> Working dir "$PWD" `pwd`
> EOF
Working dir "/home/paul" /home/paul

# <<< here string

$ base64 <<< institut-de-la-pme    # equivalent of echo institut-de-la-pme | base64
aW5zdGl0dXQtZGUtbGEtcG1lCg==
$ base64 -d <<< aW5zdGl0dXQtZGUtbGEtcG1lCg==
institut-de-la-pme

# Quickest way to clear a file
$ >text.txt

🚀 HANDS-ON

  1. Make sure you have two shells open on the same computer. Create an empty tailing.txt file. Then type tail -f tailing.txt. Use the second shell to append a line of text to that file. Verify that the first shell displays this line.
  2. Create a file that contains the names of five people. Use cat and output redirection to create the file and use a here document to end the input.

PIPES AND COMMANDS

FILTERS

cat

$ tac count.txt | cat | cat | cat | cat | cat
five
four
three
two
one
$

When between two pipes, the cat command does nothing (except putting stdin on stdout).

tee

$ tac count.txt | tee temp.txt | tac
one 
two
three
four
five
$ cat temp.txt
five
four
three
two
one
$ 

Puts stdin on stdout and also into a file. Useful on long pipes to store intermediate results.

grep

$ cat tennis.txt
Amelie Mauresmo, Fra
Kim Clijsters, BEL
Justine Henin, Bel
Serena Williams, usa
Venus Williams, USA
$ cat tennis.txt | grep Williams
Serena Williams, usa
Venus Williams, USA
$ grep Bel tennis.txt
Justine Henin, Bel
$ grep -i Bel tennis.txt # insensitive way
Kim Clijsters, BEL
Justine Henin, Bel
$ grep -v Fra tennis.txt # outputs lines not matching the string
Kim Clijsters, BEL
Justine Henin, Bel
Serena Williams, usa
Venus Williams, USA

The most well-known filter! 

cut

# Using colon as a delimiter, selecting fields 1 and 3
$ cut -d: -f1,3 /etc/passwd | tail -4 
syslog:498
jira:34064
omsagent:497
nxautomation:496

# Using a speace as the delimiter for cut, you have to quote the space
$ cut -d" " -f1 tennis.txt
Amelie
Kim
Justine
Serena
Venus

# Display the second to the seventh character of /etc/passwd
$ cut -c2-7 /etc/passwd | tail -4 
yslog:
ira:x:
msagen
xautom

Filter can select columns from files, depending on a delimiter.

tr

$ cat tennis.txt | tr 'e' 'E'
AmEliE MaurEsmo, Fra
Kim ClijstErs, BEL
JustinE HEnin, BEl
SErEna Williams, usa
VEnus Williams, USA

$ cat tennis.txt | tr 'a-z' 'A-Z'
AMELIE MAURESMO, FRA
KIM CLIJSTERS, BEL
JUSTINE HENIN, BEL
SERENA WILLIAMS, USA
VENUS WILLIAMS, USA

$ cat count.txt | tr '\n' ' '
one two three four five

Used for translating characters.

wc

$ wc tennis.txt
  5  15 100 tennis.txt

$ wc -l tennis.txt
5 tennis.txt

$ wc -w tennis.txt
15 tennis.txt

$ wc -c tennis.txt
100 tennis.txt

Counting words, lines and characters.

sort

$ cat music.txt
Queen
Brel
Led Zeppelin
Abba

$ sort music.txt
Abba
Brel
Led Zeppelin
Queen

# Sorting on column
$ sort -k1 country.txt
Belgium, Brussels, 10
France, Paris, 60
Germany, Berlin, 100
Iran, Teheran, 70
Italy, Rome, 50

uniq

$ cat music.txt
Queen
Brel
Queen
Abba

$ sort music.txt
Abba
Brel
Queen
Queen

$ sort music.txt |uniq
Abba
Brel
Queen

# uniq can also count occurences
$ sort music.txt |uniq -c
  1 Abba
  1 Brel
  2 Queen

sed

$ echo level5 | sed 's/5/42/'
level42

$ echo level5 | sed 's/level/jump/'
jump5

$ echo level5 level7 | sed 's/level/jump/'
jump5 level7

# Add g for global replacements (all occurences of the string per line)
$ echo level5 level7 | sed 's/level/jump/g'
jump5 jump7

# With d you can remove lines from a stream containing a character.
$ cat tennis.txt
Venus Williams, USA
Martina Hingis, SUI
Justine Henin, BE

$ cat tennis.txt | sed '/BE/d'
Venus Williams, USA
Martina Hingis, SUI

The stream editor sed can perform editing functions in the stream.

pipe examples

# How many users are logged on to this system ?
$ who 
root tty1 Jul 25 10:50
paul pts/0 Jul 25 09:29 (laika)
Harry pts/1 Jul 25 12:26 (barry)
paul pts/2 Jul 25 12:26 (pasha)
$ who | wc -l 
4

# Display a sorted list of logged on users
$ who | cut -d' ' -f1 | sort
Harry
paul
paul
root

# Display a sorted list of logged on users, but every user only once.
$ who | cut -d' ' -f1 | sort | uniq
Harry
paul
root

🚀 HANDS-ON

  1. Put a sorted list of all bash users in bashusers.txt
  2. Put a sorted list of all logged on users in onlineusers.txt
  3. Make a list of all filenames in /etc that contain the string conf in their filename.
  4. Make a sorted like of all files in /etc that contain the case insensitive string conf in their filename
  5. Look at the output of /sbin/ifconfig. Write a line that displays only ip address and the subnet mask.
  6. Write a line that removes all non-letters from a stream.
  7. Write a line that receives a text file, and outputs all words on a separate line.

PIPES AND COMMANDS

BASIC UNIX TOOLS

find

# Find all files in /etc and put the list in etcfiles.txt
$ find /etc > etcfiles.txt

# Find all files of the entire system and put the list in allfiles.txt
$ find / > allfiles.txt

# Find all files that en in .conf in the current directory (an all subdirs)
$ find . -name "*.conf"

# Find files of type file (not directory, pipe ...) that en in .conf
$ find . -type f -name "*.conf"

# Find files that are newer than file42.txt
$ find . -newer file42.txt

# Find *.odf files in /data and copy them to /backup/
$ find /data -name "*.odf" -exec cp {} /backup/ \;

Can be very useful at the start of a pipe to search for files.

Add 2>/dev/null to avoid cluttering your screen with errors.

locate

$ locate Samba
warning: locate: could not open database: /var/lib/slocate/slocate.db:...
warning: You need to run the 'updatedb' command (as root) to create th...
Please have a look at /etc/updatedb.conf to enable the daily cron job.
$ updatedb
fatal error: updatedb: You are not authorized to create a default sloc...
$ su -
Password:
$ updatedb
$

The locate tool is very different from find in that it uses an index to locate files.

This is faster than traversing all the directories but it also means that it's always outdated. If the index does not exist yet, then you have to create it (as root on RHEL).

date

$ date 
Thu Jan 11 13:06:40 UTC 2018

$ date +'%A %d-%m-%Y'
Thursday 11-01-2018

$ date +%s
1515676031

# When will this seconds counter read two thousand milion ?
$ date -d '1970-01-01 + 2000000000 seconds'

To display date, time, time zone and more

cal

$ cal
    January 2018
Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6
 7  8  9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31

$ cal 2 1970
    February 1970
Su Mo Tu We Th Fr Sa
 1  2  3  4  5  6  7
 8  9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28

sleep

$ sleep 5
$ 

Sometimes used in scripts to wait a number of seconds.

time

$ time date
Thu Jan 11 13:12:30 UTC 2018

real    0m0.002s
user    0m0.001s
sys     0m0.000s

# The sleep 5 command takes five real seconds to execute it
$ time sleep 5

real    0m5.002s
user    0m0.001s
sys     0m0.001s

Can display how long it takes to execute a command.

gzip - gunzip - zcat - zmore 

$ ls -lh
total 10M
-rw-r--r-- 1 paul users 10M Jan 11 13:25 10mb.txt
$ gzip 10mb.txt
$ ls -lh
total 12K
-rw-r--r-- 1 paul users 10K Jan 11 13:25 10mb.txt.gz
$ gunzip 10mb.txt.gz
$ ls -lh
total 10M
-rw-r--r-- 1 paul users 10M Jan 11 13:25 10mb.txt

# Files that are compressed with gzip can be viewed with zcat and zmore

Users never have enough disk space so compression comes in handy.

bzip2 - bunzip2 - bzcat - bzmore

🚀 HANDS-ON

  1. Explain the difference between these two commands
    a) find /data -name "*.txt"
    b) find /data -name *.txt
  2. Write a find command that finds all files created after January 30th 2010.
  3. Count the number of *.conf files in /etc and all its subdirs.
  4. Create a file called loctest.txt. Can you find this file with locate ? Why not ? How do you make locate find this file ?
  5. Use find and -exec to rename all .htm files to .html.

vi

Introduction to vi

The vi editor is installed on almost every Unix.

Linux will very often install vim (vi improved) which is similar.

 

EVERY SYSTEM ADMINISTRATOR SHOULD KNOW VI(M) BECAUSE IT IS AN EASY TOOL TO SOLVE PROBLEMS.

 

vi cheat sheet

SCRIPTING

SCRIPTING

SCRIPTING INTRODUCTION

hello world

$ echo echo Hello World > hello_world
$ chmod +x hello_world
$ ./hello_world
Hello World
$ 

she-bang

#!/bin/bash
Hello World

#! is called a she-bang.

Used because you can never be sure which shell a user is running.

comment

#!/bin/bash
#
# Hello World Script
#
echo Hello World

variables

#!/bin/bash
#
# simple variable in script
#
var1=4
echo var1 = $var1
$ echo $var1

$ ./vars
var1 = 4
$ echo $var1

$ 

sourcing a script

$ source ./vars
var1 = 4
$ echo $var1
4
$ 

Luckily, you can force a script to run in the same shell; this is called sourcing a script.

🚀 HANDS-ON

Give each script a different name, keep them for later!

  1. Write a script that outputs the name of a city.
  2. Make sure the script runs in the bash shell.
  3. Makes sure the script runs in the Korn shell.
  4. Create a script that defines two variables, and outputs their value.
  5. The previous script does not influence your current shell. Now run the script so that it influences your current shell.
  6. Is there a shorter way to source the script ?
  7. Comment your scripts so that you know what they're doing.

SCRIPTING

SCRIPTING LOOPS

test []

$ test 10 -gt 55 ; echo $?
1
$ test 56 -gt 55 && echo true || echo false
true
$ [ 6 -gt 55 ] && echo true || echo false
false

# [ -d foo ] Does the directory foo exist ?
# [ -e bar ] Does the file bar exist ?
# [ '/etc' = $PWD ] Is the string /etc equal to the variable $PWD ?
# [ $1 != 'secret' ] Is the first parameter different from secret ?
# [ 55 -lt $bar ] Is 55 less than the value of $bar ?
# [ $foo -ge 1000 ] Is the value of $foo greater or equal to 1000 ?
# [ "abc" < $bar ] Does abc sort before the value of $bar ?
# [ -f foo ] Is foo a regular file ?
# [ -r bar ] Is bar a readable file ?
# [ foo -nt bar ] Is file foo newer than file bar ?

The test command can test whether something is true or false.

if then else

#!/bin/bash

if [ -f isit.txt ]
then echo isit.txt exists!
else echo isit.txt not found!
fi

If a certain condition is met, then execute something, else execute something else.

$ ./choice
isit.txt not found!
$ touch isit.txt
$ ./choice
isit.txt exists!
$ 

if then elif

#!/bin/bash
count=42
if [ $count -eq 42 ]
then
  echo "42 is correct."
elif [ $count -gt 42 ]
then
  echo "Too much."
else
  echo "Not enough."
fi

You can nest a new if inside an else with elif.

for loop

for i in 1 2 4
do
  echo $i
done
for counter in {1..20}
do
  echo counting from 1 to 20, now at $counter
  sleep 1
done

while loop

i=100
while [ $i -ge 0 ] ;
do
  echo Counting down, from 100 to 0, now at $i;
  let i--;
done

until loop

let i=100;
until [ $i -le 0 ];
do
  echo Counting down, from 100 to 1, now at $i;
  let i--;
done

🚀 HANDS-ON

  1. Write a script that uses
    • A for loop to count from 3 to 7.
    • A while loop to count from 1 to 17000.
    • An until loop to count down from 8 to 4.
  2. Write a script that counts the number of file ending in .txt in the current directory.
  3. Wrap an if statement around the script so it is also correct when there are zero files ending in .txt.

SCRIPTING

SCRIPTING PARAMETERS

script parameters

#!/bin/bash
echo The first argument is $1
echo The second argument is $2
echo The third argument is $3

echo \$ $$ PID of the script
echo \# $# count arguments
echo \? $? last return code
echo \* $* all the arguments
$ ./pars one two three
The first argument is one
The second argument is two
The third argument is three
$ 5610 PID of the script
# 3 count arguments
? 0 last return code
* one two three all the arguments

runtime input

#!/bin/bash
echo -n Enter a number:
read number

You can ask the user for input.

sourcing a config file

$ cat myApp.conf
# The config file of myApp
# Enter the path here
myAppPath=/var/myApp
# Enter the number of quines here
quines=5

The source can be used to source a config file.

$ cat myApp.bash
#!/bin/bash
# ...
. ./myApp.conf
echo There are $quines quines
$ ./myApp.bash
There are 5 quines
$ 

get script options with getopts

$ cat options.ksh
#!/bin/ksh

while getopts ":afz" option;
do
  case $option in
    a)
      echo received -a
      ;;
    f)
      echo received -f
      ;;
    z)
      echo received -z
      ;;
    *)
      echo "invalid option -$OPTARG"
      ;;
    esac
done

🚀 HANDS-ON

  1. Write a script that receives four parameters, and outputs them in reverse order.
  2. Write a script that receives two parameters (two filenames) and outputs whether those files exist.
  3. Write a script that asks for a filename. Verify existence of the file, then verify that you own the file, and whether it is writable. If not, then make it writable.
  4. Make a configuration file for the previous script. Put a logging switch in the config file, logging means writing detailed output of everything the script does to a log file in /tmp.

SCRIPTING

SCRIPTING PARAMETERS

eval

# This is horrible but :
$ answer=42
$ word=answer
$ eval x=\$$word ; echo $x
42

eval reads arguments as input to the shell. Allows using the value of a variable as a variable.

(( ))

$ (( 42 > 33 )) && echo true || echo false
true
$ (( 42 > 1201 )) && echo true || echo false
false
$ var42=42
$ (( 42 == var42 )) && echo true || echo false
false
$ (( 42 == $var42 )) && echo true || echo false
true
$ var42=33
$ (( 42 == $var42 )) && echo true || echo false
false

The (( )) allows for evaluation of numerical expressions.

let

$ let x="3 + 4" ; echo $x
7
$ let x="10 + 100/10" ; echo $x
20
$ let x="10-2+100/10" ; echo $x
18
$ let x="10*2+100/10" ; echo $x
30

# 
                Care! let is not variable value assignation :
$ dec=15 ; oct=017 ; hex=0x0f
$ echo $dec $oct $hex
15 017 0x0f
$ let dec=15 ; let oct=017 ; let hex=0x0f
$ echo $dec $oct $hex
15 15 15
            

The let built-in shell function instructs the shell to perform an evaluation of aritmetic expressions.

case

$ ./help
What animal did you see ? lion
You better start running fast!
$ cat help
#!/bin/bash
#...
echo -n "What animal did you see ? "
read animal
case $animal in
  "lion" | "tiger")
    echo "You better start running fast!"
    ;;
  "cat")
    echo "Let that mouse go..."
    ;;
  "dog")
    echo "Don't worry, give it a cookie."
    ;;
  *)
    echo "You discovered an unknown animal, name it!"
    ;;
esac
$

shell functions

$ cat funcs.ksh
#!/bin/ksh

function greetings {
  echo Hello World!
  echo and hello to $USER to!
}
echo We will now call a function
greetings
echo The end
$ ./funcs.ksh
We will now call a function
Hello World!
and hello to paul to!
The end

🚀 HANDS-ON

  1. Write a script that asks for two numbers, and outputs the sum and product.
  2. Improve the previous script to test that the numbers are between 1 and 100, exit with an error if necessary.
  3. Improve the previous script to congratulate the user is the sum is equals to the product.

LOCAL USER MANAGEMENT

LOCAL USER MANAGEMENT

INTRODUCTION TO USERS

whoami

$ whoami
paul
$

Tells you your username.

who - who am i

$ who
root    pts/0     2014-10-10 23:07 (10.104.33.101)
paul    pts/1     2014-10-10 23:30 (10.104.33.101)
laura   pts/2     2014-10-10 23:34 (10.104.33.96)
tania   pts/3     2014-10-10 23:39 (10.104.33.91)

$ who am i
paul    pts/1     2014-10-10 23:30 (10.104.33.101)

Give you information about who is logged on the system.

w

$ w
23:34:07 up 31 min, 2 users, load average: 0.00, 0.01, 0.02
USER     TTY     LOGIN@     IDLE       JCPU     PCPU     WHAT
root     pts/0    23:07     15.00s    0.01s     0.01s     top
paul     pts/1    23:30     7.00s     0.00s     0.00s     w
$

Shows you who is logged on and what they are doing

id

$ id
uid=1000(paul) gid=1000(paul) groups=1000(paul)

# On RHEL : you will also get SELinux context information
$ id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r\
:unconfined_t:s0-s0:c0.c1023

Give you your user id, primary group id, and a list of the groups that you belong to

su

# To another user
laura@centos:~$ su tania
Password:
tania@centos:/home/laura$

# To root
laura@centos:~$ su root
Password:
root@centos:/home/laura#

# As root
root@centos:~# id
uid=0(root) gid=0(root) groups=0(root)
root@centos:~# su - valentina
valentina@centos:~$

# Use su - to get the target user's environment !
$ su - # will assume root is the target

Allow a user to run a shell as another user

run a program as another user

# Example
# User paul received the right to run useraddwith the credentials of root. 
# This allows paul to create new users on the system without becoming root and without knowing the root password.
paul@centos:~$ /usr/sbin/useradd -m valentina
useradd: Permission Dernied.
useradd: cannot lock /etc/passwd; try again later.

# But with sudo it works:
paul@centos:~$ sudo /usr/sbin/useradd -m valentina
[sudo] password for paul:
$
  • The sudo program allows a user to start a program with the credentials of another user.
  • Before this works, the system admin has to set up the /etc/sudoers file.
  • This can be useful to delegate administrative tasks to another user.

visudo

$ apropos visudo
visudo        (8)    - edit the sudoers file
$ 

Check the man page of visudo before playing with the /etc/sudoers file !

sudo logging

$ sudo su -
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for paul:
paul is not in the sudoers file. This incident will be reported.
$

Using sudo without autorization will result in a severe warning

Check /var/log/secure ;-)

🚀 HANDS-ON

  1. Run a command that displays only your currently logged on user name.
  2. Display a list of all logged users.
  3. Display your user name and your unique user identification (userid).
  4. Use su to switch to another user account and get back to the previous account.
  5. Now use su - to switch to another user and notice the differences.

LOCAL USER MANAGEMENT

USER MANAGEMENT

/etc/passwd

$ tail /etc/passwd
apache:x:48:48:Apache:/var/www:/sbin/nologin
BIRT:x:30138:508::/home/BIRT:/bin/bash
postgres:x:26:26:PostgreSQL Server:/var/lib/pgsql:/bin/bash

The local user database on Linux (and most Unixes).

The columns contain the username, an x, the user id, the primary group id, a description, the name of the home directory, and the login shell.

$ man 5 passwd

root

$ head -1 /etc/passwd
root:x:0:0:root:/root:/bin/bash

The root user always has userid 0 (regardless of the name of the account).

useradd - userdel

[root@centos ~]# useradd -m -d /home/alexis -c "alexis plantin" alexis
[root@centos ~]#  tail -1 /etc/passwd
alexis:x:529:529:alexis plantin:/home/alexis:/bin/bash

Add users with the useradd command.

The user named alexis received userid 529 and primary group id 529.

#To delete the user alexis and remove also his home directory
[root@centos ~]# userdel -r alexis 

usermod

$ tail -1 /etc/passwd
harry:x:516:520:harry potter:/home/harry:/bin/bash
$ usermod -c 'wizard' harry
$ tail -1 /etc/passwd
harry:x:512:520:wizard:/home/harry:/bin/bash

Modify properties of a user with the usermod command.

creating home directories

$ mkdir /home/laura
$ chown laura:laura /home/laura
$ chmod 700 /home/laura
  • Easiest way : add -m option with useradd
  • Less easy way :

/etc/skel

$ ls -al /etc/skel/
total 32
drwxr-xr-x.   2 root root  4096 Oct 23 14:16 .
drwxr-xr-x. 109 root root 12288 Jan 10 04:30 ..
-rw-r--r--    1 root root    18 Mar 23  2017 .bash_logout
-rw-r--r--    1 root root   176 Mar 23  2017 .bash_profile
-rw-r--r--    1 root root   124 Mar 23  2017 .bashrc
-rw-r--r--    1 root root   500 Nov 27  2014 .emacs

When useradd the -m option is used, the /etc/skel directory is copied to the newly created home directory.

/etc/skel contains somes usually files that contain profil settings and default values for applications.

deleting home directories

-r option of userdel will make sure that the home directory is deleted.

chsh

Users can change their login shell with the chsh command.

$ chsh -l
/bin/sh
/bin/bash
...
...
$ chsh -s /bin/ksh
Changing shell for laura.
Password:
Shell changed.

🚀 HANDS-ON

  1. Create a user account names serena, including a home directory and a description tha reads Serena Williams. Do all this in one single command.
  2. Create a user named venus, including home directory, bash shell, a description that reads Venus Williams all in one single command.
  3. Verify that both users have correct entriers in /etc/passwd, /etc/shadow and /etc/group.
  4. Verify that their home directory was created.
  5. Create a user named einstime with /bin/date as default logon shell. What happens when you log on ?

LOCAL USER MANAGEMENT

USER PASSWORDS

passwd

Update users passwords.

[paul@centos ~]$ passwd
Changing password for user paul.
Changing password for paul.
(current) UNIX password:
New password:
BAD PASSWORD: The password is shorter than 8 characters
New password:
BAD PASSWORD: The password is a palindrome
New password:
BAD PASSWORD: The password is too similar to the old one
passwd: Have exhausted maximum number of retries for service

# As root:
root@centos:~# passwd paul
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

shadow file

User passwords are encrypted and kept in /etc/shadow.

[root@centos ~]# tail -4 /etc/shadow
omsagent:!!:17450::::::
apache:!!:17452::::::
nxautomation:!!:17536::::::
f298530:$6$vtRFTYzA$ufKddy.ws6oNfro/aDxND5.ngVxjY50.7ttEfxf7vWRzySx.Mka.CZ1:17546:0:99999:7:::

/etc/login.defs

Contains some default settings for user passwords.

$ grep PASS /etc/login.defs
#       PASS_MAX_DAYS   Maximum number of days a password may be used.
#       PASS_MIN_DAYS   Minimum number of days allowed between password changes.
#       PASS_MIN_LEN    Minimum acceptable password length.
#       PASS_WARN_AGE   Number of days warning given before a password expires.
PASS_MAX_DAYS   99999
PASS_MIN_DAYS   0
PASS_MIN_LEN    5
PASS_WARN_AGE   7
$ 

chage

Set an expiration date for a user account ( -E ), minimum ( -m ) and maximum ( -M ) password age ...

$ chage -l paul
Last password change                                    : Jan 15, 2018
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

disabling a password

Locking, disabling or suspending a user account.

[root@centos ~]# grep paul /etc/shadow | cut -c1-70
paul:$6$vtRFTYzA$ufKddy.ws6oNfro/aDxND5.ngVKDKUBxjY50.7ttEfxf7vWRzy
[root@centos ~]# usermod -L paul # Locking a user
[root@centos ~]# grep paul /etc/shadow | cut -c1-70
paul:!$6$vtRFTYzA$ufKddy.ws6oNfro/aDxND5.ngVKDKUBxjY50.7ttEfxf7vWRz
[root@centos ~]# usermod -U paul # Unlocking a user
[root@centos ~]# grep paul /etc/shadow | cut -c1-70
paul:$6$vtRFTYzA$ufKddy.ws6oNfro/aDxND5.ngVKDKUBxjY50.7ttEfxf7vWRzy

🚀 HANDS-ON

  1. Set the password for serena to hunter2.
  2. Also set a password for venus and then lock the venus user account with usermod. Verify the locking in /etc/shadow before and after you lock it.
  3. Use passwd -d to disable the serena password. Verify the serena line in /etc/shadow before and after disabling.
  4. What is the difference between locking a user account and disabling a user account's password like we just did with usermod -L and passwd -d ?
  5. Make sure serena has to change her password in 10 days.

LOCAL USER MANAGEMENT

USER PROFILES

system profile

Both the bash and the ksh shell will verify the existence of /etc/profile and source it if it exists.

~/.bash_profile

When this file exists in home directory, then bash will source it.

$  cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

~/.profile

When neither ~/.bash_profile and ~/.bash_login exist, then bash will verify the existence of ~/.profile and execute it.

$ cat .profile
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
alias vi="vim"

~/.bashrc

Leaves room for custom aliases and functions.

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions

~/.bash_logout

When exiting bash, it can execute ~/.bash_logout.

# ~/.bash_logout

RHEL overview

script su su - ssh
~/.bashrc yes yes yes
~/.profile no yes yes
/etc/profile no yes yes
/etc/bashrc yes yes yes

🚀 HANDS-ON

  1. Make a list of all the profile files on your system.
  2. Read the contents of each of these, often they source extra scripts.
  3. Put a unique variable, alias and function in each of those files.
  4. When an application depends on a setting in $HOME/.profile, does it matter whether $HOME/.bash_profile exists or not ?

LOCAL USER MANAGEMENT

GROUPS

groupadd

Groups can be created with the groupadd command.

# groupadd tennis
# groupadd football
# groupadd smoker
# groupadd formula1
# groupadd salsa

group file

Users can be a member of several groups. Group membership is defined by the /etc/group file.

# tail -5 /etc/group
ipme:x:1000:
omiusers:x:993:nxautomation
omsagent:x:992:nxautomation
apache:x:48:
nxautomation:x:991:

Group name ; group's (encrypted) password ; group identification ; list of members

groups

A user can type the groups command to see a list of groups where the user belongs to.

[paul@centos ~]$ groups
harry sports

usermod

Group membership can be modified with the useradd or usermod command

root@centos:~# usermod -a -G tennis inge
root@centos:~# usermod -a -G tennis katrien
root@centos:~# usermod -a -G salsa katrien
root@centos:~# usermod -a -G snooker sandra
root@centos:~# usermod -a -G formula1 annelies
root@centos:~# tail -5 /etc/group
tennis:x:1006:inge,katrien
football:x:1007:
snooker:x:1008:sandra
formula1:x:1009:annelies
salsa:x:1010:katrien
root@centos:~# 

groupmod

You can change the group name with groupmod command.

# Rename the football group to foot
root@centos:~# groupmod -n foot football
root@centos:~# tail -5 /etc/group
tennis:x:1006:inge,katrien
foot:x:1007:
formula1:x:1009:annelies
salsa:x:1010:katrien
darts:x:1008:sandra
root@centos:~# 

groupdel

You can permanently remove a group with the groupdel command.

root@centos:~# groupdel tennis
root@centos:~# 

🚀 HANDS-ON

  1. Create the groups tennis, football and sports.
  2. In one command, make venus a member of tennis and sports.
  3. Rename the football group to foot.
  4. Use vi to add serena to the tennis group.
  5. Use the id command to verify that serena is a member of tennis.

FILE SECURITY

FILE SECURITY

STANDARD FILE PERMISSIONS

user & group owner

Every file has a user owner and a group owner.

paul@centos:~/owners$ ls -lh
total 636K
-rw-r--r--. 1 paul snooker 1.1K Apr 8 18:47 data.odt
-rw-r--r--. 1 paul paul 626K Apr 8 18:46 file1
-rw-r--r--. 1 root tennis 185 Apr 8 18:46 file2
-rw-rw-r--. 1 root root 0 Apr 8 18:47 stuff.txt

listing user accounts

You can use the following command to list all local user accounts.

paul@centos:~/owners$ $ cut -d: -f1 /etc/passwd | column
root                    ftp                     postfix
bin                     nobody                  ntp
daemon                  systemd-bus-proxy       chrony
adm                     systemd-network         tcpdump
lp                      dbus                    michelin
sync                    polkitd                 omsagent
shutdown                libstoragemgmt          apache
halt                    abrt                    nxautomation
mail                    rpc                     ipme
operator                sshd
games                   tss

chgrp

You can change the group owner of a file using the chgrp command.

root@centos:/home/paul/owners# ls -l file2
-rw-r--r--. 1 root tennis 185 Apr 8 18:46 file2
root@centos:/home/paul/owners# chgrp snooker file2
root@centos:/home/paul/owners# ls -l file2
-rw-r--r--. 1 root snooker 185 Apr 8 18:46 file2
root@centos:/home/paul/owners#
                

chown

The user owner of a file can be changed with chown command.

root@centos:/home/paul# ls -l FileForPaul
-rw-r--r-- 1 root paul 0 2008-08-06 14:11 FileForPaul
root@centos:/home/paul# chown paul FileForPaul
root@centos:/home/paul# ls -l FileForPaul
-rw-r--r-- 1 paul paul 0 2008-08-06 14:11 FileForPaul
# You can also use chown to change both the user owner and the group owner.
root@centos:/home/paul# ls -l FileForPaul
-rw-r--r-- 1 paul paul 0 2008-08-06 14:11 FileForPaul
root@centos:/home/paul# chown root:project42 FileForPaul
root@centos:/home/paul# ls -l FileForPaul
-rw-r--r-- 1 root project42 0 2008-08-06 14:11 FileForPaul

list of special files

When you use ls -l, the first character tells us the type of file.

first character file type
- normal file
d directory
l symbolic link
p named pipe
b block device
c character device
s socket

permissions

The nine characters following the file type denote the permissions in three triplets.

permission on a file on a directory
​r (read) read file contents (cat) read directory content (ls)
​w (write) change file contents (vi) create files in (touch)
​x (execute) execute the file enter the directory (cd)

three sets of rwx

position characters function
1 - this is a regular file
2-4 rwx permissions for the user owner
5-7 r-x permissions for the group owner
8-10 r-- permissions for others
root@centos:/home/paul# ls -l 
-rwxr-xr-- 1 root paul 0 2008-08-06 14:11 FileForPaul

chmod

[paul@centos]$ ls -l
total 0
-rw-r--r--. 1 paul users 0 Jan 15 14:18 permissions.txt
[paul@centos]$ chmod u+x permissions.txt
[paul@centos]$ ls -l
total 0
-rwxr--r--. 1 paul users 0 Jan 15 14:18 permissions.txt
[paul@centos]$ chmod o-r permissions.txt
[paul@centos]$ ls -l
total 0
-rwxr-----. 1 paul users 0 Jan 15 14:18 permissions.txt
[paul@centos]$ chmod a+w permissions.txt
[paul@centos]$ ls -l
total 0
-rwxrw--w-. 1 paul users 0 Jan 15 14:18 permissions.txt
[paul@centos]$ chmod u=rw,g=rw,o=r permissions.txt
[paul@centos]$ ls -l
total 0
-rw-rw-r--. 1 paul users 0 Jan 15 14:18 permissions.txt

Permissions can be changed with chmod.

setting octal permissions

[paul@centos]$ ls -l
-rw-r--r--. 1 paul users 0 Jan 15 14:18 permissions.txt
[paul@centos]$ chmod 674 permissions.txt
[paul@centos]$ ls -l
-rw-rwxr--. 1 f298530 users 0 Jan 15 14:18 permissions.txt
binary octal permission
000 0 ---
001 1 --x
010 2 -w-
011 3 -wx
100 4 r--
101 5 r-x
110 6 rw-
111 7 rwx

🚀 HANDS-ON

  1. As normal user, create a directory ~/permissions. Create a file owned by yourself in there.
  2. Copy a file owned by root from /etc/  to your permissions dir, who owns this file now ?
  3. As root, create a file in the users ~/permissions directory.
  4. As normal user, look at who owns this file created by root.
  5. Change the ownership of all files in ~/permissions to yourself.
  6. Make sure you have all rights to these files, and others can only read.

FILE SECURITY

ADVANCED FILE PERMISSIONS

sticky bit on directory

Prevent users from removing files that they do not own as a user owner.

[paul@centos]$ ls -ld ipme/
drwxr-xr-x. 2 paul users 6 Jan 15 14:33 ipme/
[paul@centos]$ chmod +t ipme/ # Octal syntax : chmod 1775 ipme/
[paul@centos]$ ls -ld ipme/
drwxr-xr-t. 2 paul users 6 Jan 15 14:33 ipme/

You typically find the sticky bit on the /tmp directory.

setgid bit on directory

To make sure that all files inside the directory are owned by the group owner of the directory.

[root@centos]# ll
total 0
drwxr-xr-t. 2 paul users 6 Jan 15 14:33 ipme
[root@centos]# groupadd proj55
[root@centos]# chown root:proj55 ipme/
[root@centos]# chmod 2775 ipme/
[root@centos]# touch ipme/test.txt
[root@centos]# ls -l ipme/
total 0
-rw-r--r--. 1 root proj55 0 Jan 15 14:40 test.txt

set(u/g)id bit on regular file

These two permissions cause an executable file to be executed with the permissions of the file owner instead of the executing owner.

[root@centos]# ls -l /etc/shadow
----------. 1 root root 982 Jan 15 12:40 /etc/shadow
# Changing your password required an update of this file, so how can normal
# non-root users do this ?
[root@centos]# ls -l /usr/bin/passwd
-rwsr-xr-x. 1 root root 27832 Jun 10  2014 /usr/bin/passwd
# When running the passwd program, you are executing it with root credentials.

In most case, setting the setuid bit on executables is sufficient. Setting the setgid bit will result in these programs to run with the credentials of their group owner.

🚀 HANDS-ON

  1. Set up a directory, owned by the group sports.
  2. Members of the sports group should be able to create files in this directory.
  3. All files created in this directory should be group-owned by the sports group.
  4. Users should be able to delete only their own user-owned files.
  5. Test that this works!

FILE SECURITY

ACCESS CONTROL LISTS

acl in /etc/fstab

File systems that support access control lists have to be mounted with the acl option listed in /etc/fstab.

root@centos:~# tail -4 /etc/fstab
/dev/sda1      /           ext3  acl,relatime    0 1
/dev/sdb2      /home/data  auto  noacl,defaults  0 0
pasha:/home/r  /home/pasha nfs   defaults        0 0
wolf:/srv/data /home/wolf  nfs   defaults        0 0

getfacl

Reading acls can be done with /usr/bin/getfacl.

[paul@centos]$ touch bidule
[paul@centos]$ getfacl bidule
# file: bidule
# owner: paul
# group: users
user::rw-
group::r--
other::r--

setfacl

Writing or changing acls can be done with /usr/bin/setfacl.

# Add user sandra with octal permission 7 to the acl
[paul@centos]$ setfacl -m u:sandra:7 bidule

# Add the group tennis with octal permission 6 to the acl of the same file
[paul@centos]$ setfacl -m g:tennis:6 bidule

# Result is visible with getfacl 
[paul@centos]$ getfacl bidule
# file: bidule
# owner: paul
# group: users
user::rw-
user:sandra:rwx
group::r--
group:tennis:rw-
mask::rwx
other::r--

remove an acl entry

# Remove an acl entry
$ setfacl -m u:sandra:7 bidule
$ getfacl bidule | grep sandra
user:sandra:rwx
$ setfacl -x sandra bidule
$ getfacl bidule | grep sandra

# Remove the complete acl
$ setfacl -b bidule
$ getfacl bidule
# file: bidule
# owner: paul
# group: paul
user::rw-
group::r--
other::r--

FILE SECURITY

FILE LINKS

inode contents

An inode is a data structure that contains metadata about a file. When the FS stores a new file on the HD, it stores not only the contents of the file but also extra properties (name, creation date, its permissions, owner ...).

# The ls -l command will display some of the inode contents
root@centos~# ls -ld /home/project42/
drwxr-xr-x 4 root pro42 4.0K Mar 27 14:29 /home/project42/

inode table

The inode table contains all of the inodes and is created when you create the FS (with mkfs).

[paul@centos]$ df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda2      15472128 169689 15302439    2% /
devtmpfs         243306    346   242960    1% /dev
tmpfs            245737      1   245736    1% /dev/shm
tmpfs            245737    396   245341    1% /run
tmpfs            245737     16   245721    1% /sys/fs/cgroup
/dev/sda1        256000    335   255665    1% /boot
/dev/sdb1        655360     12   655348    1% /mnt/resource
tmpfs            245737      1   245736    1% /run/user/1001

inode number

Each inode has a unique number (the inode number). You can see the inode numbers with the ls -li command.

[paul@centos]$ touch file1
[paul@centos]$ touch file2
[paul@centos]$ touch file3
[paul@centos]$ ls -li
total 0
 3235 -rw-r--r--. 1 paul paul 0 Jan 15 16:08 file1
18362 -rw-r--r--. 1 paul paul 0 Jan 15 16:08 file2
18372 -rw-r--r--. 1 paul paul 0 Jan 15 16:08 file3

All the information you see with this ls command resides in the inode, except for the filename (contained in the directory).

inode and file contents

Let's put some data in one of the files.

[paul@centos]$ ls -li
total 0
 3235 -rw-r--r--. 1 paul paul 0 Jan 15 16:08 file1
18362 -rw-r--r--. 1 paul paul 0 Jan 15 16:08 file2
18372 -rw-r--r--. 1 paul paul 0 Jan 15 16:08 file3
[paul@centos]$ cat file2
It is winter now and it is very cold.
We do not like the cold, we prefer hot summer nights.
[paul@centos]$ 

The data that is displayed by the cat command is not in the inode, but somewhere else on the disk. The inode contains a pointer to that data.

a directory is a table

A directory is a special kind of file that contains a table which maps filenames to inodes.

[paul@centos]$ ls -al
total 0
drwxr-xr-x. 2 paul    users  45 Jan 15 16:08 .
drwx------. 5 paul    users 119 Jan 15 14:18 ..
-rw-r--r--. 1 paul    paul    0 Jan 15 16:08 file1
-rw-r--r--. 1 paul    paul    0 Jan 15 16:08 file2
-rw-r--r--. 1 paul    paul    0 Jan 15 16:08 file3

hard links

Create a hard link to a file with ln.

[paul@centos]# ln file2 hardlink_to_file2
[paul@centos]# ls -al
total 0
drwxr-xr-x. 2 paul    users  69 Jan 15 16:22 .
drwx------. 5 paul    users 119 Jan 15 14:18 ..
-rw-r--r--. 1 paul    paul    0 Jan 15 16:08 file1
-rw-r--r--. 2 paul    paul    0 Jan 15 16:08 file2
-rw-r--r--. 1 paul    paul    0 Jan 15 16:08 file3
-rw-r--r--. 2 paul    paul    0 Jan 15 16:08 hardlink_to_file

Both files have the same inode => same permissions, owner, content.

Both file are equal, meaning you can safely remove the original file, the hardlinked file will remain.

symbolic links

Sometimes called soft links, they are not linked to inodes.

[paul@poc-dgad-mpo test]# ls -al
total 0
drwxr-xr-x. 2 paul users  70 Jan 15 16:28 .
drwx------. 5 paul users 119 Jan 15 14:18 ..
-rw-r--r--. 1 paul    paul    0 Jan 15 16:28 file1
-rw-r--r--. 1 paul    paul    0 Jan 15 16:28 file2
-rw-r--r--. 1 paul    paul    0 Jan 15 16:28 file3
lrwxrwxrwx. 1 paul    paul    5 Jan 15 16:28 symbolic_to_file2 -> file2

Permissions on a symbolic link have no meaning, since the permissions of the target apply.

Hard link are limited to their own partition, symbolic links can link anywhere.

🚀 HANDS-ON

  1. Create two files named winter.txt and summer.txt, put some text in them.
  2. Create a hard link to winter.txt named hlwinter.txt.
  3. Display the inode numbers of these three files.
  4. Use the find command to list the two hardlinked files.
  5. Everything about a file is in the inode, except two things : name them!
  6. Create a symbolic link to summer.txt called slsummer.txt.
  7. Find all files with inode number 2. What does this information tell you ?

THE END.

Made with Slides.com