How to develop distributed applications on Docker platform
Marco Federighi, Developer@Nephila
An open platform for distributed applications for developers and sysadmins
https://www.docker.com/
Advantages
Not a new concept/technology !
e.g. LXC (https://linuxcontainers.org/)
operating-system-level virtualization environment
What a "Container" is ?
Docker add a layer on the Linux operating-system-level virtualization, using some external interface as LXC or the internal Docker interfaces libcontainer
cgroups + kernel namespaces + Union file systems
Isolated resources + shared kernel
Almost 0 overhead!
But...
Containers depend from the host kernel
Isolation and safety is NOT strong as full virtualized system (VMs)
A read-only template!
It defines and describes how to build a container instance
An image can include one or more other images (e.g. OS images, DBMS images etc.), starting from a base image
Simply a set of instructions!
e.g. run a command, create environment variabiles etc.
docker commit creates also a new image, but Dockerfile is often a better way
Put these sequence of commands in a file called... Dockerfile
https://github.com/DjangoBeer/os-o
https://hub.docker.com/r/fmarco/os-o/
FROM debian:jessie
RUN echo 'Installing base dependencies...'
RUN apt-get -y update && apt-get install -y python-dev libcurl4-gnutls-dev libexpat1-dev gettext libz-dev apache2 x11vnc xvfb iceweasel openssh-server
RUN echo 'Done.\n\n'
RUN echo 'Stopping apache...'
RUN apache2ctl stop
RUN echo 'Done.\n\n'
RUN echo 'Installing nginx...'
RUN apt-get install -y nginx
RUN echo 'Done.\n\n'
RUN echo 'Stopping nginx...'
RUN service nginx stop
RUN echo 'Done.\n\n'
RUN echo 'Installing pip, ipython, bpython and supervisor...'
RUN apt-get install -y git python-pip
RUN pip install -U pip
RUN pip install ipython && pip install bpython && pip install virtualenv && pip install supervisor --pre
RUN echo 'Done.\n\n'
RUN echo 'Installing postgresql...'
RUN apt-get install -y postgresql postgresql-contrib
RUN echo 'Done.\n\n'
RUN echo 'Setup some final stuffs...'
RUN mkdir ~/.vnc
RUN x11vnc -storepasswd 1234 ~/.vnc/passwd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]
RUN echo 'Done.\n\n'.
RUN echo 'Installing ssh daemon...'
RUN mkdir /var/run/sshd
RUN echo 'root:1234' | chpasswd
RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config
# SSH login fix. Otherwise user is kicked off after login
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
ENV NOTVISIBLE "in users profile"
RUN echo "export VISIBLE=now" >> /etc/profile
RUN echo 'Done.\n\n'.Create the Dockerfile and add this line to:
FROM debianBuild an image called debian_only with the following command:
sudo docker build -t debian_only .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM debian
latest: Pulling from library/debian
1565e86129b8: Pull complete
a604b236bcde: Pull complete
Digest: sha256:b111538daad89ea8514b9bbf34edb221a58e50a74f12f1908949fcfc397d4bc1
Status: Downloaded newer image for debian:latest
---> a604b236bcde
Successfully built a604b236bcde
And now, let's create and run a container!
docker run -i -t debian_only /bin/bashDocker creates a container based on your image and launch a bash terminal
root@8649d6b04a95:/# ls
bin dev home lib64 mnt proc run srv tmp var
boot etc lib media opt root sbin sys usr
root@8649d6b04a95:/# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 20228 2008 ? Ss 21:15 0:00 /bin/bash
root 17 0.0 0.0 17484 1128 ? R+ 21:28 0:00 ps -aux
root@8649d6b04a95:/# uname -r
3.13.0-68-generic
root@8649d6b04a95:/#
You can list the containers
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8649d6b04a95 debian_only "/bin/bash" 15 minutes ago Up 15 minutes hopeful_swanson
or the images
docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
gaetan/dockercraft latest a20fa6c36db0 5 days ago 832.9 MB
fmarco/os-o latest 53741c79bc36 2 weeks ago 896.8 MB
postgres 9.4 d198c0e3de2a 4 weeks ago 265.1 MB
fmarco/docker-whale latest 034d41ca24b9 5 weeks ago 274.1 MB
hello-world latest 0a6ba66e537a 5 weeks ago 960 B
ubuntu latest 0a17decee413 6 weeks ago 188.4 MB
docker/whalesay latest ded5e192a685 6 months ago 247 MB
create a new directory inside the running container
root@a43cb7a237d2:/# cd
root@a43cb7a237d2:~# mkdir new_dir_created_by_me
root@a43cb7a237d2:~# ls
new_dir_created_by_me
root@a43cb7a237d2:~#
check the differences
> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a43cb7a237d2 debian_only "bin/bash" 2 minutes ago Up 2 minutes determined_poincare
> docker diff determined_poincare
C /root
A /root/new_dir_created_by_me
create a new directory inside the running container
docker commit determined_poincare debian_only_modified
f2cdc54be5f95b7f76c1d460072189bc8e612c7899f06c154d95942da07e7719
new image created!
docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
debian_only_modified latest f2cdc54be5f9 About a minute ago 125.1 MB
debian latest a604b236bcde 4 days ago 125.1 MB
...
Useful client commands
A server application that stores images (Apache license)
Dockerhub is the official public registry (similar to github)
You can push and pull images
You can tag your images! (Versioning)
You can host your own registry!
docker run postgres
Unable to find image 'postgres:latest' locally
latest: Pulling from library/postgres
1565e86129b8: Pull complete
a604b236bcde: Pull complete
e86beb43b578: Pull complete
...
d991122a71c9: Pull complete
06712d9ad68d: Pull complete
Digest: sha256:540b1486294ea46c82f16f267b7d36954faa346fe392268684f919e4b07aea7e
Status: Downloaded newer image for postgres:latest
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
postgres -D /var/lib/postgresql/data
or
pg_ctl -D /var/lib/postgresql/data -l logfile start
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
****************************************************
WARNING: No password has been set for the database.
This will allow anyone with access to the
Postgres port to access your database. In
Docker's default configuration, this is
effectively any other container on the same
system.
Use "-e POSTGRES_PASSWORD=password" to set
it in "docker run".
****************************************************
waiting for server to start....LOG: database system was shut down at 2015-11-24 09:47:25 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
ALTER ROLE
/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
LOG: received fast shutdown request
LOG: aborting any active transactions
LOG: autovacuum launcher shutting down
waiting for server to shut down....LOG: shutting down
LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
LOG: database system was shut down at 2015-11-24 09:47:28 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: received smart shutdown request
LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut downdocker pull mysql
Using default tag: latest
latest: Pulling from library/mysql
2a1fefc8d587: Pull complete
f9519f46a2bf: Pull complete
35e21079caee: Pull complete
b08f3d26d732: Pull complete
ced60f6188d8: Pull complete
93148df18c31: Pull complete
8587beca5034: Pull complete
2593a05c664f: Pull complete
1483cfdf29e5: Pull complete
1366ffba5c3b: Pull complete
4e07f9c5fa5e: Pull complete
1c19ce71381b: Pull complete
d6a18d40a940: Pull complete
0f1a7d5b17f5: Pull complete
Digest: sha256:d5471349b65f83021ae382e59636408ba172e26a8a6c773fec0017a7a298b0f3
Status: Downloaded newer image for mysql:latest
Why ?
Keep persistent data separated from containers life cycle
storing persistent databases, configuration files, data files etc
bypass the Union File System !
How?
directory within one or more containers,
eventually mapped to some host directory (the latter is NOT portable!)
> docker run -d -ti --name debian_with_volume -v /i_am_a_volume debian_only /bin/bash
8c649a855217625b865a81fd26b0162028d63681d0265639eb1e885ef0aa6b40
>docker exec -ti debian_with_volume /bin/bash
root@8c649a855217:/# ls
bin boot dev etc home i_am_a_volume lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
root@8c649a855217:/# cd i_am_a_volume/
root@8c649a855217:/i_am_a_volume# ls
root@8c649a855217:/i_am_a_volume# mkdir new_dir
root@8c649a855217:/i_am_a_volume# cd new_dir/
root@8c649a855217:/i_am_a_volume/new_dir# touch file.txt
root@8c649a855217:/i_am_a_volume/new_dir# echo 'something' > file.txt
> docker inspect debian_with_volume
...
"Mounts": [
{
"Name": "4f0ca188e9855455cbbe35fe0af1af6cfb8cd2649f41153d68379c2f11a4df40",
"Source": "/var/lib/docker/volumes/4f0ca188e9855455cbbe35fe0af1af6cfb8cd2649f41153d68379c2f11a4df40/_data",
"Destination": "/i_am_a_volume",
"Driver": "local",
"Mode": "",
"RW": true
}
],
...
> sudo ls /var/lib/docker/volumes/4f0ca188e9855455cbbe35fe0af1af6cfb8cd2649f41153d68379c2f11a4df40/_data
new_dir
> sudo cat /var/lib/docker/volumes/4f0ca188e9855455cbbe35fe0af1af6cfb8cd2649f41153d68379c2f11a4df40/_data/new_dir/file.txt
something
Map a local host path
> docker run -d -ti --name debian_with_host_volume -v /absolute/host/path:/i_am_a_host_volume debian_only /bin/bash
Same results, but i'm can read and write on a host directory!
You can mount even a single file!
Create not (active) running container that act as logical mount point to keep data separated from application contianer
How to:
Handling manually
Using external tools
https://github.com/ClusterHQ/flocker
https://github.com/paimpozhil/docker-volume-backup
https://github.com/discordianfish/docker-backup
Tool to handle multiple containers/services
a yaml file!
simple commands to do a lot of stuffs!
web:
build: .
ports:
- "8000:8000"
volumes:
- .:/project
links:
- postgres
postgres:
image: postgresContainer can be connected together with:
Port mapping is nice, but you occupy ports and only one container at time can occupy a port
-P with no arguments -> all port
-p host_port:container_port -> specific ports
Linking is the solution !
Use the flag
--link <name or id>:aliasSafer!
No more waste of ports!
Ingredients
https://github.com/DjangoBeer/dockerized_django
1 - Create the project folder
> mkdir docker_django && cd docker_django2 - Create your Dockerfile
FROM python:2.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /project
WORKDIR /project
ADD requirements.txt /project/
RUN pip install -r requirements.txt
ADD . /project/3 - Create the requirements.txt
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/project
ports:
- "8000:8000"
links:
- db
4 - Create the docker-compose.yml
Django
django-cms
psycopg2
> docker-compose run web django-admin.py startproject dockerized_django .
Creating dockerdjango_db_1...
Building web...
Step 1 : FROM python:2.7
---> 58f3d9f818bf
Step 2 : ENV PYTHONUNBUFFERED 1
---> Running in ece6698ab043
---> cb4f31f7f7b4
Removing intermediate container ece6698ab043
Step 3 : RUN mkdir /project
---> Running in d44f7ac76ac6
---> 446dc8d4aa2a
Removing intermediate container d44f7ac76ac6
Step 4 : WORKDIR /project
---> Running in 68d0d802546e
---> 8a13c82320b8
Removing intermediate container 68d0d802546e
Step 5 : ADD requirements.txt /project/
---> c7e9fcb37618
Removing intermediate container e7fb09ac91a9
Step 6 : RUN pip install -r requirements.txt
---> Running in a77ab9331b57
Collecting Django (from -r requirements.txt (line 1))
Downloading Django-1.8.7-py2.py3-none-any.whl (6.2MB)
Collecting django-cms (from -r requirements.txt (line 2))
Downloading django-cms-3.2.0.tar.gz (4.6MB)
Collecting psycopg2 (from -r requirements.txt (line 3))
Downloading psycopg2-2.6.1.tar.gz (371kB)
Collecting django-classy-tags>=0.5 (from django-cms->-r requirements.txt (line 2))
Downloading django_classy_tags-0.6.2-py2-none-any.whl
Collecting html5lib!=0.9999,!=0.99999,>=0.90 (from django-cms->-r requirements.txt (line 2))
Downloading html5lib-0.9999999.tar.gz (889kB)
Collecting django-formtools>=1.0 (from django-cms->-r requirements.txt (line 2))
Downloading django_formtools-1.0-py2.py3-none-any.whl (132kB)
Collecting django-treebeard==3.0 (from django-cms->-r requirements.txt (line 2))
Downloading django-treebeard-3.0.tar.gz (93kB)
Collecting django-sekizai>=0.7 (from django-cms->-r requirements.txt (line 2))
Downloading django_sekizai-0.8.2-py2.py3-none-any.whl
Collecting djangocms-admin-style (from django-cms->-r requirements.txt (line 2))
Downloading djangocms-admin-style-1.0.6.tar.gz (1.5MB)
Collecting six (from html5lib!=0.9999,!=0.99999,>=0.90->django-cms->-r requirements.txt (line 2))
Downloading six-1.10.0-py2.py3-none-any.whl
Building wheels for collected packages: django-cms, psycopg2, html5lib, django-treebeard, djangocms-admin-style
Running setup.py bdist_wheel for django-cms
Stored in directory: /root/.cache/pip/wheels/98/ab/65/55b1f05bb44efbb436b471942f9fc2afba5d9779843333c430
Running setup.py bdist_wheel for psycopg2
Stored in directory: /root/.cache/pip/wheels/e2/9a/5e/7b620848bbc7cfb9084aafea077be11618c2b5067bd532f329
Running setup.py bdist_wheel for html5lib
Stored in directory: /root/.cache/pip/wheels/e1/62/7e/25731b7df4b5b564f5dc89ba14e220c238dd8b115ffea7b4b6
Running setup.py bdist_wheel for django-treebeard
Stored in directory: /root/.cache/pip/wheels/73/8e/8b/3d72e01575557ffeeb1ef648581006d468a415ed868abbfb5d
Running setup.py bdist_wheel for djangocms-admin-style
Stored in directory: /root/.cache/pip/wheels/50/79/31/85a3840f747dabaa54f27e35d467a59ff960a97d935fda548e
Successfully built django-cms psycopg2 html5lib django-treebeard djangocms-admin-style
Installing collected packages: Django, django-classy-tags, six, html5lib, django-formtools, django-treebeard, django-sekizai, djangocms-admin-style, django-cms, psycopg2
Successfully installed Django-1.8.7 django-classy-tags-0.6.2 django-cms-3.2.0 django-formtools-1.0 django-sekizai-0.8.2 django-treebeard-3.0 djangocms-admin-style-1.0.6 html5lib-0.9999999 psycopg2-2.6.1 six-1.10.0
---> 1b13f362cd46
Removing intermediate container a77ab9331b57
Step 7 : ADD . /project/
---> 2e3bdfffd8d2
Removing intermediate container 2decbb5bae46
Successfully built 2e3bdfffd8d2
4 - Create a blank project
> sudo chown -R $USER:$USER .
5 - Change the permission
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'postgres',
'USER': 'postgres',
'HOST': 'db',
'PORT': 5432,
}
}
etc.6 - Edit your settings.py (remember the CMS settings too!)
(Container run as root!)
> docker-compose up
docker-compose up
dockerdjango_db_1 is up-to-date
Creating dockerdjango_web_1...
Attaching to dockerdjango_db_1, dockerdjango_web_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
db_1 | initializing pg_authid ... ok
db_1 | initializing dependencies ... ok
db_1 | creating system views ... ok
db_1 | loading system objects' descriptions ... ok
db_1 | creating collations ... ok
db_1 | creating conversions ... ok
db_1 | creating dictionaries ... ok
db_1 | setting privileges on built-in objects ... ok
db_1 | creating information schema ... ok
db_1 | loading PL/pgSQL server-side language ... ok
db_1 | vacuuming database template1 ... ok
db_1 | copying template1 to template0 ... ok
db_1 | copying template1 to postgres ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | postgres -D /var/lib/postgresql/data
db_1 | or
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | ****************************************************
db_1 | WARNING: No password has been set for the database.
db_1 | This will allow anyone with access to the
db_1 | Postgres port to access your database. In
db_1 | Docker's default configuration, this is
db_1 | effectively any other container on the same
db_1 | system.
db_1 |
db_1 | Use "-e POSTGRES_PASSWORD=password" to set
db_1 | it in "docker run".
db_1 | ****************************************************
db_1 | waiting for server to start....LOG: database system was shut down at 2015-11-26 09:44:19 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | done
db_1 | server started
db_1 | ALTER ROLE
db_1 |
db_1 |
db_1 | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | LOG: received fast shutdown request
db_1 | LOG: aborting any active transactions
db_1 | LOG: autovacuum launcher shutting down
db_1 | waiting for server to shut down....LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | LOG: database system was shut down at 2015-11-26 09:44:22 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
web_1 | Performing system checks...
web_1 |
web_1 | System check identified no issues (0 silenced).
web_1 |
web_1 | You have unapplied migrations; your app may not work properly until they are applied.
web_1 | Run 'python manage.py migrate' to apply them.
web_1 | November 26, 2015 - 09:55:53
web_1 | Django version 1.8.7, using settings 'dockerized_django.settings'
web_1 | Starting development server at http://0.0.0.0:8000/
web_1 | Quit the server with CONTROL-C.
7 - Start! (http://localhost:8000)
> docker-compose run web python manage.py migrate
Starting dockerdjango_db_1...
Operations to perform:
Synchronize unmigrated apps: staticfiles, messages
Apply all migrations: admin, contenttypes, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying sessions.0001_initial... OK
8 - Wait! We didn't migrate! ;-)
https://nanobox.io/
http://subuser.org/
https://github.com/progrium/dokku
https://github.com/fmarco