Abdullah Fathi

www.fotia.com.my

Pautan Muat Turun

Muat Turun Slaid

Panduan DevOps

Recap: Agile

Penyelesaiannya adalah DevOps

Pasukan Pembangun

Pasukan Operasi

Agile + DevOps

Currently the most heard buzz word in IT industry

  • Relatively new concept
  • Has been gaining popularity

"DevOps"

Proses yang Sama: Deliver aplikasi kepada pengguna akhir

Sentiasa menjadi matlamat utama

Deliver aplikasi kepada pengguna

Application Release Process

Release to Market

Development

Operations

Same Process of: Deliver application to the end user

This is always the main goal!

Deliver application to end users

Typical Software Release Process

Idea

Requirements

Code it

Test it

Build & Package

Deploy it

  • Installing tools
  • Deploy application
  • Configure firewall

Configure the server

Typical Software Release Process

Idea

Requirements

Code it

Test it

Build & Package

Deploy it

  • Any problems with the application?
  • Are users experiencing any issues?
  • Can the application handle high user load?

Operate & Monitor

Typical Software Release Process

  • Add new features!
  • Fix bugs
  • Optimize performance

Initial Launch

Improvements

Idea

Requirements

Code it

Test it

Build & Package

Deploy it

Operate & Monitor

Make it accessible to user immediately

Software versioning

Code it

Test it

Build & Package

Version 1

Code it

Test it

Build & Package

Version 2

Code it

Test it

Build & Package

Version 3

Code it

Test it

Build & Package

Version 4

Replaced Framework

New Feature

1.0.0

1.1.0

1.1.1

1.1.2

1  .  5  .  2

Major

Minor

Patch

Bug fix

Bug fix

Delivery Process

Implement

Idea

Test

Build

Deploy

Observe

  • Making the process fast
  • Minimal Bugs

Quickly Delivering with High Quality Code

CI/CD is in the heart of DevOps

Test it

Build & Package

Deploy it

Push Code
Changes

Artifact
Repository

CI

CD

CD of CI/CD

Deploy to
PROD

  • We don't deploy directly to PRODUCTION environment
  • We can't be sure that the changes won't break anything
  • We deploy the application in stages
    • ​To test the application extensively on multiple levels

Continuous Deployment

Deploy to
PROD

Deploy to
DEV

Deploy to
STAGING

Unit Tests,
functional tests
Code analysis tests, lint tests

Load tests,
Regression tests

  • Every code change is built, tested and deployed to the different environments
  • Deployment to production happens automatically without explicit manual approval

Continuous Delivery

  • Automated test not 100% reliable
  • Not enough time or too difficult to test each and every aspect of the application
  • Not confident enough to release automatically to production

Continuous Delivery

Deploy to
PROD

Deploy to
DEV

Deploy to
STAGING

Unit Tests,
functional tests
Code analysis tests, lint tests

Load tests,
Regression tests

  • Pipeline is automated only until the non-production environment
  • Update to production happens only with a manual approval

CI of CI/CD

remote git
repository

Test it

Build & Package

Push Code
Changes

Continuous Integration

  • Developer commits code changes to remote git repository
  • Triggers the CI/CD Pipeline

Why continuously integrating the code changes is important?

  • Each developer works with their own local copy of app code
  • Changes need to be integrated ("merged") into central shared code
  • Handle conflict, if some code updated
  • Run automated tests: The bigger the code changes, the more likely that something in the application breaks

Rangka Kerja DevOps Sektor Awam - Muka Surat: 75

Infrastruktur Pipeline DevOps

Tools DevOps

Gitlab The DevOps Platform

  • Git: Version Control System for:
    • Tracking code changes
    • Tracking who made changes
    • Coding collaboration
  • Gitlab, Github, Bitbucket: Services that provide remote access to git repositories

Git Introduction

  • Projects generally have multiple developers working in parallel, Git is needed to ensure there are no code conflicts between the developers
  • The requirements in such projects change often. So a version control system allows developers to revert and go back to an older version of the code.
  • Sometimes several projects which are being run in parallel involve the same codebase. In such a case, the concept of branching in Git is very important.

Why Git?

Advantages

  • Good distributed model
  • Branching capabilities and merging are easy
  • Free and open-source
  • Good and faster network performance and superior disk utilization

Disadvantages

  • Poor GUI and usability
  • GIT needs multiple branches to support parallel developments used by the developers

Git Advantages vs Disadvantages

Git Installation

https://git-scm.com/

VSC

Git Commands

# Check Git Version
git --version

# Create a new empty git repository
git init

# Create git repository
git init <project-name>

# Configure initial branch name to
# use in all of your new repositories
git config --global init.defaultBranch main

# Rename the branch of current git repository to main
git branch -m main

# Verify folder is git repository
# Tells you the current status of git repository
git status

Git Commands

# Set author username
git config --global user.name "USERNAME"

# Verify configured user name
git config --global user.name

# Set email address
git config --global user.email "test@fotia.com.my"

# Verify configured email address
git config --global user.email

Add git username/email to identify the author

Review History of Repository

# Show a history of changes
git log
git log --all --oneline
git log --patch

Git Commands

# State changes for commit
git add .
# or add multiple selected files
git add 'file1' 'file2' 'file3' 

# Commit staged changes
git commit -m "message-here"

Commit

working directory

staging area

repository

git add

git commit

Git Commands

# Create branch
git branch <branch-name>
# or create branch and checkout to that branch
$ git checkout -b <branch-name>

# View Branches
git branch

# Switch branch
git checkout <branch-name-here>

Branch

Git Commands

Rebase, Resolve Conflict

# Rebase to sync with the master/main branch
git rebase master

# Resolve conflict when merge request
(on branch feature)
git checkout main
git pull origin main
git checkout <branch_feature>
git rebase main
(solve conlict)
git add .
git commit -m "solve conflict"
git rebase --continue
git push origin feature --force

# Resolve Conflict when merge request
git checkout main
git pull origin main
git checkout <branch_feature>
git merge main (conflict)
<solve conflict>
git add .
git commit -m "message"
git push origin <branch_feature>

# Abort merge conflict
git merge --abort

Git Commands

# Push file to stash
git stash push

# List stash files
git stash list

# Reapply change to working directory
# Does not remove this change from the stash
git stash apply

# Pop change of the stash
# Automatically remove from stash list
git stash pop

# Clear stash
git stash clear

Stash

Git
Version Control System

Gitlab
Source Code Management System

Git VS Gitlab

  • Open Source software development platform
  • Source Code Management System
  • CI/CD
  • Complete DevOps platform

Gitlab Overview

  • Host your application's code
  • Enables collaboration among a team of developers with Gitlab Flow
  • Built-in CI/CD functionality
  • Can integrate with other tools
  • Granular access controls
  • File locking, preventing conflicts
  • Many more other great features..

Why Gitlab?

User Profile -> preferences -> SSH Keys

ssh-keygen -t ed25519 -C "Gitlab Key Pair"

# test authenticate to gitlab account
ssh -T git@gitlab.com

SSH is a way to authenticate without exposing your username and password

SSH Keys

User Profile -> preferences -> Access Tokens

git clone https://<git-project-name-here>:<access-token-here>@<git-project-url-here>

Authenticate with Personal Access Token in place of your password

Access Token

  • Manage settings across multiple projects
  • Logically categorize multiple users or projects
  • Provide a cross-project view

Gitlab: Group

  • A container for a Git repository
  • Built in CI/CD
  • Issue Tracking
  • Collaboration Tools

Gitlab: Project

  • Users or Groups that have access to a project
  • Members are assigned to roles
  • Member roles include permissions to perform actions on projects or groups

Gitlab: Member

  • A request to merge one branch into another
  • Merge Requests provide a space to have a conversation with the team about the changes on a branch
  • It is the central place through which changes are reviewed and verified
  • Merge Request Role:
    • Assignee: A person who is directly responsible for creating or merging a merge request
    • Reviewer: A team member who may only be involved in one aspect of the merge request, such as a peer review

Gitlab: Merge Request

  • An issue is a way to track work related to a Gitlab project
  • Issues can be used to report bugs, track tasks, request new features, ask a question and more
  • Your development workflow should begin with an issue

Gitlab: Issue Tracking

Home Page

Profile

Menu -> Groups -> Create Group

Manage one or more related projects at the same time

Create Group

Groups

Menu -> Projects -> Create New Project

Files and code are saved in projects, and most features are in the scope of projects

Create New Blank Project

Initialize repo with readme file if there is no existing code base

Projects

Side Menu-> Issues -> Milestones

A way to track issue and merge requests created to achieve a broader goal in a certain period of time

Can be used as:

  1. Agile Sprint
  2. Releases

Milestones

Use issues to collaborate on ideas, solve problems, and plan work.

Purpose of Issue:

  • Discuss the implementation of an idea.
  • Track tasks and work status.
  • Accept feature proposals, questions, support requests, or bug reports.
  • Elaborate on code implementations.

Side Menu-> Issues

Issues

CSV Quick Action

Provide efficient email support to customers

Type of email for service desk:

  • Bug Reports
  • Feature Request
  • General Feedback

Side Menu-> Issues -> Service Desk

Custom Email:
Settings -> General -> Service Desk

Server Config

Service Desk

Project Documentation

Gitlab wiki content format:

  • Markdown
  • RDoc
  • AsciiDoc
  • Org

Side Menu-> Wiki

https://docs.gitlab.com/ee/user/markdown.html

Wiki

Store and share bits of code and text with other users

Two types of snippets:

  • Personal Snippets
  • Project Snippets
    (Depends on Project Visibility)

Side Menu-> Snippets

Snippets

CI/CD Platform
(Gitlab)

  • Integrate various tools to do the testing, building, etc.
  • Platform to build CI/CD Pipeline
  • Covering the complete software development and release cycle

One platform for all your DevOps capabilities

Gitlab CI/CD
VS
Jenkins

JENKINS: Currently the most well-known and widely used tool for building CI/CD pipelines

  • Still an industry leader
  • Powerful, Open source, big community
  • Flexible integrations through 1000s of community plugin
  • Old tool, which was not designed for the new container age
  • Need to install, configure or maintain additional plugins
  • Gitlab is a DevOps platform
  • Keeps up with the industry development
  • Many features built-in: Self-Monitoring, Container Registry, Docker CI Runner, ...
  • Allows keeping CI/CD & code management in the same place
  • Only CI/CD Tool
  • Self-hosting is the only option
  • All in one solution
  • Self-Hosted or SaaS (managed)
  • Using CI/CD without overhead of setting it up yourself

Git Branching Strategy

  • A software Development workflow within the context of Git
  • Describes how a development team will create, collaborate on, and merge branches of source code in a codebase
  • Enable concurrent development in the codebase

Branching Strategy

Common Git Branching Strategies

  • GitHub Flow
  • Git Flow
  • Gitlab Flow

Github Flow (Simplest Workflow)

Git Flow (Most Complex)

Gitlab Primary Branching Strategy

2 Variation:

  • Environment Branches
  • Release Branches

Gitlab Flow

Gitlab Flow (Environment Branches)

Gitlab Flow (Release Branches)

  1. Create Issue
  2. Create Production Branch
  3. Set/Add production branch as protected branch in setting
  4. Clone Project to your machine
  5. *** git pull
  6. Create and checkout feature branch
  7. Begin making modification to the codebase
  8. Add and Commit the changes
  9. Push the changes to Remote repository (Gitlab): git push -u origin <branch name>
  10. Create a merge request from feature branch to main branch
  11. Review the code changes in gitlab
  12. Approve Merge Request to Main Branch
  13. Create a merge request from main branch to production branch
  14. Create a git tag for versioning
  15. Source Codes assets is available in Deployment -> Releases
  16. Git pull main branch to sync remote repository to local repository
  17. Remove feature branch *if existed in local repo: git branch -d <feature branch name>
  18. git branch --all to check remote branch and local branch
  19. *if feature branch still exist in remote branch: git pull --prune
  20. Close the issue in gitlab
  • With the continuous method of software development, you continuously build, test, and deploy iterative code changes.
  • This iterative process helps reduce the chance that you develop new code based on buggy or failed previous versions.
  • Strive to have less human intervention or even no intervention at all, from the development of new code until its deployment.

Gitlab CI/CD: Introduction

The three primary approaches for the continuous method  :-

  1. Continuous Integration
  2. Continuous Delivery
  3. Continuous Deployment

 

Gitlab CI/CD: Introduction

  1. Continuous Integration
  • Consider an application that has its code stored in a Git repository in GitLab. Developers push code changes every day, multiple times a day. For every push to the repository, you can create a set of scripts to build and test your application automatically. These scripts help decrease the chances that you introduce errors in your application.
  • Each change submitted to an application, even to development branches, is built and tested automatically and continuously. These tests ensure the changes pass all tests, guidelines, and code compliance standards you established for your application.

Gitlab CI/CD: Introduction

 

2. Continuous Delivery

Not only is your application built and tested each time a code change is pushed to the codebase, the application is also deployed continuously. However, with continuous delivery, you trigger the deployments manually.

 

3. Continuous Deployment

Similar to Continuous Delivery. The difference is that instead of deploying your application manually, you set it to be deployed automatically. Human intervention is not required

Gitlab CI/CD: Introduction

Gitlab Architecture

Configuration File

  • .gitlab-ci.yml file
    • Instructions for Gitlab CI/CD jobs
    • Lives in the root of the repository
  • Gitlab Runner
    • Lightweight agent that runs CI/CD jobs
  • An Application that works with GitLab CI/CD to pick and execute CI/CD Jobs.
  • Open-source and written in Go Language.
  • You can add or remove runner into your GitLab architecture
  • GitLab offer several shared runners available to every project in a GitLab instance.
  • The GitLab runner can be installed on any platform or architecture that your own or manage.
    Linux, macOS, Windows, FreeBSD, Cloud Provider, Bare Metal,Your work station and Docker
  • The GitLab runner can test any programming language:
    .Net, Java, Python, C, PHP and others.

 

What is Gitlab Runner?

  • How do we build connection between gitlab runner server and gitlab instance server?

    • By register installed runner to gitlab server

 

Gitlab Runner

A runner can be:

Gitlab Runner

Shared VS Specific Runner

Tagged VS Untagged

Protected VS Unprotected

  1. Shell - Directly run commands as if writing them into terminal (bash or sh) or command prompt (cmd) or powershell.
  2. Docker - “main” machine scales up runners with *any* executor on demand Typical in cloud deployments.
  3. Docker - Execute inside of a docker image.
  4. Kubernetes - Runs as a pod in a K8s cluster. Can also feature auto scaling
  5. SSH - Similar to shell, but not as many features (bash only, no caching) Does allow you to SSH and execute commands on a machine you might not want to install runner on.
  6. Parallels - Parallels is a nice platform on top of VirtualBox.
  7. VirtualBox - Base VM for runner Main “main” creates a new VM for each needed runner

Runner Executor

Core Concept of Gitlab CI/CD

 

  • Pipelines are the top-level component of continuous integration, delivery and deployment.
  • Pipeline comprises of two things: Jobs and Stages
  • Job defines what to do. Ex: A job test the code or to compile the code.
  • Stages are collection of jobs. It define the order of the jobs.
  • Pipeline are executed automatically and require no intervention once created.
  • Gitlab also allows you to manually interact with a pipeline.
  • Four usual stages of a CI/CD pipeline are test/build/stage/prod.

CI/CD Pipeline

CI/CD Pipeline

 

  • To use CI/CD feature, there are two pre-requisite:
    1. Application code that host in Gitlab repository.
    2. .gitlab-ci.yml file that define your pipeline.

CI/CD Pipeline

  • Jobs define what we want to accomplish in our pipeline.
    • Executed by Runners
    • Executed in Stages
  • Stages define when and how to run jobs.
    • Stages that run tests after stages that compile the code.
  • Jobs in each stage are executed in parallel
    • If all jobs in a stage succeed, the pipeline moves on to the next stage.
    • if one job in a stage fails, the next stage is not (usually) executed

Jobs

Basic building block of pipeline

  • Jobs are the most fundamental building block of a .gitlab-ci.yml file

job

job

job

job

Pipeline

.gitlab-ci.yml

  • Basic building block of pipelines
  • Job defined what to do

Jobs

  • You can define arbitrary names for your job
  • Must contain at least the script clause
  • Script specify the command to execute
  • You can define as many jobs as you want

before_script

is used to define a command that should be  run before each job, including deploy jobs,  but after the restoration of any artifacts

after_script

is used to define the command that will be run  after each job, including failed ones.

Run before and after the script defined in  each job

  • Can update the image with the  latest version of components
  • They run within the job and can  interact with the job

before_script and after_script

GitLab CI/CD pipeline configuration reference

  • A job is defined as a list of keywords that  define the job’s behavior.
  • Configuration options for your Gitlab .gitlab-ci.yml file.
  • The keywords available for jobs are:
    https://docs.gitlab.com/ee/ci/yaml/

●image

●services

●script

●before_script &  after_script

●variables

●Environment

●cache

●artifacts

●rules

●tags

●when

Stages

Group jobs

  • You can group multiple jobs into stages that run in a defined order
  • Multiple jobs in the same stage are executed in parallel
  • Only if all jobs in a stage succeed, the pipeline moves on to the next stage
  • If any job in a stage fails, the next stage is not executed and the pipeline ends
  • Default Stages: Build,  Test, Review, & Deploy

  • User can define  custom stages & any  number of jobs per  stage

Stages seperate jobs into logical sections while Jobs perform the  actual tasks

Stages

Needs

  • Execute jobs in a certain order
  • If it fails, it should skip the other dependent jobs

Relationship between jobs

Allow Failure

Keyword to continue the pipeline execution and report a success even if the job fails

"allow_failure:true"

Script

Inline shell commands and executing shell script

  • Specify commands to execute
  • Eg: OS commands, like Linux commands

Script

Jobs and Scripts

Only/Except

  • By default, pipeline is triggered automatically for all branches
  • Pipeline configuration applies to all git branches
  • Only/Except: Job keywords to control when jobs are executed
  • only: Define when a job runs
  • except: Define when a job does not run

The rules syntax is an improved, more powerful solution for defining when jobs should run or not. Consider using rules instead of only/except to get the most out of your pipelines.

Only/Except

Basic Parameter

Predefined Variables

https://docs.gitlab.com/ee/ci/variables/predefined_variables.html

Predefined CI/CD variables are available in every Gitlab CI/CD pipeline

CI Tags

  • Tags are used to select a specific runner
    • CI tags are different from Git tags

Workflow Rules

"workflow"

A global keyword, which configures the whole pipelines behavior

Workflow Rules

  • Controls whether or not the whole pipeline is created
  • "rules" can also be used on the job level as replacement for only/except

"workflow:rules"

Rules

Using variables to Determine What Triggered the Pipeline

Common Rules Use Cases

Rules

When is a Job created in Pipeline?

Rules Syntax

Rules

When is a Job NOT created in Pipeline?

Manual Execution

Rules Examples 1

Rules Examples 2

Rules Examples 3

Rules Examples 4

Variables

Variables Processing Order

  • What is a template?
    • The way to share CI/CD capabilities  with other teams
    • The  way to consume CI/CD capabilities from other teams
    • The way that GitLab engineering provides capabilities via templates

  • Templates are 0% magical, they are  GitLab CI/CD YAML files

  • Templates are always executed into a CI/CD pipeline through an include statement in the project’s .gitlab-ci.yml

  • Template jobs are created  in your CI/CD pipeline based on their defined stage (and any applicable rules)

Template

Template

To take advantage of SAST capabilities from GitLab engineering, we will use include:template

Template

Share Pipeline Definitions

Share Pipeline Definitions

Share Pipeline Definitions

Extending Jobs

Example Using extends

Artifacts

  • Job artifacts are directories and files created by a job that you want to save after the job is finished; for example binaries that were built or reports generated by a test job.
  • These artifacts are uploaded to GitLab and you can access them in a variety of ways

Environment
Preparation

Cache and Artifacts Download

Running
Scripts

Cache and Artifacts Upload

Cleanup

Gitlab job execution phases

Artifact Download

Example Build Artifact

Artifact VS Cache

Image

Environments

  • Have this environment information in the pipeline
  • Have an overview of which version deployed to which environment
  • Link directly to the deployment endpoint

The environment keyword defines where the app is deployed and is defined by 3  parts.

When triggers jobs & stages manually (e.g. deploy to  production)

Environments

Testing Tools

Testing

There are multiple type of testing that we can execute on stage Test

  1. Unit Test
  2. Code Quality Test
  3. Browser Performance Test
  4. Accessibility Test
  5. Load Performance Test

Unit Test

  • Using your own language unit test.
  • There are four types of results:-
    1. Newly failed tests: Test cases which passed on base branch and failed on head branch
    2. Newly encountered errors: Test cases which passed on base branch and failed due to a test error on head branch
    3. Existing failures: Test cases which failed on base branch and failed on head branch
    4. Resolved failures: Test cases which failed on base branch and passed on head branch

●

●

View Gitlab Documentation

Code Quality Test

 

  • Gitlab uses plugins supported by Code Climate, which are free and open source. Code Quality does not require a Code Climate subscription.
  • To ensure your project’s code stays simple, readable, and easy to contribute to

View Gitlab Documentation

Static Application Security Test
(SAST)

 

Gitlab SAST use analyzer to check source code for any known vulnerabilities

View Gitlab Documentation

include:
    - template: Jobs/SAST.gitlab-ci.yml
    
 
#Static Application Security Test (SAST)
sast:
    stage: test
    artifacts:
        paths: [gl-sast-report.json]
        reports:
            sast: gl-sast-report.json

Browser Performance Test

  • GitLab uses Sitespeed.io, a free and open source tool, for measuring the rendering performance of web sites.
  • The Sitespeed plugin that GitLab built outputs the performance score for each page analyzed in a file called browser-performance.json this data can be shown on Merge Requests.

View Gitlab Documentation

Accessability Test

  • GitLab integrates Pa11y into a CI job template.
  • The a11y job analyzes a defined set of web pages and reports accessibility violations, warnings, and notices in a file named accessibility.

View Gitlab Documentation

Load Performance Test

  • GitLab uses k6, a free and open source tool, for measuring the system performance of applications under load.
  • To perform various types of load tests against application endpoints such as APIs, Web Controllers, and so on.

View Gitlab Documentation

Monitoring Tools

Application Performance Monitoring

Built on top of Elastic Stack to monitor application performance and services

CI/CD for Microservices

Monolithic Architecture

  • All components are part of a single unit
  • Everything is developed, deployed and scaled as 1 unit
  • App must be written with 1 tech stack
  • 1 single artifact, so you must redeploy the entire application on each update

Challenges of Monolithic Architecture

  • Application is too large and complex
  • You can only scale the entire app, instead of a specific service
  • Difficulty if services need different dependency versions
  • Release process take longer
  • On every change, the entire application needs to be tested
  • Entire application needs to be build and deployed
  • Bug in any module can potentially bring down the entire application

Microservices

  • Split your application into smaller, independent services
    • Split based on business functionalities
  • Services can be built and deployed separately
  • Each microservice has its own version

Communication between Microservices

  • Each services has its own API
  • By calling the respective API endpoint, they can talk to each other
  • Each microservice team can choose its own tech stack
  • Each team can develop the service independently, without affecting others
  • Added complexity just by the fact that a microservices application is distributed system
    • Configure the communication between services
    • More difficult to monitor with multiple instances of each service distributed across servers

Challenges of Microservices Architecture

Monorepo

MS

MS

MS

MS

MS

MS

Microservices

Monorepo

1 Git repository that contains many projects

Monorepo

  • 1 Codebase with folders
    • A directory for each service/project
  • Makes the code management and development easier
  • Clone and work only with 1 repo
  • Changes can be tracked together, tested together and released together
  • Share code and configurations

User

Thread

Post

Monorepo

Monorepo Challenges

  • Tight coupling of projects
  • Easier to break this criterion and develop more tightly coupled code
  • Big source code, means git interactions becomes slow
  • Additional logic necessary to make sure only service is build and deployed which had changes

Polyrepo

  • 1 repository for each service
    • Code is completely isolated
    • Clone and work on them separately
  • Gitlab Groups:
    • Projects to configure connected project together (Groups)
    • Helps to keep an overview
    • Create shared secrets, CI/CD variables, Runners
  • CI/CD: Own pipeline for each repository

User

Polyrepo

Thread

Post

Polyrepo Challenges

  • Switching between projects tedious
  • Searching, testing and debugging is more difficult
  • Sharing resources more difficult

User

Thread

Post

Monorepo

User

Polyrepo

Thread

Post

For smaller microservice application

  • Separate teams / better access control
  • Have complete isolation
  • Smaller code bases
  • Own pipelines

Polyrepo VS Monorepo

Download Presentation Slide

Download Slide (PDF)

Download Presentation Slide

Download Slide (PDF)

Gitlab Slide (PDF)

THANK YOU

Gitlab CI/CD Training

By Abdullah Fathi

Gitlab CI/CD Training

  • 688

More from Abdullah Fathi