Convex Sets and Functions

 

Shen Shen

Feb 23, 2021

Convex Sets

Def:   A set \(\Omega\) is a convex set if

\(\lambda x+(1-\lambda) y \in \Omega, \) \(\forall x,y \in \Omega, \forall \lambda\in[0,1]\)

convex combination of \(x\) and \(y\)

convex combination coefficient

Simple examples

Convex Sets

Simple examples

Non-convex Sets

because...

Common Convex Sets from Optimization

- Hyperplanes: \(\left\{x \mid a^{T} x=b\right\}\left(a \in \mathbb{R}^{n}, b \in \mathbb{R}, a \neq 0\right)\)

- Halfspaces: \(\left\{x \mid a^{T} x \leq b\right\} \left(a \in \mathbb{R}^{n}, b \in \mathbb{R}, a \neq 0\right)\)

\(n=3: 3x_1+4x_2+5x_3=1\)

\(n=2: 3x_1+4x_2+\leq 2\)

Common Convex Sets from Optimization

- Euclidean balls: \(\left\{x \mid \left||x-x_{c}\right|| \leq r\right\} \) \( (x_{c} \in \mathbb{R}^{n}, r \in \mathbb{R},\|.\|\) 2-norm)

- Ellipsoids: \(\left\{x \mid\left(x-x_{c}\right)^{T} P\left(x-x_{c}\right) \leq r\right\}\left(x_{c} \in \mathbb{R}^{n}, r \in \mathbb{R}, P \succ 0\right)\)

\(n=2\)

\(n=3\)

Common Convex Sets from Optimization

- The set of PSD matrices

- The set of co-positive matrices (HW)

A common recipe

1. Pick two arbitrary members, say \(x\) and \(y\), from \(\Omega\)

2. Write out the convex combination \(c\)

3. Make use of that \(x\) and \(y\) are from \(\Omega\), and \(\lambda \in [0,1]\)

4. Claim the \(c\) also belongs in \(\Omega\)

Proof of the PSD claim:

Pick \(A\succeq 0, B\succeq 0\)

N.t.s \(\lambda A+(1-\lambda) B \succeq 0\)

i.e., \(x^T(\lambda A+(1-\lambda) B)x \geq 0 \forall x\)

\(=\lambda x^{T} A x+(1-\lambda) x^{T} B x\)

\(x^T(\lambda A+(1-\lambda) B)x\)

\(\geq 0\)

\(\geq 0\)

\(\geq 0\)

Intersections of convex sets

Useful fact: \(\Omega_{1}\) convex, \(\Omega_{2}\) convex \(\Rightarrow \Omega_{1} \cap \Omega_{2}\) convex.

 

- Proof in last lecture.

- The union of convex sets may not be convex

- Polyhedrons: \(\{x \mid A x \leq b\}\) are convex sets

e.g. m =4 (4 halfspaces), n=2 (2d)

\(A=\left[\begin{array}{rr}-1 & 0 \\ 0 & -1 \\ 0 & 1 \\ 1 & 1\end{array}\right], b=\left[\begin{array}{l}0 \\ 0 \\ 1 \\ 3\end{array}\right]\)

Convex Functions

Def:   A function \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}\) is convex if its domain is a convex set and

\(f(\lambda x+(1-\lambda) y) \leq \lambda f(x)+(1-\lambda) f(y)\) \(\forall x,y \in \text{domain}(f), \forall \lambda\in[0,1]\)

Simple examples

Convex functions

Non-convex functions

\(f\) is called a concave function if \(-f\) is convex

Common convex functions from optimization

- All affine functions \(f(x)=a^{T} x+b \quad\) (for any \(\left.a \in \mathbb{R}^{n}, b \in \mathbb{R}\right)\)

- Some quadratic functions (details later)

Common convex functions from optimization

- All norms

Recall norm is a function \(f\) that satisfies:

a. \(f(\alpha x)=|\alpha| f(x), \forall \alpha \in \mathbb{R}\)
b. \(f(x+y) \leq f(x)+f(y)\)
c. \(f(x) \geq 0, \forall x, f(x)=0 \Rightarrow x=0\)

\(f(\lambda x+(1-\lambda) y) \)      

\(= \lambda f(x)+(1-\lambda) f(y)\)

b.c. (b)

b.c. (a) and \(\lambda \geq 0\)

Proof (convexity of any norm):

\(\leqslant  f(\lambda x)+f((1-\lambda) y)\)

Connecting Convex Sets and Convex Functions

Theorem: If a function \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}\) is convex, then all its sublevelsets are convex sets.

Remarks:

- Recall the sub-levelsets definition:

   The \(\alpha\) -sublevel set of a function \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}\) is the set \(S_{\alpha}=\{x \in \operatorname{domain}(f) \mid f(x) \leq \alpha\}\)

Connecting Convex Sets and Convex Functions

Theorem: If a function \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}\) is convex, then all its sublevelsets are convex sets.

Remarks:

- Recall the sub-levelsets definition:

   The \(\alpha\) -sublevel set of a function \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}\) is the set \(S_{\alpha}=\{x \in \operatorname{domain}(f) \mid f(x) \leq \alpha\}\)

 

 

 

 

 

 

 

- Instead, a function whose sub-levelsets are convex sets is called quasiconvex.

- Converse is not true:

First and second order characterizations of convex functions

Theorem:  Suppose\(f: \mathbb{R}^{n} \rightarrow \mathbb{R}\) is twice differentiable over its domain. Then, the following are equivalent:

(i) \(f\) is convex.

(ii) \(f(y) \geq f(x)+\nabla f^{T}(x)(y-x), \quad \forall x, y \in \operatorname{dom}(f)\)
 

(iii) \(\nabla^{2} f(x) \succeq 0, \quad \forall x \in \operatorname{dom}(f)\) (i.e., the Hessian is psd \(\left.\forall x \in \operatorname{dom}(f)\right)\).

Intuition for (ii) \(f(y) \geq f(x)+\nabla f^{T}(x)(y-x), \quad \forall x, y \in \operatorname{dom}(f)\)
 

Intuition for (iii) \(\nabla^{2} f(x) \succeq 0, \quad \forall x \in \operatorname{dom}(f)\) (i.e., the Hessian is psd \(\left.\forall x \in \operatorname{dom}(f)\right)\).
 

Function has nonnegative curvature everywhere: "it curves up"

Recall we said, some quadratic functions are convex, this second order characterization makes identifying the "some" really easy

In one dimension: \(f^{\prime \prime}(x) \geq 0, \forall x \in \operatorname{dom}(f)\)

Recall theorem says \(f\) convex \(\Leftrightarrow \nabla^{2} f(x) \succeq 0, \quad \forall x \in \operatorname{dom}(f)\)

Consider a quadratic function \(f(x)=x^{T} A x+b x+c \quad\) (A symmetric)

when is \(f\) convex?

we know \(\nabla^{2} f(x)=2 A\)

Text

\(f\) convex \(\Leftrightarrow A \succeq 0\) (independent of \(b, c)\)

Convexity of quadratic functions

Convexity-preserving Rules

(for convex functions)

  • Think of as shortcuts or stepping-stones
  • Rules take in some convex functions, perform certain operations, and spit out a convex function
  • Makes identifying convex functions a lot easier
  • We'll present 4 such rules

Rule 1: Nonnegative weighted sum

If \(f_{1}, \ldots, f_{n}\) are convex functions and \(\omega_{1}, \ldots, \omega_{n} \geq 0,\) then
\(f(x)=\omega_{1} f_{1}(x)+\cdots+\omega_{n} f_{n}(x)\) is also convex.

Rule 2: Point-wise maximum

If \(f_{1}, \ldots, f_{m}\) are convex functions then their pointwise maximum
\(f(x)=\max \left\{f_{1}(x), f_{2}(x), \ldots f_{m}(x)\right\}\)
with \(\operatorname{dom}(f)=\operatorname{dom}\left(f_{1}\right) \cap \operatorname{dom}\left(f_{2}\right) \cap \cdots \cap \operatorname{dom}\left(f_{m}\right),\) is also convex.

Rule 3: Composition with an affine mapping

Suppose \(f: \mathbb{R}^{\mathrm{n}} \rightarrow \mathbb{R}, \mathrm{A} \in \mathbb{R}^{\mathrm{n} \times \mathrm{m}}\) and \(b \in \mathbb{R}^{\mathrm{n}}\). Define \(g: \mathbb{R}^{\mathrm{m}} \rightarrow \mathbb{R}\) by

\(g(x)=f(A x+b)\)

with \(\operatorname{dom}(g)=\{x \mid A x+b \in \operatorname{dom}(f)\}\). Then, if \(f\) is convex, so is \(g\); if \(f\) is concave so is \(g\).

Rule 4: Restriction to a line

Let \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}\) be a convex function and fix some \(a, b, \in \mathbb{R}^{n} .\) Then, the function \(g: \mathbb{R} \rightarrow \mathbb{R}\) given by \(g(x)=f(ax+b)\) is convex.