Homomorphic encryption and applications to deep learning

featuring Alice, Bob, and Eve

People want to share information...

...but not with everybody

Generals most of all!

Caesar cipher

We can use a mutual key (Symmetric encryption)

PK

But using two keys is safer (Asymmetric encryption)

PK

Often we do not want to just send data, but also process it

Local computing is not always viable...

...but cloud computing elicits privacy concerns

f(    )

Quick math (and greek) course: homomorphism

   ὁμός (homo's) = "same"    (watch the accent!)

+ μορφή (morphe') = "form", "shape"

Homomorphic Encryption (HE)

f(    )

\( enc(\cdot)\)

\( dec(\cdot)\)

\( f(\cdot)\)

HE types

• Fully Homomorphic Encryption (FHE)
  • Supports arbitrary number of operations (bootstrapping).
  • Very slow.
• Leveled Homomorphic Encryption (LHE)
  • Faster.
  • Must know depth/complexity of circuit in advance.
• Partial Homomorphic Encryption (PHE)
  • Faster.
  • More secure than FHE for equal runtime budget.
  • Only supports addition, multiplication (e.g., Paillier encryption).

That's cool but can I do deep learning with it?

"bored Yann Lecun"

https://twitter.com/boredyannlecun

Not all operations are supported

Cryptosystems support integers

Solution: Integer approximation of model parameters before encryption, decode back to floats after decryption.

Only inference is supported

• (Pre)Training on plaintext data.
• Training is unstable because of approximation errors.

We may want to keep the model private too!

GELU highlights

1. Paillier encryption

2. Split into linear and non-linear components and distribute computations to non-colluding parties.

3. No approximation of ReLUs.

4. Privacy-preserving backpropagation.

1. GELU-Net: A Globally Encrypted, Locally Unencrypted Deep Neural Network for Privacy-Preserved Learning, Zhang et al., IJCAI 2018.
2. Public-Key Cryptosystems Based on Composite Degree Residuosity Classes, Paillier, Eurocrypt 1999.

Globally Encrypted

\(enc(x_1)\)

\(enc(x_2)\)

\(enc(x_3)\)

\( x_1 \)

\( x_2 \)

\( x_3 \)

Locally Unencrypted

\(w*enc(x_1)\)

\( ReLU(dec(w*enc(x_3)) \)

\(w*enc(x_2)\)

\(w*enc(x_3)\)

\( ReLU(dec(w*enc(x_2)) \)

\( ReLU(dec(w*enc(x_1)) \)

Not as slow as it sounds

Computation time of activation (ms)

Computation time for LeNet on MNIST (8192 image batch)

Privacy-preserving training

Privacy-preserving training

Potential contributions

1. Better encryption schemes.

2. Improve performance for single (non-batched) inputs.

3. Exploit advances on binary neural networks (BNNs) [1].

4. Optimize privacy-preserving training.

1. TAPAS: Tricks to Accelerate (encrypted) Prediction as a Service, Sanval et. al., 2018.

Resources

Papers

1. CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy

2. CryptoDL: Deep Neural Networks over Encrypted Data

3. Learning to protect communications with adversarial neural cryptography

4. GELU-Net: A globally encrypted, locally unencrypted deep neural networks for privacy-preserved learning

5. 2D-PNN Privacy Preserving Deep Neural Networks Based on Homomorphic Cryptosystem

Code/libraries

• GELU-Net Code

• Paillier cryptosystem in Python

• OpenMined: private AI resources