Metal³

Baremetal Host Provisioning for Kubernetes

metal3.io

What is Metal³

  • Baremetal Host Provisioning for Kubernetes
  • Kubernetes native API
  • An Infrastructure provider for the Cluster API (K8S SIG life-cycle)
  • Self-hosted
  • Self-managed
  • CNCF sandbox project

     

Metal³

Management
cluster

Cluster API

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure gcp

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

What's Cluster API

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure aws

Target

cluster

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure azure

Target

cluster

Target

cluster

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

Target

cluster

Target

cluster

Target
cluster

 

What's Cluster API

Metal³

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure metal3

Metal³

Master

Worker

Worker

Metal³

Master

Machine

GCP

Machine

Baremetal

Operator

Metal3

Machine

AWS

Machine

BareMetalHost

Metal³ Stack

Metal³

management

storage

compute

network

Metal³ Stack

Metal³

management

storage

compute

network

Metal³ Stack

Metal³

Ironic documentation : https://docs.openstack.org/ironic/latest/

management

storage

compute

network

Ironic

Baremetal Operator

+

Metal³ Stack

Metal³

management

storage

compute

network

Ironic documentation : https://docs.openstack.org/ironic/latest/

Ironic

Baremetal Operator

+

Cluster-api- provider-metal3

Cluster API

Metal³

Metal³

Custom Controllers and Objects

Metal³

Ironic

Metal³

  • Bare metal provisioning and management service developed under the OpenStack umbrella
  • Supports a variety of technologies and standards: IPMI, Redfish, (i)PXE, virtual media, UEFI
  • Wide vendor support: HPE, Dell, Fujitsu, Huawei, Lenovo
  • Provides RESTful API, supported in GopherCloud

Ironic overview

GopherCloud: https://github.com/gophercloud/gophercloud

Metal³

  • Hardware inspection, inventory collection
  • Disk erasure (NVMe, SATA secure erase, shredding) [*]
  • Firmware (BIOS/UEFI) settings [**]
  • Hardware [**] and software RAID
  • Firmware updates [*] [**]
  • BMC reset [*] [**]

* Not exposed in Metal3 yet

** Vendor-specific

Ironic extra features

Metal³

When a BareMetalHost is created:

  • Bare metal machine enrollment, verify BMC access
  • Boot the deployment agent on the machine
  • Inspection: collect hardware inventory
  • Cleaning:
    • Erase partitioning table
    • Optionally: build RAID
    • Optionally: configure firmware settings
  • Ready for provisioning!

Ironic workflow: enrollment

Metal³

When a BareMetalHost is deployed on:

  • Fetch, cache and (optionally) convert the requested image
  • Start the deployment agent (if not already running)
  • The agent:
    • Fetch and convert (if needed) the image via HTTP
    • Calculate the target disk using RootDeviceHints
    • Write to the target disk
    • Configure UEFI boot record (if needed)
  • Set the boot device to disk permanently
  • Enable secure boot (if requested)

Ironic workflow: provisioning

Metal³

Baremetal Operator

Metal³

1. Host Baseboard Management Controller (BMC) credentials,

    BMC address

2. Host MAC address

 

Prerequisites

Metal³

Baremetal

Operator

Metal³

Secret

Baremetal

Operator

Metal³

Secret

# Secret with BMC credentials
apiVersion: v1
kind: Secret
metadata:
  name: example-host-secret
type: Opaque
data:
  username: YWRtaW4=       # base64
  password: cGFzc3dvcmQ=   # base64

Baremetal

Operator

Metal³

Secret

Baremetal

Operator

# Secret with BMC credentials
apiVersion: v1
kind: Secret
metadata:
  name: example-host-secret
type: Opaque
data:
  username: YWRtaW4=       # base64
  password: cGFzc3dvcmQ=   # base64
  
---
# BareMetalHost CR
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: example-host
spec:
  online: true
  bootMACAddress: 00:34:61:e6:0d:81
  bootMode: legacy
  bmc:
    address: ipmi://192.168.111.1:6230
    credentialsName: example-host-secret

BareMetalHost

Metal³

reconcile

BareMetalHost

Baremetal

Operator

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  finalizers:
  - baremetalhost.metal3.io
  generation: 1
  labels:
    cluster.x-k8s.io/cluster-name: test1
  name: node-0
  namespace: metal3
  ownerReferences:
  - apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
    controller: true
    kind: Metal3Machine
    name: test1-controlplane-s6tdz
spec:
  bmc:
    address: ipmi://192.168.111.1:6230
    credentialsName: node-0-bmc-secret
  bootMACAddress: 00:8e:50:0e:e8:3a
  bootMode: legacy
  consumerRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
    kind: Metal3Machine
    name: test1-controlplane-s6tdz
    namespace: metal3
  image:
    checksum: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8.md5sum
    checksumType: md5
    format: qcow2
    url: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2
  metaData:
    name: test1-controlplane-s6tdz-metadata
    namespace: metal3
  networkData:
    name: test1-controlplane-s6tdz-networkdata
    namespace: metal3
  online: true
  userData:
    name: test1-wxcvt
    namespace: metal3
  rootDeviceHints:
    deviceName: /dev/sda
status:
  goodCredentials:
    credentials:
      name: node-0-bmc-secret
      namespace: metal3
    credentialsVersion: "4854"
  hardware:
    cpu:
      arch: x86_64
      clockMegahertz: 2494
      count: 4
      flags:
      - aes
      ...
      model: Intel Xeon E3-12xx v2 (Ivy Bridge)
    firmware:
      bios:
      ...
    hostname: node-0
    nics:
    - ip: 192.168.111.20
      mac: 00:8e:50:0e:e8:3c
      model: 0x1af4 0x0001
      name: enp2s0
      pxe: false
      speedGbps: 0
      vlanId: 0
    - ip: 172.22.0.43
      mac: 00:8e:50:0e:e8:3a
      model: 0x1af4 0x0001
      name: enp1s0
      pxe: true
      speedGbps: 0
      vlanId: 0
    ramMebibytes: 4096
    storage:
    - hctl: "0:0:0:0"
      model: QEMU HARDDISK
      name: /dev/sda
      rotational: true
      serialNumber: drive-scsi0-0-0-0
      sizeBytes: 53687091200
      vendor: QEMU
    systemVendor:
      manufacturer: QEMU
  lastUpdated: "2020-10-31T02:43:10Z"
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: "2020-10-31T02:03:33Z"
      start: "2020-10-31T01:58:50Z"
    provision:
      end: "2020-10-31T02:20:30Z"
      start: "2020-10-31T02:15:54Z"
    register:
      end: "2020-10-31T02:43:10Z"
      start: "2020-10-31T02:43:07Z"
  operationalStatus: OK
  poweredOn: true
  provisioning:
    ID: b84d7118-f5ca-4dea-a65a-8487c9f68d07
    bootMode: legacy
    image:
      checksum: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8.md5sum
      checksumType: md5
      format: qcow2
      url: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2
    rootDeviceHints:
      deviceName: /dev/sda
    state: provisioned
  triedCredentials:
    credentials:
      name: node-0-bmc-secret
      namespace: metal3
    credentialsVersion: "4854"

BareMetalHost

Metal³

Metal³

Cluster-api-provider-metal3

Cluster-api

Cluster
Controller

BareMetal

Host
 

Metal3

Machine

Machine

Metal3

Cluster

Cluster

Kubeadm

Config

Machine
Controller

Metal3
Cluster
Controller

Metal3
Machine
Controller

Cluster-api-

provider-metal3

Baremetal
Operator

Cluster API
Provider
Kubeadm

Object Refernce

Reconcile

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: Metal3Cluster
metadata:
  name: example_cluster
spec:
  controlPlaneEndpoint:
    host: 192.168.111.249
    port: 6443

Metal3Cluster

Metal³

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: Metal3Machine
metadata:
  annotations:
    metal3.io/BareMetalHost: metal3/node-0
  finalizers:
  - metal3machine.infrastructure.cluster.x-k8s.io
  name: test1-controlplane-s6tdz
  namespace: metal3
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1alpha3
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: test1-jntbq
spec:
  hostSelector: {}
  image:
    checksum: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8.md5sum
    checksumType: md5
    format: raw
    url: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2
  providerID: metal3://fc5847cc-7cde-46f5-a4f4-99024439f6a4
status:
  addresses:
  - address: 192.168.111.20
    type: InternalIP
  - address: 172.22.0.66
    type: InternalIP
  - address: node-0
    type: Hostname
  - address: node-0
    type: InternalDNS
  ready: true

Metal3Machine

Metal³

Let's see a

demo

Let's see a

Metal³

Metal³

Master

Worker

Worker

Target cluster

Metal³

bare metal servers

Master

Worker

Worker

Target cluster

Metal³

Minikube

#1

Management cluster

Management cluster == Source cluster

Metal³

#1

BMO

CAPM3

CAPI

BMO -    Baremetal Operator

CAPM3 -   Cluster-api-provider-metal3

CAPI -    Cluster-api

Management cluster == Source cluster

Minikube

Management cluster

Metal³

BMO

CAPM3

CAPI

BMO -    Baremetal Operator

CAPM3 -   Cluster-api-provider-metal3

CAPI -    Cluster-api

Libvirt Virtual Machines

Management cluster

Minikube

#2

Metal³

Libvirt Virtual Machines

BareMetal

Host

Metal3

Machine

Metal3

Cluster

Metal3

Cluster

Cluster

Management cluster

BMO

CAPM3

CAPI

Metal3

Machine

Minikube

#3

BMO -    Baremetal Operator

CAPM3 -   Cluster-api-provider-metal3

CAPI -    Cluster-api

Metal³

#4

Master

Worker

Worker

Target cluster

Management cluster

Minikube

CAPI

CAPM3

BMO

Metal³

Interested to contribute...

  • Documentations
  • A new feature request
  • Bug report
  • Bug fixes
  • Reviews
  • Talks/presentations/blog posts
  • Questions/feedback

We welcome you very much!

How to Contribute:  https://github.com/metal3-io/cluster-api-provider-metal3/blob/master/CONTRIBUTING.md

Metal³

Metal³ Community 

Contributors:  Red Hat, Ericsson, Mirantis, Dell EMC, Fujitsu, AT&T

#cluster-api-baremetal channel on K8S slack

 https://groups.google.com/g/metal3-dev

Community meetings in Zoom. Every Wednesday, @13:00 UTC

Github: https://github.com/metal3-io

Meeting recordings & Demos: Metal³ YouTube channel

Mailing list:

Website: https://metal3.io

@metal3_io

Made with Slides.com