Metal³

Baremetal Host Provisioning for Kubernetes

metal3.io

What is Metal³

Baremetal Host Provisioning for Kubernetes
Kubernetes native API
An Infrastructure provider for the Cluster API (K8S SIG life-cycle)
Self-hosted
Self-managed
CNCF sandbox project

Metal³

Management
cluster

Cluster API

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure gcp

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

What's Cluster API

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure aws

Target

cluster

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure azure

Target

cluster

Target

cluster

What's Cluster API

Metal³

Cluster API: https://github.com/kubernetes-sigs/cluster-api

Management
cluster

Target

cluster

Cluster API

Target

cluster

Target

cluster

Target
cluster

What's Cluster API

Metal³

clusterctl init \
 --core cluster-api:v0.3.11 \
 --bootstrap kubeadm:v0.3.11 \
 --control-plane kubeadm:v0.3.11 \
 --infrastructure metal3

Metal³

Master

Worker

Metal³

Master

Machine

GCP

Machine

Baremetal

Operator

Metal3

Machine

AWS

Machine

BareMetalHost

Metal³ Stack

Metal³

management

storage

compute

network

Metal³ Stack

Metal³

management

storage

compute

network

Metal³ Stack

Metal³

Ironic documentation : https://docs.openstack.org/ironic/latest/

management

storage

compute

network

Ironic

Baremetal Operator

Metal³ Stack

Metal³

management

storage

compute

network

Ironic documentation : https://docs.openstack.org/ironic/latest/

Ironic

Baremetal Operator

Cluster-api- provider-metal3

Cluster API

Metal³

Custom Controllers and Objects

Metal³

Ironic

Metal³

Bare metal provisioning and management service developed under the OpenStack umbrella
Supports a variety of technologies and standards: IPMI, Redfish, (i)PXE, virtual media, UEFI
Wide vendor support: HPE, Dell, Fujitsu, Huawei, Lenovo
Provides RESTful API, supported in GopherCloud

Ironic overview

GopherCloud: https://github.com/gophercloud/gophercloud

Metal³

Hardware inspection, inventory collection
Disk erasure (NVMe, SATA secure erase, shredding) [*]
Firmware (BIOS/UEFI) settings [**]
Hardware [**] and software RAID
Firmware updates [*] [**]
BMC reset [*] [**]

* Not exposed in Metal3 yet

** Vendor-specific

Ironic extra features

Metal³

When a BareMetalHost is created:

Bare metal machine enrollment, verify BMC access
Boot the deployment agent on the machine
Inspection: collect hardware inventory
Cleaning:
- Erase partitioning table
- Optionally: build RAID
- Optionally: configure firmware settings
Ready for provisioning!

Ironic workflow: enrollment

Metal³

When a BareMetalHost is deployed on:

Fetch, cache and (optionally) convert the requested image
Start the deployment agent (if not already running)
The agent:
- Fetch and convert (if needed) the image via HTTP
- Calculate the target disk using RootDeviceHints
- Write to the target disk
- Configure UEFI boot record (if needed)
Set the boot device to disk permanently
Enable secure boot (if requested)

Ironic workflow: provisioning

Metal³

Baremetal Operator

Metal³

1. Host Baseboard Management Controller (BMC) credentials,

BMC address

2. Host MAC address

Prerequisites

Metal³

Baremetal

Operator

Metal³

Secret

Baremetal

Operator

Metal³

Secret

# Secret with BMC credentials
apiVersion: v1
kind: Secret
metadata:
  name: example-host-secret
type: Opaque
data:
  username: YWRtaW4=       # base64
  password: cGFzc3dvcmQ=   # base64

Baremetal

Operator

Metal³

Secret

Baremetal

Operator

# Secret with BMC credentials
apiVersion: v1
kind: Secret
metadata:
  name: example-host-secret
type: Opaque
data:
  username: YWRtaW4=       # base64
  password: cGFzc3dvcmQ=   # base64
  
---
# BareMetalHost CR
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: example-host
spec:
  online: true
  bootMACAddress: 00:34:61:e6:0d:81
  bootMode: legacy
  bmc:
    address: ipmi://192.168.111.1:6230
    credentialsName: example-host-secret

BareMetalHost

Metal³

reconcile

BareMetalHost

Baremetal

Operator

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  finalizers:
  - baremetalhost.metal3.io
  generation: 1
  labels:
    cluster.x-k8s.io/cluster-name: test1
  name: node-0
  namespace: metal3
  ownerReferences:
  - apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
    controller: true
    kind: Metal3Machine
    name: test1-controlplane-s6tdz
spec:
  bmc:
    address: ipmi://192.168.111.1:6230
    credentialsName: node-0-bmc-secret
  bootMACAddress: 00:8e:50:0e:e8:3a
  bootMode: legacy
  consumerRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
    kind: Metal3Machine
    name: test1-controlplane-s6tdz
    namespace: metal3
  image:
    checksum: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8.md5sum
    checksumType: md5
    format: qcow2
    url: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2
  metaData:
    name: test1-controlplane-s6tdz-metadata
    namespace: metal3
  networkData:
    name: test1-controlplane-s6tdz-networkdata
    namespace: metal3
  online: true
  userData:
    name: test1-wxcvt
    namespace: metal3
  rootDeviceHints:
    deviceName: /dev/sda
status:
  goodCredentials:
    credentials:
      name: node-0-bmc-secret
      namespace: metal3
    credentialsVersion: "4854"
  hardware:
    cpu:
      arch: x86_64
      clockMegahertz: 2494
      count: 4
      flags:
      - aes
      ...
      model: Intel Xeon E3-12xx v2 (Ivy Bridge)
    firmware:
      bios:
      ...
    hostname: node-0
    nics:
    - ip: 192.168.111.20
      mac: 00:8e:50:0e:e8:3c
      model: 0x1af4 0x0001
      name: enp2s0
      pxe: false
      speedGbps: 0
      vlanId: 0
    - ip: 172.22.0.43
      mac: 00:8e:50:0e:e8:3a
      model: 0x1af4 0x0001
      name: enp1s0
      pxe: true
      speedGbps: 0
      vlanId: 0
    ramMebibytes: 4096
    storage:
    - hctl: "0:0:0:0"
      model: QEMU HARDDISK
      name: /dev/sda
      rotational: true
      serialNumber: drive-scsi0-0-0-0
      sizeBytes: 53687091200
      vendor: QEMU
    systemVendor:
      manufacturer: QEMU
  lastUpdated: "2020-10-31T02:43:10Z"
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: "2020-10-31T02:03:33Z"
      start: "2020-10-31T01:58:50Z"
    provision:
      end: "2020-10-31T02:20:30Z"
      start: "2020-10-31T02:15:54Z"
    register:
      end: "2020-10-31T02:43:10Z"
      start: "2020-10-31T02:43:07Z"
  operationalStatus: OK
  poweredOn: true
  provisioning:
    ID: b84d7118-f5ca-4dea-a65a-8487c9f68d07
    bootMode: legacy
    image:
      checksum: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8.md5sum
      checksumType: md5
      format: qcow2
      url: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2
    rootDeviceHints:
      deviceName: /dev/sda
    state: provisioned
  triedCredentials:
    credentials:
      name: node-0-bmc-secret
      namespace: metal3
    credentialsVersion: "4854"

BareMetalHost

Metal³

Cluster-api-provider-metal3

Cluster-api

Cluster
Controller

BareMetal

Host

Metal3

Machine

Metal3

Cluster

Kubeadm

Config

Machine
Controller

Metal3
Cluster
Controller

Metal3
Machine
Controller

Cluster-api-

provider-metal3

Baremetal
Operator

Cluster API
Provider
Kubeadm

Object Refernce

Reconcile

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: Metal3Cluster
metadata:
  name: example_cluster
spec:
  controlPlaneEndpoint:
    host: 192.168.111.249
    port: 6443

Metal3Cluster

Metal³

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: Metal3Machine
metadata:
  annotations:
    metal3.io/BareMetalHost: metal3/node-0
  finalizers:
  - metal3machine.infrastructure.cluster.x-k8s.io
  name: test1-controlplane-s6tdz
  namespace: metal3
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1alpha3
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: test1-jntbq
spec:
  hostSelector: {}
  image:
    checksum: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8.md5sum
    checksumType: md5
    format: raw
    url: https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2
  providerID: metal3://fc5847cc-7cde-46f5-a4f4-99024439f6a4
status:
  addresses:
  - address: 192.168.111.20
    type: InternalIP
  - address: 172.22.0.66
    type: InternalIP
  - address: node-0
    type: Hostname
  - address: node-0
    type: InternalDNS
  ready: true

Metal3Machine

Metal³

Let's see a

demo

Let's see a

Metal³

Master

Worker

Target cluster

Metal³

bare metal servers

Master

Worker

Target cluster

Metal³

Minikube

Management cluster

Management cluster == Source cluster

Metal³

BMO

CAPM3

CAPI

BMO - Baremetal Operator

CAPM3 - Cluster-api-provider-metal3

CAPI - Cluster-api

Management cluster == Source cluster

Minikube

Management cluster

Metal³

BMO

CAPM3

CAPI

BMO - Baremetal Operator

CAPM3 - Cluster-api-provider-metal3

CAPI - Cluster-api

Libvirt Virtual Machines

Management cluster

Minikube

Metal³

Libvirt Virtual Machines

BareMetal

Host

Metal3

Machine

Metal3

Cluster

Metal3

Cluster

Management cluster

BMO

CAPM3

CAPI

Metal3

Machine

Minikube

BMO - Baremetal Operator

CAPM3 - Cluster-api-provider-metal3

CAPI - Cluster-api

Metal³

Master

Worker

Target cluster

Management cluster

Minikube

CAPI

CAPM3

BMO

Metal³

Interested to contribute...

Documentations
A new feature request
Bug report
Bug fixes
Reviews
Talks/presentations/blog posts
Questions/feedback

We welcome you very much!

How to Contribute: https://github.com/metal3-io/cluster-api-provider-metal3/blob/master/CONTRIBUTING.md

Metal³

Metal³ Community

Contributors: Red Hat, Ericsson, Mirantis, Dell EMC, Fujitsu, AT&T

#cluster-api-baremetal channel on K8S slack

https://groups.google.com/g/metal3-dev

Community meetings in Zoom. Every Wednesday, @13:00 UTC

Github: https://github.com/metal3-io

Meeting recordings & Demos: Metal³ YouTube channel

Mailing list:

Website: https://metal3.io

@metal3_io

KubeCon 2021: Metal³: Kubernetes-native bare metal host management

By fmuyassarov

KubeCon 2021: Metal³: Kubernetes-native bare metal host management

Metal³ (“metal kubed”) is an open-source bare metal host provisioning tool created to enable Kubernetes-native infrastructure management. With Metal³ bare metal hosts can be managed via custom resources through the Kubernetes API. The Metal³ project is also building integration with the Kubernetes cluster-api project, allowing Metal³ to be used as an infrastructure backend for Machine objects from the Cluster API. This presentation will introduce the project and its motivations and will provide an overview of what has been accomplished so far.

Metal³

metal3.io

Metal³

Custom Controllers and Objects

Ironic

Baremetal Operator

Cluster-api-provider-metal3

KubeCon 2021: Metal³: Kubernetes-native bare metal host management

More from fmuyassarov