• formal verification is hard
• counterexamples are often better at explaining possible problems
• we want to identify errors as early as possible, so that our systems can evolve rapidly
• still, we want to have strong guarantees

Introduction

As we have seen during the course,
PBT is convenient because:

• it's cheap and automatic
• programs are tested against specifications
• there is no need to maintain test suites thanks to random data generation

Introduction

When PBT exhibits a counterexample, it means we can be in one of the following situations:

• we have a bug in our program
• a checker has not been properly defined

In both cases we can fix the problem and continue our development.

Introduction

When no counterexamples are produced,
still there can be problems in our program.

• We may have bugs in PBT code too
• Our program might have not been stimulated enough to exhibit a wrong behaviour

Introduction

• Generators may not cover all the possible cases
• Checkers may have specified too strong preconditions or they can be wrong w.r.t the original specification
• There can even be the case that the PBT framework is bugged, although we generally trust it

Introduction

Model

• Data Structures

• Functions

Inductive color := Red | Black.

Inductive tree :=
| Leaf : tree
| Node : color → tree → nat → tree → tree.

Fixpoint insert (x: nat) (t: tree): tree := ...

Model

Declarative Definitions

Model

RedBlack Tree Properties

• The root is always black
• The leaves are empty and black
• For each node, the path to each possible leaf has the same number of black nodes
• Red nodes can only have black children

Inductive is_redblack : tree → color → nat → Prop :=
| IsRB_leaf: ∀ c, is_redblack Leaf c 0
| IsRB_r: ∀ n tl tr h,
      is_redblack tl Red h →
      is_redblack tr Red h →
      is_redblack (Node Red tl n tr) Black h
| IsRB_b: ∀ c n tl tr h,
      is_redblack tl Black h →
      is_redblack tr Black h →
      is_redblack (Node Black tl n tr) c (S h).

Specifications

Propositions

we expect that hold in our system

Definition insert_preserves_redblack :=
  ∀ x t h, is_redblack t Red h →
    ∃ h', is_redblack (insert x t) Red h'

Implementation

Executable Definitions

Decisional procedures that check
whether a property of the system holds

Fixpoint is_redblack_dec (t: tree) (c: color): bool :=
match t with
| Leaf ⇒ true
| Node c' tl _ tr ⇒
  match c' with
  | Black ⇒
    (black_height_dec tl == black_height_dec tr) &&
    is_redblack_dec tl Black &&
    is_redblack_dec tr Black
  | Red ⇒
    match c with
    | Black ⇒
      (black_height_dec tl == black_height_dec tr) &&
      is_redblack_dec tl Red &&
      is_redblack_dec tr Red
    | Red ⇒ false
    end
  end
end.

Equivalence Proof

between
Model and Implementation

Lemma is_redblack_exP:
    ∀ (t: tree) (c: color),
        reflect (∃ n, is_redblack t c n)
                (is_redblack_dec t c).

Generators

Procedural generation
of values

Program Fixpoint genRBTree_height (hc : nat*color) {wf wf_hc hc} : G tree :=
  match hc with
  | (0, Red) => returnGen Leaf
  | (0, Black) => oneOf [returnGen Leaf;
                    (do! n <- arbitrary; returnGen (Node Red Leaf n Leaf))]
  | (S h, Red) => liftGen4 Node (returnGen Black) (genRBTree_height (h, Black))
                                        arbitrary (genRBTree_height (h, Black))
  | (S h, Black) => do! c' <- genColor;
                    let h' := match c' with Red => S h | Black => h end in
                    liftGen4 Node (returnGen c') (genRBTree_height (h', c'))
                                       arbitrary (genRBTree_height (h', c')) end.

Definition genRBTree := bindGen arbitrary (fun h => genRBTree_height (h, Red)).

Generators

• Soundness: generate only values for which the property holds

• Completeness: generate all the values for which the property holds

• Correctness: soundness + completeness

Generators

Set as Logical Predicates

x ∈ P <-> P x

{ x | P x }

Definition Pred (A: Type): Type := A → Prop.

Definition set_eq {A} (s s': Pred A) := ∀ A: s A ←→ s' A.

Generators

Set of Outcomes Semantics

x ∈ G <-> Prob { x <- G } > 0

{ x | x <- G }

Generators

Generator correctness

{ x | x <- G } = { x | P x }

Lemma genRBTree_correct:
    genRBTree ←→ (fun t ⇒ ∃ h, is_redblack t Red h).

Checkers

Implementation of properties

using QuickChick's combinators

Section Checker.

  Context { Gen: Type → Type }
          { H: GenMonad Gen }.

  Definition insert_is_redblack_checker: Property Gen :=
    forAll arbitraryNat (fun n ⇒
    forAll genRBTree (fun t ⇒
      is_redblack_dec t Red ==>
        is_redblack_dec (insert n t) Red)).

End Checker.

Checkers

Correctness

Whether we are testing
the model's high level specification

Checkers

Equivalence Proof

By considering checkers
as generators of testing results

(* High Level Specification *)
Definition insert_preserves_redblack :=
  ∀ x t h, is_redblack t Red h →
    ∃ h', is_redblack (insert x t) Red h'

(* Checker Correctness *)
Lemma insert_is_redblack_checker_correct :
  semChecker insert_is_redblack_checker
  ↔ insert_preserves_redblack.