$TP = \sum^T_{t=1}\ell_t\hat{\ell}_t$,

$FP=\sum^T_{t=1}(1-\ell_t)\hat{\ell}_t$,

$FN=\sum^T_{t=1}\ell_t(1-\hat{\ell}_t)$.

$\ell_t$: true label for pair $t$.

$\hat{\ell}_t$: estimated label for pair $t$.

$TP = \sum^T_{t=1}\ell_t\hat{\ell}_t$,

$FP=\sum^T_{t=1}(1-\ell_t)\hat{\ell}_t$,

$FN=\sum^T_{t=1}\ell_t(1-\hat{\ell}_t)$.

$\ell_t$: true label for pair $t$.

$\hat{\ell}_t$: estimated label for pair $t$.

$\alpha = 0$: recall,

$\alpha = 1$: precision.

• Two data source: $\mathcal{D}_1$ and $\mathcal{D}_2$,
• Candidate pairs: $\mathcal{Z} = \mathcal{D}_1 \times \mathcal{D}_2$,
• A similarity function: $s: \mathcal{Z} \rightarrow \mathbb{R}$,
• Randomized Oracle: $Oracle(z): \mathcal{Z} \rightarrow \{0,1\}$.
• An estimation $\{\hat{\ell}_1, \hat{\ell}_2,...\}$.

• Approximate $\mathit{F}_{\alpha,T}$
• Minimize # Oracle query.

• Ground truth $\{\ell_1,\ell_2,...\}$.
• Enough money.

• Consistency: $\hat{F}_\alpha \to F_\alpha$, where $F_\alpha = \lim_{T \to \infty} F_{\alpha,T}$
• Minimal variance: The variance of estimation should be minimized

With the purpose of getting $F_\alpha$, sampling pairs and asking Oracle to estimate is a good choice.

Since the original distribution is imbalanced, uniformly sampling doesn't work here because of how we calculate $F_\alpha$ (Why?).

Since the original distribution is imbalanced, uniformly sampling doesn't work here because of how we calculate $F_\alpha$ (Why?).

Instead of sampling directly on $p(x)$ to estimate $\theta = E[f(X)]$,

i.e. $\hat{\theta} = \frac{1}{T}\sum^T_{i=1}f(x_i)$,

we draw samples from $q(x)$.

Interestingly, we can still do estimation by $\hat{\theta}^{IS} = \frac{1}{T}\sum^T_{i=1}\frac{p(x_i)}{q(x_i)}f(x_i)$.

Rewrite $\textit{F}_{\alpha,T} = \frac{TP}{\alpha(TP+FP) + (1-\alpha)(TP+FN)}$

to $\textit{F}_{\alpha}^{AIS} = \frac{\sum^T_{t=1} w_t \ell_t \hat{\ell}_t}{\alpha \sum^T_{t=1}w_t\hat{\ell}_t + (1-\alpha)\sum^T_{t=1}w_t \ell_t}$

Rewrite $\textit{F}_{\alpha,T} = \frac{TP}{\alpha(TP+FP) + (1-\alpha)(TP+FN)}$

to $\textit{F}_{\alpha}^{AIS} = \frac{\sum^T_{t=1} w_t \ell_t \hat{\ell}_t}{\alpha \sum^T_{t=1}w_t\hat{\ell}_t + (1-\alpha)\sum^T_{t=1}w_t \ell_t}$

Find $q(z_t)$, how?

$q^* \in \arg \min_q Var(\hat{F}_\alpha^{AIS}[q])$

Previous work: $q^*(z) \propto p(z) \cdot A(F_\alpha,p_{Oracle}(1|z_t))$

$w_t = \frac{p(z_t)}{q(z_t)}$

Note: $A(\cdot)$ here stands for ellipsis

$q^* \in \arg \min_q Var(\hat{F}_\alpha^{AIS}[q])$

$w_t = \frac{p(z_t)}{q(z_t)}$

Note: $A(\cdot)$ here stands for ellipsis

As long as we got $F_\alpha$ and $p_{Oracle}(1|z_t)$, problem solved.

Previous work: $q^*(z) \propto p(z) \cdot A(F_\alpha,p_{Oracle}(1|z_t))$

$q(z) = \epsilon \cdot p(z) + (1-\epsilon) \cdot q^*(z)$.

$F_\alpha$ and $p(1|z)$ - unknown!

Approximate them iteratively: For each step $t+1$, use $F_\alpha$ and $p(1|z)$ in step $t$.

2. $F_\alpha$ - Simple

Intuitively, we can just use $\hat{F}^{AIS}_\alpha$ instead of $F_\alpha$.

3. $p(1|z)$ - Big problem

There's no way we can do to get the distribution of the oracle without asking them for every $z$. 

3. $p(1|z)$ - Big problem

There's no way we can do to get the distribution of the oracle without asking them for every $z$. 

Similarity func $s: \mathcal{Z} \to \mathbb{R}$ kicks in.

For each $z$ in $P_k$, we can use $p(1|P_k)$ instead of $p(1|z)$ now.

We treat each $p(1|P_k) \sim Bernoulli(\pi_k)$, so $\pi_k \sim Beta(\alpha, \beta)$.

How to update $\alpha$ and $\beta$ iteratively? Easy.

if $\ell_t = 1$, $\alpha += 1$

if $\ell_t = 0$, $\beta += 1$

Now we get $F_\alpha$ and $p(1|z)$