VOL. 01 — A DESIGN WORKSHOP

Zhou et al.  |  ICLR 2024

Web

Arena

A Realistic Web Environment for Building Autonomous Agents

01 — Inspiration

New Web Agent Benchmark?

I.

II.

III.

Simplified Environment

Current environments over-simplified real-world situations. The action set is quite limited, and tasks were lack of complexity and diversity

Static Resources

The website contains only static resources, wont react to agents' actions.

Agent can only access cached states of the website.

Action Matching

Different trajectories may lead to same success state, but some previous researches match the action sequences.

{{#}} / {{total}}

Real-world website?

Content, UI, status changes, CAPTCHA, anti-robot. Not a stable testing environment.

Principles

Web

Arena

Realistics x Reproductive

01

02

03

Environment

Self-hosted websites with atomic action sets

Tasks and Benchmark

812 complex, creative tasks in natural languages

Evaluation

Mainly based on actual result, Accepting various trajectories

03 — WebArena Environment

Web applications:

• E-commerce: OneStopShop
• Social Forum: Reddit
• Collaborative Software Development: GitLab
• Content Management System: CMS

Toolbox: calculator, notebook, map

Knowledge Resources: offline Wikipedia, general info, website usage manual

{{#}} / {{total}}

FIG. 1 — MAIN ENTRANCE, AFTER

WebArena Environment

Reward Func \(=r(a_1^T, s_1^T)\)

04 & 05 — Observation, Action space

{{#}} / {{total}}

Observation Space

ScreenShot, Dom Tree, A11y Tree

Action Space

Select element by ID / Position

Action decision based on previous observations & actions

06 & 07 — Tasks Collections, Evaluation methods

Tasks

{{#}} / {{total}}

• Abstraction & High-level
• Creative
• Template Based
  • 241 templates, 821 tasks
• Unachievable Tasks

Evaluation

r_info: Information Seeking

r_prog: Site Navigation, Content & Config

• locator: DB, API, Selector
• Condition Checker

08 — Workflow

Baseline & stratagy

{{#}} / {{total}}

{{#}} / {{total}}