Ply: Visual Regression Pruning for Web Design Source Inspection

Sarah Lim

Northwestern University

Novice web developers take inspiration from professional examples

Novices think in terms of visual output

div {
    padding: 10px;
    color: red;
}
<div>
    <p>Hi!</p>
</div>
<p>Goodbye</p>

Novices

Experts

In theory, any website can be an example

Inspection tools impose high cognitive load

Browser-calculated specificity != visual significance

Ineffective Properties

{

How might we quantify and leverage visual significance to help novice developers locate relevant source code more quickly?

Quantify change through visual regression

  • Regression testing: "Software still performs correctly after some change to the underlying codebase"
  • Software == visual output of webpage

Not statistical regression!

Visual regression: an example

width: 100%;

Using

width: 100%;

After disabling

Visual Regression

Ply

  • Modification of Chrome Developer Tools element inspector

  • ​Developed through needfinding with 6 novice developers

  • Only displays CSS properties whose removal causes visual regression

User Evaluation

  1. Original stylesheet: 2 participants, 45 minutes

  2. Pruned stylesheet: 2 participants, 45 minutes

Task:  "Replicate a component from a professional website"

.position--relative {
  position: relative!important;
}

div {
  display: block;
}

.fare-estimate-input {
  position: absolute;
  top: 23px;
  bottom: 21px;
  left: 15.5px;
  width: 1px;
  background-color: #C6C6C6;
}

.fare-estimate-input::before {
  content: '';
  position: absolute;
  left: -2px;
  width: 5px;
  height: 5px;
  border-radius: 50%;
}

.fare-estimate-input::after {
  content: '';
  position: absolute
  left: -2px;
  width: 5px;
  height: 5px;
  border-radius: 50%;
}

.autocomplete-input {
  height: 44px;
  width: 100%;
  border: 1px solid #E5E5E4;
}

.autocomplete div input {
  width: 100%;
  text-overflow: ellipsis;
}

input {
  line-height: normal;
}

input, {
  color: inherit;
  font: inherit;
}

input, {
  text-rendering: auto;
  letter-spacing: normal;
  word-spacing: normal;
  text-transform: none;
  text-indent: 0px;
  display: inline-block;
  text-align: start;
}
.outside {
  background-color: #FFF!important;
}

.next {
  
}

.connector:before {
    
}

.

.pickup {
    height: 44px;
    width: 100%;
    border: 1px solid #E5E5E4;
    padding: 0 16px 0 0;
    overflow: hidden;
    white-space: nowrap;
    text-overflow: ellipsis;
    padding: 8px 12px 6px 29px;
    margin-bottom: 14px;
  
}

.destination {
    height: 44px;
    width: 100%;
    border: 1px solid #E5E5E4;
    overflow: hidden;
    white-space: nowrap;
    text-overflow: ellipsis;
    padding: 8px 12px 6px 29px;
}

*, {
    box-sizing: border-box;
}

div {
    display: block;
}


html {
    font-family: ff-clan-web-pro,"Helvetica Neue",Helvetica,sans-serif!important;
    font-weight: 400;
}
html {
    color: #000;
    -webkit-font-smoothing: antialiased;
}
html {
    -ms-text-size-adjust: 100%;
    -webkit-text-size-adjust: 100%;
    font-size: .875em;
    line-height: 2;
    background-color: #fff;
    min-height: 100%;
    overflow-y: visible;
}

Without pruning: "try everything"

  • LOC copied: 64, 64
  • Irrelevant properties: 14, 15 
.btn{
  width: 44px;
  position: absolute;
  right: 0;
  background-color: #C6C6C6;
  border-color: #c6c6c6;
  font-size: 20px;
  line-height: 1.75;
  
}

.container {
  background-color: white;
  border: 1px solid grey;
  padding: 8px 12px 6px 29px;
}

.pickup {
  margin-bottom: 12px;
}

input{
  border: none;
}


.fare-estimate_input-connector{
  position: absolute;
  top: 23px;
  bottom: 1140px;
  left: 15.5px;
  width: 1px;
  background-color: #C6C6C6;
}

.fare-estimate {
  height: 50px;
  width: 200px;
  border: 1px solid #E5E5E4;
  margin-bottom: 12px;
}
.destination {
  height: 50px;
  width: 200px;
  border: 1px solid #E5E5E4; 
}

With pruning: targeted

  • LOC copied: 20, 24
  • Irrelevant properties: 0

Case Study: Airbnb Booking Form

  • 30 elements total

  • 13.5 properties removed per element, on average

  • 49.79% decrease per element, on average

Case Study: Airbnb Booking Form

43 properties

20 properties

Default

*Later prototype also captures rounded corners, for 21 total properties

Future Work

  • Distinguishing between changes that matter vs. changes that don't matter 
    • background-color
  • Pruning DOM nodes and other domains with visual output
  • Using code reduction as a starting point for semantic tools to support authentic practice

Acknowledgements

  • Haoqi Zhang, Josh Hibschman, Nell O'Rourke
  • Design, Technology, and Research (DTR) community
  • User testing participants

Backup Slides

Computed properties (cf. WebCrystal)

Browser computes values for all declared styles

div {
    font-family: 'Comic Sans';
    color: blue;
}

p {
    color: red;
}

div

h1

h2

font-family: 'Comic Sans';

color: blue;
<div>
    <h1>Never</h1>
    <h2>Gonna</h2>
    <p>Give you up</p>
</div>
color: red;

p

Rendered output

"Used properties" don't generalize

Used value

Declared value

width: 75%;
width: 43.333px;

Ply UI changes

DOM focus affordances

Hide unnecessary selectors

Chrome Developer Tools

Ply

49% average decrease in properties, across component

Element # Pruned % Decrease
button 35  34.75%
div 6.47 54.01%
input 36.33 35.03%
label 10.4 57.80%
select 40 35.71%
span 12 38.1%

Why inspection

Inspection vs. tutorials

  • Authentic learning (Shaffer and Resnick 1999): activities and materials should reflect real-world problems of personal interest to learners
  • Web developers are motivated by project demands, up-to-date standards (Dorn and Guzdial, CHI '10)

Goal: do both!

Rendering

Existing browser tools expose the DOM/CSSOM

div {
    padding: 10px;
    color: red;
}
<div>
    <p>Hi!</p>
</div>
<p>Goodbye</p>

Markup

Declared styles

DOM tree

CSSOM tree

Rendered output

DOM: Document Object Model

How a webpage is rendered (simplified)

<div id="rickrolled">
    <h1 class="red">Never Gonna Give You Up</h1>
    <h2>A Memoir by Rick Astley</h2>
</div>
#rickrolled {
    font-family: 'Comic Sans';
    color: blue;
}

.red {
    color: red;
}

HTML

CSS

Three "stages"

  1. Declared code

How a webpage is rendered (simplified)

<div id="rickrolled">
    <h1 class="red">Never Gonna Give You Up</h1>
    <h2>A Memoir by Rick Astley</h2>
</div>
#rickrolled {
    font-family: 'Comic Sans';
    color: blue;
}

.red {
    color: red;
}

HTML

CSS

div

h1

h2

"Never..."

"A Memoir..."

font-family: 'Comic Sans';

color: blue;
font-family: 'Comic Sans';

color: red;
font-family: 'Comic Sans';

color: blue;

Document Object Model (DOM),

CSS Object Model (CSSOM)

Three "stages"

  1. Declared code
  2. DOM/CSSOM

How a webpage is rendered (simplified)

<div id="rickrolled">
    <h1 class="red">Never Gonna Give You Up</h1>
    <h2>A Memoir by Rick Astley</h2>
</div>
#rickrolled {
    font-family: 'Comic Sans';
    color: blue;
}

.red {
    color: red;
}

HTML

CSS

div

h1

h2

"Never..."

"A Memoir..."

font-family: 'Comic Sans';

color: blue;
font-family: 'Comic Sans';

color: red;
font-family: 'Comic Sans';

color: blue;

Document Object Model (DOM),

CSS Object Model (CSSOM)

Output

Three "stages"

  1. Declared code
  2. DOM/CSSOM
  3. Rendered output

CHI 2017: Visual Regression Pruning for Web Design Source Inspection

By Sarah Lim

Made with Slides.com

CHI 2017: Visual Regression Pruning for Web Design Source Inspection

  • 3,501

More from Sarah Lim