GUIs and Event-Driven Programs

How are clothes simulated in games if cloth simulation is difficult?

Most of the time, it's simulated using completely inextensible cloth or very stretchy cloth, and without collisions.

If cloth is used in places where this doesn't work, it's often hand-animated.

Games that don't do this tend to suffer performance issues, even today.

One student noticed that any time their character's clothes stretched or deformed in Star Citizen, they went down to 30 FPS.

Is Minecraft terrain generated by Cellular Automata?

Yes. In fact, while small-scale Minecraft terrain generation is done by techniques we haven't covered (Perlin Noise and Perlin Worms), the large-scale stuff is handled by stochastic cellular automata and image kernel-like algorithms.

Vote for procgen as a subject if you want to know more!

Why did GPT play chess so badly?

A pirate found buried _____

treasure?

children?

slovenly?

Why did GPT play chess so badly?

A pirate found buried _____

treasure?

children?

slovenly?

Highly likely!

Not likely, but possible depending on what came before this sentence.

Not actually valid English, highly unlikely.

Language models try to construct likely continuations based off of what's been seen so far, using probabilities inferred from their training data.

Of course, if you always construct the most likely sentence, you'll get very boring (or sometimes undesired) results so in practice language models do a little sampling from the space of likely results. This is controlled by the temperature parameter.

Fun fact about the LLaMa models...

Chess!

Knight to B1.


...

...

...

...


Knight to A3 <OR> Knight to C3 <OR> knight to D2.


<more moves>

An example of what the model might have gotten during training.

 

Note: no hard and fast rules about knight movement, just that knight moves to B1 are usually followed up by moves to A3 or C3.

The model isn't following some hard-and-fast rule that knights have to move in a certain way. It's just using probabilities from games that it's seen before (in its training set).

So what if you wind up with a game that the model has never seen before?

So what if you wind up with a game that the model has never seen before?

Well the model knows that moves are still of the form <piece> to <square> but it doesn't necessarily know what rules it has to follow.

 

For instance, queen to C6 is a valid move, but only if the queen hasn't already been captured. If you look at it only from a language modeling point, this is a reasonable move to play.

So what if you wind up with a game that the model has never seen before?

Note that the model actually plays really well in the beginning, because openers are well-studied and written down in lots of places.

It's only once the game enters a state where you wouldn't reasonable expect to find the game state already studied in written material that the model starts to break down.

Language Model Quirks

Once you start to think of language models like this, a lot of their quirks become a lot easier to explain.

  • Language models can struggle to solve problems in languages with poor training data.
  • Many models struggle with negation (e.g. "draw me a picture that does not have an elephant in it").
  • Language models cannot count letters. Partially due to above issues and partially due to a tokenization process.

These are currently being worked on, but it is unknown if they are fundamental to the language modeling approach or flaws in our current understanding.

(╯°□°）╯︵ ┻━┻

Does this mean that language models are dumb autocomplete tools?

This is a philosophical question and one that I'm not qualified to answer.

Experimentally, the top language models tend to be much better at creative tasks than most people. Whether this is because they're truly reasoning through langauge or because they've basically memorized the entire internet is undecided.

Regardless, it's important to understand the strengths and limitations of your tools.

Note: In practice, many ML tools are difficult to understand and form for humans.

So many of these issues are "solved" in ways that just make the issues non-visible to the user, slightly degrading the quality of the tool in the process.

The Challenges of User-Facing Programs

  • Do computation
  • When results are ready, show them to the user
  • User may optionally enter more input
  • Program processes user input when it is ready, then carries out more computation

"Typical" Program

These are the kinds of programs you wrote in 303E and 313E

Program does things on its schedule!

User-Facing Programs

  • Program usually sits around and does nothing until it needs to respond to something.
  • Once external event occurs, program should respond to it as quickly as possible.
  • Might need to process multiple events quickly

Events

Events

The things we need to respond to in our program are termed "events."

 

Events can be system-generated or user-generated, but are almost always external to the program---we can't control when they happen.

 

When events occur, we execute code which alters the program state. This is termed "event-driven programming."

Taken to the extreme, this creates a paradigm known as "reactive programming."

What types of events might we need to listen for?

Examples of Events

System-Generated

  • Hardware timer fires
  • File load completes
  • Network message arrives

User-Generated

  • Keyboard press
  • Mouse movement/click
  • Shutdown initiated

Toolkits

Unless you plan to write your own user interfaces from scratch (not recommended!), you'll likely be programming interfaces using an existing toolkit.

These toolkits already know how to recognize most common events, like mouse click, mouse drag, window close, etc.

The programmer is responsible for writing functions that will be called when these events occur, and telling the toolkit which functions to call on which events.

void clickCallback(){
  playPauseSound();
  displayMenu();
  pauseGame();
}

// Without being told to, the program does
// not know it needs to run mouseClick() on
// mouse click, so we tell it this in main()
void main(){
  GUI.registerHandler(ON_CLICK, clickCallback);
}

Callbacks

A common programming technique: we register a function to be called when something happens. Example in fake-Processing:

What are the existing event handlers in Processing?

GUIs

Short for "Graphical User Interface". An interface with a visual component.

 

Designed for easier, more intuitive experience. Typically based on event-driven programming.

Did you know you can browse the web using only the keyboard? To the left is the eLinks browser.

 

I'm glad I have Firefox.

Uses

I mean, honestly. What programs don't have graphical interfaces these days?

Question: what are common interactable elements of GUIs?

Widgets

A general name given to interactable elements within a GUI, including things like:

  • Buttons
  • Checkboxes
  • Sliders
  • Radio buttons

Give us different ways of interacting with program behavior and configuring the program.

Buttons

Simplest form of widget. Allows for functionality on mouse click.

 

Buttons need to be aware of the mouse position and the button boundary.

 

  • Circles check whether mouse is within button based on distance from center.
  • Rectangles check based on width/height from corner or center.

Hands-On: Buttons

  1. Implement a Button class. This class should have a method that takes screen coordinates (as separate x, y parameters) and checks whether the coordinates are within the button or not.
  2. Create both rectangular and circular buttons, and place them in nonoverlapping positions on the screen.
  3. Add functionality so that the color of a button changes when it is clicked.

You will extend the Button for today's second Hands-On.

Build Buttons

Check Boxes

A specialized button with an "on" and "off" state.

 

What do we need to store to track a checkbox?

Radio Buttons

A specialized version of checkboxes. Only one radio button can be on at any given time---when one is set to on, all others must be set to off.

Questions

  • What other data do radio buttons have to be aware of?
  • What data structures can help us organize the radio buttons?

Scrollbars

  • Users can select based on a range of values
  • Minimum and maximum values correspond to the ends of the slider
  • Thumb, or current position, controls the assigned value
  • Allows for a “continuous” range of values

What value does this correspond to on a slider?

How about this one?

How about this one?

Does this seem like something else we've studied in this class?

Slider's value is determined by a linear interpolation between the end position of the slider.

0.0 * SLIDER_MAX

1.0 * SLIDER_MAX

0.75 * SLIDER_MAX

This also works for non-linear sliders (like knobs) with some modification.

Hands-On: More Widgets

  1. Extend your button class to create checkboxes
  2. Extend your button class to create radio buttons
  3. Create a scrollbar that either updates the x/y position along an image or updates the background color of a sketch.

Pick two of the following to implement:

Make sure you demo them in your sketch.

Other GUI Libraries

Existing libraries for Processing can simplify the GUI creation process.

  • Guido: framework for GUI component creation
  • controlP5: provides GUI components
  • G4P: provides GUI components and builder

Sketches -> Import Library -> Add Library -> [name]

Index Cards!

  1. Your name and EID.
     
  2. One thing that you learned from class today. You are allowed to say "nothing" if you didn't learn anything.
     
  3. One question you have about something covered in class today. You may not respond "nothing".
     
  4. (Optional) Any other comments/questions/thoughts about today's class.

Final project proposal due tomorrow at midnight.

Physical simulation project due Wednesday at midnight.

You are not required to write more than an quick outline of your project. However:

  1. Writing things down in detail acts as a focusing lens for your thoughts.
  2. If you write down some implementation ideas, we can evaluate them and provide feedback.

Is there a time limit for the final presentation?

This will depend on group sizes in the class. We might be looking at anything from a very strict 5 minute time limit to a setting where almost nobody needs to watch time very closely.

What info can go in the slides if we're doing a demo?

Explain what you did! There are tons of small decisions and other things to explain to someone who's never looked at your project before.

 

For reference, I've seen engaging, interesting 7 minute presentations on a 2D "throw a ball" simulator.

Also, look on Canvas for other requirements.

What other cellular automata exist?

In 1D, there are exactly 256 possible CAs that use neighbor information. Stephen Wolfram has catalogued them in his book, "A New Kind of Science."

Rule 250

Rule 90

Rule 30

In 2D, we go from 256 possible automata to \(1.3 \times 10^{154}\).

Very hard to categorize all of these, but many interesting behaviors.

Is "The Sims" just a giant Game of Life?

No, but....

What is the connection between Cellular Automata and AIs?

Automata (not cellular in nature) power most rule-based systems.

AIs in games tend to be one of two systems, both discrete automata:

  • Behavior trees
  • Finite state machines

For more general AIs, before the neural network + backpropogation paradigm was shown to be effective (ca. 2010), the primary AI tools were automata-based.

Can CAs display fractals?

Not with infinite resolution, but yes.

In fact, some of the earliest graphics programs were generated by slightly more general CAs!

Debugging is Hard

Yes.

Debugging broken code is significantly harder than writing new code. I have never heard a professional developer express an opinion contrary to this.

It's even harder in graphics because certain parts of the "regular" debugging cycle are much harder or even impossible to do, so debugging broken graphics code can become very interesting.

Debugging

Observe a problem.

Think of things that could be causing this problem.

Try to observe parts of the program that let you distinguish between your potential causes.

Fixed!

Apply change to prevent issue.

Issue disappears

Do we see something consistent with our potential cause?

Dang.

Yes.

No.

How not to Debug*

* You can do this a few times as a sort of quick-and-dirty debugging path. If it doesn't work 2 or 3 times, then STOP! Back up, take a breath, and take it slow and steady.

  1. See a problem.
  2. "I wonder if it might be X."
  3. Write a bunch of code that would fix X.
  4. It doesn't work.
  5. "I wonder if it might be Y..."

Do this too many times, and your code is undebuggable!

Example

My Game of Life was not showing any change in squares.

Other Tricks

def calculate(lst):
    total_i = 0
    total_e = 0
    for t in lst:
        if t['t'] == 'i':
            total_i += t['a']
        else:
            total_e -= t['a']
    return total_i - total_e
def total_income(transactions):
    income = 0
    for t in transactions:
        if t['type'] == 'income':
            income += t['amount']
    return income

def total_expenses(transactions):
    expenses = 0
    for t in transactions:
        if t['type'] == 'expense':
            expenses -= t['amount']
    return expenses

def calculate_net(transactions):
    income = total_income(transactions)
    expenses = total_expenses(transactions)
    income - expenses

Other Tricks

Good naming and commenting in your functions can save you a lot of grief!

Structuring code into functions can allow you to test things without suffering printspam.

void main(){
  // ...
 
  for(int i = 0; i < 100; i++){
    for(int j = 0; j < 100; j++){
      array[i * width + j] = array[i+1 * width + j] + 10;
    }
  }
 
  // ...
}
void main(){
  // ...
 
  for(int i = 0; i < 100; i++){
    for(int j = 0; j < 100; j++){
      float val = func(array, i+1, j);
      array[i * width + j] = val;
    }
  }
}

Other Tricks

When writing your code, do it as simply as you can.

If debugging is harder than writing, and you're using 100% of your cleverness when writing the code, how are you ever going to debug it when things inevitably go wrong?

NOTE: When you're still learning to program (as you are now), there is a tendency to favor cool tricks over boring code. This is fine (I would even argue that it is good) but remember that this does make it harder to debug later.

But mostly...

It takes a lot of time and experience.