Productive E2E Test Writing Using GitHub Copilot
Gleb Bahmutov
Sr Director of Engineering
Speaker: Gleb Bahmutov PhD
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
Gleb Bahmutov
Sr Director of Engineering
A typical Mercari US Cypress E2E test
2022: Mercari US disallows using AI code generation assistants pending a legal review
2023: Mercari US allows using AI code generation assistants after a legal review
Profit!!!
VSCode Copilot Extension
Which model?
Mercari US isn't paying for Copilot GH CLI extension
Tip: VSCode Copilot Chat extension
can provide the same info as GitHub CLI copilot
Tip: use "/clear" command to clear the Copilot session
Paste the Copilot answer right into VSCode built-in terminal
〞
Is GitHub Copilot
making me more productive?
〞
Measuring programming progress by lines of code is like measuring aircraft building progress by weight.
Bill Gates, MS Clippy user
Testing Productivity
🚨 "AI, write me 10 tests"
✅ "AI, help me write this test faster"
🎁 "AI, debug this failed test"
GitHub Copilot in Action
- how to ask GitHub Chat questions
- context and session
- use commands suggested by the Chat
- asking Chat for authoritative sources
- running the first Cypress test
⚠️⚠️⚠️
The Most Important Copilot Trick
⚠️⚠️⚠️
Clear Chat session and add prettier
- how to clear Copilot Chat conversion session
- how to insert suggested code snippets into a file
- format source code using Prettier
Copilot Suggestions
https://github.com/bahmutov/cypress-copilot-example
branch a3, might need "npm install"
- how to always show the Copilot suggestions bar
Copilot Suggestions
https://github.com/bahmutov/cypress-copilot-example
branch a3, might need "npm install"
- how to always show the Copilot suggestions bar
- seeing multiple suggestions at once using the Completions Panel
- tweaking the code suggestions using comments
- using Chat to suggest a better solution
Tip: Add a comment like "// format the price using USD currency"
to see better suggestions
Writing Unit Tests Using Copilot
- how to prompt Copilot to generate a unit test file
- how to adjust the generated test file to use Cypress globals and Chai assertions
- how to generate additional unit tests
Guide Copilot Using Comments
- the generated test quality is determined by the local context
- adding more comments with concrete values helps generate the correct test code
Guiding A Human QA The Same Way
- visit the page
- the page finishes loading 👎
- the body element has class "loaded" 👍
- find the input element with class "new-todo"
- type "Hello, World!" + Enter
- the list should have 1 item with "Hello, World!" text
imprecise for the computer
Test Adding Todos
- naming spec file
- accepting the full Copilot suggestion vs the next word
- guiding the Copilot to generate a comment vs a
cycommand chain - keep Cypress open to see the test result
- reset the data using beforeEach hook
Test Deleting Todos
- generating JSON fixtures
- inspecting the generated test code
- setting up the initial test data by loading a fixture and making an API call
Spy on Network Calls
- how Copilot can explain source code
- spying on the network calls using the cy.intercept command
- waiting on network calls using the cy.wait command
⚠️ if you see "cy.route" suggestion,
it is obsolete!
Modify Code In Place
- modifying the existing code using the Command-I tool
network-calls.cy.js
Fix Code In Place
- fixing the selected code using the Copilot
/fixcommand - generate the commit messages based on the changed source files
Document your code
- using
/doccommand to generate JSDoc comments - how to document your Cypress tests using
cy.logcommands
Generate Better Code By Example
- how to remove duplicated selectors in the generated queries
- how Copilot can use local variables and functions
- how you ultimately can dramatically improve the final test code quality
adding-todos.cy.js
Generate Test Data
- how you can generate arrays of objects following a simple schema
- how Copilot update the generate list to refine it further
Copilot Edits
https://github.com/bahmutov/cypress-copilot-example
branch bonus02
- a new mode for the "prompt + review proposed code diff"
Copilot Edits
https://github.com/bahmutov/cypress-copilot-example
branch bonus02
- a new mode for the "prompt + review proposed code diff"
Also: GitHub Copilot Workspace
Copilot 👍
- Easy test data generation
- Good starting point
- Follows the comments
- Speeds up typing
Copilot 🔥
- Hallucinations
- Outdated information
- Unorthodox suggestions
- Complex suggestions
- ❌ writing code that I do not know how to write myself (new language or framework)
- ❌ write big chunks of code (generate 10 tests at once)
- ✅ speeding up writing code guided by comments
- ✅ test data generation
- ✅ code or data in-place transformations
- 😒 commit message generation
GitHub Copilot
❤️ the "/explain" command
GitHub Copilot is the "Y" key when playing Mario
Local "Copilot"
- Install https://ollama.com/
Download public models (almost like Docker!)
Local "Copilot"
- Install https://ollama.com/
- Install VS Code extension "Continue"
Local "Copilot"
{
"models": [
{
"title": "DeepSeek Coder 6.7b",
"provider": "ollama",
"model": "deepseek-coder:6.7b"
},
{
"model": "claude-3-5-sonnet-latest",
"provider": "anthropic",
"apiKey": "",
"title": "Claude 3.5 Sonnet"
}
],
"tabAutocompleteModel": {
"title": "DeepSeek Coder 6.7b",
"provider": "ollama",
"model": "deepseek-coder:6.7b"
},
"embeddingsProvider": {
"provider": "ollama",
"model": "nomic-embed-text"
}
}Local "Copilot"
- Install https://ollama.com/
- Install VS Code extension "Continue"
- Configure local / remote models
- Local chat and code completions!
Local "Copilot"
- Install https://ollama.com/
- Install VS Code extension "Continue"
- Configure local / remote models
- Local chat and code completions!
I found local models
to be pretty
inaccurate