Open Sourced Tests: A Retrospective
John Hill
About Me
Web UI Test Engineer,
Space Mission Control Software, KBR, Inc
@ NASA Ames Research Center
Playwright Ambassador
https://playwright.dev/community/ambassadors
VIPER
In this talk
- Open MCT e2e Test Initiative
- Getting from 0 to 1 test
- Challenges with e2e Testing Projects
- Challenges unique to this e2e testing project
- Call to Action!
Open e2e Test Initiative Goals
-
Increase our test coverage. (OSC already increased it by 3%!)
-
Provide an opportunity for the testing community to directly contribute to this NASA Open Source project
-
Learn from folks who demonstrate some new approaches to old testing challenges with Playwright.
-
Ensure that Open MCT testing is accessible to everyone
History
- The project already has a healthy open source developer community. (GraphQL Example)
- The project already had an existing Open Source contribution process and CLA
- e2e testing came online July 21
- Open e2e Testing Initiative came online October 21
Getting from 0 to 1 Test
- Start with our Github Discussion.
- Playwright? We leverage official playwright.dev/docs/intro
- Getting started with Open MCT's implementation of Playwright. (Let's run through this together!)
- Gain Open MCT Domain knowledge
- Picking a test to write
What did we do?
- Utilized Playwright to mimic Mission Control Operator Behavior to interact with Open MCT
- We demo'd how easy it is to get going with the codegen and --debug.
- How Open MCT reenforces evolving best practices into the onboarding process.
- Test Stubs and test.fixme()
- Invoked failure and utilized the triage tooling
Picking a test to write
- Test stubs -> Real Tests
- Originate from Open MCT dev team
- Bugs and enhancements
//If you run out of time to write new tests, please stub in the missing tests in place
//with a test.fixme and BDD-style test steps. Someone will carry the baton!
test.fixme('Can Rename Timer Object from Tree', async ({ page }) => {
//Create a new object
//Copy this object
//Delete first object
//Expect copied object to persist
});Challenges with any e2e project
- Onboarding
- AKA How to get from 0 to 1 test contributed.
- Solution: Re-enforcing example template, lint, and JSDoc
- Solution 2: Test Promotion.
- Start with @unstable tag to skip per commit.
- "Promote" behind the scenes
- Web-dev-isms
- Generally, JS web devs focus on clever, elegant, and concise implementations of test frameworks
- QA is more interested in saving 3 hours of triage time than 3 lines of code
- Solution: Failure must be obvious in tracefile and error logs
Unique Opportunities (Challenges)
- Open Source tests are re-used in downstream projects
- ExampleTemplate suite needs to run against closed source projects.
- More abstraction and fewer assumptions
- No Page Objects?! (Solution: AppActions)
- "UI Contract Tests" and Performance
- No assertions, just Contracts
- Similar to Pact API Contracts
- Dev and Test (me) Time is shared
- Incentive to make app testable
CLA Stall :(
Please, no drive-bys 🙏
App Actions and a11y
- No Common DOM = No Common POM (Page Objects)
- App Action Pattern combines repeated user behavior into common functions (not fixtures)
- Instead, we leverage a11y locators
- Aria and roles for screen readers
- Now the preferred way of interacting with Chrome Dev Tools Recorder
- If dev and test time is shared, where would you spend that time to make the app more testable?
Call to Action!
I am here [Playwright Slack] because of your Open MCT project and I liked what I was doing so much I brought it to my employer. Now I code Playwright all day!
~ Open MCT e2e contributor
Summary
- Open MCT e2e Test Initiative
- 0 to 1
- Challenges with any e2e Project
- Challenges unique to this e2e Project
- Call to Action?!
Cutting Room Floor
Why Playwright?
- First-class support for Automated Performance Testing
- "Official" Chrome, Chrome Canary,
and iPad Capabilities - Support for Browserless.io to run tests
in a "hermetically sealed" environment - Ability to generate code coverage reports
Open Source Tests: A Retrospective
By John Hill
