Software testing

Agenda

  • recap why we write tests & journey
  • facing the challenges
  • testing in general & best practices
  • useful patterns
  • coding

The usefulness of testing

  1. you implement a new feature
  2. you manually test it locally, on staging, etc.
  3. other dev needs to change it weeks later
  4. other dev forgets to test the same previous use cases
  5. the feature gets broken

Real world example

https://github.com/polygence/boshi/pull/12506

(is it even possible without good test coverage?)

In short, we need tests to develop and maintain such a large application

0. phase: zero tests

1. phase: the beginning

  • let's start writing some test cases
  • it doesn't matter how we write them
  • just to cover more and more business logic
  • increase coverage

2. phase: high coverage

  • the number of tests is high
  • tests form a mass in the code base
  • there is an impact of tests on development
  • things are getting interesting

Statistics

  • ~9000 backend tests
  • ~1200 backend test files
  • more than 100.000 lines of tests (backend)
  • 87% backend test coverage

We experience more

  • sometimes test cases (E2E + backend)
  • not helpful, but breaking test cases (during development)
  • broken CI because of missing tests (annoying)
  • end result: frustration

Contradiction

It is possible to resolve this contradiction!

3. phase: trusted test suite

  • test cases based on the same pattern
  • trust towards the test suite is high
  • tests are not flaky, not falsey, sometimes test cases are rare
  • easy to write test cases, because of the same pattern
  • easy to maintain the test suite
  • easy to release

Black-box testing

  • testing the behaviour of the software, not its implementation
  • tests should not depend on the implementation

Bad


expected_url = get_frontend_url(f"/dashboard/hermes/{self.meeting.workspace.id}/meeting-summary/{self.meeting.id}/review")

self.assertEqual(session_summary_url, expected_url)

Good


expected_url = f"/dashboard/hermes/{self.meeting.workspace.id}/meeting-summary/{self.meeting.id}/review"

self.assertTrue(session_summary_url.endswith(expected_url))


expected_url = f"/dashboard/hermes/{self.meeting.workspace.id}/meeting-summary/{self.meeting.id}/review"

self.assertTrue(session_summary_url.endswith(expected_url))
self.assertGreater(len(expected_url), len(session_summary_url))


expected_url = f"/dashboard/hermes/{self.meeting.workspace.id}/meeting-summary/{self.meeting.id}/review"

self.assertUrl(session_summary_url, expected_url)

Tests should be isolated

  • test has to be independent from each other
  • setUp?

def setUp


class WorkspaceGroupUpdateViewTestCase(APITestCase):
    def setUp(self):
        self.default_email = Configuration.get(ConfigurationOptions.PATHFINDER_DEFAULT_USHER_EMAIL)
        self.admin: User = AdminMentorFactory.create(email=self.default_email)
        self.workspace_group: WorkspaceGroup = WorkspaceGroupFactory.create_for_launchpad()
        self.launchpad_projects = [
            ProjectFactory.create_for_pathfinder_launchpad(),
            ProjectFactory.create_for_pathfinder_launchpad(),
            ProjectFactory.create_for_pathfinder_launchpad(),
        ]
        self.workspace_group.workspaces.set([project.workspace for project in self.launchpad_projects])
        self.student_profile = self.workspace_group.student

resolution


def test_does_not_trigger_reflection_survey_email_if_workspace_group_is_not_launchpad(self, trigger_process_for: MagicMock):
    self.workspace_group.finance_type = WorkspaceGroupFinanceTypes.PATHFINDER_PPP
    self.workspace_group.save()

    self._update_workspace_group(user=self.admin, payload={"status": WorkspaceGroupStatuses.COMPLETED})

    trigger_process_for.assert_not_called()


def test_responds_with_ok_for_valid_payload(self, _: MagicMock):
    response = self._update_workspace_group(user=self.admin, payload={"status": WorkspaceGroupStatuses.COMPLETED})

    self.assertEqual(response.status_code, status.HTTP_200_OK)

resolution


def test_does_not_trigger_reflection_survey_email_if_workspace_group_is_not_launchpad(self, trigger_process_for: MagicMock):
    self.workspace_group.finance_type = WorkspaceGroupFinanceTypes.PATHFINDER_PPP
    self.workspace_group.save()

    self._update_workspace_group(user=self.admin, payload={"status": WorkspaceGroupStatuses.COMPLETED})

    trigger_process_for.assert_not_called()


def test_responds_with_ok_for_valid_payload(self, _: MagicMock):
    response = self._update_workspace_group(user=self.admin, payload={"status": WorkspaceGroupStatuses.COMPLETED})

    self.assertEqual(response.status_code, status.HTTP_200_OK)

Arrange-Act-Assert (AAA) Pattern


class LaunchpadReflectionProcessServiceTriggerProccessForTestCase(TestCase):
    def setUp(self):
        self.workspace = WorkspaceFactory.create_with_project()
        self.student_profile = self.workspace.project.students.first()
        self.workspace_group = WorkspaceGroupFactory.create_for_launchpad()
        self.workspace_group.workspaces.add(self.workspace)

    def test_does_nothing_without_student(self, send_launchpad_reflection_complete_survey: MagicMock):
      	# Arrange
        self.student_profile.delete()
		
        # Act
        LaunchpadReflectionProcessService.trigger_process_for(self.workspace_group)

        # Assert
        send_launchpad_reflection_complete_survey.create_survey_for.assert_not_called()

Test one behaviour

at a time

(one expectation)

Bad


@patch("hermes.services.room.RoomService.send_student_inactivity_notification")
def test_sends_correct_email_at_fourth_nudge(self, mocked_send_notification: MagicMock):
  	# ...

    InactiveStudentEscalationService.escalate_inactivity_reports(
      self.inactivity_report.inactive_user.id, self.project
    )

    self.assertEqual(
      HubSpotTransactionalEmail.objects.filter_by_email_template(
        HubSpotEmailTemplate.STUDENT_INACTIVE_FLOW_FOURTH_NOTIFICATION
      ).count(),
      original_email_count + 1,
    )
    email = HubSpotTransactionalEmail.objects.last()
    self.assertEqual(email.to, self.inactivity_report.inactive_user.email)
    self.assertEqual(
      email.custom_properties,
      {
        "link_to_chatroom": get_room_url(self.project.workspace),
        "mentor_name": self.project.mentors.last().first_name,
      },
    )
    self.inactivity_report.refresh_from_db()
    self.assertIsNotNone(self.inactivity_report.fourth_nudge_sent_at)
    mocked_send_notification.assert_called()

Good


def test_response_created_after_successful_request(self, _):
    response = self._send_email_change_request(self._payload, self.user)

    self.assertEqual(response.status_code, HTTP_201_CREATED)
    self.assertEqual(response.json()["email"], NEW_EMAIL)

def test_create_email_change_request_object(self, _):
    self._send_email_change_request(self._payload, self.user)

    email_change_request = EmailChangeRequest.objects.first()

    self.assertEqual(email_change_request.old_email, OLD_EMAIL)
    self.assertEqual(email_change_request.email, NEW_EMAIL)

Understandable names

tests document the code


class <EntityName>TestCase(TestCase):
    def test_<method_name>_<behaviour>_<condition>(self):
        # expected data & setup data
        
        # action
        
        # expectations


class <EntityName><MethodName>TestCase(TestCase):
    def test_<behaviour>_<condition>(self):
        # expected data & setup data
        
        # action
        
        # expectations

Use mocks when...

  • mocked logic is well tested
  • do not duplicate the tests over layers

def test_schedules_next_nudge_email_after_7_days(self, schedule_action: MagicMock):
    current_time = timezone.now()
    expected_eta = current_time + timezone.timedelta(days=7)

    with time_machine.travel(current_time, tick=False):
      LaunchpadReflectionProcessService.notify_student_about_survey_again(
		self.student_profile, self.workspace_group, 1
      )

    schedule_action.assert_called_once_with(
      "hermes.tasks.tasks.send_launchpad_reflection_survey_nudge",
      [self.student_profile.id, self.workspace_group.id, 2],
      eta=expected_eta,
    )

Summary

  • test behaviour, not implementation
  • isolate tests from each other (happy path setUp)
  • Arrange-Act-Assert pattern
  • test one thing at once
  • tests document the code (good names)
  • trust in tested methods, then mock

Our best practices

use factories


class LaunchpadReflectionProcessServiceTriggerProccessForTestCase(TestCase):
    def setUp(self):
        self.workspace = WorkspaceFactory.create_with_project()
        self.student_profile = self.workspace.project.students.first()
        self.workspace_group = WorkspaceGroupFactory.create_for_launchpad()
        self.workspace_group.workspaces.add(self.workspace)

mock time


def test_sets_first_completed_at_and_completed_at_when_student_application_is_completed_for_the_first_time(self):
    expected_completed_at = timezone.now()

    self.student_application.completed_at = None
    self.student_application.first_completed_at = None
    self.student_application.save()

    with time_machine.travel(expected_completed_at, tick=False):
      self.update_student_application({"status": StudentApplicationStatuses.COMPLETED})

	self.student_application.refresh_from_db()
    self.assertEqual(self.student_application.completed_at, expected_completed_at)
    self.assertEqual(self.student_application.completed_at, self.student_application.first_completed_at)

parameterized


@parameterized.expand([StudentApplicationStatuses.DECLINE_PENDING, StudentApplicationStatuses.DECLINED])
def test_check_missing_decline_pending_at_returns_nothing_for_application_with_check_decline_pending_at(self, status):
  	StudentApplicationFactory.create(status=status, decline_pending_at=timezone.now())

	self.assertEqual(StudentApplicationChecksService.check_missing_decline_pending_at().count(), 0)

subTest

def test_returns_project_with_completed_at(self):
  for project in self.projects:
    project.completed_at = timezone.now()
    project.save()

    with self.subTest(project_id=project.id):
      expected_completed_at = project.completed_at.strftime("%Y-%m-%dT%H:%M:%S.%fZ")

      returned_project_data = StudentJourneyTileProjectDataService.get_data(project)

      self.assertEqual(returned_project_data["completed_at"], expected_completed_at)

override settings


@override_settings(GOOGLE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS="account_credentials")
def test_returns_my_drive_when_credentials_configured(self):
  drive_type = GoogleDriveFactory.get_drive_type(workspace=self.workspace)

  self.assertEqual(drive_type, GoogleDriveType.MY_DRIVE)

outgoing requests testing


class UnstructuredServiceTestCase(SimpleTestCase):
    @responses.activate
    def test_get_chunks_from_url_returns_page_chunks(self):
        expected_url = "https://arxiv.org/pdf/2303.09232v2"
        expected_page_chunks = {"pages": [{"content": "content", "page_number": 0, "page_image_base64": "image"}]}

        responses.post(
            url=f"{UNSTRUCTURED_API_BASE_URL}/api/v1/pages/",
            match=[responses.matchers.json_params_matcher({"url": expected_url})],
            status=HTTP_200_OK,
            json=expected_page_chunks,
        )

        page_chunks = UnstructuredService.get_chunks_from_url(expected_url)

        for index, expected_page_chunk in enumerate(expected_page_chunks.get("pages")):
            for key, expected_value in expected_page_chunk.items():
                self.assertEqual(page_chunks[index][key], expected_value)

API tests

private helper methods


def _update_workspace_group(
  self, user: User | None = None, workspace_group_id: int | None = None, payload: dict | None = None
):
  if user:
    self.client.force_authenticate(user)

    return self.client.patch(
      f"/zeus/workspace-groups/{workspace_group_id if workspace_group_id else self.workspace_group.id}/",
      data=payload if payload else {},
      format="json",
    )

Proper mocking

mock objects at usage

SERVICE_PATH = "survey.services.launchpad_reflection_process"


@patch(f"{SERVICE_PATH}.SchedulerService.schedule_action")
@patch(f"{SERVICE_PATH}.LaunchpadWorkspaceGroupService.set_reflection_survey_first_email_timestamp")
@patch(f"{SERVICE_PATH}.HermesMailUtils.send_launchpad_reflection_complete_survey")
class LaunchpadReflectionProcessServiceTriggerProccessForTestCase(TestCase):

Task

work in pairs if you want to


@base_task()
def update_graduate_student_uuid_for_parents():
    current_datetime = django_timezone.now()
    current_year = current_datetime.year
    
    # ...

Thanks!

Importance of testing

By Róbert Beretka

Made with Slides.com

Importance of testing

Presentation for Email Channel Crew about NodeJS project setup with TypeScript and best practices.

  • 63

More from Róbert Beretka