Lessons learned about testing and TDD

Marco Buttu, EuroPython 2015


Marco Buttu @ EuroPython 2015, Bilbao

Just a Few Years Ago

The Software Goes in Production...

A System Without Regression Tests

Development Workflow


The Sardinia Radio Telescope

A User Story

As an astronomer, I want to interact with the radiotelescope through a command-line interface. When I execute a setup, I want to know if the setup is actually in progress. I also want to know if the setup is done, and in that case I want to know if the telescope is ready to operate. Eventyally, I want to get the actual and the commanded setup.

Pinco Pallino Jansky

User Acceptance Test

            $ setup KK # Set the K-band receiver (wait a bit...)
            $ is_starting
            $ commanded_setup 
            $ actual_setup 
            $ is_ready # At some point the setup is completed
            $ actual_setup 

Functional Test: User Point Of View

User typing

Functional Test

            def component():
                from myframework import services 
                return services.get_component('component_name')

            def test_setup(component):
                component.cli('setup KK')
                assert component.cli('is_starting') == 'True'
                assert component.cli('commanded_setup') == 'KK'
                assert component.cli('actual_setup') == 'None'
                assert component.cli('is_ready') == 'True'
                assert component.cli('actual_setup') == 'KK'

Integration Test: System Point Of View

Integration test

Integration Test

            def component():
                from myframework import get_component
                mycomponent = get_component('component_name')
                return mycomponent

            def test_is_starting(component):
                assert component.is_starting()
                assert not component.is_starting()

Unit Tests

            def component():
                from myframework import get_component
                mycomponent = get_component('component_name')
                return mycomponent

            def test_setup_set_device_position(component):
                expected_pos = component.conf.position()
                actual_pos = component.get_position()
                assert math.isclose(actual_pos, expected_pos) # Py3.5

Component Implementation

            class MyComponent(object):

                def __init__(self):
                    self.device = get_component('low_level_device')
                    self.conf = Configuration()

                def setup(self, setup_code):
                    # ... At some point:
                    p = self.conf.get_position(setup_code)

Unit Tests Should Run Offline

Sometimes the external resources are not available during development.

Unit Tests With Mock Objects

            def component():
                from myframework import get_component
                mycomponent = get_component('component_name')
                mycomponent.device.set_position = unittest.mock.MagicMock()
                return component

            def test_setup_set_device_position(component):
                expected = component.conf.position('KK')

Tests With Simulators


No TDD and more Functional Tests?

The current fanatical TDD experience leads to a primary focus on the unit tests, because those are the tests capable of driving the code design (the original justification for test-first). I don't think that's healthy... Less emphasis on unit tests, because we're no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests.

David Heinemeier Hansson, creator of Ruby on Rails

Are functional tests worse than useless?

Unlike unit tests, the functional tests don't tell you what is broken or where to locate the failure in the code base. They just tell you something is broken. That something could be the test, the browser, or a race condition. There is no way to tell because functional tests, by definition of being end-to-end, test everything.

Chirag Doshi and Rachel Laycock (ThoughtWorks)

A Recent Story: the Airbus A350

As with any new plane, the early design phases were riddled with uncertainty. Would the materials be light enough and strong enough? Would the components perform as Airbus desired? Would parts fit together? Would it fly the way simulations predicted? To produce a working aircraft, Airbus had to systematically eliminate those risks using a process it calls a "testing pyramid".

Jeff Wise (BusinessWeek)

The Pyramid

  1. Cover your code mostly with unit tests
  2. Verify the APIs behave as expected
  3. Ensure the user expectations
  4. Reduce manual sessions
  5. Distrust DHH: do TDD

Lessons Learned