My Favorite Testing Approach in Flutter

Over the years I’ve applied many different testing approaches.

• Widget testing
• Unit testing
• Integration testing

But there is one approach that, to me, has been one of the most valuable testing approaches.

Golden tests

Some Background

While working on a large codebase, I noticed a page where items in a list would sometimes have different padding.

This would get noticed in production.

The issue was caused by a shared component being modified in one part of the app, which unintentionally changed it in another part of the app.

As a result, I often had to fix the bug in production, and then deal with the slow app store review cycle. Or even worse have the app store deny publishing the app because they found an unrelated thing they want us to fix.

All for something that could be automatically resolved.

The solution

The way I mitigated this issue was by using a golden test

Golden tests are a way to compare your UI to a generated image of that UI also sometimes called a “golden file” or “golden image”.

I wrote the test, which I may add is one of the simplest tests to write.

testWidgets('renders page correctly', (WidgetTester tester) async {
  await tester.pumpWidget(
    TestableWidget(
      child: const TheBuggyPage(),
    ),
  );

  await expectLater(find.byType(TheBuggyPage), matchesGoldenFile('the_buggy_page.png'));
});

That was all.

I had a working test, whenever anyone would change the component the page was using, the test would fail.

I put it in the CI process and realized something.

It failed.

It was working fine locally but not in CI.

It didn’t work

I asked myself why.

After diving deeper, I understood that the way golden tests work is that when I generate images using flutter test --update-goldens, Flutter uses LocalFileComparator to compare the image from the file system byte-to-byte when running the tests.

Well, when you generate the images, the resulting image slightly differs depending on the OS you are running on, whether MacOS, Windows, or Linux.

I generated the image on MacOS and the CI was running Linux, there was a 0.4% diff in the golden test.

I want to mention that I go deeper into testing in The Best Flutter Course On The Internet, but of course, I will share the solution here as well.

Want to learn Flutter?

We teach you all you need to confidently build apps at hungrimind.com/learn/flutter

The actual solution

There are two ways to solve this.

1. make sure local and CI are using the same OS
2. ignore the slight variance

I chose option 2 to prevent others on the team, especially those using Linux, from encountering the same issue.

If you remember LocalFileComparator well as it turns out in coding, you can extend and override the functionality.

I created a GoldenFileComparator this would allow me to override the compare logic to add a tolerance. In simple terms, if there is a 1% diff, I could allow it as a pass in the CI.

With all said and done I could now update the test with the new implementation.

testWidgets('renders page correctly', (WidgetTester tester) async {
  await tester.pumpWidget(
    TestableWidget(
      child: const TheBuggyPage(),
    ),
  );

  await expectGoldenMatches(
    find.byType(TheBuggyPage),
    'the_buggy_page.png',
    tolerancePercentage: 0.5,
 );
});

I never had an issue again

As a result of all of this work which in reality was probably about 20-30 minutes of work. I never saw an issue with that page ever again.

I went from fixing this page 4 times in production to never doing it again.