We're pretty much standardized on FastAPI (https://fastapi.tiangolo.com/) for all new web apps. We were struggling with testing, and in this post I'll share what we saw and what we learned.

Using pydantic's "BaseSettings" for settings management in FastAPI makes a ton of sense, since Pydantic is used all over the place. When I'm talking about "settings" I'm thinking about all the info you need to provide to your app: Database credentials, urls for other services it needs to call, which port(s) to listen on, etc. We generally try and keep a single settings file that is used thrughout the app, it might look something like this:

lru-cache is a simple way of in-memory caching the settings object, so that Pydantic doesn't have to re-read environment variables, config files, etc every time a module asks for settings. This becomes extra critical when you start adding more advanced logic to your settings. Here's how we load database credentials from AWS Secrets Manager (mounted using the Kubernetes Secrets Store CSI Driver) for example:

You don't want to run the "secrets loader" code everytime a part of your app asks for settings, since that may slow the entire app down. So, lru-caching is good. However.

This is not the way

If you look at https://github.com/trondhindenes/fastapi-pydantic-testing/tree/this-is-not-the-way you'll see that we attempt to test the app using different variations of a setting, and it fails. Even worse, the tests become non-deterministic; a single test might succeed where if it is run as part of all tests it may fail. That's bad. This happens because lru-cache has already cached the method and so any subsequent environment variable changes didn't have any effect.

At first we experimented with dynamically yanking loaded modules from sys.modules to force re-importing them and it worked somewhat. We also tested clearing the lru cache to force environment variables to get re-evaluated. However, it was very tedious.

Where to get settings

We also originally put the "get_settings" call into the module level of any part of our app that needed it, like so:

However, since the get_settings call is returned from cache anyway, we've started placing it inside any method relying on settings:

This means that we have less side-effects when monkey-patching settings should we need it. However, we mostly rely on just patching environment variables in order to manipulate settings as shown below:

This is the way

I've adjusted the "bad" example above into a structure better suited for testing, which you'll find at https://github.com/trondhindenes/fastapi-pydantic-testing/tree/main.

If you look at the settings file, it now looks like this:

Notice how we now can control the "cachiness" of the settings object using the "USE_CACHED_SETTINGS" environment variable. If it is not present, the settings will behave as before.

In order to automatically set it, we use the magic of conftest.py - this will will always be loaded when running tests, and never otherwise:

With a few simple tweaks, we now force settings to be loaded everytime they're needed when under test. Sure, this will slow down instantiating the settings object, but you're hopefully okay with trading a few milliseconds of test performance for more a more robust and easy-to-use test suite.

With this new "anticache" in place, we're free to manipulate our settings as often as we'd like:

It's a simple thing, but for us it made all the difference in terms making our tests easier to manage. We can now trust that tests are deterministic and behave as expected.

Feel free to check out out the repo https://github.com/trondhindenes/fastapi-pydantic-testing and its two branches (the "good" and "bad").