The Problem:#

I ran into an interesting problem today when trying to handle a particularly hellish scenario¹. The situation is thus:

• I'm spinning up a new API service (using ASP.NET)
• We are re-using a bunch of existing infrastructure, including:
  • Our business-modelling graph database (we use Memgraph), and
  • Some C# libraries that define our interface with that DB, and a bunch of API helper services
• For primarily historical reasons², said graph database models most entities that are accessible via the API as nodes that have two IDs.
  • One is an internal ID (arbitrary, auto-assigned by various databases over time)
  • The other is a publicly-shared UUID³ (that we give customers and is used in our API's I/O)
• This internal ID is necessary to reference these entities in any system other than the core graph database, and the external ID is only used in customer-facing APIs.
• Multiple other systems already rely on this setup, and it is generally considered to be A Good Thing that customer-facing IDs are disjoint from canonical IDs, so ditching the dual IDs is not feasible.

If you need to access any system other than the DB itself, this means that you have to perform a mapping from externalId<->internalId when receiving an API request, and then again in the other direction in the response payload.

There are no functional issues with this, but the problem comes into play when attempting to test scenarios involving this logic, especially so when mocking the database calls out. We will take a better look at this in a moment.

Solution 1: The Way Things Were#

The way we've done performed this mapping in previous APIs is to just write a controller that manually requests the entity details using the external-facing ID, like so:

[ApiController]
public class EntitiesController(IDataService dataService) : ControllerBase
{
    [HttpGet]
    public async Task<EntityResult> GetEntity(string entityId)
    {
        var request = new GraphRequest()
        {
            Entity = new() { { "ExternalId", entityId } }
        };

        var entity = await dataService.GetEntityAsync(request)
	        ?? throw new ResourceNotFound404Exception();
        var entityInternalId = entity.internalId;

        // Then do the actual action's work
    }
}

This is, of course, serviceable and pretty unambiguous — you can look at it and immediately see what's going on. Up til now, this was fine, and there were no issues with this, apart from a little bit of boilerplate per action. The DB request itself is cached on first-use, so it's not particularly expensive (and is required one way or another, somewhere in the stack).

However — as alluded to earlier, mocking this requires some fantastically chunky setup to catch the GraphRequest, check its contents, and then return a mocked node matching the given details.

We can do better. But first, things get worse.

The Problem: Part 2 — It Gets Worse#

Previously, we had not performed any integration testing on our API, figuring⁴ we could get away with extensive unit and integration testing of the underlying logic, rather than testing the ASP magic sprinkled on top. With this new API came additional layers of parameter binding complication, and I spent the time to look into whether we could easily run integration tests of the entire application, including ASP.NET, and if so, do it.

Spoiler: it is possible. We're using Alba, but this is just a convenience layer on top of the ASP.NET test package, making it init, write, and call test scenarios. In combination with our existing integration test approach for other systems using testcontainers to spin up instances of Postgres with our migrations applied⁵, we were off to the races.

The latent issue with the above code wasn't immediately apparent to us until this point; that is, that this scenario conflates two things — (a) the need to perform this ID mapping, and (b) the use of the database service to do this task.

This means that as soon as we attempt to test this method⁶, we hit this wall: in order to mock out the ID mapping, you have to catch a very specific DB call with a specific constraint specified in a dictionary, and only then can you return a mocked entity object.

Solution 2: string-to-string Binding#

An almost trivial solution is to just whack an IExternalIdMapper service on top of our database, and use that instead. With minimal overhead (an interface and a lightweight class), this provides much more straightforward single-responsibility interface, which is easily mocked out for test runs.

var mapper = new Mock<IExternalIdMapper>();
mapper.Setup(x => x.FromExternalId(It.IsAny<string>()))
	  .ReturnsAsync((string externalId) => new EntityIdPair(externalId, externalId + "_internal"));

But, seeing as we're re-examining our approach: why not go further? Let's use this as an excuse to improve our general approach, instead of just taking the shortest path to a testable solution.

ASP.NET does a lot of things, and most of them are magic: middleware, content negotiation, exception handling, authentication and authorisation, attribute-based everything, model validation and binding. Model binding, however, is some fantastic wizardry — an HTTP request is received, with a query string and/or headers, and it magically resolves those to our statically-typed action parameters, parsing them as-required in most cases.

What if we leveraged this system to accept an external ID, but remap this to our internal ID, and our action itself never has to worry about doing it manually?

public class EntityLookupBinder(IExternalIdMapper _mapper) : IModelBinder
{
	public async Task BindModelAsync(ModelBindingContext bindingContext)
	{
            // Validate expected types
            if (bindingContext.ModelType != typeof(string))
            {
                bindingContext.Result = ModelBindingResult.Failed();
                return;
            }

            // Validate that we actually have a tangible value to bind against
            var valueResult = bindingContext.ValueProvider.GetValue(bindingContext.ModelName);
            var externalId = valueResult.FirstValue;
            if (string.IsNullOrWhiteSpace(externalId))
            {
                bindingContext.Result = ModelBindingResult.Success(null);
                return;
            }

            bindingContext.ModelState.SetModelValue(bindingContext.ModelName, valueResult);

		// Now lookup the actual desired value
		var mapped = await _mapper.GetInternalIdAsync(externalId, bindingContext.HttpContext.RequestAborted)
            ?? throw new ResourceNotFound404Exception();

		bindingContext.Result = ModelBindingResult.Success(mapped);
	}
}

Couple this with a quick custom attribute:

[AttributeUsage(AttributeTargets.Parameter)]
public sealed class EntityLookupAttribute : ModelBinderAttribute
{
	public EntityLookupAttribute() : base(typeof(EntityLookupBinder))
	{
		BindingSource = BindingSource.Query;
	}
}

And we can annotate our action parameter:

[HttpGet]
public async Task<EntityResult> GetEntity([EntityLookup] string entityId) { }

And boom, the internal ID is automatically injected into our action, no manual mapping required. From here, the action just has to ensure that it still returns the expected external ID in any result objects, rather than the internal ID.
But since the action never sees the external ID, how can it return it?

We can hit two birds with one stone, here. Firstly, the above problem.
We actually want to inject both the IDs into the controller, so it can pick and choose depending on the context. Secondly, remapping from a string to another string is a little fraught with confusion. I can easily see someone⁷ coming along, looking at the code, and getting very confused about why the ID they're sending is suddenly, magically, some other value once it enters the action itself.

By adding a wrapper object, we can pass both values in, and make it abundantly clear that this is not the exact value that was sent by the caller.

public record EntityIdPair(string ExternalId, string InternalId);

and then, in the binder:

var mapped = await _mapper.GetInternalIdAsync(externalId, bindingContext.HttpContext.RequestAborted);
var details = new EntityIdPair(externalId, mapped);

bindingContext.Result = ModelBindingResult.Success(details);

and our action now binds to the wrapper object:

[HttpGet]
public async Task<EntityResult> GetEntity([EntityLookup] EntityIdPair entityId)
{
    // ...
}

if we wanted, we could even go so far as to register our binder against our wrapper type, and then it could automatically apply to any use of that type, using a ModelBinderProvider or, since we're defining the bound type locally, you could just apply a [ModelBinder(BinderType = typeof(InternalIdLookupBinder))] to the EntityIdPair record.

Solution 3: Full-Fat Model Binding#

But what if we actually need additional information about the entity, that we would be able to access if we'd looked it up ourselves?
Well, why not just bind the entire entity object? This gives you the best of both worlds — both IDs, and (if you need it) any additional info available on the entity, all while moving the dependency out a layer, allowing our testing strategy to easier mock it out.

Well, here's the answer: ASP.NET makes it relatively easy to bind across names using the ModelBinderAttribute.Name property — for instance, to have a Entity entity parameter in code bind to an entityId: string query parameter.
However, Swashbuckle⁸ will fight tooth and nail to not generate the correct spec with that rebinding. Compared to the payoff, the effort to coerce it into doing what we desire in this situation seems almost intractable. I also only have so many hours in the day that I can spend on improving⁹ code architecture before my manager starts asking to see the Actual Work that I'm supposed to be doing instead.

BUT! Oh woe! It turns out that Swashbuckle also chokes on treating our types (whether Entity or EntityIdPair) as a single string ID, and decomposes it into its own properties as additional query parameters. Instead of a single entityId query parameter on our action, it instead reports two - ExternalId and InternalId. How heathen! So we would still require a (rather verbose) IOperationFilter to use any custom type here. Thanks, Swashbuckle.

Summary#

The problem I experienced was with "correctly"¹⁰ decomposing the database access in a way that was testable (both in our integration⁶ tests and also validation tests).

On the way, we also investigated possible improvements to the way we bind our models in ASP.NET, allowing more robust type validation and simpler boilerplate for lookups required by many endpoints.

After implementing this, I found myself foiled at the last hurdle by Swashbuckle's resistance to presenting a query parameter of a complex type as a simple, singular, string. This is a surmountable hurdle (with enough investment), but I consider the required IOperationFilter logic to be pretty arduous, messy, and (worst-of-all), complex.

I do think the core premise of the approaches outlined above are promising, and I can foresee scenarios where I may revisit these approaches and it would be worth the time and trade-offs to make this work, but it would take a different set of requirements for me to walk further down that path — e.g. an API with a much larger surface area that's doing this operation constantly, it requires the full-fat object upfront, or doing the same fetch for multiple, different, strongly-typed entities.

The best solution for our situation turned out to be my very first instinct to solving the initial problem — hide the database access behind a focused interface. This minimises magic, keeps the flow of logic obvious (and accessible for debugging), and is low-maintenance, while making mocking (the original problem) trivial.

Once we integrate the graph database using another testcontainer, the mock goes away, and nullifies the issue entirely — which raises the question, how much value does an abstraction buy you, if it doesn't change anything for anyone?

Footnotes#