Posted On: 2020-11-23
As mentioned in the previous part, today's post is about a kind of testing that, until recently, I didn't even know existed. It wasn't until watching a talk from the folks at Unity that I became aware of Content Validation as a third kind of automated testing. Today's post will dive into that idea: exploring what Automated Content Validation is, why I've never heard of it before, and, most importantly, why it's so useful in Unity.
For the purposes of this post, Content Validation is the act of validating that the content created by content designers conforms with the expectations of the systems designers. To unpack that idea a bit: when systems are created, they have a number of assumptions and limitations built in. As a simple example, consider a health system: characters have a certain amount of health, losing health lowers that amount, and reaching zero causes the character to die. Such a system has a number of assumptions baked in - for example, health cannot be negative*. Designers who create content for those systems (such as enemy and encounter designers) need to work within those constraints - otherwise they may design content that behaves incorrectly, such as creating an impossible situation or crashing the program. Ideally, content designers will always remain within the constraints, but, in sufficiently large and rapidly changing projects, mistakes are inevitable. To prevent such mistakes from reaching users, one can review the content to make sure it does not violate any constraint - but such efforts are tedious and time-consuming, if one does not use automation to assist.
Savvy software developers may have noticed a flaw in the example I provided earlier: to keep content designers from using negative health values, system designers can (and arguably should) use an unsigned integer to store health values. Unsigned integers, by definition, cannot contain a negative number - so that constraint is guaranteed to never be violated*. Alternatively, system designers could enforce the constraints within the implementation of the systems themselves - for example, by coding it to change all negative numbers to be zero before they are used.
This solution - designing the systems themselves to enforce their own constraints - was really the only approach available in my previous line of work. We didn't have any content per-se - the closest thing we had was that we supported domain experts configuring or modifying the behavior of the systems, but such changes occurred without developers' direct involvement. As such, we treated such "content" as user input - subject to the same robust system needs and validation concerns as any other user-facing system. In some sense, you could say that I did not know about Content Validation since I hadn't really interacted with any "content" before using Unity*.
In contrast with my prior experiences, Unity makes prolific use of content. From objects in a scene to prefabs to scriptable objects to asset import settings - the vast majority of what makes up a user's experience is pre-authored content*. While in many cases it is possible to use system constraints to enforce the correct configuration of content, doing so may come with the cost of increased development time or decreased performance. Perhaps most important of all, however, is many such constraints are beyond the scope of what Unity's inspector supports** - thereby commiting system developers not just to maintaining the systems themselves but also to maintaining the tools content developers use to interact with them.
Fortunately, automated content validation is both a lower cost and better supported approach to dealing with non-conformant content. While not as nice has having the issue prevented outright, such tests can quickly detect and flag content that needs revision, enabling content authors to resolve the issue well before building or releasing the project. What's more, the tests can generally be quite focused, keeping the maintainence burden for developers light*.
For an example of how valuable content tests can be in Unity, I'll draw on my personal experience with it. In my current project, I use a "faction" system, which describes the relationships between various actors. Two actors that are friendly to one another may belong to the same faction, or they might belong to factions that have a friendly "faction relationship". Conversely, characters that are hostile to one another would belong to factions that have an aggressive relationship. The relationships between factions are static content, authored in-editor (using scriptable objects.)
All of the systems that rely on the faction system* assume that every faction has a valid entry for its relationship with another faction. Unfortunately, since I don't have this constraint enforced through the faction system itself**, I've already encountered issues where code misbehaved due to a missing faction relationship. When I became aware of automated content validation, I immediately thought of using it to validate that the faction relationships were setup correctly.
Implementing the tests turned out to be far simpler than I'd expected. Designed as editor-only unit tests, I could easily query the asset database for every faction (and its associated relationships.) From there, I could easily loop through, validating that every faction had relationship records for every other faction.
For those that are curious, here's the exact code (minus my AssetDatabaseUtilities helper, the source of which is documented in an earlier blog post.)
[Test]
public void AllFactionsHaveRelationsToEachother()
{
var factionRecords = AssetDatabaseUtilities.LoadAllInFolder<ScriptableFaction>("Assets/").ToList();
foreach (Faction faction in Enum.GetValues(typeof(Faction)))
{
var matches = factionRecords.Where(x => x.Faction == faction);
//All non-null factions should have exactly 1 record
var expectedMatches = faction != Faction.Null ? 1 : 0;
Assert.AreEqual(expectedMatches, matches.Count(),
$"Incorrect number of faction records for {faction}: expected {expectedMatches}, but found {matches.Count()}.");
}
foreach (var searchedFaction in factionRecords)
{
foreach (var item in factionRecords)
{
var matchingFactionCount = item.Relations.Count(x => x.Key == searchedFaction.Faction);
Assert.AreEqual(1, matchingFactionCount,
$"{item.name} has an incorrect number of relations with {searchedFaction.name}: expected 1, but found {matchingFactionCount}");
}
}
}This concludes my three-part series on automated testing in Unity. As you can see, each of the three kinds of tests (unit, integration, and content) brings its own unique advantages. Additionally, while unit tests may be more difficult to use in Unity, the improved support for integration and content testing can help fill in many of the gaps that creates. While I haven't covered the technical details of how to implement automated tests in Unity*, it is my hope that what I have provided has been useful, both for understanding automated testing in general and specifically how it differs in Unity (compared to, say, enterprise software.)