LiteDB And Unity

Posted On: 2018-12-10

By Mark

I recently used LiteDB to save data for my current prototype, "Bypass" , and, although I eventually removed it from the prototype, I expect that the skills and knowledge I have gained will be useful the next time I use LiteDB. As a complete beginner with LiteDB (and generally inexperienced with NoSql), I had quite a bit to learn (often through my own mistakes) and I thought I should document some of it here, in case it is useful for anyone else.

Why LiteDB?

As a bit of background, I'll go into why I thought LiteDB would be the right fit for Bypass. The game allowed players to move objects around a room to solve puzzles, as well as jump inside the various objects to move into new rooms. Since the position (and rotation) of objects are essential for the puzzle-solving part of the game, I needed to preserve that information for a large amount of objects across a large number of rooms. (As an aside, the way the code was designed should have allowed for an infinite number of rooms, but building a system to procedurally generate an infinite number of fun puzzles was never in scope for the prototype.) Ordinarily I use json files to store data (since they are easy to use and human readable) but I knew that wouldn't scale well if I had a large number of rooms. On the other hand, the game is single-player/offline only, so a traditional database (hosted on a server somewhere) would have a large amount of extra complexity and overhead. Those constraints helped me narrow my choice down to LiteDB or SqLite, and I decided to start by trying LiteDB. (This was mostly personal preference - I personally conceptualize affordances on an per-instance basis, rather than a per-type basis, so I generally find it more intuitive to work with an unstructured data store.)

Getting started

Adding LiteDB to my Unity project was about as painless as any other third-party dependency. First, I used the NuGet package manager to install the LiteDB package. Then I had to work around Unity not supporting NuGet by copying the newly referenced dlls (and xml documentation) into my Assets folder. Finally, I closed the solution and re-opened it from Unity (to make sure I have Unity's auto-generated project file, rather than the one that references the NuGet package folder.)

Once I had it referenced properly I was able to use it right away. LiteDB has the (imho very reasonable) convention of automatically creating the database file and/or collection if they don't already exist, so I could immediately write a simple hello-world-esque command for it:

using(var db = new LiteDatabase(new ConnectionString(Path.Combine(UnityEngine.Application.persistentDataPath, "test.db")))
{
    var collection = db.GetCollection<RoomData>();
    collection.Insert(new RoomData{ DebugMessage: "Hello World!"});
    var room = collection.FindAll().FirstOrDefault();
    UnityEngine.Debug.Log(room?.Message ?? "Expected message is missing!");
 }

Note: this sample is simpler than what I actually wrote at the time - for example, I didn't realize that GetCollection<T> automatically uses nameof(typeof(T)) for the name - so I kept writing it by hand until I eventually found that out.

First Lesson: Use an Id

This segues nicely into the first lesson of using LiteDB: anything that is added directly to a collection (an instance of RoomData in this case) must have an Id property. (You don't technically have to call it "Id", you can instead use an attribute to define any particular property, but I prefer convention over configuration, and "Id" is a pretty reasonable convention.) If LiteDB can't find an Id, it won't let you insert the record. Additionally, if your Id is nullable, it won't auto-increment (instead it throws errors about null not being allowed.) The nullable limitation is a bit of an inconvenience, so while I am just accepting it for now, it's on my list for things to revisit to try to find a better approach for (the null-coalescing ?? and null-conditional ?./ ?[]operators make defensive coding with nullable objects painless, so I try to make everything nullable whenever I can.)

Second Lesson: Object Mapping

After sorting out the Id issues, I learned a second lesson about working with LiteDB and Unity: LiteDB can't serialize Unity objects without custom configuration. This limitation appears to be a product of Unity's programming style: in Unity, it is expected that data should be stored in fields, and those fields are (usually) modifiable in the Unity Editor. All of Unity's built-in data types (such as Vectors) use this same convention of preferring to store data in fields. In contrast, LiteDB (by default) assumes that fields are not supposed to be serialized, and instead only serializes properties. Additionally, Unity makes heavy use of read-only properties, such as Vector3.zero being a read-only property on Vector3 instances. LiteDB, automatically tries to serialize these read-only properties, which is both unwanted and often impossible (since many result in infinite loops.) The solution to this is to write custom object mapping to convert the object into a BsonValue, the data type that LiteDB uses for data storage. For example, this is the type registration I used for a Vector3:

mapper.RegisterType(
    vector => new BsonArray(new BsonValue[] { vector.x, vector.y, vector.z }),
    value => new Vector3((float)value.AsArray[0].AsDouble, (float)value.AsArray[1].AsDouble, (float)value.AsArray[2].AsDouble)
);

Although I eventually switched over to using System.Numerics.Vectors for my Vector3 class, the same object mapping skills proved useful there as well. (This is another thing on my list of things to look into later: I would expect the Numerics vectors to work without a mapping, since they use public properties to store data, and LiteDB is supposed to work automatically when that is the case.)

Third Lesson: Unlearn Relational Database Habits

The third lesson I learned with DBLite was that I would paint myself into a corner if I kept thinking relationally. This is best illustrated with an example (specifically what I experienced that taught me this lesson): Since each room is filled with RoomItems, I added a List<RoomItem> to the RoomData object. This immediately worked and gave me the expected results, but, since I still thought about problems relationally, I assumed that I would need an Id on each RoomItem so that I could individually update the records. That led me to creating a separate collection and using DBRef to connect the two collections together. The result, unfortunately, was broken code and workarounds. Using DBRef caused the code to require that I specifically include the relevant properties from the RoomItem. Any property that was not included would be left at the default value (so at first I was working with a list of empty RoomItem objects.) IncludeAll() proved to be inadequate, as it only populated the Id for each record. After coding numerous workarounds (including manually querying for all the RoomItems and merging them into the existing collection) I finally realized that I didn't need to update RoomItems by Id at all: I can (and should) persist a reference to the RoomItem on the GameObject it applies to (so that I can make changes to its properties), and then bulk update all the items in the room by performing an update using the original RoomData object (which should reflect any modifications made using object references).

Applying this pattern (GameObjects containing object references to parts of the object I will later save) will require a bit more discipline and caution than I ordinarily use. My typical pattern treats objects as externally immutable (for example, a GameObject would create and maintain its own copy of the data, rather than modifying a reference it received from somewhere else), since that allows me to reap many of the simplicity benefits of associated with functional programming. As a result, I don't usually consider whether modifying data at any given moment will have consequences beyond the GameObject I am currently working on. Considering all of this, I expect that, at some point in the future, I will find myself debugging some kind of race condition, on account of this change to retain and reuse object reference.

Fourth Lesson: Use a Separate Library to Write Queries

While there are many ways to interpret the hours I spent trying to get my first query to work, I think the most important thing is how much faster I was able to iterate once I moved out of Unity and into class libraries and unit tests. Essentially, the problem I was facing was that I was working with outdated information (which suggested using MongoDB syntax) and that I really needed to read and apply the documentation for expressions. Working within Unity, however, prevented me from discovering this because the iteration loop was too slow: starting and stopping the game between each attempt prevented me from seeing the bigger picture: that none of my attempts would work because I was referencing the wrong documentation. Fortunately, I eventually moved the code I was working on into a class library, and wrote some quick unit tests to describe what I was expecting. Once the tests failed, I promptly attached the debugger and used the immediate window to repeatedly try different queries. After several dozen attempts (in a matter of seconds) I was finally on the right track, since I realized that it wasn't a logic issue with my query but rather that there was something more fundamental that was wrong. (It is worth mentioning - the immediate window is technically available when debugging Unity, but it is very limited. I've been able to use it to inspect existing variables, but whenever I try to code directly in it, such as creating new Query objects to use on LiteDB, it just throws cryptic errors.) Once I finally used the correct query syntax, the query I needed was rather simple:

Query.EQ("RoomItems[*].DiveTargetRoomId", currentRoomId);
//Or a more future-proof version:
Query.EQ($"{nameof(RoomData.RoomItems)}[*].{nameof(RoomItem.DiveTargetRoomId)}", currentRoomId);

As an added bonus, using a unit test means that I now have an automated test, so even if I refactor the code I know this query will continue to behave correctly. While I generally don't write unit tests for my prototypes, I expect developing this habit will be useful once I am using LiteDB in a larger project.

Conclusion

After learning (and applying) all of these lessons, players of Bypass could move into and out of rooms, and everything remained exactly where they left it. Even better, when players exited the game and came back later, everything remained exactly where they left it. Although I eventually abandoned this particular use of LiteDB (for gameplay reasons) I am still glad that I learned how to use it, and I fully expect that I will use it again in the future (being able to save the position of items across play sessions will likely open up a lot of interesting game design.)