Directory-Based / Dynamic sharding

There are four common types of sharding strategies.

1. Horizontal or range based.
2. Vertical.
3. Key-based (Algorithmic).
4. Directory-based (Dynamic).

In this blog, we will only discuss directory-based sharding a.k.a dynamic sharding.

Directory based shard partitioning involves placing a lookup/locator service in front of the sharded databases. The lookup service knows the current partitioning scheme and keeps a map of each entity and which database shard it is stored on. The lookup service is usually implemented as a web service.

To read and write data, clients need to consult the locator service first. Operation by primary key becomes fairly trivial. Other queries also become efficient depending on the structure of locators.

But in this case, we have kept it inside our demo project to make things easier to grasp. We are using:

1. ASPNET CORE 6.
2. EF CORE 6.
3. IN MEMORY DATABASE / SQL SERVER / POSTGRESQL / MYSQL

Let's begin.

At first, we will have to store a map for our configurations. Like which data needs to go onto which shard. We are naming it Tenant. For simplicity, we are keeping it inside of a json file. In our case we are storing it inside of appsettings.json file.

Sharding configuration in json will be like this:

"ShardingConfigurations": [
    {
      "TenantIds": [ "", "" ],
      "DbConnectionString": ""
    },
    {
      "TenantIds": [ "", "" ],
      "DbConnectionString": ""
    }
]

this is a very basic way to store multi-tenant configurations. It can vary from usage to usage. Mainly we are storing two things.

1. An identifier. (can be multiple)
2. Database connection information.

Let's translate this json configuration inside of a C# class.

public class Tenant
{
    public string[] TenantIds { get; set; }
    public string DbConnectionString { get; set; }
}

To serialize it from json we have to create an option class.

public class TenantOptions
{
    public List<Tenant> TenantConfigurations { get; set; }
}

Now it's time for our locator service. For the locator service to work functionally it will need the configurations of TenantOptions. Here comes the LookupService.

public class LookupService
{
  private readonly TenantOptions _options;
  public LookupService(TenantOptions options)
  {
      _options = options;
  }
}

Now it's time to get and set the method for the lookup service to work.

private string _tenantId;

public Tenant GetTenant()
{
    if (string.IsNullOrEmpty(_tenantId)) return null;
    if (_options == null) return null;

    Tenant appropiateTenant = _options.TenantConfigurations
        .FirstOrDefault(x => x.TenantIds.Any(x => x.Equals(_tenantId)));
    return appropiateTenant;
}

public void SetTenant(string tenantId)
{
    if (string.IsNullOrEmpty(tenantId)) return;
    _tenantId = tenantId;
}

SetTenant has to be called before any database operation so that in the database configuration layer a GetTenant call will be sufficient. We will see this in action later.

We have wrapped this lookupservice class with an interface so that we can use it onwards.

public interface ILookupService
{
    Tenant GetTenant();
    void SetTenant(string tenantId);
}

In the DatabaseContext class where we define our dbsets, we need to inject this lookup service so that we can access the tenant configurations.

public class NormalDbContext : DbContext
{
    private readonly ILookupService _lookupService;

    public NormalDbContext(
        DbContextOptions<NormalDbContext> options,
        ILookupService lookupService) : base(options)
    {
        _lookupService = lookupService;
    }
}

This DbContext class has a method called OnConfiguring for any configuration. We will override this method and fit it in our case.

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    Tenant tenant = _lookupService.GetTenant();
    if (tenant == null || string.IsNullOrEmpty(tenant.DbConnectionString)) return;

    optionsBuilder.UseInMemoryDatabase(tenant.DbConnectionString);
}

See, here are calling GetTenant method to get a configuration so that we can initialize our database connection accordingly.

Let's inject it into the program.cs class to our newly created aspnetcore webapi project.

var options = builder.Configuration.Get<TenantOptions>();

builder.Services.AddSingleton(options);
builder.Services.AddDbContext<NormalDbContext>();
builder.Services.AddScoped<ILookupService, LookupService>();

Horray! Now just inject the lookupservice into your repositories or broker or wherever you need to connect to a database. set it and get it from your desired repository or service or broker.

public class Controller {  

    private readonly ILookupService _lookupService;
Controller(ILookupService lookup) => _lookupService = lookup;
    public IActionResult Get() { 

       _lookupService.SetTenant("your identification");
        // call the database (service/ repository)
     }
}

However, I have created a demo on ** github.

Thanks for reading 👏. Happy sharding.

N.B: There is an infinite way of doing dynamic-based sharding. Some people will create a web service for LookupService itself to separate the responsibility. It will help along the way if the project gets big. It's just a starter version.