#4 v0.4 index schema generation (was: Single-structure database schema)

The singlestructschema branch includes an experimental rewrite that should, in theory, drastically improve the ergonomics of using the library. This includes first-class support for indices, injective one-to-many entity relationships, and less complicated/externally-visible metaprogramming for query building.

Current status:

• rowid extraction and WithID<T>-esque wrapper
• Index schema generation
• AssocMap construction
• AssocMap lookups
• AssocMap insertions
• AssocMap additions, removals
• IDMap and AssocMap target deletion
• move ::entity and ::datum into ::schema
• ON DELETE CASCADE for assoc tables
• entity updates!
• improved entity insertion ergonomics
• foreign key constraints for EntityID fields
• [ ] specific ConstraintViolation error to allow for more specific application-side error recovery moved to #6

As it stands, the next major step (that affects all other parts of the rewrite) is to figure out what entity relationships to support. currently, the prototype supports only unidirectional one-to-non-exclusive-many relationships (which also includes unidirectional one-to-exclusive-many and one-to-one relationships), but expanding this to (optionally) bidirectional relationships is certainly desirable.

For a place where this is relevant, consider a simple schema:

#[derive(Entity)]
struct User {
    #[unique]
    name: String,
    groups: AssocMap<Group>,
}

#[derive(Entity)]
struct Group {
    #[unique]
    name: String,
    members: AssocMap<User>,
}

#[derive(Database)]
struct UserGroupDB {
    users: IDMap<User>,
    groups: IDMap<Group>,
}

While there are two separate AssocMaps here, ideally these would both be the same table under the hood with an additional index added to allow for reverse lookups.

The 'obvious' way to do this would be to introduce another type that AssocMap takes as a parameter, and using the same type in two AssocMaps means that they're intended to be inverse maps of each other. (an #[attribute] would also do the trick, similarly to how #[unique] currently functions.)

One reason I dislike this approach is the fact that it kicks the validity-checking down to runtime. One variant approach that has come to mind is to do something like the following:

struct UserGroupRelation;
impl microrm::schema::EntityRelation for UserGroupRelation {
    type Base = User;
    type Target = Group;
}

#[derive(Entity)]
struct User {
    #[unique]
    username: String,
    groups: AssocMapBase<UserGroupRelation>,
}

#[derive(Entity)]
struct Group {
    #[unique]
    groupname: String,
    members: AssocMapTarget<UserGroupRelation>,
}

This allows for the type checking to be done at compile time instead of runtime, which is superior. It does, however, require even more extra code than just a single tag type.

Upon some reflection, one thing the EntityRelation trait approach outlined above has in its favour is the ability to add additional relationship property tweaks. For example:

trait EntityRelation {
    type Base : Entity;
    type Target : Entity;

    /// if false, each Target has at most one Base related to it
    fn allow_overlapping() -> bool;

    /// allows the name of the sqlite table to be customized
    fn table_name_override() -> Option<&'static str> { None }
}

I think I prefer this approach over something like trying to match attribute strings between fields in different structs.

For the updated query interface that comes along with this, there are a few opportunities and options. This describes one of them.

Consider a relatively simple schema:

struct UserGroupRelation;
impl Relation for UserGroupRelation {
    type Domain: User;
    type Range: Group;
    const NAME: &'static str = "UserGroupRelation";
}

struct GroupRoleRelation;
impl Relation for GroupRoleRelation {
    type Domain: Group;
    type Range: Role;
    const NAME: &'static str = "GroupRoleRelation";
}

#[derive(Entity,Debug,Default)]
struct User {
    #[unique]
    name: String,

    groups: AssocDomain<UserGroupRelation>
}

#[derive(Entity,Debug,Default)]
struct Group {
    #[unique]
    name: String,

    users: AssocRange<UserGroupRelation>,
    roles: AssocDomain<GroupRoleRelation>,
}

#[derive(Entity,Debug,Default)]
struct Role {
    #[unique]
    name: String,

    groups: AssocRange<GroupRoleRelation>,
}

#[derive(Database)]
struct DB {
    users: IDMap<User>,
    groups: IDMap<Group>,
    roles: IDMap<Role>,
}

There are a few queries that we might want to run against this schema.

Simple authorization check

• Determine if a given username transitively possesses a specific role name

  fn auth_check_no_joins(db: &DB, username: &String, role: &String) -> DBResult<bool> {
      if let Some(user) = db.users.lookup_unique(username)? {
          for group in user.groups.get_all()? {
              if group.roles.lookup_unique(role)?.is_some() {
                  return Ok(true)
              }
          }
      }
      Ok(false)
  }
  
  fn auth_check_with_joins(db: &DB, username: &String, role: &String) -> DBResult<bool> {
      Ok(db
          .users
          .unique(username)
          .join(User::Groups)
          .join(Group::Roles)
          .unique(role)
          .count()? > 0)
  }
  

Check which users are authorized

• Look up all users authorized in a single role

  fn all_authorized_with_joins(db: &DB, role: &String) -> DBResult<Vec<IDWrap<User>>> {
      Ok(db
          .roles
          .unique(role)
          .groups
          .join(Group::Users)
          .get())
  }
  

Significant progress on new query interface implemented in ab06e1f3c2. With this, the rewrite is nearing completion, pending some nontrivial dogfooding.

regarding entity updates, ideally something like the following would be available:

let mut entity = db.entitymap.unique(somekey).get()?.unwrap();
entity.field += 1;
entity.sync(); # update changes back into database

However, in order to implement sync(), a reference to the database would need to be held by the entity, which is not feasible with the derive proc-macro for Entity. A struct attribute could fix that, as attribute proc-macros can modify the fields of structs.

One solution here would be to toss the database reference into the IDWrap struct (which would possibly necessitate its renaming), which also gives way to a relatively nice insertion-and-return API:

let mut e : IDWrap<Entity> = db.entitymap.insert_and_return(Entity { field: 42 })?;

Perhaps IDWrap could be renamed to Stored, Remote, or something similar? Then it's clear that it's not just holding an ID, and I haven't liked the WithID or IDWrap names very much...

The synchronization could even happen automatically if e.g. Remote had a Drop impl that could compare a "last-known" hash against drop-time hash, with manual sync() calls available for if you want to keep a reference around but also synchronize with the datastore.

The last major item on the TODO list is index schema generation, which I'm now renaming the issue to reflect.

As of e3e2259b7f, index schemata are generated correctly (though currently all indices are currently assumed to be unique) and can be used for querying (with type safety, even!).