#4 v0.4 index schema generation (was: Single-structure database schema)

Închis
deschis 8 luni în urmă cu kestrel · 7 comentarii

The singlestructschema branch includes an experimental rewrite that should, in theory, drastically improve the ergonomics of using the library. This includes first-class support for indices, injective one-to-many entity relationships, and less complicated/externally-visible metaprogramming for query building.

Current status:

  • rowid extraction and WithID<T>-esque wrapper
  • Index schema generation
  • AssocMap construction
  • AssocMap lookups
  • AssocMap insertions
  • AssocMap additions, removals
  • IDMap and AssocMap target deletion
  • move ::entity and ::datum into ::schema
  • ON DELETE CASCADE for assoc tables
  • entity updates!
  • improved entity insertion ergonomics
  • foreign key constraints for EntityID fields
  • [ ] specific ConstraintViolation error to allow for more specific application-side error recovery moved to #6
The `singlestructschema` branch includes an experimental rewrite that should, in theory, drastically improve the ergonomics of using the library. This includes first-class support for indices, injective one-to-many entity relationships, and less complicated/externally-visible metaprogramming for query building. Current status: - [x] rowid extraction and `WithID<T>`-esque wrapper - [x] ~~Index schema generation~~ - [x] ~~`AssocMap` construction~~ - [x] ~~`AssocMap` lookups~~ - [x] ~~`AssocMap` insertions~~ - [x] ~~`AssocMap` additions, removals~~ - [x] ~~`IDMap` and `AssocMap` target deletion~~ - [x] ~~move ::entity and ::datum into ::schema~~ - [x] ~~`ON DELETE CASCADE` for assoc tables~~ - [x] ~~entity updates!~~ - [x] ~~improved entity insertion ergonomics~~ - [x] ~~foreign key constraints for `EntityID` fields~~ - ~~[ ] specific `ConstraintViolation` error to allow for more specific application-side error recovery~~ moved to #6
kestrel a comentat 8 luni în urmă
Proprietar

As it stands, the next major step (that affects all other parts of the rewrite) is to figure out what entity relationships to support. currently, the prototype supports only unidirectional one-to-non-exclusive-many relationships (which also includes unidirectional one-to-exclusive-many and one-to-one relationships), but expanding this to (optionally) bidirectional relationships is certainly desirable.

For a place where this is relevant, consider a simple schema:

#[derive(Entity)]
struct User {
    #[unique]
    name: String,
    groups: AssocMap<Group>,
}

#[derive(Entity)]
struct Group {
    #[unique]
    name: String,
    members: AssocMap<User>,
}

#[derive(Database)]
struct UserGroupDB {
    users: IDMap<User>,
    groups: IDMap<Group>,
}

While there are two separate AssocMaps here, ideally these would both be the same table under the hood with an additional index added to allow for reverse lookups.

The 'obvious' way to do this would be to introduce another type that AssocMap takes as a parameter, and using the same type in two AssocMaps means that they're intended to be inverse maps of each other. (an #[attribute] would also do the trick, similarly to how #[unique] currently functions.)

One reason I dislike this approach is the fact that it kicks the validity-checking down to runtime. One variant approach that has come to mind is to do something like the following:

struct UserGroupRelation;
impl microrm::schema::EntityRelation for UserGroupRelation {
    type Base = User;
    type Target = Group;
}

#[derive(Entity)]
struct User {
    #[unique]
    username: String,
    groups: AssocMapBase<UserGroupRelation>,
}

#[derive(Entity)]
struct Group {
    #[unique]
    groupname: String,
    members: AssocMapTarget<UserGroupRelation>,
}

This allows for the type checking to be done at compile time instead of runtime, which is superior. It does, however, require even more extra code than just a single tag type.

As it stands, the next major step (that affects all other parts of the rewrite) is to figure out what entity relationships to support. currently, the prototype supports only unidirectional one-to-non-exclusive-many relationships (which also includes unidirectional one-to-exclusive-many and one-to-one relationships), but expanding this to (optionally) bidirectional relationships is certainly desirable. For a place where this is relevant, consider a simple schema: ```rust #[derive(Entity)] struct User { #[unique] name: String, groups: AssocMap<Group>, } #[derive(Entity)] struct Group { #[unique] name: String, members: AssocMap<User>, } #[derive(Database)] struct UserGroupDB { users: IDMap<User>, groups: IDMap<Group>, } ``` While there are two separate `AssocMap`s here, ideally these would both be the same table under the hood with an additional index added to allow for reverse lookups. The 'obvious' way to do this would be to introduce another type that `AssocMap` takes as a parameter, and using the same type in two `AssocMap`s means that they're intended to be inverse maps of each other. (an `#[attribute]` would also do the trick, similarly to how `#[unique]` currently functions.) One reason I dislike this approach is the fact that it kicks the validity-checking down to runtime. One variant approach that has come to mind is to do something like the following: ```rust struct UserGroupRelation; impl microrm::schema::EntityRelation for UserGroupRelation { type Base = User; type Target = Group; } #[derive(Entity)] struct User { #[unique] username: String, groups: AssocMapBase<UserGroupRelation>, } #[derive(Entity)] struct Group { #[unique] groupname: String, members: AssocMapTarget<UserGroupRelation>, } ``` This allows for the type checking to be done at compile time instead of runtime, which is superior. It does, however, require even more extra code than just a single tag type.
kestrel a comentat 8 luni în urmă
Proprietar

Upon some reflection, one thing the EntityRelation trait approach outlined above has in its favour is the ability to add additional relationship property tweaks. For example:

trait EntityRelation {
    type Base : Entity;
    type Target : Entity;

    /// if false, each Target has at most one Base related to it
    fn allow_overlapping() -> bool;

    /// allows the name of the sqlite table to be customized
    fn table_name_override() -> Option<&'static str> { None }
}

I think I prefer this approach over something like trying to match attribute strings between fields in different structs.

Upon some reflection, one thing the `EntityRelation` trait approach outlined above has in its favour is the ability to add additional relationship property tweaks. For example: ```rust trait EntityRelation { type Base : Entity; type Target : Entity; /// if false, each Target has at most one Base related to it fn allow_overlapping() -> bool; /// allows the name of the sqlite table to be customized fn table_name_override() -> Option<&'static str> { None } } ``` I think I prefer this approach over something like trying to match attribute strings between fields in different structs.
kestrel a comentat 8 luni în urmă
Proprietar

For the updated query interface that comes along with this, there are a few opportunities and options. This describes one of them.

Consider a relatively simple schema:


struct UserGroupRelation;
impl Relation for UserGroupRelation {
    type Domain: User;
    type Range: Group;
    const NAME: &'static str = "UserGroupRelation";
}

struct GroupRoleRelation;
impl Relation for GroupRoleRelation {
    type Domain: Group;
    type Range: Role;
    const NAME: &'static str = "GroupRoleRelation";
}

#[derive(Entity,Debug,Default)]
struct User {
    #[unique]
    name: String,

    groups: AssocDomain<UserGroupRelation>
}

#[derive(Entity,Debug,Default)]
struct Group {
    #[unique]
    name: String,

    users: AssocRange<UserGroupRelation>,
    roles: AssocDomain<GroupRoleRelation>,
}

#[derive(Entity,Debug,Default)]
struct Role {
    #[unique]
    name: String,

    groups: AssocRange<GroupRoleRelation>,
}

#[derive(Database)]
struct DB {
    users: IDMap<User>,
    groups: IDMap<Group>,
    roles: IDMap<Role>,
}

There are a few queries that we might want to run against this schema.

Simple authorization check

  • Determine if a given username transitively possesses a specific role name

    fn auth_check_no_joins(db: &DB, username: &String, role: &String) -> DBResult<bool> {
        if let Some(user) = db.users.lookup_unique(username)? {
            for group in user.groups.get_all()? {
                if group.roles.lookup_unique(role)?.is_some() {
                    return Ok(true)
                }
            }
        }
        Ok(false)
    }
    
    fn auth_check_with_joins(db: &DB, username: &String, role: &String) -> DBResult<bool> {
        Ok(db
            .users
            .unique(username)
            .join(User::Groups)
            .join(Group::Roles)
            .unique(role)
            .count()? > 0)
    }
    

Check which users are authorized

  • Look up all users authorized in a single role

    fn all_authorized_with_joins(db: &DB, role: &String) -> DBResult<Vec<IDWrap<User>>> {
        Ok(db
            .roles
            .unique(role)
            .groups
            .join(Group::Users)
            .get())
    }
    
For the updated query interface that comes along with this, there are a few opportunities and options. This describes one of them. Consider a relatively simple schema: ```rust struct UserGroupRelation; impl Relation for UserGroupRelation { type Domain: User; type Range: Group; const NAME: &'static str = "UserGroupRelation"; } struct GroupRoleRelation; impl Relation for GroupRoleRelation { type Domain: Group; type Range: Role; const NAME: &'static str = "GroupRoleRelation"; } #[derive(Entity,Debug,Default)] struct User { #[unique] name: String, groups: AssocDomain<UserGroupRelation> } #[derive(Entity,Debug,Default)] struct Group { #[unique] name: String, users: AssocRange<UserGroupRelation>, roles: AssocDomain<GroupRoleRelation>, } #[derive(Entity,Debug,Default)] struct Role { #[unique] name: String, groups: AssocRange<GroupRoleRelation>, } #[derive(Database)] struct DB { users: IDMap<User>, groups: IDMap<Group>, roles: IDMap<Role>, } ``` There are a few queries that we might want to run against this schema. ### Simple authorization check - Determine if a given username transitively possesses a specific role name ```rust fn auth_check_no_joins(db: &DB, username: &String, role: &String) -> DBResult<bool> { if let Some(user) = db.users.lookup_unique(username)? { for group in user.groups.get_all()? { if group.roles.lookup_unique(role)?.is_some() { return Ok(true) } } } Ok(false) } fn auth_check_with_joins(db: &DB, username: &String, role: &String) -> DBResult<bool> { Ok(db .users .unique(username) .join(User::Groups) .join(Group::Roles) .unique(role) .count()? > 0) } ``` ### Check which users are authorized - Look up all users authorized in a single role ```rust fn all_authorized_with_joins(db: &DB, role: &String) -> DBResult<Vec<IDWrap<User>>> { Ok(db .roles .unique(role) .groups .join(Group::Users) .get()) } ```
kestrel a comentat 8 luni în urmă
Proprietar

Significant progress on new query interface implemented in ab06e1f3c2. With this, the rewrite is nearing completion, pending some nontrivial dogfooding.

Significant progress on new query interface implemented in ab06e1f3c2. With this, the rewrite is nearing completion, pending some nontrivial dogfooding.
kestrel a comentat 8 luni în urmă
Proprietar

regarding entity updates, ideally something like the following would be available:


let mut entity = db.entitymap.unique(somekey).get()?.unwrap();
entity.field += 1;
entity.sync(); # update changes back into database

However, in order to implement sync(), a reference to the database would need to be held by the entity, which is not feasible with the derive proc-macro for Entity. A struct attribute could fix that, as attribute proc-macros can modify the fields of structs.

One solution here would be to toss the database reference into the IDWrap struct (which would possibly necessitate its renaming), which also gives way to a relatively nice insertion-and-return API:

let mut e : IDWrap<Entity> = db.entitymap.insert_and_return(Entity { field: 42 })?;

Perhaps IDWrap could be renamed to Stored, Remote, or something similar? Then it's clear that it's not just holding an ID, and I haven't liked the WithID or IDWrap names very much...

The synchronization could even happen automatically if e.g. Remote had a Drop impl that could compare a "last-known" hash against drop-time hash, with manual sync() calls available for if you want to keep a reference around but also synchronize with the datastore.

regarding entity updates, ideally something like the following would be available: ```rust let mut entity = db.entitymap.unique(somekey).get()?.unwrap(); entity.field += 1; entity.sync(); # update changes back into database ``` However, in order to implement `sync()`, a reference to the database would need to be held by the entity, which is not feasible with the derive proc-macro for `Entity`. A struct attribute _could_ fix that, as attribute proc-macros can modify the fields of structs. One solution here would be to toss the database reference into the `IDWrap` struct (which would possibly necessitate its renaming), which _also_ gives way to a relatively nice insertion-and-return API: ```rust let mut e : IDWrap<Entity> = db.entitymap.insert_and_return(Entity { field: 42 })?; ``` Perhaps `IDWrap` could be renamed to `Stored`, `Remote`, or something similar? Then it's clear that it's not just holding an ID, and I haven't liked the `WithID` or `IDWrap` names very much... The synchronization could even happen automatically if e.g. `Remote` had a `Drop` impl that could compare a "last-known" hash against drop-time hash, with manual `sync()` calls available for if you want to keep a reference around but also synchronize with the datastore.
kestrel a comentat 6 luni în urmă
Proprietar

The last major item on the TODO list is index schema generation, which I'm now renaming the issue to reflect.

The last major item on the TODO list is index schema generation, which I'm now renaming the issue to reflect.
kestrel a comentat 6 luni în urmă
Proprietar

As of e3e2259b7f, index schemata are generated correctly (though currently all indices are currently assumed to be unique) and can be used for querying (with type safety, even!).

As of e3e2259b7f3f22554410cf8b7a3dc0c4b09b1bdb, index schemata are generated correctly (though currently all indices are currently assumed to be unique) and can be used for querying (with type safety, even!).
Autentificați-vă pentru a vă alătura acestei conversații.
Fără etichetă
Nu există Milestone
Fără destinatar
1 Participanți
Se încarcă...
Anulare
Salvează
Nu există încă niciun conținut.