Hello Oryers! So, yesterday I did something dumb a...
# ory-network
c
Hello Oryers! So, yesterday I did something dumb and nuked your keto with requests (sorry, my b, I should know better than to run x sequencial requests in parallel with the only limit being how fast the CPU of the server is 🙃 ). However, this resulted in a net negative 1500 entries in Keto, which is kinda weird, since none of the requests were deletes - although some of them were patches, but with both an insert and a remove. Anyways, the question is not about why this nuke happened, I totally have myself to blame for basically ddos'ing you. However! Is there a way to rollback to a given point in time in Keto? Seeing as keto uses cockroach db (i think), which is built more like a ledger / append only db, I feel like saying "reset to the state you were in 5 minutes ago" is kind of a viable option, right?
s
I will track this as a feature request. We could maybe allow to pass a timestamp to run queries at using
AS OF SYSTEM TIME
Full backups and rollbacks are probably a bigger story though.
🙌 1
c
That's fair. If we could use AS OF SYSTEM TIME, we could get the state as it was x minutes ago, and then recreate it using patches - which serves the purpose anyways.
On that note, what is the recommended max size of a patch (or even a page)?
s
we saw 1000 to still work well, but you can just try a bit
btw the total count could have been reduced in case you had duplicates that were removed
in that case all duplicates are deleted
c
Aight, thanks dude!
Oh yeah, that's true, but the count should be around 4600 total, got reduced to like 3200
Every time we change an entry, we do this:
Copy code
userRoles, err := oc.ListUserPermissions(ctx, &client.Session{
		Identity: client.Identity{
			Id: userId,
		},
	}, tenant, api.RolesNamespace)
	if err != nil {
		return nil, fmt.Errorf("%w '%s': %s", ErrGetExistingUserRoles, userId, err)
	}
	patch := make([]client.RelationshipPatch, len(userRoles)+1)
	sid := ory.EnsureTenantPrefix(tenant, userId)
	if !strings.HasPrefix(role, gorm.GlobalTenant+".") {
		role = ory.EnsureTenantPrefix(tenant, role)
	}

	// Exit early if we try to set the same role the user already has
	if len(userRoles) == 1 {
		rel := userRoles[0]
		if rel.Object == role && *rel.SubjectId == sid {
			return &rel, nil
		}
	}
	for i, userRole := range userRoles {
		relation := userRole
		patch[i] = client.RelationshipPatch{
			Action:        &PatchOpDelete,
			RelationTuple: &relation,
		}
		logger.Infof("Revoking role '%s' from user '%s'", userRole.GetObject(), userRole.GetSubjectId())
	}
	relationship := &client.Relationship{
		Namespace: api.RolesNamespace,
		Object:    role,
		Relation:  api.RoleMemberRelation,
		SubjectId: &sid,
	}
	patch[len(userRoles)] = client.RelationshipPatch{
		Action:        &PatchOpInsert,
		RelationTuple: relationship,
	}
	logger.Infof("Granting role '%s' to user '%s'", role, sid)

	if len(patch) > 0 {
		if err := pc.PatchPermissions(ctx, patch); err != nil {
			return nil, fmt.Errorf("%w: %s", ErrPatchPermissions, err)
		}
	}
Which means we always patch rather than calling the DeleteRelationships endpoint. I thought the patch would be atomic, but something tells me it may not be?
s
it definitely is atomic
c
Okok, that's cool
s
we recently used it to migrate all permissions on Ory network (for the console & API keys), and no one noticed because it allowed for a zero downtime migration
c
Yeah, that's the general idea behind why we want to use it too, when changing users roles
In that case I have no idea why some entries would go missing, but I'm running a script now to get it back in sync. I won't do it in parallel this time, I swear 😇
s
could it be that sometimes the tuples are inserted more than once initially? in case you do have duplicates, all of them are deleted, so a patch with one insert and one delete can still result in a net -1 count if initially the deleted tuple was there twice
c
Yeah, that would make sense, however we have been mega careful not to create duplicates, and I have never seen one either
What I fear is that we may have passed in a blank field somewhere, and deleted like everything in an entire namespace - this has happened before