When trying to build a relay compliant API in golang for one of my recent projects, I found that one of the most complicated parts to construct was the relay-compliant connections for listing connected objects with a backend database using the GORM ORM. For simple cases the github.com/graphql-go/relay library is fantastic. It provides methods which greatly simplify the creation of connections.
The below example is taken from the examples section of the library repository:
shipConnectionDefinition := relay.ConnectionDefinitions(relay.ConnectionConfig{
Name: "Ship",
NodeType: shipType,
})
factionType = graphql.NewObject(graphql.ObjectConfig{
Name: "Faction",
Description: "A faction in the Star Wars saga",
Fields: graphql.Fields{
"id": relay.GlobalIDField("Faction", nil),
"name": &graphql.Field{
Type: graphql.String,
Description: "The name of the faction.",
},
"ships": &graphql.Field{
Type: shipConnectionDefinition.ConnectionType,
Args: relay.ConnectionArgs,
Resolve: func(p graphql.ResolveParams) (interface{}, error) {
// convert args map[string]interface into ConnectionArguments
args := relay.NewConnectionArguments(p.Args)
// get ship objects from current faction
ships := []interface{}{}
if faction, ok := p.Source.(*Faction); ok {
for _, shipId := range faction.Ships {
ships = append(ships, GetShip(shipId))
}
}
// let relay library figure out the result, given
// - the list of ships for this faction
// - and the filter arguments (i.e. first, last, after, before)
return relay.ConnectionFromArray(ships, args), nil
},
},
},
Interfaces: []*graphql.Interface{
nodeDefinitions.NodeInterface,
},
})Note that when we have the entire list of possible (ship) objects in memory we can easily wrap the array with:
relay.ConnectionFromArray(ships, args)and the relay library will handle all the of connection filters and pagination for us.
For large data sets, loading the entire data set into memory is not performant or even desirable. For situations like this the graphql-relay-go library provides the ConnectionFromArraySlice method
func ConnectionFromArraySlice(
arraySlice []interface{},
args ConnectionArguments,
meta ArraySliceMetaInfo,
) *ConnectionIn this case we must simply pass in a slice of the array that covers at least the entire range being requested. From the graphql-relay-go docs:
/*
Given a slice (subset) of an array, returns a connection object for use in
GraphQL.
This function is similar to `ConnectionFromArray`, but is intended for use
cases where you know the cardinality of the connection, consider it too large
to materialize the entire array, and instead wish pass in a slice of the
total result large enough to cover the range specified in `args`.
*/By using this function we gain the freedom to load exactly what we want to load from the DB and implement custom filters and sorts.
Unfortunately, the documentation on how to use this function is somewhat lacking and we are left to our own devices to determine how to use it.
The ConnectionArguments object is passed into our graphql.FieldResolveFn as a the Args field, but of type map[string]interface{} . We must convert the args to relay-args using
relayArgs = relay.NewConnectionArguments(args)so we now have args of type relay.ConnectionArguments .
The relay.ArraySliceMetaInfo we must construct from our own information about how much data there is that fulfills whatever custom filters we have on the data. With GORM you can do something like
var count int64
db := DB.Model(&Membership{}).Where("memberships.organization_id = ?", thisOrganization.ID)
result := db.Count(&count)to count the valid records in our connection. We can also determine the relay.ArraySliceMetaInfo.SliceStart from the offset cursor that is passed into as connection args. We will need this to offset the records we grab from the database as well.
I found the relay.ConnectionArgument type difficult to work with because it save the offsets as cursors that must be decoded to determine offset, so I created my own custom connection arguments to use while calculating sorts and offsets. I also created a helper method to translate the relay connection arguments to my custom connection arguments type. In this type the before and after are integer indices instead of opaque string cursors.
type customConnectionArguments struct {
Before int
After int
First int
Last int
}
func toCustomConnection(old relay.ConnectionArguments) (customConnectionArguments, error) {
cca := customConnectionArguments{-1, -1, old.First, old.Last}
var err error
if old.After != "" {
cca.After, err = relay.CursorToOffset(old.After)
if err != nil {
return cca, err
}
}
if old.Before != "" {
cca.Before, err = relay.CursorToOffset(old.Before)
if err != nil {
return cca, err
}
}
if cca.First != -1 && cca.Last != -1 {
return cca, errors.New("cannot specify both first and last")
}
return cca, nil
}Now that we better understand the relay library interface and have our setup, we can worry about solving the actual problem of implementing the relay connection pagination on the DB side. The relay connection spec requires that we be able to use cursors to offset from either the beginning or the end of our data a certain amount, while maintaining the data in the same order. SQL has no REVERSE OFFSET, so in order to offset from the end of results we must first reverse the sorts on the data, do the offset, then reverse the sort again to get our original sort, then do the forwards offset.
Fortunately, we can do something like this programatically with GORM without too much difficulty through the use for GORM scopes. This we we can apply filters, and specify sorts, then call a single function to handle the SQL pagination, then return the results with our parameters.
Because we must reverse the sorts, I have added a simple way to specify to our SQL Paginator what sorts we wish to apply and in what order through a Sort type.
//Sort is sort that we wish to apply to the results
type Sort struct {
Field string
Asc bool
}We can pass in an ordered array of these to the pagination function so it knows how to add the reverse sorts.
Our pagination function will return a GORM scope function (count is the count of possible records from above)
func Paginate(cca customConnectionArguments, sort []Sort, count int) func(db *gorm.DB) *gorm.DB {
return func(db *gorm.DB) *gorm.DB {I have also added a default sort of id in case we do not wish to specify a sort
sort := append(sort, Sort{"id", true})Passed into the scope function we get the starting DB with any necessary filters applied, then we apply the sorts, offsets, and limits as specified by the connection args. We are working from the back of the results to the front here, so we can specify how far from the end we wish to go and how many results from the end we desire.
//Reverse order, do offset and limit
var subDB = db
//reverse order
for _, s := range sort {
st := " asc"
if s.Asc {
st = " desc"
}
subDB = subDB.Order(subDB.Statement.Table + "." + s.Field + st)
}
//Do Offset
if cca.Before != -1 {
subDB = subDB.Offset(count - cca.Before)
}
//Do Limit
if cca.Last != -1 {
subDB = subDB.Limit(cca.Last)
} else {
subDB = subDB.Limit(count)
}Then the magic part is we use the results of that query as the starting point for our next results by doing
db = DB.Table("(?) as u", subDB)We can then reapply sorts, offsets, and limits going in the forwards direction
//reverse order again.
for _, s := range sort {
st := " asc"
if !s.Asc {
st = " desc"
}
db = db.Order(s.Field + st)
}
//Limit to the desired amount
if cca.First != -1 {
db = db.Limit(cca.First)
} else {
db = db.Limit(count)
}
//Offset
db = db.Offset(cca.After + 1)
return dbHaving applied this double reverse sort, offset, and limit, we can then run our find and returned the desired results already filtered and sorted as we have specified.
The last piece of the puzzle is being able to specify to the graphql-relay-go library what the first member of our slice is, which we must determine based on the amount of results we received and whether we are going from the end of the beginning of the total result set. That would look something like
metaInfo = relay.ArraySliceMetaInfo{cca.After + 1, int(count)}
if cca.Last != -1 {
if cca.Before == -1 {
metaInfo.SliceStart = int(count) - len(returnedObjects)
} else {
metaInfo.SliceStart = cca.Before - len(returnedObjects)
}
}where count is the length of the set of all results and returnedObjects is the array of objects returned from the db query.
All together, the code could look something like the following
//First create the relay connection definition to add as a field on the schema type
membershipsConnectionDefinition := relay.ConnectionDefinitions(relay.ConnectionConfig{
Name: "OrganizationMembership",
NodeType: MembershipType,
})
//Then add the field to the schema specifying the resolve function, and connection type
OrganizationType.AddFieldConfig("memberships", &graphql.Field{
Type: membershipsConnectionDefinition.ConnectionType,
Args: relay.ConnectionArgs,
Resolve: organizationResolve(resolveOrganizationMemberships),
})
type resolveOrganizationFunc func(context.Context, *models.Organization, map[string]interface{}) ([]interface{}, relay.ArraySliceMetaInfo, relay.ConnectionArguments, error)
//This lets us keep our code more DRY
func organizationResolve(thisFunc resolveOrganizationFunc) graphql.FieldResolveFn {
return func(p graphql.ResolveParams) (interface{}, error) {
val, ok := p.Source.(*models.Organization)
if !ok {
return nil, errors.New("organization source not organization")
}
data, metaInfo, relayArgs, err := thisFunc(p.Context, val, p.Args)
if err != nil {
return nil, err
}
return relay.ConnectionFromArraySlice(data, relayArgs, metaInfo), nil
}
}
//We have to transform whatever is returned into an []interface{}
func resolveOrganizationMemberships(ctx context.Context, thisOrganization *models.Organization, args map[string]interface{}) (data []interface{}, metaInfo relay.ArraySliceMetaInfo, relayArgs relay.ConnectionArguments, err error) {
thisResponse, metaInfo, relayArgs, err := GetOrganizationMemberships(ctx, thisOrganization, args)
for _, val := range thisResponse {
data = append(data, val)
}
return data, metaInfo, relayArgs, err
}
//GetOrganizationMemberships is to get the Organization Memberships specifying pagination sorts
func GetOrganizationMemberships(ctx context.Context, thisOrganization *Organization, args map[string]interface{}) (thisMemberships []*Membership, metaInfo relay.ArraySliceMetaInfo, relayArgs relay.ConnectionArguments, err error) {
//transform first to relay args then to custom connection args
relayArgs = relay.NewConnectionArguments(args)
cca, err := toCustomConnection(relayArgs)
if err != nil {
return
}
//add any sorts we want to use here
sort := []Sort{}
var count int64
//set up filters we want on the data
db := DB.Model(&Membership{}).Where("memberships.organization_id = ?", thisOrganization.ID)
//count the amount of data included in these filters
result := db.Count(&count)
if result.Error != nil {
err = result.Error
return
}
if count == 0 {
err = ErrNotAuthorized
return
}
//Then do the magic offsets and limits in the SQL before the run the query
result = db.Scopes(Paginate(cca, sort, int(count))).Find(&thisMemberships)
metaInfo = relay.ArraySliceMetaInfo{cca.After + 1, int(count)}
if cca.Last != -1 {
if cca.Before == -1 {
metaInfo.SliceStart = int(count) - len(thisMemberships)
} else {
metaInfo.SliceStart = cca.Before - len(thisMemberships)
}
}
return thisMemberships, metaInfo, relayArgs, result.Error
}
//Our custom connection type
type customConnectionArguments struct {
Before int
After int
First int
Last int
}
func toCustomConnection(old relay.ConnectionArguments) (customConnectionArguments, error) {
cca := customConnectionArguments{-1, -1, old.First, old.Last}
var err error
if old.After != "" {
cca.After, err = relay.CursorToOffset(old.After)
if err != nil {
return cca, err
}
}
if old.Before != "" {
cca.Before, err = relay.CursorToOffset(old.Before)
if err != nil {
return cca, err
}
}
if cca.First != -1 && cca.Last != -1 {
return cca, errors.New("cannot specify both first and last")
}
return cca, nil
}
//Sort is a sort field
type Sort struct {
Field string
Asc bool
}
//Paginate is used to paginate
func Paginate(cca customConnectionArguments, sort []Sort, count int) func(db *gorm.DB) *gorm.DB {
return func(db *gorm.DB) *gorm.DB {
sort := append(sort, Sort{"id", true})
//Reverse order, do offset and limit
var subDB = db
//reverse order
for _, s := range sort {
st := " asc"
if s.Asc {
st = " desc"
}
subDB = subDB.Order(subDB.Statement.Table + "." + s.Field + st)
}
//Do Offset
if cca.Before != -1 {
subDB = subDB.Offset(count - cca.Before)
}
//Do Limit
if cca.Last != -1 {
subDB = subDB.Limit(cca.Last)
} else {
subDB = subDB.Limit(count)
}
db = DB.Table("(?) as u", subDB)
//reverse order again.
for _, s := range sort {
st := " asc"
if !s.Asc {
st = " desc"
}
db = db.Order(s.Field + st)
}
//Limit to the desired amount
if cca.First != -1 {
db = db.Limit(cca.First)
} else {
db = db.Limit(count)
}
//Offset
db = db.Offset(cca.After + 1)
return db
}
}This code gives us the ease of specifying whatever filters and sorts are necessary for our relay compliant connection, with the hard work being done by the DB, graphql-relay-go and GORM.
Whether you're building your own GraphQL server or you have a new application idea you're looking to find the right technology partner for, Aptus help!
Contact Lindsay to schedule a FREE 30-min introductory consultation with a solutions architect to help you figure out how to bring your ideas to life!
