TeamSnap
&
GraphQL

 

Silver bullet or over-hyped technology?

Dustin McCraw

10/11/19

Agenda:

  • 15 minutes on the challenges facing our platform
     
  • 15 minutes explain what GraphQL is
    and if it can address those problems
     
  • 15 minutes for demos and discussion

I need your help!

I'm really looking to either disprove or validate these ideas.

 

I want to either be able to pursue these ideas or drop them and so I need your help to validate these solutions or
find issues that I haven't thought about.

This presentation will be a success if...

You come away with an understand of some of the challenges facing our platform and some ways to solve them.

Api's and Client's

In 5 years I've written code for:

Snapi, McFeely, Dozer,

org-api, org-service, soylent, accounts-api

classic, league-frontend, org-frontend,

payments-frontend, NextJenn

 

I have worked all over the stack at TeamSnap and I have seen some of challenges when it comes to developing and consuming our REST apis.

Challenge #1

Challenge #1

Challenge #1

Challenge #1

 

How do clients interact with our Microservices Architecture?

 

  • Are they expected to know each of them individually?
  • Mobile already has to connect to Snapi, Hoyle and Payments-Api.
  • What about when they need to connect to accounts-api or org-api?

 

Challenge #2

Snapi 8 events

Snapi 0 events

2957 lines of json
40,192 bytes

412 lines of json

6,356 bytes

Challenge #2

 

How large can we expect our JSON payloads?

collection+json

With 80% of our DAU's coming from mobile,
payload size is incredibly import.

Our users can be on 4G, or even worse 3G.

The smaller payload we send the better.

Clients have to parse all of that code, the more they parse, the slower they perform.

Challenge #3

{
  "collection": {
    "version": "3.822.1",
    "href": "https://api.teamsnap.com/v3/events",
    "rel": "events",
    "template": {
      "data": [
        {
          "name": "type",
          "value": "event"
        },
        {
          "name": "additional_location_details",
          "value": null
        },
        {
          "name": "browser_time_zone",
          "value": null
        },
        {
          "name": "division_location_id",
          "value": null
        },
        {
          "name": "doesnt_count_towards_record",
          "value": null
        },
        {
          "name": "duration_in_minutes",
          "value": null
        },
        {
          "name": "game_type_code",
          "value": null
        },
        {
          "name": "icon_color",
          "value": null
        },
        {
          "name": "is_canceled",
          "value": null
        },
        {
          "name": "is_game",
          "value": null
        },
        {
          "name": "is_overtime",
          "value": null
        },
        {
          "name": "is_shootout",
          "value": null
        },
        {
          "name": "is_tbd",
          "value": null
        },
        {
          "name": "label",
          "value": null
        },
        {
          "name": "location_id",
          "value": null
        },
        {
          "name": "minutes_to_arrive_early",
          "value": null
        },
        {
          "name": "name",
          "value": null
        },
        {
          "name": "notes",
          "value": null
        },
        {
          "name": "notify_opponent",
          "value": null
        },
        {
          "name": "notify_opponent_contacts_email",
          "value": null
        },
        {
          "name": "notify_opponent_contacts_name",
          "value": null
        },
        {
          "name": "notify_opponent_notes",
          "value": null
        },
        {
          "name": "notify_team",
          "value": null
        },
        {
          "name": "notify_team_as_member_id",
          "value": null
        },
        {
          "name": "opponent_id",
          "value": null
        },
        {
          "name": "points_for_opponent",
          "value": null
        },
        {
          "name": "points_for_team",
          "value": null
        },
        {
          "name": "repeating_include",
          "value": null,
          "prompt": "When updating a repeating event, this is a required field. Values are: \"all\" - updates all events in this series, \"future\" - updates this event and all that occur after, \"none\" - only updates a single event."
        },
        {
          "name": "repeating_type_code",
          "value": null,
          "prompt": "A code for the frequency of the repeated event, this is required with the \"repeating_include\" attribute when creating a repeating event. Valid values are: \"1\" - repeat an event daily, \"2\" - repeat an event weekly."
        },
        {
          "name": "repeating_until",
          "value": null,
          "prompt": "A date when the repeating event should end, this is inclusive so an event will be created on this day if it falls before the next event specified by \"repeating_type_code\". This attribute is required with \"repeating_type_code\" when creating a repeating event."
        },
        {
          "name": "results",
          "value": null
        },
        {
          "name": "results_url",
          "value": null
        },
        {
          "name": "shootout_points_for_opponent",
          "value": null
        },
        {
          "name": "shootout_points_for_team",
          "value": null
        },
        {
          "name": "start_date",
          "value": null
        },
        {
          "name": "team_id",
          "value": null
        },
        {
          "name": "time_zone",
          "value": null
        },
        {
          "name": "tracks_availability",
          "value": null
        },
        {
          "name": "uniform",
          "value": null
        }
      ]
    },
    "links": [
      {
        "rel": "assignments",
        "href": "https://api.teamsnap.com/v3/assignments"
      },
      {
        "rel": "availabilities",
        "href": "https://api.teamsnap.com/v3/availabilities"
      },
      {
        "rel": "division_location",
        "href": "https://api.teamsnap.com/v3/division_locations"
      },
      {
        "rel": "event_lineups",
        "href": "https://api.teamsnap.com/v3/event_lineups"
      },
      {
        "rel": "event_statistics",
        "href": "https://api.teamsnap.com/v3/event_statistics"
      },
      {
        "rel": "location",
        "href": "https://api.teamsnap.com/v3/locations"
      },
      {
        "rel": "opponent",
        "href": "https://api.teamsnap.com/v3/opponents"
      },
      {
        "rel": "statistic_data",
        "href": "https://api.teamsnap.com/v3/statistic_data"
      },
      {
        "rel": "team",
        "href": "https://api.teamsnap.com/v3/teams"
      },
      {
        "rel": "root",
        "href": "https://api.teamsnap.com/v3/"
      },
      {
        "rel": "self",
        "href": "https://api.teamsnap.com/v3/events/search?is_game=true&page_number=1&page_size=1&sort_start_date=desc&started_before=2019-10-01T19:18:45-07:00&team_id=6618318"
      },
      {
        "rel": "first",
        "href": "https://api.teamsnap.com/v3/events/search?is_game=true&page_number=1&page_size=1&sort_start_date=desc&started_before=2019-10-01T19%3A18%3A45-07%3A00&team_id=6618318"
      },
      {
        "rel": "last",
        "href": "https://api.teamsnap.com/v3/events/search?is_game=true&page_number=0&page_size=1&sort_start_date=desc&started_before=2019-10-01T19%3A18%3A45-07%3A00&team_id=6618318"
      }
    ],
    "queries": [
      {
        "rel": "search",
        "sort": [
          "start_date"
        ],
        "href": "https://api.teamsnap.com/v3/events/search",
        "data": [
          {
            "name": "team_id",
            "value": null
          },
          {
            "name": "user_id",
            "value": null
          },
          {
            "name": "location_id",
            "value": null
          },
          {
            "name": "opponent_id",
            "value": null
          },
          {
            "name": "started_after",
            "value": null
          },
          {
            "name": "started_before",
            "value": null
          },
          {
            "name": "repeating_uuid",
            "value": null
          },
          {
            "name": "id",
            "value": null
          },
          {
            "name": "is_game",
            "value": null
          },
          {
            "name": "updated_since",
            "value": null
          },
          {
            "name": "page_size",
            "value": null,
            "prompt": "The number of items to return for each page. Sending this parameter with the query will enable paging for the returned collection."
          },
          {
            "name": "page_number",
            "value": null,
            "prompt": "The number of the page to be returned. This requires that paging be turned on by also providing the page_size parameter."
          },
          {
            "name": "sort_start_date",
            "value": null,
            "prompt": "Sort the returned dataset based on the start_date field, valid values are 'asc' or 'desc'."
          }
        ]
      },
      {
        "rel": "search_games",
        "href": "https://api.teamsnap.com/v3/events/search_games",
        "data": [
          {
            "name": "team_id",
            "value": null
          },
          {
            "name": "page_size",
            "value": null,
            "prompt": "The number of items to return for each page. Sending this parameter with the query will enable paging for the returned collection."
          },
          {
            "name": "page_number",
            "value": null,
            "prompt": "The number of the page to be returned. This requires that paging be turned on by also providing the page_size parameter."
          }
        ]
      },
      {
        "rel": "overview",
        "href": "https://api.teamsnap.com/v3/events/overview",
        "data": [
          {
            "name": "team_id",
            "value": null
          }
        ]
      }
    ],
    "commands": [
      {
        "rel": "send_availability_reminders",
        "href": "https://api.teamsnap.com/v3/events/send_availability_reminders",
        "prompt": "members_to_notify = [member_id, member_id]",
        "data": [
          {
            "name": "id",
            "value": null
          },
          {
            "name": "members_to_notify",
            "value": null
          },
          {
            "name": "notify_team_as_member_id",
            "value": null
          }
        ]
      },
      {
        "rel": "update_final_score",
        "href": "https://api.teamsnap.com/v3/events/update_final_score",
        "prompt": "Update the final score for an event",
        "data": [
          {
            "name": "id",
            "value": null
          },
          {
            "name": "points_for_team",
            "value": null
          },
          {
            "name": "points_for_opponent",
            "value": null
          },
          {
            "name": "shootout_points_for_team",
            "value": null
          },
          {
            "name": "shootout_points_for_opponent",
            "value": null
          },
          {
            "name": "is_overtime",
            "value": null
          },
          {
            "name": "is_shootout",
            "value": null
          },
          {
            "name": "results",
            "value": null
          },
          {
            "name": "results_url",
            "value": null
          }
        ]
      },
      {
        "rel": "bulk_create",
        "href": "https://api.teamsnap.com/v3/events/bulk_create",
        "prompt": "event_ids = [event_id, event_id]",
        "data": [
          {
            "name": "templates",
            "value": null
          },
          {
            "name": "team_id",
            "value": null
          },
          {
            "name": "notify_team_as_member_id",
            "value": null
          },
          {
            "name": "notify_team",
            "value": null
          }
        ]
      }
    ],
    "page_info": {
      "total_items": 0,
      "page_size": 1,
      "page_number": 1
    }
  }
}
{ 
  "time":"2019-10-02T02:50:31.001347Z",
  "status":200,
  "metadata":null,
  "error":null,
  "data":{ 
    "uuid":"person-3192a9a0-2d87-0137-24ba-00163eab79c2",
    "user_id":4209826,
    "updated_at":"2019-03-20T21:43:59.000000",
    "last_name":"McCraw",
    "is_age_hidden":null,
    "inserted_at":"2019-03-20T21:43:59.000000",
    "id":143944,
    "gender":null,
    "first_name":"Dustin",
    "contact_infos":null,
    "birthdate":null
  }
}
{ 
  "data":[ 
    { 
      "account":{ 
        "address":{ 
          "city":null,
          "country":"US",
          "line1":null,
          "line2":null,
          "postal_code":null,
          "state":null
        },
        "business_type":"non_profit",
        "charges_enabled":true,
        "company":{ 
          "name":"Stripe Org",
          "tax_id_provided":false
        },
        "country":"US",
        "created_at":"2019-05-06T20:11:55Z",
        "currency":"usd",
        "details_submitted":false,
        "is_setup_complete":false,
        "payouts_enabled":false,
        "requirements":{ 
          "current_deadline":null,
          "currently_due":[ 
            "external_account"
          ],
          "disabled_reason":"requirements.past_due",
          "eventually_due":[ 
            "company.tax_id",
            "external_account"
          ],
          "past_due":[ 
            "external_account"
          ],
          "pending_verification":[ 

          ]
        },
        "stripes__account_id":"acct_1EXDAEE99KWL2RR4",
        "tos_accept_date":"2019-02-14T15:21:22Z"
      },
      "bank_account":null,
      "id":"148",
      "is_setup_complete":false,
      "person":null,
      "resource_type":"payment_account",
      "stripe_account_id":"acct_1EXDAEE99KWL2RR4",
      "teamsnap_stripe_account_id":"148"
    }
  ],
  "meta":{ 
    "code":"ok",
    "status_code":200,
    "timestamp":"2019-10-02T02:53:04.348694Z",
    "version":"1.0.0",
    "has_more":false
  }
}

Challenge #3

 

What JSON format should our clients understand?

  • Our services are returning different JSON formats.
  • SNAPI, Accounts-api, Payments-Api, Hoyle
  • All different formats!!!
  • Every time a client accesses a new service we fire up, they must learn that service's specific json format.
  • Because these are non-standard formats we can't use open source libraries for backend or clients.

Challenge #4

https://api.teamsnap.com/v3/team_preferences/search
https://api.teamsnap.com/v3/plans/search
https://api.teamsnap.com/v3/sports/search
https://api.teamsnap.com/v3/teams/search
https://api.teamsnap.com/v3/events/search
https://api.teamsnap.com/v3/locations/search
https://api.teamsnap.com/v3/opponents/search
https://api.teamsnap.com/v3/assignments/search
https://api.teamsnap.com/v3/availabilities/search
https://api.teamsnap.com/v3/event_lineup/search
https://api.teamsnap.com/v3/sport_positions/search

Challenge #4

What mobile has to download to display a single event.

https://api.teamsnap.com/v3/team_preferences/search
https://api.teamsnap.com/v3/plans/search
https://api.teamsnap.com/v3/sports/search
https://api.teamsnap.com/v3/teams/search
https://api.teamsnap.com/v3/events/search
https://api.teamsnap.com/v3/locations/search
https://api.teamsnap.com/v3/opponents/search
https://api.teamsnap.com/v3/assignments/search
https://api.teamsnap.com/v3/availabilities/search
https://api.teamsnap.com/v3/event_lineup/search
https://api.teamsnap.com/v3/sport_positions/search

Challenge #4

https://api.teamsnap.com/v3/broadcast_emails/search
https://api.teamsnap.com/v3/teams/search
https://api.teamsnap.com/v3/members/search
https://api.teamsnap.com/v3/member_email_addresses/search
https://api.teamsnap.com/v3/contacts/search
https://api.teamsnap.com/v3/contact_email_addresses/search
https://api.teamsnap.com/v3/division_member_email_addresses
/search
https://api.teamsnap.com/v3/division_contact_email_addresses
/search

Challenge #4

https://api.teamsnap.com/v3/broadcast_emails/search
https://api.teamsnap.com/v3/teams/search
https://api.teamsnap.com/v3/members/search
https://api.teamsnap.com/v3/member_email_addresses/search
https://api.teamsnap.com/v3/contacts/search
https://api.teamsnap.com/v3/contact_email_addresses/search
https://api.teamsnap.com/v3/division_member_email_addresses
/search
https://api.teamsnap.com/v3/division_contact_email_addresses
/search

What McFeely has to download to send a team email.

Challenge #4

 

How many endpoints do clients need to download to get all of our interconnected data?

  • How  much time are we adding to every render for all the multiple round trips with the latency for each call?
  • Clients have to store all of this interconnected data
  • Clients then stitch it back together

Challenge #5

Challenge #5

/schemas
49142 lines
661,485 bytes

Challenge #5

 

How do we document our apis for both internal and external clients?

  • How do we document all of our services and make that data discoverable?

5 Challenges

  1. Multiple services/apis
  2. Large Collection+JSON payloads
  3. Custom JSON payload
  4. Multiple server/api round trips
  5. Api/Service Documentation

GraphQL Api Gateway

Let's wrap all of our services behind a GraphQL server which serves as an api gateway giving clients one documented entry point into all our our data.

What is an api gateway?

- An API gateway is programming that sits in front of an application programming interface (API) and acts as a single point of entry for a defined group of microservices.

What is an GraphQL?

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data.

GraphQL

 

  • GraphQL is a query language created at Facebook in 2012
  • Open source way to define an API
  • Standard way to provide the description of the data of your api and it lets clients request exactly what they need
  • Came from the conflict between users' wanting to get all the information in one query vs the requirement to isolated REST calls

GraphQL

https://graphql.org/users/

Title Text

  • Defines a data shape
  • Hierarchical/Graph
  • Strongly typed
  • Protocol
  • Introspective

Defines a Data Shape

The server defines a schema.

Clients compose queries against that schema and the response mirrors that query.

This allow clients determine what data they need and query for it.

Hierarchical/Graph/Relational

GraphQL defines the relationships between the objects in the schema.

Clients can choose what relationships they need for their specific use case.

Strongly Typed

Each level of a query corresponds to a type.
Each type has a set of available fields.

Define a Schema using scalar types.

i.e. A Character has a non null name and
a non null list of Episodes.

Schema takes care of all validation.

Protocol

Open source standard between server and clients.

 

Can be written in any backend language
using any datastore.

 

Each GraphQL field has its own resolver so a single GraphQL object can come from multiple services.

 

It's not a technology, it's a standard.

Introspective

A GraphQL server can be queried for the types it supports.
This creates a powerful platform for client software to build atop this information like code generation in statically typed languages

GraphQL

Advantages

  • Retrieve only the data you need on consumer side
  • Reduce the data volume returned by the server because you retrieve only what you need
  • Reduce the number of calls to retrieve data by linking resources and aggregating queries
  • Discover the schema you are querying

5 Challenges

  1. Multiple services/apis
  2. Large Collection+JSON payloads
  3. Custom JSON payload
  4. Multiple server/api round trips
  5. Api/Service Documentation

Challenge #1

Multiple services/api

By wrapping all of our services inside GraphQL, every client that interacts with TeamSnap only has to go to one endpoint to get all of our data.

Challenge #1

Multiple services/api

Alternatives:

A restful API gateway - This still has all the downsides of REST like multiple trips from the client

 

Challenge #2

Large Collection+JSON payloads

Since graphql responses mirror the query the clients send, it's almost the smallest payload possible.

Challenge #2

Large Collection+JSON payloads

Alternatives:

Use a smaller json format - This still has all the downsides of REST like multiple trips from the client. Doesn't allow clients to select the fields they want.

Challenge #3

Custom JSON payload

GraphQL is an open source standard. Every client/server framework conforms to the same format.

This allows us to use a lot of existing open source software like Absinthe and ApolloJS.

This allows amazing tooling like GraphQL Playground.

Challenge #3

Custom JSON payload

Alternatives:

Convert all of our services/api to use the same json format or JSON-Api - This still has all the downsides of REST like multiple trips from the client. Doesn't have the ability for clients to select the data they need.

Challenge #4

Multiple server/api round trips

The relationships between data allows client to pull related data together in a single query. Clients can even query completely different data that isn't related by just having multiple queries in the request.

Challenge #4

Alternatives:

Bulk load endpoint - Few round trips but doesn't have the ability for clients to select the data they need so the payload is still larger than it needs to be.

Stripe like expanding fields - Fewer round trips but creates a custom solution for expanding fields.

Multiple server/api round trips

Challenge #5

Api/Service Documentation

The Absinthe GraphQL framework allows for co-location of documentation and code so as you write code you write documentation. 

Documentation is highly encouraged because it can easily be introspected by tools like GraphQL Playground.

Challenge #5

Alternatives:

Swagger - requires a lot of custom code to generate those documents

Api/Service Documentation

Want to learn more?

  • #graphql channel

  • https://github.com/teamsnap/fore_api_gateway

  • I am always free to talk graphql!

Thank you!

TeamSnap and GraphQl

By Dustin McCraw

Made with Slides.com

TeamSnap and GraphQl

  • 686

More from Dustin McCraw