Django Dynamic REST

Dynamic API extensions for Django REST Framework

See http://dynamic-rest.readthedocs.org for full documentation.

Table of Contents

• Overview
• Maintainers
• Requirements
• Installation
• Demo
• Features
  • Linked relationships
  • Sideloaded relationships
  • Embedded relationships
  • Inclusions
  • Exclusions
  • Filtering
  • Ordering
  • Directory panel
  • Optimizations
• Settings
• Compatibility
• Contributing
• License

Overview

Dynamic REST (or DREST) extends the popular Django REST Framework (or DRF) with API features that empower simple RESTful APIs with the flexibility of a graph query language.

DREST classes can be used as a drop-in replacement for DRF classes, which offer the following features on top of the standard DRF kit:

• Linked relationships
• Sideloaded relationships
• Embedded relationships
• Inclusions
• Exclusions
• Filtering
• Sorting
• Directory panel for your Browsable API
• Optimizations

DREST was originally written to complement Ember__, but it can be used to provide fast and flexible CRUD operations to any consumer that supports JSON over HTTP.

Maintainers

• Anthony Leontiev
• Savinay Nangalia
• Christina D'Astolfo

Contributors

• Ernesto González

Requirements

• Python (3.6, 3.7, 3.8)
• Django (2.2, 3.1, 3.2)
• Django REST Framework (3.11, 3.12)

Installation

1. Install using pip:

    pip install dynamic-rest

(or add dynamic-rest to requirements.txt or setup.py)

2. Add rest_framework and dynamic_rest to INSTALLED_APPS in settings.py:

    INSTALLED_APPS = (
        ...
        'rest_framework',
        'dynamic_rest'
    )

3. If you want to use the Directory panel, replace DRF's browsable API renderer with DREST's in your settings:

REST_FRAMEWORK = {
    'DEFAULT_RENDERER_CLASSES': [
        'rest_framework.renderers.JSONRenderer',
        'dynamic_rest.renderers.DynamicBrowsableAPIRenderer',
    ],
}

Demo

This repository comes with a tests package that also serves as a demo application. This application is hosted at https://dynamic-rest.herokuapp.com but can also be run locally:

1. Clone this repository:

    git clone [email protected]:AltSchool/dynamic-rest.git
    cd dynamic-rest

2. From within the repository root, start the demo server:

    make serve

3. Visit localhost:9002 in your browser.

4. To load sample fixture data, run make fixtures and restart the server.

Features

To understand the DREST API features, let us consider a demo model with a corresponding viewset, serializer, and route. This will look very familiar to anybody who has worked with DRF:

# The related LocationSerializer and GroupSerializer are omitted for brevity

# The Model
class User(models.Model):
    name = models.TextField()
    location = models.ForeignKey('Location')
    groups = models.ManyToManyField('Group')

# The Serializer
class UserSerializer(DynamicModelSerializer):
    class Meta:
        model = User
        name = 'user'
        fields = ("id", "name", "location", "groups")

    location = DynamicRelationField('LocationSerializer')
    groups = DynamicRelationField('GroupSerializer', many=True)

# The ViewSet
class UserViewSet(DynamicModelViewSet):
    serializer_class = UserSerializer
    queryset = User.objects.all()

# The Router
router = DynamicRouter()
router.register('/users', UserViewSet)

Linked relationships

One of the key features of the DREST serializer layer is the ability to represent relationships in different ways, depending on the request context (external requirements) and the code context (internal requirements).

By default, a "has-one" (or "belongs-to") relationship will be represented as the value of the related object's ID. A "has-many" relationship will be represented as a list of all related object IDs.

When a relationship is represented in this way, DREST automatically includes relationship links for any has-many relationships in the API response that represents the object:

-->
    GET /users/1/
<--
    200 OK
    {
        "user": {
            "id": 1,
            "name": "John",
            "location": 1,
            "groups": [1, 2],
            "links": {
                "groups": "/users/1/groups"
            }
        }
    }

An API consumer can navigate to these relationship endpoints in order to obtain information about the related records. DREST will automatically create the relationship endpoints -- no additional code is required:

-->
    GET /users/1/groups
<--
    200 OK
    {
        "groups": [{
            "id": 1,
            "name": "Family",
            "location": 2,
        }, {
            "id": 2,
            "name": "Work",
            "location": 3,
        }]
    }

Sideloaded relationships

Using linked relationships provides your API consumers with a "lazy-loading" mechanism for traversing through a graph of data. The consumer can first load the primary resource and then load related resources later.

In some situations, it can be more efficient to load relationships eagerly, in such a way that both the primary records and their related data are loaded simultaneously. In Django, this can be accomplished by using prefetch_related or select_related.

In DREST, the requirement to eagerly load (or "sideload") relationships can be expressed with the include[] query parameter.

For example, in order to fetch a user and sideload their groups:

-->
    GET /users/1/?include[]=groups.*
<--
    200 OK
    {
        "user": {
            "id": 1,
            "name": "John",
            "location": 1,
            "groups": [1, 2]
        },
        "groups": [{
            "id": 1,
            "name": "Family",
            "location": 2
        }, {
            "id": 2,
            "name": "Work",
            "location": 3
        }]
    }

The "user" portion of the response looks nearly identical to the first example; the user is returned top-level, and the groups are represented by their IDs. However, instead of including a link to the groups endpoint, the group data is present within the respones itself, under a top-level "groups" key.

Note that each group itself contains relationships to "location", which are linked in this case.

With DREST, it is possible to sideload as many relationships as you'd like, as deep as you'd like.

For example, to obtain the user with groups, locations, and groups' locations all sideloaded in the same response:

-->
    GET /users/1/?include[]=groups.location.*&include[]=location.*
<--
    200 OK
    {
        "user": {
            "id": 1,
            "name": "John",
            "location": 1,
            "groups": [1, 2]
        },
        "groups": [{
            "id": 1,
            "name": "Family",
            "location": 2,
        }, {
            "id": 2,
            "name": "Work",
            "location": 3,
        }],
        "locations": [{
            "id": 1,
            "name": "New York"
        }, {
            "id": 2,
            "name": "New Jersey"
        }, {
            "id": 3,
            "name": "California"
        }]
    }

Embedded relationships

If you want your relationships loaded eagerly but don't want them sideloaded in the top-level, you can instruct your serializer to embed relationships instead.

In that case, the demo serializer above would look like this:

# The Serializer
class UserSerializer(DynamicModelSerializer):
    class Meta:
        model = User
        name = 'user'
        fields = ("id", "name", "location", "groups")

    location = DynamicRelationField('LocationSerializer', embed=True)
    groups = DynamicRelationField('GroupSerializer', embed=True, many=True)

... and the call above would return a response with relationships embedded in place of the usual ID representation:

-->
    GET /users/1/?include[]=groups.*
<--
    200 OK
    {
        "user": {
            "id": 1,
            "name": "John",
            "location": 1,
            "groups": [{
                "id": 1,
                "name": "Family",
                "location": 2
            }, {
                "id": 2,
                "name": "Work",
                "location": 3
            }]
        }
    }

In DREST, sideloading is the default because it can produce much smaller payloads in circumstances where related objects are referenced more than once in the response.

For example, if you requested a list of 10 users along with their groups, and those users all happened to be in the same groups, the embedded variant would represent each group 10 times. The sideloaded variant would only represent a particular group once, regardless of the number of times that group is referenced.

Inclusions

You can use the include[] feature not only to sideload relationships, but also to load basic fields that are marked "deferred".

In DREST, any field or relationship can be marked deferred, which indicates to the framework that the field should only be returned when requested by include[]. This could be a good option for fields with large values that are not always relevant in a general context.

For example, a user might have a "personal_statement" field that we would want to defer. At the serializer layer, that would look like this:

# The Serializer
class UserSerializer(DynamicModelSerializer):
    class Meta:
        model = User
        name = 'user'
        fields = ("id", "name", "location", "groups", "personal_statement")
        deferred_fields = ("personal_statement", )

    location = DynamicRelationField('LocationSerializer')
    groups = DynamicRelationField('GroupSerializer', many=True)

This field will only be returned if requested:

-->
    GET /users/1/?include[]=personal_statement
<--
    200 OK
    {
        "user": {
            "id": 1,
            "name": "John",
            "location": 1,
            "groups": [1, 2],
            "personal_statement": "Hello, my name is John and I like........",
            "links": {
                "groups": "/users/1/groups"
            }
        }
    }

Note that include[]=personal_statement does not have a . following the field name as in the previous examples for embedding and sideloading relationships. This allows us to differentiate between cases where we have a deferred relationship and want to include the relationship IDs as opposed to including and also sideloading the relationship.

For example, if the user had a deferred "events" relationship, passing include[]=events would return an "events" field populated by event IDs, passing include[]=events. would sideload or embed the events themselves, and by default, only a link to the events would be returned. This can be useful for large has-many relationships.

Exclusions

Just as deferred fields can be included on demand with the include[] feature, fields that are not deferred can be excluded with the exclude[] feature. Like include[], exclude[] also supports multiple values and dot notation to allow you to exclude fields on sideloaded relationships.

For example, if we want to fetch a user with his groups, but ignore the groups' location and user's location, we could make a request like this:

-->
    GET /users/1/?include[]=groups.*&exclude[]=groups.location&exclude[]=location
<--
    200 OK
    {
        "user": {
            "id": 1,
            "name": "John",
            "groups": [1, 2],
            "links": {
                "location": "/users/1/location"
            }
        },
        "groups": [{
            "id": 1,
            "name": "Family",
            "links": {
                "location": "/groups/1/location"
            }
        }, {
            "id": 2,
            "name": "Work",
            "links": {
                "location": "/groups/2/location"
            }
        }]
    }

exclude[] supports the wildcard value: *, which means "don't return anything". Why is that useful? include[] overrides exclude[], so exclude[]=* can be combined with include[] to return only a single value or set of values from a resource.

For example, to obtain only the user's name:

-->
    GET /users/1/?exclude[]=*&include[]=name
<--
    200 OK
    {
        "user": {
            "name": "John",
            "links": {
                "location": "/users/1/location",
                "groups": "/users/1/groups"
            }
        }
    }

Note that links will always be returned for relationships that are deferred.

Filtering

Tired of writing custom filters for all of your fields? DREST has your back with the filter{} feature.

You can filter a user by his name (exact match):

-->
    GET /users/?filter{name}=John
<--
    200 OK
    ...

... or a partial match:

-->
   GET /users/?filter{name.icontains}=jo
<--
    200 OK
    ...

... or one of several names:

-->
    GET /users/?filter{name.in}=John&filter{name.in}=Joe
<--
    200 OK

... or a relationship ID:

-->
    GET /users/?filter{groups}=1
<--
    200 OK

... or lack thereof:

-->
    GET /users/?filter{-groups}=1
<--
    200 OK

... or a relationship field:

-->
    GET /users/?filter{groups.name}=Home
<--
    200 OK

... or multiple criteria:

-->
    GET /users/?filter{groups.name}=Home&filter{name}=John
<--
    200 OK

... or combine it with include[] to filter the sideloaded data (get all the users and only sideload certain groups):

-->
    GET /users/?include[]=groups.*&filter{groups|name.icontains}=h
<--
    200 OK

The sky is the limit! DREST supports just about every basic filtering scenario and operator that you can use in Django:

• in
• icontains
• istartswith
• range
• lt
• gt ...

See the full list here.

Ordering

You can use the sort[] feature to order your response by one or more fields. Dot notation is supported for sorting by nested properties:

-->
    GET /users/?sort[]=name&sort[]=groups.name
<--
    200 OK
    ...

For descending order, simply add a - sign. To sort by name in descending order for example

-->
    GET /users/?sort[]=-name
<--
    200 OK
    ...

Directory panel

We love the DRF browsable API, but wish that it included a directory that would let you see your entire list of endpoints at a glance from any page. DREST adds that in:

Optimizations

Supporting nested sideloading and filtering is expensive and can lead to very poor query performance if implemented naively. DREST uses Django's Prefetch object to prevent N+1 query situations and guarantee that your API is performant. We also optimize the serializer layer to ensure that the conversion of model objects into JSON is as fast as possible.

How fast is it? Here are some benchmarks that compare DREST response time to DRF response time. DREST out-performs DRF on every benchmark:

Linear benchmark: rendering a flat list

Quadratic benchmark: rendering a list of lists

Cubic benchmark: rendering a list of lists of lists

Settings

DREST is configurable, and all settings should be nested under a single block in your settings.py file. Here are our defaults:

DYNAMIC_REST = {
    # DEBUG: enable/disable internal debugging
    'DEBUG': False,

    # ENABLE_BROWSABLE_API: enable/disable the browsable API.
    # It can be useful to disable it in production.
    'ENABLE_BROWSABLE_API': True,

    # ENABLE_LINKS: enable/disable relationship links
    'ENABLE_LINKS': True,

    # ENABLE_SERIALIZER_CACHE: enable/disable caching of related serializers
    'ENABLE_SERIALIZER_CACHE': True,

    # ENABLE_SERIALIZER_OPTIMIZATIONS: enable/disable representation speedups
    'ENABLE_SERIALIZER_OPTIMIZATIONS': True,

    # DEFER_MANY_RELATIONS: automatically defer many-relations, unless
    # `deferred=False` is explicitly set on the field.
    'DEFER_MANY_RELATIONS': False,

    # MAX_PAGE_SIZE: global setting for max page size.
    # Can be overriden at the viewset level.
    'MAX_PAGE_SIZE': None,

    # PAGE_QUERY_PARAM: global setting for the pagination query parameter.
    # Can be overriden at the viewset level.
    'PAGE_QUERY_PARAM': 'page',

    # PAGE_SIZE: global setting for page size.
    # Can be overriden at the viewset level.
    'PAGE_SIZE': None,

    # PAGE_SIZE_QUERY_PARAM: global setting for the page size query parameter.
    # Can be overriden at the viewset level.
    'PAGE_SIZE_QUERY_PARAM': 'per_page',

    # ADDITIONAL_PRIMARY_RESOURCE_PREFIX: String to prefix additional
    # instances of the primary resource when sideloading.
    'ADDITIONAL_PRIMARY_RESOURCE_PREFIX': '+',

    # Enables host-relative links.  Only compatible with resources registered
    # through the dynamic router.  If a resource doesn't have a canonical
    # path registered, links will default back to being resource-relative urls
    'ENABLE_HOST_RELATIVE_LINKS': True
}

Compatibility

We actively support the following:

• Python: 3.6, 3.7, 3.8
• Django: 2.2, 3.1, 3.2
• Django Rest Framework: 3.11, 3.12

Note: Some combinations are not supported. For up-to-date information on actively supported/tested combinations, see the tox.ini file.

Contributing

See Contributing.

License

See License.