RFC 100: Enhancing headless support in Wagtail core

[thibaudcolas]

View as an HTML document. This RFC attempts to set a direction for Wagtail’s future headless support improvement. It covers:

1. A definition of Wagtail’s support goals when it comes to headless support. Long-term, that’ll help us decide what headless-related capabilities should be in Wagtail itself, vs. in packages, vs. for site implementers to build. This is the most crucial thing to validate as part of the RFC process.
2. An overview of the current areas of interest and known opportunities for improvement. This is meant more as a recap of current points of friction, gaps, opportunities, than a mandate that we must address them all.

I’ve also published a 2024 Wagtail headless survey so we get input from a bigger group of developers. Some of you might recall the 2022 survey, which helped us tremendously back then in documenting the current state. Those survey results will help us understand where it’d be most helpful to direct headless support contributions.

[ahosgood]

I had to add extra code to get redirects to work in the API - would that be helpful to have included?

[thibaudcolas]

@ahosgood I would assume so? Redirects is a contrib module so not as "core" as some of the more fundamental aspects of the CMS, but certainly something that many sites would consider core CMS functionality.

[lb-]

I think it would be good to somehow include the pattern for JSON rendering using normal Page routing. Essentially each Page path can have a different request header and then return JSON instead of HTML.

It's quite an intuitive approach and aligns with how other CMSs (e.g. Adobe Experience Manager) can provide their APIs. It's not going to make sense for every installation but worth reviewing as part of this RFC.

In the abstract, this also could align with applications that may want to build out HTMX style applications, still returning HTML but providing a partial render of 'inner' HTML based on request headers.

See wagtail/wagtail#11752 & wagtail/wagtail#8374

[saevarom]

Headless was on the agenda for Wagtail Space NL but this only involved incorporating the areweheadlessyet.org website into the Wagtail documentation. The resulting PR is here: wagtail/wagtail#12039
The sprint topic discussion is here: https://paper.dropbox.com/doc/Sprint-topics-Wagtail-Space-NL-2024--CUVkcdc_H1K~k2BWz8SS9UUbAg-3rkJ5imATtyajKY2Ey1Cr#:uid=172696553243645599787735&h2=Headless-Wagtail

[lb-]

Another long-standing issue with the current API is that we cannot easily generate an OpenAPI specification

See wagtail/wagtail#6209

I would say that we must have a documented or even out of the box way to generate specifications for our APIs if we want to truly say we have headless support.

[thibaudcolas]

Thank you everyone for the feedback! I’ve heavily updated the RFC. Now’s the time for further reviews and feedback (approval?) The RFC now has two well-separated sections:

1. A definition of Wagtail’s support goals when it comes to headless support. Long-term, that’ll help us decide what headless-related capabilities should be in Wagtail itself, vs. in packages, vs. for site implementers to build. This is the most crucial thing to validate as part of the RFC process.
2. An overview of the current areas of interest and known opportunities for improvement. This is meant more as a recap of current points of friction, gaps, opportunities, than a mandate that we must address them all.

I’ve also published a 2024 Wagtail headless survey so we get input from a bigger group of developers. Some of you might recall the 2022 survey, which helped us tremendously back then in documenting the current state. Those survey results will help us understand where it’d be most helpful to direct headless support contributions.

If you want to help

1. Review the RFC. You’re here already so you’re aware of how this all works and how important input is.
2. Take the survey and share it with colleagues. Should be a quick one.
3. Upvote headless issues?

We’re not big on issue emoji reactions / votes as a way to make decision, but there’s a big corpus of existing tickets here so could be interesting. Here are all open issues sorted by theme:

Headless-related existing issues

• REST API
  • Allow unlimited API results with limit=-1 #12604
  • Wagtail API: Greater than/less than filters #7497
  • Wagtail API: OR filtering related models #5755
  • Add ability to include and filter via detail fields on pages' list endpoints #11404
  • Random order in the API leading to an AttributeError #3512
  • Wagtail API should have a browsable root URL #1921
  • Duplicate db query on api detail page #6133
  • Adding per-subclass filter backend/class to the v2 API Pages Endpoint #3403
  • Future release enhancement: Implement APIField for listing_default_fields, too #3562
  • Document nested_default_fields in APIv2 #11587
  • Add a UUIDField to the Page model #6162
  • Support customizing/encoding IDs in the API #6917
• API schemas
  • Support OpenAPI Schema generation for Wagtail API #6209.
• Images
  • ImagesAPIViewSet query params for rendition? #7973
  • No Image URL in the API for ImageChooserBlock #2087
  • Feature request: Include image's title in ImageRenditionField #5435
  • Documentation for using the "Dynamic image serve" view for generating image renditions on a headless site #6113
• Page URL Routing
  • Keep queries other than html_path in /api/v2/pages/find/ #6577
  • Finding pages by HTML Path - redirect missing port #7595
  • Improvements to API router reverse lookups. #11310
  • Add ability to support custom renderers for Page views (e.g. JSON API response at page URLs) #11752
  • Accept page change in before_serve_page handling #5753
• Redirects
  • Refactor redirect middleware to use its lookup logic for redirects API #11524.
• Rich Text
  • API should provide access to the HTML-converted versions of rich text fields #2695
• Multi-site support
• Form submissions
  • Ability to accept Form Builder Page as a POST request to the API (headless) #6950
• Password-protected Pages
• Internationalisation
  • Support internationalized routes in API pages/find/?html_path view #6679.
• StreamField
  • Override serializing StreamBlocks to JSON api #3454
  • Apply custom-per-block-type StreamField serializers #5174
  • parent_context or just "parent Page" for Block.get_api_representation() #7976

---

Last thing, re long-standing issues and schema specifications: At this stage I think we need to move past the No True Scotsman chain of thought, asking ourselves whether Wagtail is truly headless or not. Wagtail is a hybrid system, it’s pretty clear. Some features aren’t headless-compatible, but clearly there’s hundreds if not thousands of headless sites out there built with Wagtail, some pretty high-profile. So yes it’s truly headless. We don’t want to mislead people, so there are gaps to fill (docs in particular), but it’s already a thing and has been for years.

[itzomen]

A frontend client like wagtail-js will make integrating wagtail for many frontend developers easier.

[thibaudcolas]

I have updated the RFC based on the Results of the 2024 Wagtail headless survey. Thank you to everyone who took part in that – I think it was exactly the type of input I was hoping we would get as far as setting priorities.

@laymonage @ahosgood @allcaps @lb- @dopry @zerolab would you be ok to re-review the RFC and approve or otherwise provide further feedback? It’s a pretty high-level RFC as it is, more of a statement of intentions than an implementation plan. So to be a bit more specific, in #106 we have three follow-up items shaping up that will depend on feedback here:

• A plan to bring headless preview and userbar support in core
• "Quick win" API and documentation improvements based on priority order expressed here
• An official headless demo site – primarily so our contributors and maintainers can more easily work on headless improvements

[Scotchester]

• A plan to bring headless preview and userbar support in core
• "Quick win" API and documentation improvements based on priority order expressed here
• An official headless demo site – primarily so our contributors and maintainers can more easily work on headless improvements

These sounds like three really great next steps to me!

[dopry]

Looks good. Personally I'd prefer to see GraphQL move towards first class treatment. I don't use REST for any headless sites anymore. The network overhead of multiple requests to assemble complex views adds a lot of latency that negatively impacts the UX. With GraphQL I get excellent tooling for code generation, specs are built in, and there is GraphIQL for exploring the model. Even if it remains an External Package it would be nice to see that it is something that is getting stronger consideration from the core team. @zerolab does a great job of supporting it. It could use quite a bit of documentation love and it could be a lot easier to work with when it comes to custom blocks, search, multisite handling. In my own project I've had to modify a lot to get it to be fully functional in my use case.. While I contribute a lot of that back, there are still pain points. It would be nice to annotate the ideal state as Improved External Package.

[auvipy]

Looks good. Personally I'd prefer to see GraphQL move towards first class treatment. I don't use REST for any headless sites anymore. The network overhead of multiple requests to assemble complex views adds a lot of latency that negatively impacts the UX. With GraphQL I get excellent tooling for code generation, specs are built in, and there is GraphIQL for exploring the model. Even if it remains an External Package it would be nice to see that it is something that is getting stronger consideration from the core team. @zerolab does a great job of supporting it. It could use quite a bit of documentation love and it could be a lot easier to work with when it comes to custom blocks, search, multisite handling. In my own project I've had to modify a lot to get it to be fully functional in my use case.. While I contribute a lot of that back, there are still pain points. It would be nice to annotate the ideal state as Improved External Package.

I don't think graphQL would be the best API for headless wagtail, but that could be an option. I would suggest following JSON:API 1.0/1.1 standard for headless REST API. Here is a old related issues #22

[dopry]

@auvipy

I don't think graphQL would be the best API for headless wagtail, but that could be an option. I would suggest following JSON:API 1.0/1.1 standard for headless REST API. Here is a old related issues #22

I disagree with your assertion regarding what would be the best API, but I also wasn't suggesting abandoning REST. It still has a userbase whose needs should be met, and it is low overhead for the core team to maintain. I just prefer GraphQL, as do a lot of other frontend developers, and would prefer to see it get more consideration than it currently does.

[auvipy]

You can do graph like query with json:api

[laymonage]

The reasoning, compromises, and the plan seems solid to me. I think having this RFC in is a good first step before we start any work on the actual improvements in Wagtail. Thanks for the great write-up!

[laymonage]

If we're considering moving away from DRF for the next version of the API, I'd suggest making whatever package we use be optional. Right now DRF is a mandatory requirement for Wagtail installation, even if you don't use it at all. Granted, there are internal admin views that are built with DRF, but I'm sure it can be refactored to use plain Django just fine.

And then if people want to use the API, they can install Wagtail with an optional dependency group, e.g. wagtail[api].

[thibaudcolas]

Sounds great to me!

[thibaudcolas]

@dopry @auvipy thanks both! I didn’t think GraphQL would be a good candidate for core simply because it doesn’t seem that would have big advantages over the status quo. I’ve also heard good things about other implementations than Graphene. But certainly we need to invest more into it, package or no.

For the first time ever, I have an open source implementation of GraphQL I can point people to: bakerydemo-nextjs. I hope this will make it easier for me to make progress on docs at least.

[dopry]

@dopry @auvipy thanks both! I didn’t think GraphQL would be a good candidate for core simply because it doesn’t seem that would have big advantages over the status quo. I’ve also heard good things about other implementations than Graphene. But certainly we need to invest more into it, package or no.

For the first time ever, I have an open source implementation of GraphQL I can point people to: bakerydemo-nextjs. I hope this will make it easier for me to make progress on docs at least.

I've never really had a problem with Graphene, I've had problems with graphene helpers, but ultimately it hasn't been a problem working with it. You need to do some optimization with your resolvers and prefetch/select_related when it's called for.

[lb-]

@thibaudcolas Thank you for the incredible work here, taking on feedback and cross-linking various issues.

I am more than happy to approve in it's current state, I have added a few small comments though if they can get looked at.

[lb-]

Would we consider providing a mechanism for sending the password via a HTTP header instead? Especially as this 'password' is actively moving towards more of a shared secret. I know this is not the same as an API key but a namespaced header could be a suitable approach.

[dopry]

That wouldn't work too well for a Static Site Generation scenario where the frontend may render an .htaccess or implement it's own authentication methodology. It would probably be better to transmit the hash and hashing details to the headless front end.

[lb-]

I would like to recommend that as part of our documentation tasks, we also aim to expose the full API for our Wagtail Guide https://guide.wagtail.org/en-latest/

This is a great way to show the features of the API (as is, and evolving) and may also allow developers to create new content/remixes of that Guide site.

[thibaudcolas]

Thank you @lb-! I’ll make a few more tweaks based on further feedback but will move this as "Final comment period" as it seems we’re overall happy with the approach.

[andreasnuesslein]

Just my two cents: I've tried GraphQL extensively with a big project of ours for a while. I tried Graphene, Strawberry and ultimately landed on Ariadne.

It was .. fine.. but oh my god so much extra headache. So many abstraction layers between django and graphql (because of GraphQLs "flexibility" to say which fields you would like..). In the end, I custom built a whole "database-http-proxy" basically and then had the nightmare of permissions. I couldn't just use Graphene with all its abstraction because it was way too slow, instead I needed to create custom .values() with django-orm-JSONObject stuff.

None of the customers (academic folk) actually enjoyed adopting it over the old REST approach. After a few tedious years of trying it, we went back to DRF and are so happy to be back.

I think Fireship had a good video about it, along the lines of "what the f* were we all thinking" but I'm not finding it right now.

Now, I know this can easily turn into a flame war and yes: I think GraphQL doesnt solve many problems but rather create new ones, but my main point I want to make is: Maybe before implementing, try to find out how big the need is, including a question along the lines of "so you say you would like GraphQL. have you actually used it before and think it has merit? Where would you see the advantage over the current model?"

[allcaps]

I have similar experience(s). Although it is from a while ago, and the capabilities of these GrapQL tooling might have improved, it is just not worth it.

Wagtail is primarily used to build websites. That means you have a URL and need some page context. How hard can it be? The unique selling point of GraphQL (create your own queries to collect only the data you need) is something that sounds appealing, but in practice is simply not needed. Developing REST API endpoints is so trivial and give full control. It is all you need.

I also woud never advice to use GraphQL with Wagtail. Just my 2ct.

[andreasnuesslein]

Like @lb- is already pointing out: I would 100% argue for OpenAPI-types for all the given Pages (and Streamfield Blocks). I heavily use DRF-Spectacular these days in almost all projects and I love the fact that all my models are just typed in the frontend. However for my custom Wagtail Pages I write all the types from hand (or omit it completely). Having this included in the already existing drf-integration would be ever-so-swell.

[thibaudcolas]

Thank you @andreasnuesslein for the feedback. re GraphQL, it is already implemented in the sense that there are existing packages: wagtail-grapple, and an experimental integration with Strawberry. And yes we’ve already assessed how much usage it gets / where it fits, via the 2024 Wagtail headless survey. Clearly there are people happy to use it and room for improvements. Probably all of that can happen outside of Wagtail core, which is what this RFC states.

Re OpenAPI schemas, we need people to pick this up (Support OpenAPI Schema generation for Wagtail API #6209) and figure out which changes to make in Wagtail, which changes might make more sense to do in drf-spectacular.

---

It looks like this RFC is in a good state to merge so I’ll do that now. Feel free to keep commenting here if you have more broad feedback, or on the Wagtail Slack in the #headless channel. Or on the issues linked from the RFC. Thank you to everyone who provided feedback :)

[andreasnuesslein]

So,

I was fed up with DRF and DRF-Spectacular.

https://github.com/sinnwerkstatt/wagtail-ninja

and

https://pypi.org/project/wagtail-ninja/

for a first alpha.

basically it should be as simple as

from wagtail_ninja.api import api as ninja_api


urlpatterns += [
    path("api/wagtail/v3/", ninja_api.urls),
]

it should follow the logic of the existing wagtail api pages endpoint at auto-discovering a bunch of fields.

not included right now is the support for APIField() but that might be something that needs to be changed since DRF-Serializer != Pydantic Model.

would be happy for feedback

[dopry]

@thibaudcolas where would one request to add block previews for parity with 6.4 block preview feature?