Supporting shader-based filters in p5js

SUMMARY a

• Faster image filters (before_and_after.mp4)
• Less boilerplate for shader code (before_and_after.jpg)

This page details what I completed during Google Summer of Code 2023. I contributed to p5.js, which is a friendly, accessible Javascript library for creative coding. People use it to make code sketches in the browser, using functions like createCanvas(), draw(), circle(), text(), image(), cursor(), and noise(), just to name a few.

My work centered around the filter() function because it was too slow. I made it faster by using shaders, which are programs that use the GPU for image processing. Now people can use effects like blur without noticing such a drop in performance.

There were some secondary accomplishments as well. There's a new function createFilterShader() that makes writing shaders easier. I helped revise a few documentation pages. I also tried to contribute some benchmarking tools, but that was a bit ambitious.

These changes make shaders and image filters more accessible. Look for them in the next p5js release, expected October 2023. Here's some web editor sketches to demonstrate the changes:

Potential improvements: (1) I think there should be some refactoring to cut down on all the graphics layers swimming around - maybe by using framebuffers, or maybe just by understanding and wrangling the canvases better. (2) There's some alternative blur filters worth investigating. (3) Mobile performance needs attention too.

I enjoyed the experience and learned a lot. In fact, I learned so much that I wrote another post. That post is more about the GSoC experience in general, especially aimed at newcomers, and not so much about p5js details. But the reflections section could make sense on this page too.

Many thanks to my mentors for all their support.

WORK COMPLETED a

• Initial pull request, just some documentation. Status: merged
• Second pull request, mostly internal changes to enable new shaders. Status: merged
• Third pull request, replacing the old filters with new shader filters. Status: merged

Public documentation pages (which may get updated in the future, but at least for now they include some of my contributions):

• WEBGL - Status: merged, page created
• filter() - Status: merged, page not updated yet
• createFilterShader() - Status: merged, page not created yet

Performance measuring helpers:

• JSPerf benchmarks for comparing two different p5js builds. Status: flopped. It was difficult to set up the tests inside the sketch, and results were inconclusive

CLOSING WEEKS a

• Drafted final writeup
• Met with @aferriss on August 25th and helped fix a few bugs with blur and vertex shaders
• Added tests for filter parameters
• Met with all mentors on August 30th and discussed demos, lightning talk prep, and future plans
• Created side-by-side comparisons of old functions vs new functions
• Got final pull request merged with some help from @aferriss
• Created slides for talk
• Tested demos on mobile (not great, but better than before)
• Collected development artifacts: a bunch of screenshots and sketches as a glimpse of the process

WEEK 12 a

• Add shaders for INVERT, OPAQUE, THRESHOLD (the last remaining filters)
• Tweaked other shaders according to feedback
• Met with @aceslowman on August 18th and discussed wrapping up final features
• Fixed feedback effect on hidden renderer
• Added tests
• Added compatibility for GLSL 3.00 ES / WebGL2 shaders
• Applied for a GSoC lightning talk
• Changed BLUR to a two-pass filter

WEEK 11 a

• Added shaders for DILATE, ERODE, BLUR
• Realized that a pixel size uniform is necessary for shaders to compare neighboring pixels (texel = 1 / width)
• Attempted to make a jsperf for side-by-side build comparisons (link)
• Added POSTERIZE after realizing it should floor() instead of round()
• Added another renderer layer so P2D sketches can use shader filters (the main goal of the project)
• Made a sketch for comparing old filters with new shader filters
• Improved filter() docs
• Met with @aferriss on August 11 and discussed shader tweaks, as well as different options for creating an effective BLUR shader (maybe downsampling and upsampling, maybe hard coding different kernel sizes, maybe compile on the fly with string concat, maybe just two canvases)

WEEK 10 a

• Started a new PR
  • Added a parameter to opt out of webGL image filters
  • Added first shader (GRAY)
• Searched for user sketches that use filter()
• Met with @aceslowman on August 4 and discussed benchmarking / profiling use cases, as well as clarifying unintended behavior of 2D filter shaders with 3D geometry

WEEK 9 a

• Met with @aferriss on July 28 and made important progress to wrap up the PR
  • Realized flipping vTexCoord.y does not flip all content, eg. mouseY; switched to flipping entire canvas instead with scale(1,-1)
  • Fixed edge case of calling filter() on a p5.Graphics
  • Moved shader uniform documentation to setUniform()
  • Brainstormed ideas for the final phase of the project, making measuring performance easier
• Reached out to @limzykenneth about updating the benchmark repo

WEEK 8 a

• Refined docs with more minimal examples and shaders
• Fiddled with blur shaders
• Met with @aceslowman July 21 and received feedback on improving docs and handling edge cases

WEEK 7 a

• Completed GSoC midterm evaluation
• Made a sketch as a public scratchpad for my wip build
• Battled with git for a bit
• Fixed transferring uniforms
• Added createFilterShader(), which seems like the best alternative to exposing a default vertex shader. Credit to @davepagurek for the idea!
• Met with @aferriss on July 14 and developed very practical TODOs, including...
  • testing edge cases like filter shader on a PGraphics or a 3D geometry
  • tidying up example shaders for docs, eg. no unnecessary functions, highp instead of mediump, applying on image instead of on plain canvas, pre-flipping vTexCoord.y for the user
  • documenting shader uniforms, incl with an example
  • testing the filter shader using a default.vert for equality with a normal shader.vert
  • thinking about BLUR and POSTERIZE and ERODE and DILATE shader implementations
• Went on a tangent and helped debug the web editor crashing

WEEK 6 a

• Got sick
• Completed test coverage for filter(shader)
• Documented filter(shader) new feature behavior
• Experimented with a demo / showcase comparison of old filters vs shader filters: midterm_demo.mp4
• Realized running two sketches on the same page, even in iframes, means that they will both draw() at the same speed, maybe because of requestAnimationFrame()
• Started copying uniforms from shader.setUniform() to secondary shader
• Met with @aceslowman on July 7 and received feedback especially about demos / showcases: that a blur filter side-by-side would be pretty hype, that it would be good to share to the gsoc / p5 community, and that his students typically use 400px or 800px canvases
• Discussed more design decisions about transferring shader uniforms, exposing a default vertex shader, and opting out of shader filters

WEEK 5 a

• Made initial p5.Graphics implementation nearly functional
• Found a successful way to call high-level functions in the source code, eg. this._pInst.texture(pg) instead of p5.prototype.texture.apply(this, pg)
• Starting implementing test suite with new insights about expected and unexpected behavior
• Troubleshot Payoneer tax forms
• Met with @aferriss June 30 and discussed next steps, discouraged some scope creeep, confirmed decent test suite, and debugged an important line of code so that the shader renders on main now (!)

WEEK 4 a

• Found a little library someone made a few weeks ago for shader filters
• Met with @aferriss and @aceslowman on June 19 and discussed potential technical challenges for shader filters, gathering feedback around my spec
• Shifted to a consistent meeting time every week, alternating between two mentors and having the third join when possible
• Agreed that p5.Graphics should be a sufficient starting approach, and framebuffer will be pursued as an alternative if necessary
• Cleaned up webgl-inline-docs branch and got the PR merged (my first real open-source contribution! Also in time for the upcoming 1.7.0 release)
• Changed docs so validateParameters accepts a p5.Shader in addition to previous parameters. Should p5.prototype.filter() be moved out of pixels.js?
• Thought up possible test ideas about releasing resources and making sure other shaders, textures, graphics, and framebuffers remain unaffected
• Created a pull request for the meaty shader filter change
• Created a sketch to help debug shaders on p5.Graphics
• Met with @aceslowman June 23 and communicated p5.Graphics difficulties, as well as confirming shared GSoC and p5js experiences that yes, the build times are annoying and yes, diving into webgl can be difficult

WEEK 3 a

• Spent a day reading through @davepagurek's framebuffer blog post, p5.Framebuffer.js, and pull requests and issues around p5.Framebuffer
• Realized that in webgl, controlling render targets means binding textures
• Realized that the strength of p5js WEBGL mode is managing settings and textures
• Finished outlining something of a spec, along with a rough demo sketch
• Confirmed environment support on a small issue while researching framebuffers

WEEK 2 a

• Started polishing shape drawing documentation with line() and arc()
• Received feedback from season of docs contributor @nickmcintyre about documentation style and overlap work
• Met with @aferriss on June 9th and realized I'm a bit fuzzy about the current and proposed webgl architecture
• Started outlining spec for implementing filter(shader)

WEEK 1 a

• Met with @aceslowman on May 29th
• Left a comment on #4820 to give a heads up that I'll be working on shader-based filters soon
• Started a pull request for WEBGL documentation, starting with summarizing differences between WEBGL and P2D mode
• Iterated once on that summary based on feedback from three contributors
• Spent a day reading through yuidoc build tasks and p5.js-website/src/templates/pages/reference/data.json
• Settled on tig as a git viewer (It works great with my terminal workflow!)

WEEK 0 a

• Reached out to mentors with initial emails
• Started a group chat for easier communication with mentors
• Forked a bunch of p5 repos and learned more about git upstream
• Met with mentors in an intro video chat