[Impeller] Implements gaussian blur that scales down before applying the blur #47576

gaaclarke · 2023-11-01T22:52:02Z

A partial implementation of the new gaussian blur effect. This should perform enough of the code to start getting some performance numbers.

Known outstanding problems:

The edges of the blur are clipped. I have notes on how I plan on expanding render space in the PR.
Animating the sigma causes some "jumping around artifacts" resulting from the downsampling (maybe the discrete nature of texture pixel size?)
Coverage hints are ignored. I think depth tests will make that not much of an issue.
We aren't ping ponging textures yet
The snapshot's transform is ignored.

issue: flutter/flutter#131580

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

gaaclarke · 2023-11-16T00:42:48Z

impeller/entity/shaders/gaussian_blur/gaussian_blur.glsl

@@ -54,7 +55,7 @@ void main() {
  // just uses 0.5 as an optimization until the gaussian coefficients are
  // calculated and passed in from the cpu.
  for (float16_t i = -blur_info.blur_radius; i <= blur_info.blur_radius;
-       i += 2.0hf) {
+       i += blur_info.step_size) {


A 2 step causes waffling that's even more noticeable with the downscale approach.

Interesting! I will dig into this afterwards. Its a nice win if we can get it working but is otherwise completely orthogonal to this change

It's the issue I reported here: flutter/flutter#138259

gaaclarke · 2023-11-16T00:44:42Z

impeller/entity/contents/filters/gaussian_blur_filter_contents_unittests.cc

+TEST_P(GaussianBlurFilterContentsTest, CalculateUVsSimple) {
+  TextureDescriptor desc = {
+      .format = PixelFormat::kB8G8R8A8UNormInt,
+      .size = ISize(100, 100),
+  };
+  std::shared_ptr<Texture> texture = MakeTexture(desc);
+  auto filter_input = FilterInput::Make(texture);
+  Entity entity;
+  Quad uvs = GaussianBlurFilterContents::CalculateUVs(filter_input, entity,
+                                                      ISize(100, 100));
+  std::optional<Rect> uvs_bounds = Rect::MakePointBounds(uvs);
+  EXPECT_TRUE(uvs_bounds.has_value());
+  if (uvs_bounds.has_value()) {
+    EXPECT_TRUE(RectNear(uvs_bounds.value(), Rect::MakeXYWH(0, 0, 1, 1)));
+  }
+}


If we used mock objects as we had originally I could test the uv values directly when rendering. This is the best I can do with real objects. cc @matanlurey @jonahwilliams

In theory you could instrument the creation of commands and render passes, and then read-back the real UVs. I might implement something like that in the future.

In the meantime I think its fair to say that based on the gaussian structure it would be too difficult to pull out the real UVs (by hooking into the cmd buffer) and that mocks may be appropriate. Factoring out the logic into a testable method is also a completely reasonable means to test IMO.

But @gaaclarke you're the one putting in the work here. If you don't feel like the static methods are sufficient then mock away.

Yea, the issue I ran into is that I kept fixing one test and breaking another and golden image tests cannot be run locally so I have to do manual testing to make sure those 2 tests kept working. Ideally I'd want to scale beyond checking 2 tests too. I'll see if I get around to testing uvs directly via mocks.

gaaclarke · 2023-11-16T01:21:26Z

@jonahwilliams @bdero are you guys interested in doing a review for this now, not being 100% done? I can keep working on it in this PR. I would like to get the coverage expanding but it will make the review easier to do if that gets split out. I think this is a good flag in the sand.

jonahwilliams · 2023-11-16T01:22:30Z

Yes, I will happily take a look at this now (well, tomorrow 😄 )

jonahwilliams · 2023-11-16T16:27:34Z

So I just tried this out. Wow 🚀 , its crazy fast compared to our current approach. I do see artifacts along the top and bottom edge. I wonder if these are coverage artifacts? I see that you've called that out as a current problem.

jonahwilliams

Just some quick notes

jonahwilliams · 2023-11-16T16:28:18Z

impeller/entity/contents/filters/filter_contents.cc

+      false;
+#endif
+
+  // TODO(https://github.com/flutter/flutter/issues/131580): Remove once the new


Can you document (here or in the issue) which cases the blur doesn't handle now?

I have them listed in the issue above. If we land this early I can copy them over to the issue.

impeller/entity/contents/filters/filter_contents.cc

jonahwilliams · 2023-11-16T16:29:30Z

impeller/entity/contents/filters/gaussian_blur_filter_contents.cc

+  sampler_desc.min_filter = filter;
+  sampler_desc.mag_filter = filter;
+  sampler_desc.width_address_mode = address_mode;
+  sampler_desc.width_address_mode = address_mode;


you set the width address mode twice here, does this need to be height address mode too? I wonder if this is causing the artifacts.

I checked and this did not fix the artifacting issue. I suspect this is probably to do with coverage then

Good catch, that definitely would have shown up at some point. The default value is nearest. Maybe that was contributing to the worse waffling we were seeing with the 2 step?

jonahwilliams · 2023-11-16T16:30:52Z

impeller/entity/contents/filters/gaussian_blur_filter_contents.cc

+    std::shared_ptr<Texture> input_texture,
+    const SamplerDescriptor& sampler_descriptor,
+    const GaussianBlurFragmentShader::BlurInfo& blur_info) {
+  // TODO(gaaclarke): This doesn't render if the blur_info.sigma == 0.


Where do we check for sigma < kEhClose enough on the old blur? Presumibly we could detect that and just make the filter return the original contents, or "optimize" it out entirely

Good call. I do have a todo to address this. It didn't occur to me yet that the solution was so simple, done.

jonahwilliams · 2023-11-16T16:31:39Z

impeller/entity/contents/filters/gaussian_blur_filter_contents.cc

+
+/// Calculate how much to scale down the texture depending on the blur radius.
+/// This curve was taken from |DirectionalGaussianBlurFilterContents|.
+Scalar CalculateScale(Scalar radius) {


If I wanted to change this scaling, is this where I would do it?

As a follow up we may align to skia's curve, which scales sigma down to 4 proportionally.

Yep, oh interesting. This was me copying the one from the other blur. I think we'll have to revisit this since I think it's a bit too aggressive (because of the artifacts when animating the sigma).

gaaclarke · 2023-11-16T17:39:28Z

So I just tried this out. Wow 🚀 , its crazy fast compared to our current approach. I do see artifacts along the top and bottom edge. I wonder if these are coverage artifacts? I see that you've called that out as a current problem.

We might want to tweak the scaling algorithm, its a bit jittery when sigma is animated. So we have that lever to change to get animation quality versus performance. I think the static results all look fine with this curve though.

Which artifacts? You talking about the cropping of the "halo" around blurred items? Thats the one I know about.

jonahwilliams · 2023-11-16T17:55:07Z

I see a dark line/flicker across the top and bottom of one of the blurs on wonderous, specifically the one that slides up on the timeline page.

gaaclarke · 2023-11-16T17:58:25Z

I see a dark line/flicker across the top and bottom of one of the blurs on wonderous, specifically the one that slides up on the timeline page.

Oh interesting. I haven't seen that. It probably isn't worth looking into until I've fixed the halo since that's going to change things around the edges.

gaaclarke · 2023-11-16T18:22:30Z

Oh I'm seeing the glitch now, we should fix that. Looking into it.

gaaclarke · 2023-11-16T18:27:00Z

I see a dark line/flicker across the top and bottom of one of the blurs on wonderous, specifically the one that slides up on the timeline page.

Fixed.

jonahwilliams · 2023-11-16T19:36:06Z

impeller/entity/contents/filters/gaussian_blur_filter_contents.h

+/// Note: This will replace `DirectionalGaussianBlurFilterContents`.
+class GaussianBlurFilterContents final : public FilterContents {
+ public:
+  GaussianBlurFilterContents(Scalar sigma = 0.0f);


nit: explicit?

jonahwilliams

LGTM (with nits)!

bdero

Awesome progress! Left some comments.

I suspect there is a problem with the way we're scaling the sampling interval because I'm seeing extreme loss of high frequency info for low sigma values. But I haven't stared at it long enough to spot it yet and we can always fix it up later.

bdero · 2023-11-16T20:12:05Z

impeller/entity/contents/filters/gaussian_blur_filter_contents.cc

+          .step_size = 1.0,
+      });
+
+  // TODO(gaaclarke): Make this pass reuse the texture from pass1.


How would we reuse the first pass texture?

We should be able to do:
Downsample(src: input, dst: texture1) -> Horizontal Blur(src: texture1, dst: texture2) -> Vertical Blur(src: texture2, dst: texture1) -> texture1

Right now we are doing:
Downsample(src: input, dst: texture1) -> Horizontal Blur(src: texture, dst: texture2) -> Vertical Blur(src: texture2, dst: texture3) -> texture3.

I haven't looked into what plumbing we'd need to make this work yet, the ol' ping-pong maneuver. Potentially it will require different synchronization but I suspect the way we are doing passes now it won't be needed. I haven't looked into it yet.

FWIW I've thought about building this into the render target cache itself, but I don't have a good way to enforce that we don't create self dependencies. Perhaps if we deferred actual texture creation (from @chinmaygarde 's suggestion) and just used descriptors, it would be easier to apply these sorts of optimizations generally.

Ah, yeah that makes sense. The first thing that popped into my mind was that you were planning to sample from the first texture for something. 😆

bdero · 2023-11-16T20:20:29Z

impeller/entity/contents/filters/filter_contents.cc

+      true;
+#else
+      false;
+#endif


Definitely no need to do this now, but it would be nice if this was a temporary runtime flag in Aiks and we just had a new FilterContents::MakeGaussianBlurV2, that way we could test the blurs in the playgrounds side-by-side in the playgrounds and allow for dirty hacks to flip the flag between the old and new blurs when testing against apps.

impeller/entity/contents/filters/gaussian_blur_filter_contents.cc

impeller/geometry/rect.h

bdero · 2023-11-16T20:39:46Z

impeller/entity/contents/filters/filter_contents.cc

@@ -55,6 +56,20 @@ std::shared_ptr<FilterContents> FilterContents::MakeGaussianBlur(
    Sigma sigma_y,
    BlurStyle blur_style,
    Entity::TileMode tile_mode) {
+  constexpr bool use_new_filter =
+#ifdef IMPELLER_ENABLE_NEW_GAUSSIAN_FILTER


Are you flipping this define on via the build config, or just appending code?

Just editing the code during development. I haven't needed anything else yet.

bdero · 2023-11-16T21:18:15Z

impeller/entity/contents/filters/gaussian_blur_filter_contents.cc

+    const Entity& entity,
+    const Matrix& effect_transform,
+    const Rect& coverage,
+    const std::optional<Rect>& coverage_hint) const {


I don't see the snapshot's transform being consumed anywhere, which will cause issues with some optimized snapshots.

The "destination rect" of TextureContents scales the texture when taking a snapshot to avoid an unnecessary render pass. I'd recommend absorbing the scale of the basis vectors in the first downsampling pass, and applying other aspects of the transform to the final returned entity.

So the texture of a snapshot may be in pass space, entity space, or something in-between.

What parameter are you editing to get that? I did see that but I didn't have a case where it manifested. I assumed that was getting absorbed elsewhere so the uv coords weren't effected.

It's the "Path XYWH" at the bottom

gaaclarke · 2023-11-16T22:01:22Z

I suspect there is a problem with the way we're scaling the sampling interval because I'm seeing extreme loss of high frequency info for low sigma values.

@bdero I think that is related to our aggressive scaling function. There is a conversation in this issue a couple places about changing it.

… applying the blur (flutter/engine#47576)

…138585) flutter/engine@aae07e9...5064aef 2023-11-16 [email protected] [Impeller] Clang tidy even more (flutter/engine#48102) 2023-11-16 [email protected] [Impeller] Implements gaussian blur that scales down before applying the blur (flutter/engine#47576) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

github-actions bot added the e: impeller label Nov 1, 2023

gaaclarke force-pushed the gaussian-blur branch 4 times, most recently from 0ac7e4b to 52cce80 Compare November 7, 2023 21:45

gaaclarke mentioned this pull request Nov 10, 2023

[Impeller] Gaussian blur exhibits banding flutter/flutter#138259

Closed

gaaclarke force-pushed the gaussian-blur branch from ea53fdd to 39f83f9 Compare November 15, 2023 08:16

gaaclarke changed the title ~~Gaussian blur~~ Implements gaussian blur that scales down before applying the blur Nov 15, 2023

gaaclarke force-pushed the gaussian-blur branch from 78cfe70 to c6ecc90 Compare November 16, 2023 00:40

gaaclarke commented Nov 16, 2023

View reviewed changes

gaaclarke changed the title ~~Implements gaussian blur that scales down before applying the blur~~ [Impeller] Implements gaussian blur that scales down before applying the blur Nov 16, 2023

jonahwilliams reviewed Nov 16, 2023

View reviewed changes

jonahwilliams approved these changes Nov 16, 2023

View reviewed changes

gaaclarke marked this pull request as ready for review November 16, 2023 19:37

gaaclarke requested a review from bdero November 16, 2023 19:38

gaaclarke added 5 commits November 16, 2023 12:29

[Impeller] added new gaussian blur

9a5bb13

added directional_gaussian unittests to help

139e800

added filtersourcecoverage

2f8aa40

added more tests

58aac4c

got a possitive rendering test

3944bed

gaaclarke added 14 commits November 16, 2023 12:29

format

0512e91

added placeholder asserts

4b029d5

split out calculateuvs function

1960fae

cleanup

465e748

added rotation test

81ca1f1

cleanup

bdbf6b0

tidy

a84f12d

removed stray files

6d9113f

removed debugging code

6e9adb7

license

76bac19

license

6375d37

updated malioc

6cfa0bb

jonah feedback

9ec72d4

fixed jittering at the edges

2dc9201

gaaclarke force-pushed the gaussian-blur branch from b9da55a to 2dc9201 Compare November 16, 2023 20:30

added explicit

fb88897

bdero approved these changes Nov 16, 2023

View reviewed changes

brandon feedback

b26d11e

gaaclarke added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 16, 2023

auto-submit bot merged commit 9113bc8 into flutter:main Nov 16, 2023

gaaclarke mentioned this pull request Nov 16, 2023

[Impeller] Scale down the Gaussian blur in both directions prior to blurring. flutter/flutter#131580

Closed

engine-flutter-autoroll mentioned this pull request Nov 16, 2023

Roll Flutter Engine from aae07e989b0a to 9113bc8f4bcd (1 revision) flutter/flutter#138584

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 16, 2023

9113bc8f4 [Impeller] Implements gaussian blur that scales down before…

44518d1

… applying the blur (flutter/engine#47576)

engine-flutter-autoroll mentioned this pull request Nov 16, 2023

Roll Flutter Engine from aae07e989b0a to 5064aeff00de (2 revisions) flutter/flutter#138585

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 16, 2023

9113bc8f4 [Impeller] Implements gaussian blur that scales down before…

9d913c2

… applying the blur (flutter/engine#47576)

zanderso mentioned this pull request Nov 23, 2023

[Impeller] Metal validation errors in blur coverage unit tests flutter/flutter#138955

Closed

gaaclarke mentioned this pull request Nov 27, 2023

[Impeller] Gaussian blur should respect the snapshot transform flutter/flutter#139085

Closed

[Impeller] Implements gaussian blur that scales down before applying the blur #47576

[Impeller] Implements gaussian blur that scales down before applying the blur #47576

Uh oh!

Conversation

gaaclarke commented Nov 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

gaaclarke Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaaclarke commented Nov 16, 2023

Uh oh!

jonahwilliams commented Nov 16, 2023

Uh oh!

jonahwilliams commented Nov 16, 2023

Uh oh!

jonahwilliams left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaaclarke commented Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonahwilliams commented Nov 16, 2023

Uh oh!

gaaclarke commented Nov 16, 2023

Uh oh!

gaaclarke commented Nov 16, 2023

Uh oh!

gaaclarke commented Nov 16, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonahwilliams left a comment

Choose a reason for hiding this comment

Uh oh!

bdero left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdero Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

gaaclarke commented Nov 1, 2023 •

edited

Loading

gaaclarke Nov 16, 2023 •

edited

Loading

gaaclarke commented Nov 16, 2023 •

edited

Loading

bdero Nov 16, 2023 •

edited

Loading