Implement SendTouchEvent for aura/native windows

[misyltoad]

I saw a CEF forum post from this user, luke1985 where they wanted this to be implemented.
https://magpcss.org/ceforum/viewtopic.php?f=8&t=20359&p=57824&hilit=+SendTouchEvent#p57824
Unfortunately I was not able to make an account there as I never got the activation e-mail, seems like it's down or something.

Maybe they will see this, and this will be useful to someone.

I actually recently implemented this for something today, and tracking down all the needed bits was kinda annoying. One key thing to note is that the touch event is not what triggers interactions, its the gestures derived from the touch event. These gestures are typically derived in the host window tree event dispatch stuff's PreDispatch handlers for each event type. In this, it's forwarded to the gesture_provider/gesture_recognisers few a few steps, and these are what makes the OnGestureEvent stuff work for the given touch events that actually end up doing the interactions.

CEF's synthetic input injection for non-OSR does not go through this PreDispatch handler as it's not actually dispatching as an event from the WindowTree.

So when implementing this, you need to manually forward the touch event to the gesture_recognizer for the given NativeWindow for the WindowWidget in order for interactions to work.

Here's an example of how that could be done:

void CefBrowserPlatformDelegateNativeAura::SendTouchEvent(
    const CefTouchEvent& event) {
  auto view = GetHostView();
  if (!view) {
    return;
  }

  if (!browser_->GetWindowWidget()) {
    return;
  }

  ui::TouchEvent ui_event = TranslateTouchEvent(event);
  ui::TouchEvent root_relative_event(ui_event);
  root_relative_event.set_location_f(ui_event.root_location_f());  

  aura::Env* env = aura::Env::GetInstance();
  aura::Window *win = browser_->GetWindowWidget()->GetNativeWindow();

  if (!env->gesture_recognizer()->ProcessTouchEventPreDispatch(
          &root_relative_event, win)) {
    return;
  }

  // This flag is set depending on the gestures recognized in the call above,
  // and needs to propagate with the forwarded event.
  ui_event.set_may_cause_scrolling(root_relative_event.may_cause_scrolling());

  view->OnTouchEvent(&ui_event);
}

// ...

ui::EventType CefBrowserPlatformDelegateNativeAura::TranslateTouchEventType(
    cef_touch_event_type_t touch_event_type, cef_gesture_type_t gesture) const {
  switch (touch_event_type) {
    case CEF_TET_RELEASED: return ui::ET_TOUCH_RELEASED;
    case CEF_TET_PRESSED: return ui::ET_TOUCH_PRESSED;
    case CEF_TET_MOVED: return ui::ET_TOUCH_MOVED;
    case CEF_TET_CANCELLED: return ui::ET_TOUCH_CANCELLED;
    }
  }
}

ui::EventPointerType CefBrowserPlatformDelegateNativeAura::TranslatePointerType(
    cef_pointer_type_t pointer_type) const {
  switch (pointer_type) {
    case CEF_POINTER_TYPE_TOUCH:   return ui::EventPointerType::kTouch;
    case CEF_POINTER_TYPE_MOUSE:   return ui::EventPointerType::kMouse;
    case CEF_POINTER_TYPE_PEN:     return ui::EventPointerType::kPen;
    case CEF_POINTER_TYPE_ERASER:  return ui::EventPointerType::kEraser;
    default:
    case CEF_POINTER_TYPE_UNKNOWN: return ui::EventPointerType::kUnknown;
  }
}

ui::TouchEvent CefBrowserPlatformDelegateNativeAura::TranslateTouchEvent(
    const CefTouchEvent &touch_event) const {
  ui::EventType type = TranslateTouchEventType(touch_event.type, touch_event.gesture);
  gfx::PointF location(touch_event.x, touch_event.y);
  // HACK(misyl): Match OSR behaviour in GetPointerPropertiesFromTouchEvent
  // which puts root_location == location...
  //
  // The touch -> gesture code works from root_location, not location.
  // There's something DEFINITELY slightly wrong either here or the other places
  // setting up root_location in CEF event injection, but this seems to work correctly...
  gfx::PointF root_location = location;
  //gfx::PointF root_location(GetScreenPoint(
  //    gfx::Point(touch_event.x, touch_event.y), /*want_dip_coords=*/false));
  base::TimeTicks time_stamp = GetEventTimeStamp();
  int flags = TranslateUiEventModifiers(touch_event.modifiers);

  ui::TouchEvent ui_touch_event(type, location, root_location, time_stamp,
      ui::PointerDetails(
        TranslatePointerType(touch_event.pointer_type), touch_event.id,
        touch_event.radius_x, touch_event.radius_y,
        touch_event.pressure),
      flags);

  return ui_touch_event;
}

On top of removing the IsWindowless check in the AlloyBrowserHostImpl and just removing the override that hooks SendTouchEvent in ChromeBrowserHostImpl

If there is interest in CEF upstream supporting native/aura synthetic input, I'd be willing to submit a PR for this support.

Hopefully this helps someone or finds its way to that guy as the forum is not sending activation emails right now.

Thanks!

[magreenblatt]

If there is interest in CEF upstream supporting native/aura synthetic input, I'd be willing to submit a PR for this support.

I agree that this functionality could be useful. We would need to see the implementation (complexity) to judge if it's something that we can support. Also, this will require very good unit test coverage.

[magreenblatt]

Note that macOS doesn't use Aura, so you would need a different (native) implementation for that platform.

[magreenblatt]

Injecting synthetic input events has proven rather unreliable in automated tests. In many cases it's likely better to inject JavaScript that simulates input events, e.g. document.getElementById('mybutton').click(). The CefExecuteJavaScriptWithUserGestureForTests function can be used (in tests) if a user gesture is required.