Changeset 245320 in webkit


Ignore:
Timestamp:
May 15, 2019 1:37:02 AM (5 years ago)
Author:
Devin Rousso
Message:

Web Automation: elements larger than the viewport have incorrect in-view center point
​https://bugs.webkit.org/show_bug.cgi?id=195696
<rdar://problem/48737122>

Reviewed by Simon Fraser.

Original patch by Brian Burg <BJ Burg>.

Source/WebCore:

Some conversion methods do not exist for FloatRect/FloatPoint. Fill them in as needed,
and export some symbols used by WebDriver code to compute an element's in-view center point
in various coordinate systems.

  • dom/TreeScope.h:
  • dom/TreeScope.cpp:

(WebCore::TreeScope::elementsFromPoint): Added.

  • page/FrameView.h:
  • page/FrameView.cpp:

(WebCore::FrameView::absoluteToLayoutViewportPoint const): Added.
(WebCore::FrameView::layoutViewportToAbsoluteRect const): Added.
(WebCore::FrameView::absoluteToLayoutViewportRect const): Added.

  • platform/ScrollView.h:
  • platform/ScrollView.cpp:

(WebCore::ScrollView::viewToContents const): Added.
(WebCore::ScrollView::contentsToView const): Added.
(WebCore::ScrollView::contentsToRootView const): Added.

  • platform/Widget.h:
  • platform/Widget.cpp:

(WebCore::Widget::convertToRootView const): Added.
(WebCore::Widget::convertFromRootView const): Added.
(WebCore::Widget::convertToContainingView const): Added.
(WebCore::Widget::convertFromContainingView const): Added.

Source/WebKit:

This seems to be an omission in the specification. While it does mention that the in-view
center point (IVCP) must be within the viewport, the algorithm never intersects the element
bounding box with the viewport rect.

  • WebProcess/Automation/WebAutomationSessionProxy.cpp:

(WebKit::WebAutomationSessionProxy::computeElementLayout):
This code is incorrect. For CoordinateSystem::LayoutViewport, coordinates should be in
root view coordinates so that it can be later converted to screen and synthesized as a HID
event in screen coordinates. Intersect the element rect and the viewport rect before finding
the center point of the part of the element that's visible in the viewport.

(WebKit::convertRectFromFrameClientToRootView): Added.
(WebKit::convertPointFromFrameClientToRootView): Added.
Added helpers to properly account for scroll contents position on iOS.

  • UIProcess/Automation/WebAutomationSession.cpp:

(WebKit::WebAutomationSession::viewportInViewCenterPointOfElement):
Now that we determine whether the element is inside the viewport much earlier, if the
element has no inViewCenterPoint, we can return a TargetOutOfBounds instead of a more
"generic" ElementNotInteractable.

(WebKit::WebAutomationSession::simulateMouseInteraction):
Rename locationInView -> locationInViewport.

(WebKit::WebAutomationSession::simulateTouchInteraction):
This code is incorrect. The unobscuredContentRect is in screen coordinates, but
we are trying to see if (x, y) is outside the size of the viewport assumed to be at (0, 0).
Grab the visual viewport rect and see if the location exceeds the viewport size.

  • UIProcess/Automation/ios/WebAutomationSessionIOS.mm:

(WebKit::operator<<):
Add logging helper for TouchInteraction enum.

(WebKit::WebAutomationSession::platformSimulateTouchInteraction):
Move local variable.

  • UIProcess/Automation/SimulatedInputDispatcher.cpp:

(WebKit::SimulatedInputDispatcher::transitionInputSourceToState):
Fix a typo in logging.

  • UIProcess/Automation/Automation.json:

Simplify enum name.

  • Platform/Logging.h:

Add logging channel to dump fully resolved interaction details.

Location:
trunk/Source
Files:
16 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r245317 r245320  
     12019-05-15  Devin Rousso  <drousso@apple.com>
     2
     3        Web Automation: elements larger than the viewport have incorrect in-view center point
     4        https://bugs.webkit.org/show_bug.cgi?id=195696
     5        <rdar://problem/48737122>
     6
     7        Reviewed by Simon Fraser.
     8
     9        Original patch by Brian Burg <bburg@apple.com>.
     10
     11        Some conversion methods do not exist for `FloatRect`/`FloatPoint`. Fill them in as needed,
     12        and export some symbols used by WebDriver code to compute an element's in-view center point
     13        in various coordinate systems.
     14
     15        * dom/TreeScope.h:
     16        * dom/TreeScope.cpp:
     17        (WebCore::TreeScope::elementsFromPoint): Added.
     18        * page/FrameView.h:
     19        * page/FrameView.cpp:
     20        (WebCore::FrameView::absoluteToLayoutViewportPoint const): Added.
     21        (WebCore::FrameView::layoutViewportToAbsoluteRect const): Added.
     22        (WebCore::FrameView::absoluteToLayoutViewportRect const): Added.
     23        * platform/ScrollView.h:
     24        * platform/ScrollView.cpp:
     25        (WebCore::ScrollView::viewToContents const): Added.
     26        (WebCore::ScrollView::contentsToView const): Added.
     27        (WebCore::ScrollView::contentsToRootView const): Added.
     28        * platform/Widget.h:
     29        * platform/Widget.cpp:
     30        (WebCore::Widget::convertToRootView const): Added.
     31        (WebCore::Widget::convertFromRootView const): Added.
     32        (WebCore::Widget::convertToContainingView const): Added.
     33        (WebCore::Widget::convertFromContainingView const): Added.
     34
    1352019-05-14  Wenson Hsieh  <wenson_hsieh@apple.com>
    236

  • trunk/Source/WebCore/dom/TreeScope.cpp

    r239427 r245320  
    442442}
    443443
     444Vector<RefPtr<Element>> TreeScope::elementsFromPoint(const FloatPoint& p)
     445{
     446    return elementsFromPoint(p.x(), p.y());
     447}
     448
    444449Element* TreeScope::findAnchor(const String& name)
    445450{

  • trunk/Source/WebCore/dom/TreeScope.h

    r233778 r245320  
    3838class Document;
    3939class Element;
     40class FloatPoint;
    4041class HTMLImageElement;
    4142class HTMLLabelElement;
    … …  
    9697    WEBCORE_EXPORT RefPtr<Element> elementFromPoint(double clientX, double clientY);
    9798    WEBCORE_EXPORT Vector<RefPtr<Element>> elementsFromPoint(double clientX, double clientY);
     99    WEBCORE_EXPORT Vector<RefPtr<Element>> elementsFromPoint(const FloatPoint&);
    98100
    99101    // Find first anchor with the given name.

  • trunk/Source/WebCore/page/FrameView.cpp

    r244731 r245320  
    47924792}
    47934793
     4794FloatPoint FrameView::absoluteToLayoutViewportPoint(FloatPoint p) const
     4795{
     4796    ASSERT(frame().settings().visualViewportEnabled());
     4797    p.scale(1 / frame().frameScaleFactor());
     4798    p.moveBy(-layoutViewportRect().location());
     4799    return p;
     4800}
     4801
    47944802FloatPoint FrameView::layoutViewportToAbsolutePoint(FloatPoint p) const
    47954803{
    … …  
    47974805    p.moveBy(layoutViewportRect().location());
    47984806    return p.scaled(frame().frameScaleFactor());
     4807}
     4808
     4809FloatRect FrameView::layoutViewportToAbsoluteRect(FloatRect rect) const
     4810{
     4811    ASSERT(frame().settings().visualViewportEnabled());
     4812    rect.moveBy(layoutViewportRect().location());
     4813    rect.scale(frame().frameScaleFactor());
     4814    return rect;
     4815}
     4816
     4817FloatRect FrameView::absoluteToLayoutViewportRect(FloatRect rect) const
     4818{
     4819    ASSERT(frame().settings().visualViewportEnabled());
     4820    rect.scale(1 / frame().frameScaleFactor());
     4821    rect.moveBy(-layoutViewportRect().location());
     4822    return rect;
    47994823}
    48004824

  • trunk/Source/WebCore/page/FrameView.h

    r244633 r245320  
    476476    float absoluteToDocumentScaleFactor(Optional<float> effectiveZoom = WTF::nullopt) const;
    477477
    478     FloatRect absoluteToDocumentRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const;
    479     FloatPoint absoluteToDocumentPoint(FloatPoint, Optional<float> effectiveZoom = WTF::nullopt) const;
     478    WEBCORE_EXPORT FloatRect absoluteToDocumentRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const;
     479    WEBCORE_EXPORT FloatPoint absoluteToDocumentPoint(FloatPoint, Optional<float> effectiveZoom = WTF::nullopt) const;
    480480
    481481    FloatRect absoluteToClientRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const;
    482482
    483483    FloatSize documentToClientOffset() const;
    484     FloatRect documentToClientRect(FloatRect) const;
     484    WEBCORE_EXPORT FloatRect documentToClientRect(FloatRect) const;
    485485    FloatPoint documentToClientPoint(FloatPoint) const;
    486486    WEBCORE_EXPORT FloatRect clientToDocumentRect(FloatRect) const;
    487487    WEBCORE_EXPORT FloatPoint clientToDocumentPoint(FloatPoint) const;
    488488
     489    WEBCORE_EXPORT FloatPoint absoluteToLayoutViewportPoint(FloatPoint) const;
    489490    FloatPoint layoutViewportToAbsolutePoint(FloatPoint) const;
     491
     492    WEBCORE_EXPORT FloatRect absoluteToLayoutViewportRect(FloatRect) const;
     493    FloatRect layoutViewportToAbsoluteRect(FloatRect) const;
    490494
    491495    // Unlike client coordinates, layout viewport coordinates are affected by page zoom.

  • trunk/Source/WebCore/platform/ScrollView.cpp

    r243919 r245320  
    843843}
    844844
     845FloatPoint ScrollView::viewToContents(const FloatPoint& point) const
     846{
     847    if (delegatesScrolling())
     848        return point;
     849
     850    return viewToContents(IntPoint(point));
     851}
     852
     853FloatPoint ScrollView::contentsToView(const FloatPoint& point) const
     854{
     855    if (delegatesScrolling())
     856        return point;
     857
     858    return contentsToView(IntPoint(point));
     859}
     860
    845861IntRect ScrollView::viewToContents(IntRect rect) const
    846862{
    … …  
    909925}
    910926
     927FloatPoint ScrollView::contentsToRootView(const FloatPoint& contentsPoint) const
     928{
     929    return convertToRootView(contentsToView(contentsPoint));
     930}
     931
    911932IntRect ScrollView::rootViewToContents(const IntRect& rootViewRect) const
    912933{
    … …  
    917938{
    918939    return viewToContents(convertFromRootView(rootViewRect));
     940}
     941
     942FloatRect ScrollView::contentsToRootView(const FloatRect& contentsRect) const
     943{
     944    return convertToRootView(contentsToView(contentsRect));
    919945}
    920946

  • trunk/Source/WebCore/platform/ScrollView.h

    r243905 r245320  
    231231
    232232    // Scroll position used by web-exposed features (has legacy iOS behavior).
    233     IntPoint contentsScrollPosition() const;
     233    WEBCORE_EXPORT IntPoint contentsScrollPosition() const;
    234234    void setContentsScrollPosition(const IntPoint&);
    235235
    … …  
    280280    WEBCORE_EXPORT IntPoint rootViewToContents(const IntPoint&) const;
    281281    WEBCORE_EXPORT IntPoint contentsToRootView(const IntPoint&) const;
     282    WEBCORE_EXPORT FloatPoint contentsToRootView(const FloatPoint&) const;
    282283    WEBCORE_EXPORT IntRect rootViewToContents(const IntRect&) const;
    283284    WEBCORE_EXPORT IntRect contentsToRootView(const IntRect&) const;
    284285    WEBCORE_EXPORT FloatRect rootViewToContents(const FloatRect&) const;
     286    WEBCORE_EXPORT FloatRect contentsToRootView(const FloatRect&) const;
    285287
    286288    IntPoint viewToContents(const IntPoint&) const;
    287289    IntPoint contentsToView(const IntPoint&) const;
     290
     291    FloatPoint viewToContents(const FloatPoint&) const;
     292    FloatPoint contentsToView(const FloatPoint&) const;
    288293
    289294    IntRect viewToContents(IntRect) const;

  • trunk/Source/WebCore/platform/Widget.cpp

    r238336 r245320  
    9696}
    9797
     98FloatRect Widget::convertToRootView(const FloatRect& localRect) const
     99{
     100    if (const ScrollView* parentScrollView = parent()) {
     101        FloatRect parentRect = convertToContainingView(localRect);
     102        return parentScrollView->convertToRootView(parentRect);
     103    }
     104    return localRect;
     105}
     106
    98107IntPoint Widget::convertFromRootView(const IntPoint& rootPoint) const
    99108{
    … …  
    109118    if (const ScrollView* parentScrollView = parent()) {
    110119        IntPoint parentPoint = convertToContainingView(localPoint);
     120        return parentScrollView->convertToRootView(parentPoint);
     121    }
     122    return localPoint;
     123}
     124
     125
     126FloatPoint Widget::convertFromRootView(const FloatPoint& rootPoint) const
     127{
     128    if (const ScrollView* parentScrollView = parent()) {
     129        FloatPoint parentPoint = parentScrollView->convertFromRootView(rootPoint);
     130        return convertFromContainingView(parentPoint);
     131    }
     132    return rootPoint;
     133}
     134
     135FloatPoint Widget::convertToRootView(const FloatPoint& localPoint) const
     136{
     137    if (const ScrollView* parentScrollView = parent()) {
     138        FloatPoint parentPoint = convertToContainingView(localPoint);
    111139        return parentScrollView->convertToRootView(parentPoint);
    112140    }
    … …  
    205233}
    206234
     235FloatRect Widget::convertToContainingView(const FloatRect& localRect) const
     236{
     237    return convertToContainingView(IntRect(localRect));
     238}
     239
    207240FloatRect Widget::convertFromContainingView(const FloatRect& parentRect) const
    208241{
    … …  
    226259}
    227260
     261FloatPoint Widget::convertToContainingView(const FloatPoint& localPoint) const
     262{
     263    return convertToContainingView(IntPoint(localPoint));
     264}
     265
     266FloatPoint Widget::convertFromContainingView(const FloatPoint& parentPoint) const
     267{
     268    return convertFromContainingView(IntPoint(parentPoint));
     269}
     270
    228271#if !PLATFORM(COCOA) && !PLATFORM(GTK) && !PLATFORM(WIN)
    229272

  • trunk/Source/WebCore/platform/Widget.h

    r240901 r245320  
    151151    IntRect convertFromRootView(const IntRect&) const;
    152152
     153    FloatRect convertToRootView(const FloatRect&) const;
    153154    FloatRect convertFromRootView(const FloatRect&) const;
    154155
    155156    IntPoint convertToRootView(const IntPoint&) const;
    156157    IntPoint convertFromRootView(const IntPoint&) const;
     158
     159    FloatPoint convertToRootView(const FloatPoint&) const;
     160    FloatPoint convertFromRootView(const FloatPoint&) const;
    157161
    158162    // It is important for cross-platform code to realize that Mac has flipped coordinates.  Therefore any code
    … …  
    187191    WEBCORE_EXPORT virtual IntRect convertToContainingView(const IntRect&) const;
    188192    WEBCORE_EXPORT virtual IntRect convertFromContainingView(const IntRect&) const;
     193    WEBCORE_EXPORT virtual FloatRect convertToContainingView(const FloatRect&) const;
    189194    WEBCORE_EXPORT virtual FloatRect convertFromContainingView(const FloatRect&) const;
    190195    WEBCORE_EXPORT virtual IntPoint convertToContainingView(const IntPoint&) const;
    191196    WEBCORE_EXPORT virtual IntPoint convertFromContainingView(const IntPoint&) const;
     197    WEBCORE_EXPORT virtual FloatPoint convertToContainingView(const FloatPoint&) const;
     198    WEBCORE_EXPORT virtual FloatPoint convertFromContainingView(const FloatPoint&) const;
    192199
    193200private:

  • trunk/Source/WebKit/ChangeLog

    r245301 r245320  
     12019-05-15  Devin Rousso  <drousso@apple.com>
     2
     3        Web Automation: elements larger than the viewport have incorrect in-view center point
     4        https://bugs.webkit.org/show_bug.cgi?id=195696
     5        <rdar://problem/48737122>
     6
     7        Reviewed by Simon Fraser.
     8
     9        Original patch by Brian Burg <bburg@apple.com>.
     10
     11        This seems to be an omission in the specification. While it does mention that the in-view
     12        center point (IVCP) must be within the viewport, the algorithm never intersects the element
     13        bounding box with the viewport rect.
     14
     15        * WebProcess/Automation/WebAutomationSessionProxy.cpp:
     16        (WebKit::WebAutomationSessionProxy::computeElementLayout):
     17        This code is incorrect. For `CoordinateSystem::LayoutViewport`, coordinates should be in
     18        root view coordinates so that it can be later converted to screen and synthesized as a HID
     19        event in screen coordinates. Intersect the element rect and the viewport rect before finding
     20        the center point of the part of the element that's visible in the viewport.
     21
     22        (WebKit::convertRectFromFrameClientToRootView): Added.
     23        (WebKit::convertPointFromFrameClientToRootView): Added.
     24        Added helpers to properly account for scroll contents position on iOS.
     25
     26        * UIProcess/Automation/WebAutomationSession.cpp:
     27        (WebKit::WebAutomationSession::viewportInViewCenterPointOfElement):
     28        Now that we determine whether the element is inside the viewport much earlier, if the
     29        element has no `inViewCenterPoint`, we can return a `TargetOutOfBounds` instead of a more
     30        "generic" `ElementNotInteractable`.
     31
     32        (WebKit::WebAutomationSession::simulateMouseInteraction):
     33        Rename `locationInView` -> `locationInViewport`.
     34
     35        (WebKit::WebAutomationSession::simulateTouchInteraction):
     36        This code is incorrect. The `unobscuredContentRect` is in screen coordinates, but
     37        we are trying to see if (x, y) is outside the size of the viewport assumed to be at (0, 0).
     38        Grab the visual viewport rect and see if the location exceeds the viewport size.
     39
     40        * UIProcess/Automation/ios/WebAutomationSessionIOS.mm:
     41        (WebKit::operator<<):
     42        Add logging helper for `TouchInteraction` enum.
     43
     44        (WebKit::WebAutomationSession::platformSimulateTouchInteraction):
     45        Move local variable.
     46
     47        * UIProcess/Automation/SimulatedInputDispatcher.cpp:
     48        (WebKit::SimulatedInputDispatcher::transitionInputSourceToState):
     49        Fix a typo in logging.
     50
     51        * UIProcess/Automation/Automation.json:
     52        Simplify enum name.
     53
     54        * Platform/Logging.h:
     55        Add logging channel to dump fully resolved interaction details.
     56
    1572019-05-14  Ross Kirsling  <ross.kirsling@sony.com>
    258

  • trunk/Source/WebKit/Platform/Logging.h

    r244849 r245320  
    4343    M(AdClickAttribution) \
    4444    M(Automation) \
     45    M(AutomationInteractions) \
    4546    M(ActivityState) \
    4647    M(BackForward) \

  • trunk/Source/WebKit/UIProcess/Automation/Automation.json

    r243340 r245320  
    3333            "enum": [
    3434                "Page",
    35                 "LayoutViewport"
     35                "Viewport"
    3636            ]
    3737        },

  • trunk/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp

    r244814 r245320  
    291291#if !LOG_DISABLED
    292292                String mouseButtonName = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(a.pressedMouseButton.value());
    293                 LOG(Automation, "SimulatedInputDispatcher[%p]: simulating MouseUp[button=%s] @ (%d, %d) for transition to %d.%d", this, mouseButtonName.utf8().data(), a.location.value().x(), a.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex);
     293                LOG(Automation, "SimulatedInputDispatcher[%p]: simulating MouseUp[button=%s] @ (%d, %d) for transition to %d.%d", this, mouseButtonName.utf8().data(), b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex);
    294294#endif
    295295                m_client.simulateMouseInteraction(m_page, MouseInteraction::Up, a.pressedMouseButton.value(), b.location.value(), WTFMove(eventDispatchFinished));
    … …  
    325325                m_client.simulateTouchInteraction(m_page, TouchInteraction::TouchDown, b.location.value(), WTF::nullopt, WTFMove(eventDispatchFinished));
    326326            } else if (a.pressedMouseButton && !b.pressedMouseButton) {
    327                 LOG(Automation, "SimulatedInputDispatcher[%p]: simulating LiftUp @ (%d, %d) for transition to %d.%d", this, a.location.value().x(), a.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex);
     327                LOG(Automation, "SimulatedInputDispatcher[%p]: simulating LiftUp @ (%d, %d) for transition to %d.%d", this, b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex);
    328328                m_client.simulateTouchInteraction(m_page, TouchInteraction::LiftUp, b.location.value(), WTF::nullopt, WTFMove(eventDispatchFinished));
    329329            } else if (a.location != b.location) {

  • trunk/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp

    r245046 r245320  
    14651465
    14661466        if (!inViewCenterPoint) {
    1467             completionHandler(WTF::nullopt, AUTOMATION_COMMAND_ERROR_WITH_NAME(ElementNotInteractable));
     1467            completionHandler(WTF::nullopt, AUTOMATION_COMMAND_ERROR_WITH_NAME(TargetOutOfBounds));
    14681468            return;
    14691469        }
    … …  
    14781478void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInteraction interaction, WebMouseEvent::Button mouseButton, const WebCore::IntPoint& locationInViewport, CompletionHandler<void(Optional<AutomationCommandError>)>&& completionHandler)
    14791479{
    1480     WebCore::IntPoint locationInView = WebCore::IntPoint(locationInViewport.x(), locationInViewport.y() + page.topContentInset());
    1481     page.getWindowFrameWithCallback([this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler), page = makeRef(page), interaction, mouseButton, locationInView](WebCore::FloatRect windowFrame) mutable {
    1482         auto clippedX = std::min(std::max(0.0f, (float)locationInView.x()), windowFrame.size().width());
    1483         auto clippedY = std::min(std::max(0.0f, (float)locationInView.y()), windowFrame.size().height());
    1484         if (clippedX != locationInView.x() || clippedY != locationInView.y()) {
     1480    page.getWindowFrameWithCallback([this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler), page = makeRef(page), interaction, mouseButton, locationInViewport](WebCore::FloatRect windowFrame) mutable {
     1481        auto clippedX = std::min(std::max(0.0f, (float)locationInViewport.x()), windowFrame.size().width());
     1482        auto clippedY = std::min(std::max(0.0f, (float)locationInViewport.y()), windowFrame.size().height());
     1483        if (clippedX != locationInViewport.x() || clippedY != locationInViewport.y()) {
    14851484            completionHandler(AUTOMATION_COMMAND_ERROR_WITH_NAME(TargetOutOfBounds));
    14861485            return;
    … …  
    14971496        callbackInMap = WTFMove(mouseEventsFlushedCallback);
    14981497
    1499         platformSimulateMouseInteraction(page, interaction, mouseButton, locationInView, OptionSet<WebEvent::Modifier>::fromRaw(m_currentModifiers));
     1498        platformSimulateMouseInteraction(page, interaction, mouseButton, locationInViewport, OptionSet<WebEvent::Modifier>::fromRaw(m_currentModifiers));
    15001499
    15011500        // If the event does not hit test anything in the window, then it may not have been delivered.
    … …  
    15141513{
    15151514#if PLATFORM(IOS_FAMILY)
    1516     if (!page.unobscuredContentRect().contains(locationInViewport)) {
     1515    WebCore::FloatRect visualViewportBounds = WebCore::FloatRect({ }, page.unobscuredContentRect().size());
     1516    if (!visualViewportBounds.contains(locationInViewport)) {
    15171517        completionHandler(AUTOMATION_COMMAND_ERROR_WITH_NAME(TargetOutOfBounds));
    15181518        return;

  • trunk/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm

    r244975 r245320  
    2929#if PLATFORM(IOS_FAMILY)
    3030
     31#import "Logging.h"
    3132#import "NativeWebKeyboardEvent.h"
    3233#import "WebAutomationSessionMacros.h"
    … …  
    176177
    177178#if ENABLE(WEBDRIVER_TOUCH_INTERACTIONS)
     179#if !LOG_DISABLED
     180static TextStream& operator<<(TextStream& ts, TouchInteraction interaction)
     181{
     182    switch (interaction) {
     183    case TouchInteraction::TouchDown:
     184        ts << "TouchDown";
     185        break;
     186    case TouchInteraction::MoveTo:
     187        ts << "MoveTo";
     188        break;
     189    case TouchInteraction::LiftUp:
     190        ts << "LiftUp";
     191        break;
     192    }
     193    return ts;
     194}
     195#endif // !LOG_DISABLED
     196
    178197void WebAutomationSession::platformSimulateTouchInteraction(WebPageProxy& page, TouchInteraction interaction, const WebCore::IntPoint& locationInViewport, Optional<Seconds> duration, AutomationCompletionHandler&& completionHandler)
    179198{
    180199    WebCore::IntPoint locationOnScreen = page.syncRootViewToScreen(IntRect(locationInViewport, IntSize())).location();
    181     _WKTouchEventGenerator *generator = [_WKTouchEventGenerator sharedTouchEventGenerator];
     200    LOG_WITH_STREAM(AutomationInteractions, stream << "platformSimulateTouchInteraction: interaction=" << interaction << ", locationInViewport=" << locationInViewport << ", locationOnScreen=" << locationOnScreen << ", duration=" << duration.valueOr(0_s).seconds());
    182201
    183202    auto interactionFinished = makeBlockPtr([completionHandler = WTFMove(completionHandler)] () mutable {
    184203        completionHandler(WTF::nullopt);
    185204    });
    186    
     205
     206    _WKTouchEventGenerator *generator = [_WKTouchEventGenerator sharedTouchEventGenerator];
    187207    switch (interaction) {
    188208    case TouchInteraction::TouchDown:

  • trunk/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp

    r244033 r245320  
    478478}
    479479
    480 static Optional<WebCore::FloatPoint> elementInViewClientCenterPoint(WebCore::Element& element, bool& isObscured)
    481 {
    482     // §11.1 Element Interactability.
    483     // https://www.w3.org/TR/webdriver/#dfn-in-view-center-point
    484     auto* clientRect = element.getClientRects()->item(0);
    485     if (!clientRect)
    486         return WTF::nullopt;
    487 
    488     auto clientCenterPoint = WebCore::FloatPoint::narrowPrecision(0.5 * (clientRect->left() + clientRect->right()), 0.5 * (clientRect->top() + clientRect->bottom()));
    489     auto elementList = element.treeScope().elementsFromPoint(clientCenterPoint.x(), clientCenterPoint.y());
    490     if (elementList.isEmpty()) {
    491         // An element is obscured if the pointer-interactable paint tree at its center point is empty,
    492         // or the first element in this tree is not an inclusive descendant of itself.
    493         // https://w3c.github.io/webdriver/webdriver-spec.html#dfn-obscured
    494         isObscured = true;
    495         return clientCenterPoint;
    496     }
    497 
    498     auto index = elementList.findMatching([&element] (auto& item) { return item.get() == &element; });
    499     if (index == notFound)
    500         return WTF::nullopt;
    501 
    502     if (index) {
    503         // Element is not the first one in the list.
    504         auto firstElement = elementList[0];
    505         isObscured = !firstElement->isDescendantOf(element);
    506     }
    507 
    508     return clientCenterPoint;
    509 }
    510 
    511480static WebCore::Element* containerElementForElement(WebCore::Element& element)
    512481{
    … …  
    535504}
    536505
     506static WebCore::FloatRect convertRectFromFrameClientToRootView(FrameView* frameView, WebCore::FloatRect clientRect)
     507{
     508    if (!frameView->delegatesScrolling())
     509        return frameView->contentsToRootView(frameView->clientToDocumentRect(clientRect));
     510
     511    // If the frame delegates scrolling, contentsToRootView doesn't take into account scroll/zoom/scale.
     512    auto& frame = frameView->frame();
     513    clientRect.scale(frame.pageZoomFactor() * frame.frameScaleFactor());
     514    clientRect.moveBy(frameView->contentsScrollPosition());
     515    return clientRect;
     516}
     517
     518static WebCore::FloatPoint convertPointFromFrameClientToRootView(FrameView* frameView, WebCore::FloatPoint clientPoint)
     519{
     520    if (!frameView->delegatesScrolling())
     521        return frameView->contentsToRootView(frameView->clientToDocumentPoint(clientPoint));
     522
     523    // If the frame delegates scrolling, contentsToRootView doesn't take into account scroll/zoom/scale.
     524    auto& frame = frameView->frame();
     525    clientPoint.scale(frame.pageZoomFactor() * frame.frameScaleFactor());
     526    clientPoint.moveBy(frameView->contentsScrollPosition());
     527    return clientPoint;
     528}
     529
    537530void WebAutomationSessionProxy::computeElementLayout(uint64_t pageID, uint64_t frameID, String nodeHandle, bool scrollIntoViewIfNeeded, CoordinateSystem coordinateSystem, CompletionHandler<void(Optional<String>, WebCore::IntRect, Optional<WebCore::IntPoint>, bool)>&& completionHandler)
    538531{
    … …  
    544537    }
    545538
    546     String frameNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::FrameNotFound);
    547     String nodeNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NodeNotFound);
    548     String notImplementedErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NotImplemented);
    549 
    550539    WebFrame* frame = frameID ? WebProcess::singleton().webFrame(frameID) : page->mainWebFrame();
    551540    if (!frame || !frame->coreFrame() || !frame->coreFrame()->view()) {
     541        String frameNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::FrameNotFound);
    552542        completionHandler(frameNotFoundErrorType, { }, WTF::nullopt, false);
    553543        return;
    … …  
    556546    WebCore::Element* coreElement = elementForNodeHandle(*frame, nodeHandle);
    557547    if (!coreElement) {
     548        String nodeNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NodeNotFound);
    558549        completionHandler(nodeNotFoundErrorType, { }, WTF::nullopt, false);
    559550        return;
    … …  
    570561    WebCore::FrameView* frameView = frame->coreFrame()->view();
    571562    WebCore::FrameView* mainView = frame->coreFrame()->mainFrame().view();
    572     WebCore::IntRect frameElementBounds = roundedIntRect(coreElement->boundingClientRect());
    573     WebCore::IntRect rootElementBounds = mainView->rootViewToContents(frameView->contentsToRootView(frameElementBounds));
     563
    574564    WebCore::IntRect resultElementBounds;
     565    Optional<WebCore::IntPoint> resultInViewCenterPoint;
     566    bool isObscured = false;
     567
     568    auto elementBoundsInRootCoordinates = convertRectFromFrameClientToRootView(frameView, coreElement->boundingClientRect());
    575569    switch (coordinateSystem) {
    576570    case CoordinateSystem::Page:
    577         resultElementBounds = WebCore::IntRect(mainView->clientToDocumentRect(WebCore::FloatRect(rootElementBounds)));
     571        resultElementBounds = enclosingIntRect(mainView->absoluteToDocumentRect(mainView->rootViewToContents(elementBoundsInRootCoordinates)));
    578572        break;
    579573    case CoordinateSystem::LayoutViewport:
    580         // The element bounds are already in client coordinates.
    581         resultElementBounds = WebCore::IntRect(mainView->clientToLayoutViewportRect(WebCore::FloatRect(rootElementBounds)));
     574        resultElementBounds = enclosingIntRect(mainView->absoluteToLayoutViewportRect(elementBoundsInRootCoordinates));
    582575        break;
    583576    }
    584577
    585     Optional<WebCore::IntPoint> resultInViewCenterPoint;
    586     bool isObscured = false;
    587     if (containerElement) {
    588         Optional<WebCore::FloatPoint> frameInViewCenterPoint = elementInViewClientCenterPoint(*containerElement, isObscured);
    589         if (frameInViewCenterPoint.hasValue()) {
    590             WebCore::IntPoint rootInViewCenterPoint = mainView->rootViewToContents(frameView->contentsToRootView(WebCore::IntPoint(frameInViewCenterPoint.value())));
    591             switch (coordinateSystem) {
    592             case CoordinateSystem::Page:
    593                 resultInViewCenterPoint = WebCore::IntPoint(mainView->clientToDocumentPoint(rootInViewCenterPoint));
    594                 break;
    595             case CoordinateSystem::LayoutViewport:
    596                 // The point is already in client coordinates.
    597                 resultInViewCenterPoint = WebCore::IntPoint(mainView->clientToLayoutViewportPoint(rootInViewCenterPoint));
    598                 break;
    599             }
    600         }
     578    // If an <option> or <optgroup> does not have an associated <select> or <datalist> element, then give up.
     579    if (!containerElement) {
     580        String elementNotInteractableErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::ElementNotInteractable);
     581        completionHandler(elementNotInteractableErrorType, resultElementBounds, resultInViewCenterPoint, isObscured);
     582        return;
     583    }
     584
     585    // §12.1 Element Interactability.
     586    // https://www.w3.org/TR/webdriver/#dfn-in-view-center-point
     587    auto* firstElementRect = containerElement->getClientRects()->item(0);
     588    if (!firstElementRect) {
     589        String elementNotInteractableErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::ElementNotInteractable);
     590        completionHandler(elementNotInteractableErrorType, resultElementBounds, resultInViewCenterPoint, isObscured);
     591        return;
     592    }
     593
     594    // The W3C WebDriver specification does not explicitly intersect the element with the visual viewport.
     595    // Do that here so that the IVCP for an element larger than the viewport is within the viewport.
     596    // See spec bug here: https://github.com/w3c/webdriver/issues/1402
     597    auto viewportRect = frameView->documentToClientRect(frameView->visualViewportRect());
     598    auto elementRect = FloatRect(firstElementRect->x(), firstElementRect->y(), firstElementRect->width(), firstElementRect->height());
     599    auto visiblePortionOfElementRect = intersection(viewportRect, elementRect);
     600
     601    // If the element is entirely outside the viewport, still calculate it's bounds.
     602    if (visiblePortionOfElementRect.isEmpty()) {
     603        completionHandler(WTF::nullopt, resultElementBounds, resultInViewCenterPoint, isObscured);
     604        return;
     605    }
     606
     607    auto elementInViewCenterPoint = visiblePortionOfElementRect.center();
     608    auto elementList = containerElement->treeScope().elementsFromPoint(elementInViewCenterPoint);
     609    auto index = elementList.findMatching([containerElement] (auto& item) { return item.get() == containerElement; });
     610    if (elementList.isEmpty() || index == notFound) {
     611        // We hit this case if the element is visibility:hidden or opacity:0, in which case it will not hit test
     612        // at the calculated IVCP. An element is technically not "in view" if it is not within its own paint/hit test tree,
     613        // so it cannot have an in-view center point either. And without an IVCP, the definition of 'obscured' makes no sense.
     614        // See <https://w3c.github.io/webdriver/webdriver-spec.html#dfn-in-view>.
     615        String elementNotInteractableErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::ElementNotInteractable);
     616        completionHandler(elementNotInteractableErrorType, resultElementBounds, resultInViewCenterPoint, isObscured);
     617        return;
     618    }
     619
     620    // Check the case where a non-descendant element hit tests before the target element. For example, a child <option>
     621    // of a <select> does not obscure the <select>, but two sibling <div> that overlap at the IVCP will obscure each other.
     622    // Node::isDescendantOf() is not self-inclusive, so that is explicitly checked here.
     623    isObscured = elementList[0] != containerElement && !elementList[0]->isDescendantOf(containerElement);
     624
     625    auto inViewCenterPointInRootCoordinates = convertPointFromFrameClientToRootView(frameView, elementInViewCenterPoint);
     626    switch (coordinateSystem) {
     627    case CoordinateSystem::Page:
     628        resultInViewCenterPoint = roundedIntPoint(mainView->absoluteToDocumentPoint(inViewCenterPointInRootCoordinates));
     629        break;
     630    case CoordinateSystem::LayoutViewport:
     631        resultInViewCenterPoint = roundedIntPoint(mainView->absoluteToLayoutViewportPoint(inViewCenterPointInRootCoordinates));
     632        break;
    601633    }
    602634
Note: See TracChangeset for help on using the changeset viewer.