Changeset 198792 in webkit


Ignore:
Timestamp:
Mar 29, 2016, 1:19:45 PM (9 years ago)
Author:
BJ Burg
Message:

Web Automation: implement Automation.performMouseInteraction
​https://bugs.webkit.org/show_bug.cgi?id=155606
<rdar://problem/25227576>

Source/WebKit2:

Reviewed by Timothy Hatcher.

This method implements common mouse interactions by synthesizing native events.
The creation and dispatching of simulated events is implemented separately per
application/window/event framework. This patch adds an implementation for ports
that use AppKit.

The event is created synthetically in a platform-specific way and delivered to
the platform's window in the UIProcess. The event goes through all the
layers of processing that a real mouse event could go through once it reaches
WebKit itself.

  • UIProcess/Automation/Automation.json: Add new enums and command.
  • UIProcess/Automation/WebAutomationSession.cpp:

(WebKit::protocolModifierToWebEventModifier):
(WebKit::WebAutomationSession::performMouseInteraction): Added.
Clip the requested cursor position so that it stays within the selected window's
frame and doesn't wander off into other windows or applications. Fire it using
the platform-specific simulation method.

(WebKit::WebAutomationSession::platformSimulateMouseInteraction): Added.

  • UIProcess/Automation/WebAutomationSession.h:
  • UIProcess/Cocoa/WebAutomationSessionCocoa.mm:

(WebKit::WebAutomationSession::platformSimulateMouseInteraction): Added.
Convert Automation protocol values to AppKit values. Simulate and dispatch
one or more events depending on the desired interaction.

  • WebKit2Prefix.h: Include AppKitCompatibilityDeclarations.h so newer

NSEventTypes can be used.

Source/WTF:

Reviewed by Timothy Hatcher.

  • wtf/mac/AppKitCompatibilityDeclarations.h: Add missing some NSEventTypes.
Location:
trunk/Source
Files:
1 added
8 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WTF/ChangeLog

    r198778 r198792  
     12016-03-28  Brian Burg  <bburg@apple.com>
     2
     3        Web Automation: implement Automation.performMouseInteraction
     4        https://bugs.webkit.org/show_bug.cgi?id=155606
     5        <rdar://problem/25227576>
     6
     7        Reviewed by Timothy Hatcher.
     8
     9        * wtf/mac/AppKitCompatibilityDeclarations.h: Add missing some NSEventTypes.
     10
    1112016-03-29  Yusuke Suzuki  <utatane.tea@gmail.com>
    212

  • trunk/Source/WTF/wtf/mac/AppKitCompatibilityDeclarations.h

    r197819 r198792  
    7979static const NSEventType NSEventTypeMouseMoved = NSMouseMoved;
    8080static const NSEventType NSEventTypeOtherMouseDown = NSOtherMouseDown;
     81static const NSEventType NSEventTypeOtherMouseDragged = NSOtherMouseDragged;
     82static const NSEventType NSEventTypeOtherMouseUp = NSOtherMouseUp;
    8183static const NSEventType NSEventTypePeriodic = NSPeriodic;
    8284static const NSEventType NSEventTypeRightMouseDown = NSRightMouseDown;
     85static const NSEventType NSEventTypeRightMouseDragged = NSRightMouseDragged;
     86static const NSEventType NSEventTypeRightMouseUp = NSRightMouseUp;
    8387static const NSEventType NSEventTypeScrollWheel = NSScrollWheel;
    8488static const NSEventType NSEventTypeSystemDefined = NSSystemDefined;

  • trunk/Source/WebKit2/ChangeLog

    r198775 r198792  
     12016-03-28  Brian Burg  <bburg@apple.com>
     2
     3        Web Automation: implement Automation.performMouseInteraction
     4        https://bugs.webkit.org/show_bug.cgi?id=155606
     5        <rdar://problem/25227576>
     6
     7        Reviewed by Timothy Hatcher.
     8
     9        This method implements common mouse interactions by synthesizing native events.
     10        The creation and dispatching of simulated events is implemented separately per
     11        application/window/event framework. This patch adds an implementation for ports
     12        that use AppKit.
     13
     14        The event is created synthetically in a platform-specific way and delivered to
     15        the platform's window in the UIProcess. The event goes through all the
     16        layers of processing that a real mouse event could go through once it reaches
     17        WebKit itself.
     18
     19        * UIProcess/Automation/Automation.json: Add new enums and command.
     20
     21        * UIProcess/Automation/WebAutomationSession.cpp:
     22        (WebKit::protocolModifierToWebEventModifier):
     23        (WebKit::WebAutomationSession::performMouseInteraction): Added.
     24        Clip the requested cursor position so that it stays within the selected window's
     25        frame and doesn't wander off into other windows or applications. Fire it using
     26        the platform-specific simulation method.
     27
     28        (WebKit::WebAutomationSession::platformSimulateMouseInteraction): Added.
     29
     30        * UIProcess/Automation/WebAutomationSession.h:
     31        * UIProcess/Cocoa/WebAutomationSessionCocoa.mm:
     32        (WebKit::WebAutomationSession::platformSimulateMouseInteraction): Added.
     33        Convert Automation protocol values to AppKit values. Simulate and dispatch
     34        one or more events depending on the desired interaction.
     35        * WebKit2Prefix.h: Include AppKitCompatibilityDeclarations.h so newer
     36        NSEventTypes can be used.
     37
    1382016-03-29  Carlos Garcia Campos  <cgarcia@igalia.com>
    239

  • trunk/Source/WebKit2/UIProcess/Automation/Automation.json

    r198760 r198792  
    6969                { "name": "windowOrigin", "$ref": "Point", "description": "The current (x, y) position of the browsing context's window relative to the top-left of the screen." }
    7070            ]
     71        },
     72        {
     73            "id": "MouseInteraction",
     74            "type": "string",
     75            "description": "Enumerates different ways of interacting with a mouse cursor.",
     76            "enum": [
     77                "Move",
     78                "Down",
     79                "Up",
     80                "SingleClick",
     81                "DoubleClick"
     82            ]
     83        },
     84        {
     85            "id": "MouseButton",
     86            "type": "string",
     87            "description": "Enumerates different mouse buttons that can be used.",
     88            "enum": [
     89                "None",
     90                "Left",
     91                "Middle",
     92                "Right"
     93            ]
     94        },
     95        {
     96            "id": "KeyModifier",
     97            "type": "string",
     98            "description": "Enumerates different key modifiers that can remain pressed during other mouse/touch interactions.",
     99            "enum": [
     100                "CapsLock",
     101                "Control",
     102                "Shift",
     103                "Meta",
     104                "Alt"
     105            ]
    71106        }
    72107    ],
    … …  
    172207        },
    173208        {
     209            "name": "performMouseInteraction",
     210            "description": "Simulates interaction with a mouse cursor.",
     211            "parameters": [
     212                { "name": "handle", "$ref": "BrowsingContextHandle", "description": "The browsing context to be interacted with." },
     213                { "name": "position", "$ref": "Point", "description": "The requested position for the mouse cursor, specified in viewport coordinates." },
     214                { "name": "button", "$ref": "MouseButton", "description": "The button to use to perform the interaction. This parameter only has an effect for the <code>Down</code>, <code>Up</code>, <code>Click</code>, and <code>DoubleClick</code> mouse interactions." },
     215                { "name": "interaction", "$ref": "MouseInteraction", "description": "The type of interaction to be performed with the mouse cursor." },
     216                { "name": "modifiers", "type": "array", "items": { "$ref": "KeyModifier" }, "description": "Key modifiers to be active during the mouse interaction." }
     217            ],
     218            "returns": [
     219                { "name": "position", "$ref": "Point", "description": "The updated position of the mouse cursor, specified in viewport coordinates." }
     220            ]
     221        },
     222        {
    174223            "name": "resolveChildFrameHandle",
    175224            "description": "Determines the <code>FrameHandle</code> based on the ordinal, name or node handle of a child frame.",

  • trunk/Source/WebKit2/UIProcess/Automation/WebAutomationSession.cpp

    r198760 r198792  
    3636#include <WebCore/URL.h>
    3737#include <WebCore/UUID.h>
     38#include <algorithm>
    3839#include <wtf/HashMap.h>
    3940
    … …  
    632633}
    633634
     635#if USE(APPKIT)
     636static WebEvent::Modifiers protocolModifierToWebEventModifier(Inspector::Protocol::Automation::KeyModifier modifier)
     637{
     638    switch (modifier) {
     639    case Inspector::Protocol::Automation::KeyModifier::Alt:
     640        return WebEvent::AltKey;
     641    case Inspector::Protocol::Automation::KeyModifier::Meta:
     642        return WebEvent::MetaKey;
     643    case Inspector::Protocol::Automation::KeyModifier::Control:
     644        return WebEvent::ControlKey;
     645    case Inspector::Protocol::Automation::KeyModifier::Shift:
     646        return WebEvent::ShiftKey;
     647    case Inspector::Protocol::Automation::KeyModifier::CapsLock:
     648        return WebEvent::CapsLockKey;
     649    }
     650}
     651#endif // USE(APPKIT)
     652
     653void WebAutomationSession::performMouseInteraction(Inspector::ErrorString& errorString, const String& handle, const Inspector::InspectorObject& requestedPositionObject, const String& mouseButtonString, const String& mouseInteractionString, const Inspector::InspectorArray& keyModifierStrings, RefPtr<Inspector::Protocol::Automation::Point>& updatedPositionObject)
     654{
     655#if !USE(APPKIT)
     656    FAIL_WITH_PREDEFINED_ERROR_MESSAGE(NotImplemented);
     657#else
     658    WebPageProxy* page = webPageProxyForHandle(handle);
     659    if (!page)
     660        FAIL_WITH_PREDEFINED_ERROR_MESSAGE(WindowNotFound);
     661
     662    // FIXME <rdar://problem/25094106>: Specify what parameter was missing or invalid and how.
     663    // This requires some changes to the other end's error handling. Right now it looks for an
     664    // exact error message match. We could stuff this into the 'data' field on error object.
     665    float x;
     666    if (!requestedPositionObject.getDouble(WTF::ASCIILiteral("x"), x))
     667        FAIL_WITH_PREDEFINED_ERROR_MESSAGE(MissingParameter);
     668
     669    float y;
     670    if (!requestedPositionObject.getDouble(WTF::ASCIILiteral("y"), y))
     671        FAIL_WITH_PREDEFINED_ERROR_MESSAGE(MissingParameter);
     672
     673    WebCore::FloatRect windowFrame;
     674    page->getWindowFrame(windowFrame);
     675
     676    x = std::max(std::min(0.0f, x), windowFrame.size().width());
     677    y = std::max(std::min(0.0f, y), windowFrame.size().height());
     678
     679    WebCore::IntPoint viewPosition = WebCore::IntPoint(static_cast<int>(x), static_cast<int>(y));
     680
     681    auto parsedInteraction = Inspector::Protocol::parseEnumValueFromString<Inspector::Protocol::Automation::MouseInteraction>(mouseInteractionString);
     682    if (!parsedInteraction)
     683        FAIL_WITH_PREDEFINED_ERROR_MESSAGE(InvalidParameter);
     684
     685    auto parsedButton = Inspector::Protocol::parseEnumValueFromString<Inspector::Protocol::Automation::MouseButton>(mouseButtonString);
     686    if (!parsedButton)
     687        FAIL_WITH_PREDEFINED_ERROR_MESSAGE(InvalidParameter);
     688
     689    WebEvent::Modifiers keyModifiers = (WebEvent::Modifiers)0;
     690    for (auto it = keyModifierStrings.begin(); it != keyModifierStrings.end(); ++it) {
     691        String modifierString;
     692        if (!it->get()->asString(modifierString))
     693            FAIL_WITH_PREDEFINED_ERROR_MESSAGE(InvalidParameter);
     694
     695        auto parsedModifier = Inspector::Protocol::parseEnumValueFromString<Inspector::Protocol::Automation::KeyModifier>(modifierString);
     696        if (!parsedModifier)
     697            FAIL_WITH_PREDEFINED_ERROR_MESSAGE(InvalidParameter);
     698        WebEvent::Modifiers enumValue = protocolModifierToWebEventModifier(parsedModifier.value());
     699        keyModifiers = (WebEvent::Modifiers)(enumValue | keyModifiers);
     700    }
     701
     702    platformSimulateMouseInteraction(*page, viewPosition, parsedInteraction.value(), parsedButton.value(), keyModifiers);
     703
     704    updatedPositionObject = Inspector::Protocol::Automation::Point::create()
     705        .setX(x)
     706        .setY(y)
     707        .release();
     708#endif // USE(APPKIT)
     709}
     710
     711#if !USE(APPKIT)
     712void WebAutomationSession::platformSimulateMouseInteraction(WebKit::WebPageProxy&, const WebCore::IntPoint&, Inspector::Protocol::Automation::MouseInteraction, Inspector::Protocol::Automation::MouseButton, WebEvent::Modifiers)
     713{
     714}
     715#endif // !USE(APPKIT)
     716
    634717} // namespace WebKit

  • trunk/Source/WebKit2/UIProcess/Automation/WebAutomationSession.h

    r198760 r198792  
    2424 */
    2525
    26 #ifndef WebAutomationSession_h
    27 #define WebAutomationSession_h
     26#pragma once
    2827
    2928#include "APIObject.h"
    3029#include "AutomationBackendDispatchers.h"
    3130#include "Connection.h"
     31#include "WebEvent.h"
    3232#include <wtf/Forward.h>
    3333
    … …  
    4848class IntRect;
    4949}
     50
     51namespace WebCore {
     52class IntPoint;
     53}
     54
     55#if USE(APPKIT)
     56OBJC_CLASS NSArray;
     57#endif
    5058
    5159namespace WebKit {
    … …  
    95103    void reloadBrowsingContext(Inspector::ErrorString&, const String&) override;
    96104    void evaluateJavaScriptFunction(Inspector::ErrorString&, const String& browsingContextHandle, const String* optionalFrameHandle, const String& function, const Inspector::InspectorArray& arguments, bool expectsImplicitCallbackArgument, Ref<Inspector::AutomationBackendDispatcherHandler::EvaluateJavaScriptFunctionCallback>&&) override;
     105    void performMouseInteraction(Inspector::ErrorString&, const String& handle, const Inspector::InspectorObject& requestedPosition, const String& mouseButton, const String& mouseInteraction, const Inspector::InspectorArray& keyModifiers, RefPtr<Inspector::Protocol::Automation::Point>& updatedPosition) override;
    97106    void resolveChildFrameHandle(Inspector::ErrorString&, const String& browsingContextHandle, const String* optionalFrameHandle, const int* optionalOrdinal, const String* optionalName, const String* optionalNodeHandle, Ref<ResolveChildFrameHandleCallback>&&) override;
    98107    void resolveParentFrameHandle(Inspector::ErrorString&, const String& browsingContextHandle, const String& frameHandle, Ref<ResolveParentFrameHandleCallback>&&) override;
    … …  
    121130    void didResolveParentFrame(uint64_t callbackID, uint64_t frameID, const String& errorType);
    122131    void didComputeElementLayout(uint64_t callbackID, WebCore::IntRect, const String& errorType);
     132
     133    // Platform-specific helper methods.
     134    void platformSimulateMouseInteraction(WebPageProxy&, const WebCore::IntPoint& viewPosition, Inspector::Protocol::Automation::MouseInteraction, Inspector::Protocol::Automation::MouseButton, WebEvent::Modifiers);
     135
     136#if USE(APPKIT)
     137    void sendSynthesizedEventsToPage(WebPageProxy&, NSArray *eventsToSend);
     138#endif
    123139
    124140    WebProcessPool* m_processPool { nullptr };
    … …  
    155171
    156172} // namespace WebKit
    157 
    158 #endif // WebAutomationSession_h

  • trunk/Source/WebKit2/WebKit2.xcodeproj/project.pbxproj

    r198736 r198792  
    12391239                9955A6F61C7986E300EB6A93 /* AutomationProtocolObjects.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 9955A6F21C79866400EB6A93 /* AutomationProtocolObjects.cpp */; };
    12401240                9955A6F71C7986E500EB6A93 /* AutomationProtocolObjects.h in Headers */ = {isa = PBXBuildFile; fileRef = 9955A6F31C79866400EB6A93 /* AutomationProtocolObjects.h */; };
     1241                9986BDD71CA9A22C004800AA /* WebAutomationSessionCocoa.mm in Sources */ = {isa = PBXBuildFile; fileRef = 9986BDD61CA9A222004800AA /* WebAutomationSessionCocoa.mm */; };
    12411242                99C81D591C20E1E5005C4C82 /* AutomationClient.mm in Sources */ = {isa = PBXBuildFile; fileRef = 99C81D561C20DFBE005C4C82 /* AutomationClient.mm */; };
    12421243                99C81D5A1C20E7E2005C4C82 /* AutomationClient.h in Headers */ = {isa = PBXBuildFile; fileRef = 99C81D551C20DFBE005C4C82 /* AutomationClient.h */; };
    … …  
    32723273                9955A6F21C79866400EB6A93 /* AutomationProtocolObjects.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = AutomationProtocolObjects.cpp; sourceTree = "<group>"; };
    32733274                9955A6F31C79866400EB6A93 /* AutomationProtocolObjects.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AutomationProtocolObjects.h; sourceTree = "<group>"; };
     3275                9986BDD61CA9A222004800AA /* WebAutomationSessionCocoa.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WebAutomationSessionCocoa.mm; sourceTree = "<group>"; };
    32743276                99C81D551C20DFBE005C4C82 /* AutomationClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AutomationClient.h; sourceTree = "<group>"; };
    32753277                99C81D561C20DFBE005C4C82 /* AutomationClient.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = AutomationClient.mm; sourceTree = "<group>"; };
    … …  
    46324634                                E4E8648E1B1673FB00C82F40 /* VersionChecks.h */,
    46334635                                E4E8648F1B1673FB00C82F40 /* VersionChecks.mm */,
     4636                                9986BDD61CA9A222004800AA /* WebAutomationSessionCocoa.mm */,
    46344637                                1AC0273E196622D600C12B75 /* WebPageProxyCocoa.mm */,
    46354638                                7C4694CB1A4B510A00AD5845 /* WebPasteboardProxyCocoa.mm */,
    … …  
    94569459                                1AA2E56618D77508003814BD /* WKWebProcessBundleParameters.mm in Sources */,
    94579460                                BC989D81161A7E5D000D46D3 /* WKWebProcessPlugIn.mm in Sources */,
     9461                                9986BDD71CA9A22C004800AA /* WebAutomationSessionCocoa.mm in Sources */,
    94589462                                BC8F2F2A16273A2C005FACB5 /* WKWebProcessPlugInBrowserContextController.mm in Sources */,
    94599463                                1F7506AD1859161C00EC0FF7 /* WKWebProcessPlugInFrame.mm in Sources */,

  • trunk/Source/WebKit2/WebKit2Prefix.h

    r192697 r198792  
    5252#if USE(APPKIT)
    5353#import <Cocoa/Cocoa.h>
     54#import <wtf/mac/AppKitCompatibilityDeclarations.h>
    5455#endif
    5556#endif
Note: See TracChangeset for help on using the changeset viewer.