Changeset 233131 in webkit


Ignore:
Timestamp:
Jun 23, 2018 12:17:09 PM (6 years ago)
Author:
BJ Burg
Message:

Web Automation: key actions should support multiple pressed virtual keys
​https://bugs.webkit.org/show_bug.cgi?id=186899
<rdar://problem/38222248>

Reviewed by Timothy Hatcher.

Source/WebDriver:

Adopt new protocol command argument types.

  • Session.cpp:

(WebDriver::Session::performActions):

Source/WebKit:

This patch changes the protocol to allow multiple virtual keys per input source state.
Chords like Cmd-Shift-A and Shift-F12 must be represented this way as they are encoded
in the VirtualKey enum rather than as an ASCII char.

  • UIProcess/Automation/Automation.json:
  • UIProcess/Automation/SimulatedInputDispatcher.h:
  • UIProcess/Automation/SimulatedInputDispatcher.cpp:

(WebKit::SimulatedInputDispatcher::transitionInputSourceToState):

  • UIProcess/Automation/WebAutomationSession.cpp:

(WebKit::WebAutomationSession::simulateKeyboardInteraction):
(WebKit::WebAutomationSession::performKeyboardInteractions):
(WebKit::WebAutomationSession::performInteractionSequence):

  • UIProcess/Automation/WebAutomationSession.h:
  • UIProcess/Automation/gtk/WebAutomationSessionGtk.cpp:

(WebKit::WebAutomationSession::platformSimulateKeyboardInteraction):

  • UIProcess/Automation/ios/WebAutomationSessionIOS.mm:

(WebKit::WebAutomationSession::platformSimulateKeyboardInteraction):

  • UIProcess/Automation/mac/WebAutomationSessionMac.mm:

(WebKit::WebAutomationSession::platformSimulateKeyboardInteraction):
Also clean up the signature of WebAutomationSession::platformSimulateKeyboardInteraction
to use a variant instead of mutually exclusive optional values with different types.

Location:
trunk/Source
Files:
11 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebDriver/ChangeLog

    r233122 r233131  
     12018-06-21  Brian Burg  <bburg@apple.com>
     2
     3        Web Automation: key actions should support multiple pressed virtual keys
     4        https://bugs.webkit.org/show_bug.cgi?id=186899
     5        <rdar://problem/38222248>
     6
     7        Reviewed by Timothy Hatcher.
     8
     9        Adopt new protocol command argument types.
     10
     11        * Session.cpp:
     12        (WebDriver::Session::performActions):
     13
    1142018-06-23  Yusuke Suzuki  <utatane.tea@gmail.com>
    215

  • trunk/Source/WebDriver/Session.cpp

    r233122 r233131  
    23192319                    if (currentState.pressedKey)
    23202320                        state->setString("pressedCharKey"_s, currentState.pressedKey.value());
    2321                     if (currentState.pressedVirtualKey)
    2322                         state->setString("pressedVirtualKey"_s, currentState.pressedVirtualKey.value());
     2321                    if (currentState.pressedVirtualKey) {
     2322                        // FIXME: support parsing and tracking multiple virtual keys.
     2323                        Ref<JSON::Array> virtualKeys = JSON::Array::create();
     2324                        virtualKeys->pushString(currentState.pressedVirtualKey.value());
     2325                        state->setArray("pressedVirtualKeys"_s, WTFMove(virtualKeys));
     2326                    }
    23232327                    break;
    23242328                }

  • trunk/Source/WebKit/ChangeLog

    r233122 r233131  
     12018-06-21  Brian Burg  <bburg@apple.com>
     2
     3        Web Automation: key actions should support multiple pressed virtual keys
     4        https://bugs.webkit.org/show_bug.cgi?id=186899
     5        <rdar://problem/38222248>
     6
     7        Reviewed by Timothy Hatcher.
     8
     9        This patch changes the protocol to allow multiple virtual keys per input source state.
     10        Chords like Cmd-Shift-A and Shift-F12 must be represented this way as they are encoded
     11        in the VirtualKey enum rather than as an ASCII char.
     12
     13        * UIProcess/Automation/Automation.json:
     14        * UIProcess/Automation/SimulatedInputDispatcher.h:
     15        * UIProcess/Automation/SimulatedInputDispatcher.cpp:
     16        (WebKit::SimulatedInputDispatcher::transitionInputSourceToState):
     17        * UIProcess/Automation/WebAutomationSession.cpp:
     18        (WebKit::WebAutomationSession::simulateKeyboardInteraction):
     19        (WebKit::WebAutomationSession::performKeyboardInteractions):
     20        (WebKit::WebAutomationSession::performInteractionSequence):
     21
     22        * UIProcess/Automation/WebAutomationSession.h:
     23        * UIProcess/Automation/gtk/WebAutomationSessionGtk.cpp:
     24        (WebKit::WebAutomationSession::platformSimulateKeyboardInteraction):
     25        * UIProcess/Automation/ios/WebAutomationSessionIOS.mm:
     26        (WebKit::WebAutomationSession::platformSimulateKeyboardInteraction):
     27        * UIProcess/Automation/mac/WebAutomationSessionMac.mm:
     28        (WebKit::WebAutomationSession::platformSimulateKeyboardInteraction):
     29        Also clean up the signature of WebAutomationSession::platformSimulateKeyboardInteraction
     30        to use a variant instead of mutually exclusive optional values with different types.
     31
    1322018-06-23  Yusuke Suzuki  <utatane.tea@gmail.com>
    233

  • trunk/Source/WebKit/UIProcess/Automation/Automation.json

    r232150 r233131  
    291291                { "name": "sourceId", "type": "string", "description": "The input source whose state is described by this object." },
    292292                { "name": "pressedCharKey", "type": "string", "optional": true, "description": "For 'keyboard' input sources, specifies a character key that has 'pressed' state. Unmentioned character keys are assumed to have a 'released' state." },
    293                 { "name": "pressedVirtualKey", "$ref": "VirtualKey", "optional": true, "description": "For 'keyboard' input sources, specifies a virtual key that has a 'pressed' state. Unmentioned virtual keys are assumed to have a 'released' state." },
     293                { "name": "pressedVirtualKeys", "type": "array", "items": { "$ref": "VirtualKey" }, "optional": true, "description": "For 'keyboard' input sources, specifies virtual keys that have a 'pressed' state. Unmentioned virtual keys are assumed to have a 'released' state." },
    294294                { "name": "pressedButton", "$ref": "MouseButton", "optional": true, "description": "For 'mouse' input sources, specifies which mouse button has a 'pressed' state. Unmentioned mouse buttons are assumed to have a 'released' state." },
    295295                { "name": "origin", "$ref": "MouseMoveOrigin", "optional": true, "description": "For 'mouse' input sources, specifies the origin type of a mouse move transition. Defaults to 'Viewport' if omitted."},

  • trunk/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp

    r231767 r233131  
    268268    case SimulatedInputSourceType::Keyboard:
    269269        // The "dispatch a key{Down,Up} action" algorithms (§17.4 Dispatching Actions).
    270         if ((!a.pressedCharKey && b.pressedCharKey) || (!a.pressedVirtualKey && b.pressedVirtualKey))
    271             m_client.simulateKeyboardInteraction(m_page, KeyboardInteraction::KeyPress, b.pressedVirtualKey, b.pressedCharKey, WTFMove(eventDispatchFinished));
    272         else if ((a.pressedCharKey && !b.pressedCharKey) || (a.pressedVirtualKey && !b.pressedVirtualKey))
    273             m_client.simulateKeyboardInteraction(m_page, KeyboardInteraction::KeyRelease, a.pressedVirtualKey, a.pressedCharKey, WTFMove(eventDispatchFinished));
    274         else
     270        if (!a.pressedCharKey && b.pressedCharKey)
     271            m_client.simulateKeyboardInteraction(m_page, KeyboardInteraction::KeyPress, b.pressedCharKey.value(), WTFMove(eventDispatchFinished));
     272        else if (a.pressedCharKey && !b.pressedCharKey)
     273            m_client.simulateKeyboardInteraction(m_page, KeyboardInteraction::KeyRelease, b.pressedCharKey.value(), WTFMove(eventDispatchFinished));
     274        else if (a.pressedVirtualKeys != b.pressedVirtualKeys) {
     275            for (VirtualKey key : b.pressedVirtualKeys) {
     276                if (!a.pressedVirtualKeys.contains(key))
     277                    m_client.simulateKeyboardInteraction(m_page, KeyboardInteraction::KeyPress, key, WTFMove(eventDispatchFinished));
     278            }
     279
     280            for (VirtualKey key : a.pressedVirtualKeys) {
     281                if (!b.pressedVirtualKeys.contains(key))
     282                    m_client.simulateKeyboardInteraction(m_page, KeyboardInteraction::KeyRelease, key, WTFMove(eventDispatchFinished));
     283            }
     284        } else
    275285            eventDispatchFinished(std::nullopt);
    276286        break;

  • trunk/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.h

    r231767 r233131  
    5353using KeyboardInteraction = Inspector::Protocol::Automation::KeyboardInteractionType;
    5454using VirtualKey = Inspector::Protocol::Automation::VirtualKey;
     55using VirtualKeySet = HashSet<VirtualKey, WTF::IntHash<VirtualKey>, WTF::StrongEnumHashTraits<VirtualKey>>;
    5556using CharKey = char; // For WebDriver, this only needs to support ASCII characters on 102-key keyboard.
    5657using MouseButton = WebMouseEvent::Button;
    … …  
    6768struct SimulatedInputSourceState {
    6869    std::optional<CharKey> pressedCharKey;
    69     std::optional<VirtualKey> pressedVirtualKey;
     70    VirtualKeySet pressedVirtualKeys;
    7071    std::optional<MouseButton> pressedMouseButton;
    7172    std::optional<MouseMoveOrigin> origin;
    … …  
    116117        virtual ~Client() { }
    117118        virtual void simulateMouseInteraction(WebPageProxy&, MouseInteraction, WebMouseEvent::Button, const WebCore::IntPoint& locationInView, AutomationCompletionHandler&&) = 0;
    118         virtual void simulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, std::optional<VirtualKey>, std::optional<CharKey>, AutomationCompletionHandler&&) = 0;
     119        virtual void simulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, WTF::Variant<VirtualKey, CharKey>&&, AutomationCompletionHandler&&) = 0;
    119120        virtual void viewportInViewCenterPointOfElement(WebPageProxy&, uint64_t frameID, const String& nodeHandle, Function<void (std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>&&) = 0;
    120121    };

  • trunk/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp

    r233122 r233131  
    14801480}
    14811481
    1482 void WebAutomationSession::simulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, std::optional<VirtualKey> virtualKey, std::optional<CharKey> charKey, CompletionHandler<void(std::optional<AutomationCommandError>)>&& completionHandler)
     1482void WebAutomationSession::simulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, WTF::Variant<VirtualKey, CharKey>&& key, CompletionHandler<void(std::optional<AutomationCommandError>)>&& completionHandler)
    14831483{
    14841484    // Bridge the flushed callback to our command's completion handler.
    … …  
    14921492    callbackInMap = WTFMove(keyboardEventsFlushedCallback);
    14931493
    1494     platformSimulateKeyboardInteraction(page, interaction, virtualKey, charKey);
     1494    platformSimulateKeyboardInteraction(page, interaction, WTFMove(key));
    14951495
    14961496    // Wait for keyboardEventsFlushedCallback to run when all events are handled.
    … …  
    16381638        bool foundVirtualKey = interactionObject->getString("key"_s, virtualKeyString);
    16391639        if (foundVirtualKey) {
    1640             auto virtualKey = Inspector::Protocol::AutomationHelpers::parseEnumValueFromString<Inspector::Protocol::Automation::VirtualKey>(virtualKeyString);
     1640            std::optional<VirtualKey> virtualKey = Inspector::Protocol::AutomationHelpers::parseEnumValueFromString<Inspector::Protocol::Automation::VirtualKey>(virtualKeyString);
    16411641            if (!virtualKey)
    16421642                ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InvalidParameter, "An interaction in the 'interactions' parameter has an invalid 'key' value.");
    16431643
    16441644            actionsToPerform.uncheckedAppend([this, page, interactionType, virtualKey] {
    1645                 platformSimulateKeyboardInteraction(*page, interactionType.value(), virtualKey, std::nullopt);
     1645                platformSimulateKeyboardInteraction(*page, interactionType.value(), virtualKey.value());
    16461646            });
    16471647        }
    … …  
    17961796                sourceState.pressedCharKey = pressedCharKeyString.characterAt(0);
    17971797
    1798             String pressedVirtualKeyString;
    1799             if (stateObject->getString("pressedVirtualKey"_s, pressedVirtualKeyString))
    1800                 sourceState.pressedVirtualKey = Inspector::Protocol::AutomationHelpers::parseEnumValueFromString<Inspector::Protocol::Automation::VirtualKey>(pressedVirtualKeyString);
     1798            RefPtr<JSON::Array> pressedVirtualKeysArray;
     1799            if (stateObject->getArray("pressedVirtualKeys"_s, pressedVirtualKeysArray)) {
     1800                VirtualKeySet pressedVirtualKeys { };
     1801
     1802                for (auto it = pressedVirtualKeysArray->begin(); it != pressedVirtualKeysArray->end(); ++it) {
     1803                    String pressedVirtualKeyString;
     1804                    if (!(*it)->asString(pressedVirtualKeyString))
     1805                        ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InvalidParameter, "Encountered a non-string virtual key value.");
     1806
     1807                    std::optional<VirtualKey> parsedVirtualKey = Inspector::Protocol::AutomationHelpers::parseEnumValueFromString<Inspector::Protocol::Automation::VirtualKey>(pressedVirtualKeyString);
     1808                    if (!parsedVirtualKey)
     1809                        ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InvalidParameter, "Encountered an unknown virtual key value.");
     1810                    else
     1811                        pressedVirtualKeys.add(parsedVirtualKey.value());
     1812                }
     1813
     1814                sourceState.pressedVirtualKeys = pressedVirtualKeys;
     1815            }
    18011816
    18021817            String pressedButtonString;
    … …  
    19271942
    19281943
    1929 void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, std::optional<VirtualKey>, std::optional<CharKey>)
     1944void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, WTF::Variant<VirtualKey, CharKey>&&)
    19301945{
    19311946}

  • trunk/Source/WebKit/UIProcess/Automation/WebAutomationSession.h

    r233122 r233131  
    138138    // SimulatedInputDispatcher::Client API
    139139    void simulateMouseInteraction(WebPageProxy&, MouseInteraction, WebMouseEvent::Button, const WebCore::IntPoint& locationInView, AutomationCompletionHandler&&) final;
    140     void simulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, std::optional<VirtualKey>, std::optional<CharKey>, AutomationCompletionHandler&&) final;
     140    void simulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, WTF::Variant<VirtualKey, CharKey>&&, AutomationCompletionHandler&&) final;
    141141    void viewportInViewCenterPointOfElement(WebPageProxy&, uint64_t frameID, const String& nodeHandle, Function<void (std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>&&) final;
    142142
    … …  
    234234    void platformSimulateMouseInteraction(WebPageProxy&, MouseInteraction, WebMouseEvent::Button, const WebCore::IntPoint& locationInView, WebEvent::Modifiers keyModifiers);
    235235    // Simulates a single virtual or char key being pressed/released, such as 'a', Control, F-keys, Numpad keys, etc. as allowed by the protocol.
    236     void platformSimulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, std::optional<VirtualKey>, std::optional<CharKey>);
     236    void platformSimulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, WTF::Variant<VirtualKey, CharKey>&&);
    237237    // Simulates key presses to produce the codepoints in a string. One or more code points are delivered atomically at grapheme cluster boundaries.
    238238    void platformSimulateKeySequence(WebPageProxy&, const String&);

  • trunk/Source/WebKit/UIProcess/Automation/gtk/WebAutomationSessionGtk.cpp

    r230988 r233131  
    295295}
    296296
    297 void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, std::optional<VirtualKey> virtualKey, std::optional<CharKey> charKey)
     297void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, WTF::Variant<VirtualKey, CharKey>&& key)
    298298{
    299299    ASSERT(virtualKey.has_value() || charKey.has_value());
    300300
    301301    unsigned keyCode;
    302     if (virtualKey.has_value())
    303         keyCode = keyCodeForVirtualKey(virtualKey.value());
    304     else
    305         keyCode = gdk_unicode_to_keyval(g_utf8_get_char(&charKey.value()));
     302    WTF::switchOn(key,
     303        [&] (VirtualKey virtualKey) {
     304            keyCode = keyCodeForVirtualKey(virtualKey);
     305        },
     306        [&] (CharKey charKey) {
     307            keyCode = gdk_unicode_to_keyval(g_utf8_get_char(&charKey));
     308        }
     309    );
    306310    unsigned modifiers = modifiersForKeyCode(keyCode);
    307311

  • trunk/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm

    r232236 r233131  
    6666#pragma mark Commands for Platform: 'iOS'
    6767
    68 void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, std::optional<VirtualKey> virtualKey, std::optional<CharKey> charKey)
     68void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, WTF::Variant<VirtualKey, CharKey>&& key)
    6969{
    70     ASSERT(virtualKey.has_value() || charKey.has_value());
    71 
    7270    // The modifiers changed by the virtual key when it is pressed or released.
    7371    WebEventFlags changedModifiers = 0;
    … …  
    8078
    8179    // Figure out the effects of sticky modifiers.
    82     if (virtualKey.has_value()) {
    83         charCode = charCodeForVirtualKey(virtualKey.value());
    84         charCodeIgnoringModifiers = charCodeIgnoringModifiersForVirtualKey(virtualKey.value());
     80    WTF::switchOn(key,
     81        [&] (VirtualKey virtualKey) {
     82            charCode = charCodeForVirtualKey(virtualKey);
     83            charCodeIgnoringModifiers = charCodeIgnoringModifiersForVirtualKey(virtualKey);
    8584
    86         switch (virtualKey.value()) {
    87         case VirtualKey::Shift:
    88             changedModifiers |= WebEventFlagMaskShift;
    89             break;
    90         case VirtualKey::Control:
    91             changedModifiers |= WebEventFlagMaskControl;
    92             break;
    93         case VirtualKey::Alternate:
    94             changedModifiers |= WebEventFlagMaskAlternate;
    95             break;
    96         case VirtualKey::Meta:
    97             // The 'meta' key does not exist on Apple keyboards and is usually
    98             // mapped to the Command key when using third-party keyboards.
    99         case VirtualKey::Command:
    100             changedModifiers |= WebEventFlagMaskCommand;
    101             break;
    102         default:
    103             break;
     85            switch (virtualKey) {
     86            case VirtualKey::Shift:
     87                changedModifiers |= WebEventFlagMaskShift;
     88                break;
     89            case VirtualKey::Control:
     90                changedModifiers |= WebEventFlagMaskControl;
     91                break;
     92            case VirtualKey::Alternate:
     93                changedModifiers |= WebEventFlagMaskAlternate;
     94                break;
     95            case VirtualKey::Meta:
     96                // The 'meta' key does not exist on Apple keyboards and is usually
     97                // mapped to the Command key when using third-party keyboards.
     98            case VirtualKey::Command:
     99                changedModifiers |= WebEventFlagMaskCommand;
     100                break;
     101            default:
     102                break;
     103            }
     104        },
     105        [&] (CharKey charKey) {
     106            charCode = (unichar)charKey;
     107            charCodeIgnoringModifiers = (unichar)charKey;
    104108        }
    105     }
    106 
    107     if (charKey.has_value()) {
    108         charCode = (unichar)charKey.value();
    109         charCodeIgnoringModifiers = (unichar)charKey.value();
    110     }
     109    );
    111110
    112111    // FIXME: consider using UIKit SPI to normalize 'characters', i.e., changing * to Shift-8,

  • trunk/Source/WebKit/UIProcess/Automation/mac/WebAutomationSessionMac.mm

    r232807 r233131  
    426426}
    427427
    428 void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, std::optional<VirtualKey> virtualKey, std::optional<CharKey> charKey)
     428void WebAutomationSession::platformSimulateKeyboardInteraction(WebPageProxy& page, KeyboardInteraction interaction, WTF::Variant<VirtualKey, CharKey>&& key)
    429429{
    430430    // FIXME: this function and the Automation protocol enum should probably adopt key names
    431431    // from W3C UIEvents standard. For more details: https://w3c.github.io/uievents-code/
    432 
    433     ASSERT(virtualKey.has_value() || charKey.has_value());
    434432
    435433    bool isStickyModifier = false;
    … …  
    439437    std::optional<unichar> charCodeIgnoringModifiers;
    440438
    441     if (virtualKey.has_value()) {
    442         isStickyModifier = virtualKeyHasStickyModifier(virtualKey.value());
    443         changedModifiers = eventModifierFlagsForVirtualKey(virtualKey.value());
    444         keyCode = keyCodeForVirtualKey(virtualKey.value());
    445         charCode = charCodeForVirtualKey(virtualKey.value());
    446         charCodeIgnoringModifiers = charCodeIgnoringModifiersForVirtualKey(virtualKey.value());
    447     }
    448 
    449     if (charKey.has_value()) {
    450         charCode = (unichar)charKey.value();
    451         charCodeIgnoringModifiers = (unichar)charKey.value();
    452     }
     439    WTF::switchOn(key,
     440        [&] (VirtualKey virtualKey) {
     441            isStickyModifier = virtualKeyHasStickyModifier(virtualKey);
     442            changedModifiers = eventModifierFlagsForVirtualKey(virtualKey);
     443            keyCode = keyCodeForVirtualKey(virtualKey);
     444            charCode = charCodeForVirtualKey(virtualKey);
     445            charCodeIgnoringModifiers = charCodeIgnoringModifiersForVirtualKey(virtualKey);
     446        },
     447        [&] (CharKey charKey) {
     448            charCode = (unichar)charKey;
     449            charCodeIgnoringModifiers = (unichar)charKey;
     450        }
     451    );
    453452
    454453    // FIXME: consider using AppKit SPI to normalize 'characters', i.e., changing * to Shift-8,
Note: See TracChangeset for help on using the changeset viewer.