Changeset 230817 in webkit
- Timestamp:
- Apr 19, 2018, 1:37:31 PM (7 years ago)
- Location:
- trunk/Source/WebKit
- Files:
-
- 7 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/Source/WebKit/ChangeLog
r230814 r230817 1 2018-04-19 Brian Burg <bburg@apple.com> 2 3 Web Automation: simulated mouse interactions should not be done until associated DOM events have been dispatched 4 https://bugs.webkit.org/show_bug.cgi?id=184462 5 <rdar://problem/39323336> 6 7 Reviewed by Carlos Garcia Campos and Tim Horton. 8 9 Covered by existing layout tests and actions endpoints in WebDriver test suite. 10 11 In preparation for implementing the W3C WebDriver command "Perform Actions", we need a way to 12 know when a simulated mouse event has been fully processed by WebProcess and it is okay to continue 13 to dispatch more simulated events. 14 15 This patch makes mouse events go through a queue as they are delivered to WebPageProxy. The approach 16 is very similar to how key events are handled. In the key event case, lots of WebEvents can come out 17 of typing one keystroke, so these need to be queued up and retired one by one when the WebProcess has 18 finished handling each event. In some mouse event cases---particularly fake mouse moves---there can 19 also be more than one mouse event waiting to be handled by WebProcess. 20 21 In the past, these queued mouse events were tracked with several member variables as different 22 use cases emerged. These are all replaced with ordinary deque operations, such as peeking or 23 checking the queue length. 24 25 * Platform/Logging.h: Add logging channel for mouse events. 26 * UIProcess/Automation/WebAutomationSession.cpp: 27 (WebKit::AutomationCommandError::toProtocolString): Add type-safe helper class for command errors. 28 In future patches we can hide knowledge of how this is sent over the protocol by relying more on 29 the convenience constructors and .toProtocolString() method. 30 31 (WebKit::WebAutomationSession::willShowJavaScriptDialog): 32 This section needs adjustments. Since performMouseInteraction now depends on key events being processed 33 prior to returning from the command, we need to abort any key event callbacks that are pending if an 34 alert pops up as a result of sending a mousedown event. Any mouse events that are still queued will 35 be handled when the alert is dismissed and the nested run loop exits. 36 37 (WebKit::WebAutomationSession::mouseEventsFlushedForPage): 38 (WebKit::WebAutomationSession::keyboardEventsFlushedForPage): 39 Modernize this a bit. Don't spread knowledge about how commands are sent back out into event handling code. 40 Our wrapper callbacks in performXXXInteraction handle the protocol-specific details of the response. 41 42 (WebKit::WebAutomationSession::performMouseInteraction): 43 Add code similar to performKeyboardInteractions so that the command doesn't finish until the mouse 44 event has been fully handled. Unlike keyboards, sometimes mouse interactions don't turn into WebEvents 45 so we also need to handle the case where there is nothing to be waited on because hit testing did 46 not return a target to deliver the event to. 47 48 (WebKit::WebAutomationSession::performKeyboardInteractions): 49 Modernize a little bit to use generic callbacks rather than protocol-generated callbacks in the 50 event waiting/handling code. Now it matches the types used for the mouse event case. 51 52 * UIProcess/Automation/WebAutomationSession.h: 53 (WebKit::AutomationCommandError::AutomationCommandError): 54 Add a helper struct to hold an enumerated error name and an optional free-form error message. 55 56 * UIProcess/WebPageProxy.h: 57 * UIProcess/WebPageProxy.cpp: 58 (WebKit::webMouseEventTypeString): 59 (WebKit::webKeyboardEventTypeString): 60 (WebKit::WebPageProxy::handleMouseEvent): 61 (WebKit::WebPageProxy::processNextQueuedMouseEvent): 62 Split the old method into handleMouseEvent (called by other code) and processNextQueuedMouseEvent. 63 The latter sends the next mouse event to WebProcess, and can be triggered in didReceiveEvent 64 if there are more mouse events to be sent to WebProcess. 65 66 (WebKit::WebPageProxy::isProcessingMouseEvents const): Added. 67 (WebKit::WebPageProxy::currentlyProcessedMouseDownEvent): Reimplemented on top of the deque. 68 (WebKit::WebPageProxy::didReceiveEvent): 69 Unify the code paths for different mouse event types to all use the deque. They also will 70 notify the automation session if there are no more mouse events to send (i.e., interaction is over). 71 72 (WebKit::WebPageProxy::resetStateAfterProcessExited): Add handling for new map. 73 1 74 2018-04-19 Andy Estes <aestes@apple.com> 2 75 -
trunk/Source/WebKit/Platform/Logging.h
r230773 r230817 52 52 M(Layers) \ 53 53 M(Loading) \ 54 M(MouseHandling) \ 54 55 M(Network) \ 55 56 M(NetworkCache) \ -
trunk/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp
r230773 r230817 54 54 namespace WebKit { 55 55 56 String AutomationCommandError::toProtocolString() 57 { 58 String protocolErrorName = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(type); 59 if (!message.has_value()) 60 return protocolErrorName; 61 62 return makeString(protocolErrorName, errorNameAndDetailsSeparator, message.value()); 63 } 64 56 65 // §8. Sessions 57 66 // https://www.w3.org/TR/webdriver/#dfn-session-page-load-timeout … … 582 591 auto callback = m_evaluateJavaScriptFunctionCallbacks.take(key); 583 592 callback->sendFailure(unexpectedAlertOpenError); 593 } 594 } 595 596 if (!m_pendingMouseEventsFlushedCallbacksPerPage.isEmpty()) { 597 for (auto key : copyToVector(m_pendingMouseEventsFlushedCallbacksPerPage.keys())) { 598 auto callback = m_pendingMouseEventsFlushedCallbacksPerPage.take(key); 599 callback(std::nullopt); 600 } 601 } 602 603 if (!m_pendingKeyboardEventsFlushedCallbacksPerPage.isEmpty()) { 604 for (auto key : copyToVector(m_pendingKeyboardEventsFlushedCallbacksPerPage.keys())) { 605 auto callback = m_pendingKeyboardEventsFlushedCallbacksPerPage.take(key); 606 callback(std::nullopt); 584 607 } 585 608 } … … 699 722 } 700 723 724 void WebAutomationSession::mouseEventsFlushedForPage(const WebPageProxy& page) 725 { 726 if (auto callback = m_pendingMouseEventsFlushedCallbacksPerPage.take(page.pageID())) { 727 if (m_pendingMouseEventsFlushedCallbacksPerPage.isEmpty()) 728 m_simulatingUserInteraction = false; 729 730 callback(std::nullopt); 731 } 732 } 733 701 734 void WebAutomationSession::keyboardEventsFlushedForPage(const WebPageProxy& page) 702 735 { 703 736 if (auto callback = m_pendingKeyboardEventsFlushedCallbacksPerPage.take(page.pageID())) { 704 callback->sendSuccess(JSON::Object::create());705 706 737 if (m_pendingKeyboardEventsFlushedCallbacksPerPage.isEmpty()) 707 738 m_simulatingUserInteraction = false; 739 740 callback(std::nullopt); 708 741 } 709 742 } … … 1399 1432 ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InvalidParameter, "The parameter 'button' is invalid."); 1400 1433 1434 auto mouseEventsFlushedCallback = [protectedThis = WTFMove(protectedThis), callback = WTFMove(callback), page = page.copyRef(), x, y](std::optional<AutomationCommandError> error) { 1435 if (error) 1436 callback->sendFailure(error.value().toProtocolString()); 1437 else { 1438 callback->sendSuccess(Inspector::Protocol::Automation::Point::create() 1439 .setX(x) 1440 .setY(y - page->topContentInset()) 1441 .release()); 1442 } 1443 }; 1444 1445 auto& callbackInMap = m_pendingMouseEventsFlushedCallbacksPerPage.add(page->pageID(), nullptr).iterator->value; 1446 if (callbackInMap) 1447 callbackInMap(AUTOMATION_COMMAND_ERROR_WITH_NAME(Timeout)); 1448 callbackInMap = WTFMove(mouseEventsFlushedCallback); 1449 1450 // This is cleared when all mouse events are flushed. 1451 m_simulatingUserInteraction = true; 1452 1401 1453 platformSimulateMouseInteraction(page, viewPosition, parsedInteraction.value(), parsedButton.value(), keyModifiers); 1402 1403 callback->sendSuccess(Inspector::Protocol::Automation::Point::create() 1404 .setX(x) 1405 .setY(y - page->topContentInset()) 1406 .release()); 1454 1455 // If the event location was previously clipped and does not hit test anything in the window, then it will not be processed. 1456 // For compatibility with pre-W3C driver implementations, don't make this a hard error; just do nothing silently. 1457 // In W3C-only code paths, we can reject any pointer actions whose coordinates are outside the viewport rect. 1458 if (callbackInMap && !page->isProcessingMouseEvents()) { 1459 auto callbackToCancel = m_pendingMouseEventsFlushedCallbacksPerPage.take(page->pageID()); 1460 callbackToCancel(std::nullopt); 1461 } 1407 1462 }); 1408 1463 #endif // USE(APPKIT) || PLATFORM(GTK) … … 1474 1529 ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InternalError, "No actions to perform."); 1475 1530 1531 auto keyboardEventsFlushedCallback = [protectedThis = makeRef(*this), callback = WTFMove(callback), page = makeRef(*page)](std::optional<AutomationCommandError> error) { 1532 if (error) 1533 callback->sendFailure(error.value().toProtocolString()); 1534 else 1535 callback->sendSuccess(); 1536 }; 1537 1476 1538 auto& callbackInMap = m_pendingKeyboardEventsFlushedCallbacksPerPage.add(page->pageID(), nullptr).iterator->value; 1477 1539 if (callbackInMap) 1478 callbackInMap ->sendFailure(STRING_FOR_PREDEFINED_ERROR_NAME(Timeout));1479 callbackInMap = WTFMove( callback);1540 callbackInMap(AUTOMATION_COMMAND_ERROR_WITH_NAME(Timeout)); 1541 callbackInMap = WTFMove(keyboardEventsFlushedCallback); 1480 1542 1481 1543 // This is cleared when all keyboard events are flushed. -
trunk/Source/WebKit/UIProcess/Automation/WebAutomationSession.h
r230773 r230817 1 1 /* 2 * Copyright (C) 2016 , 2017Apple Inc. All rights reserved.2 * Copyright (C) 2016-2018 Apple Inc. All rights reserved. 3 3 * 4 4 * Redistribution and use in source and binary forms, with or without … … 77 77 class WebProcessPool; 78 78 79 struct AutomationCommandError { 80 public: 81 Inspector::Protocol::Automation::ErrorMessage type; 82 std::optional<String> message { std::nullopt }; 83 84 AutomationCommandError(Inspector::Protocol::Automation::ErrorMessage type) 85 : type(type) { } 86 87 AutomationCommandError(Inspector::Protocol::Automation::ErrorMessage type, const String& message) 88 : type(type) 89 , message(message) { } 90 91 String toProtocolString(); 92 }; 93 79 94 class WebAutomationSession final : public API::ObjectImpl<API::Object::Type::AutomationSession>, public IPC::MessageReceiver 80 95 #if ENABLE(REMOTE_INSPECTOR) … … 99 114 void inspectorFrontendLoaded(const WebPageProxy&); 100 115 void keyboardEventsFlushedForPage(const WebPageProxy&); 116 void mouseEventsFlushedForPage(const WebPageProxy&); 101 117 void willClosePage(const WebPageProxy&); 102 118 void handleRunOpenPanel(const WebPageProxy&, const WebFrameProxy&, const API::OpenPanelParameters&, WebOpenPanelResultListenerProxy&); … … 236 252 HashMap<uint64_t, RefPtr<Inspector::BackendDispatcher::CallbackBase>> m_pendingEagerNavigationInBrowsingContextCallbacksPerFrame; 237 253 HashMap<uint64_t, RefPtr<Inspector::BackendDispatcher::CallbackBase>> m_pendingInspectorCallbacksPerPage; 238 HashMap<uint64_t, RefPtr<Inspector::BackendDispatcher::CallbackBase>> m_pendingKeyboardEventsFlushedCallbacksPerPage; 254 HashMap<uint64_t, Function<void(std::optional<AutomationCommandError>)>> m_pendingKeyboardEventsFlushedCallbacksPerPage; 255 HashMap<uint64_t, Function<void(std::optional<AutomationCommandError>)>> m_pendingMouseEventsFlushedCallbacksPerPage; 239 256 240 257 uint64_t m_nextEvaluateJavaScriptCallbackID { 1 }; -
trunk/Source/WebKit/UIProcess/Automation/WebAutomationSessionMacros.h
r230773 r230817 39 39 #define STRING_FOR_PREDEFINED_ERROR_MESSAGE_AND_DETAILS(errorMessage, detailsString) makeString(Inspector::Protocol::AutomationHelpers::getEnumConstantValue(VALIDATED_ERROR_MESSAGE(errorMessage)), errorNameAndDetailsSeparator, detailsString) 40 40 41 #define AUTOMATION_COMMAND_ERROR_WITH_NAME(errorName) AutomationCommandError(Inspector::Protocol::Automation::ErrorMessage::errorName) 42 41 43 // Convenience macros for filling in the error string of synchronous commands in bailout branches. 42 44 #define SYNC_FAIL_WITH_PREDEFINED_ERROR(errorName) \ -
trunk/Source/WebKit/UIProcess/WebPageProxy.cpp
r230812 r230817 294 294 295 295 #if !LOG_DISABLED 296 static const char* webMouseEventTypeString(WebEvent::Type type) 297 { 298 switch (type) { 299 case WebEvent::MouseDown: 300 return "MouseDown"; 301 case WebEvent::MouseUp: 302 return "MouseUp"; 303 case WebEvent::MouseMove: 304 return "MouseMove"; 305 case WebEvent::MouseForceChanged: 306 return "MouseForceChanged"; 307 case WebEvent::MouseForceDown: 308 return "MouseForceDown"; 309 case WebEvent::MouseForceUp: 310 return "MouseForceUp"; 311 default: 312 ASSERT_NOT_REACHED(); 313 return "<unknown>"; 314 } 315 } 316 296 317 static const char* webKeyboardEventTypeString(WebEvent::Type type) 297 318 { … … 299 320 case WebEvent::KeyDown: 300 321 return "KeyDown"; 301 302 322 case WebEvent::KeyUp: 303 323 return "KeyUp"; 304 305 324 case WebEvent::RawKeyDown: 306 325 return "RawKeyDown"; 307 308 326 case WebEvent::Char: 309 327 return "Char"; 310 311 328 default: 312 329 ASSERT_NOT_REACHED(); … … 1897 1914 return; 1898 1915 1916 LOG(MouseHandling, "UIProcess: enqueued mouse event %s (queue size %zu)", webMouseEventTypeString(event.type()), m_mouseEventQueue.size()); 1917 m_mouseEventQueue.append(event); 1918 if (m_mouseEventQueue.size() == 1) // Otherwise, called from DidReceiveEvent message handler. 1919 processNextQueuedMouseEvent(); 1920 } 1921 1922 void WebPageProxy::processNextQueuedMouseEvent() 1923 { 1924 if (!isValid()) 1925 return; 1926 1927 ASSERT(!m_mouseEventQueue.isEmpty()); 1928 1929 const NativeWebMouseEvent& event = m_mouseEventQueue.first(); 1930 1899 1931 if (m_pageClient.windowIsFrontWindowUnderMouse(event)) 1900 1932 setToolTip(String()); … … 1903 1935 if (event.type() != WebEvent::MouseMove) 1904 1936 m_process->responsivenessTimer().start(); 1905 else { 1906 if (m_processingMouseMoveEvent) { 1907 m_nextMouseMoveEvent = std::make_unique<NativeWebMouseEvent>(event); 1908 return; 1909 } 1910 1911 m_processingMouseMoveEvent = true; 1912 } 1913 1914 // <https://bugs.webkit.org/show_bug.cgi?id=57904> We need to keep track of the mouse down event in the case where we 1915 // display a popup menu for select elements. When the user changes the selected item, 1916 // we fake a mouse up event by using this stored down event. This event gets cleared 1917 // when the mouse up message is received from WebProcess. 1918 if (event.type() == WebEvent::MouseDown) 1919 m_currentlyProcessedMouseDownEvent = std::make_unique<NativeWebMouseEvent>(event); 1920 1937 1938 LOG(MouseHandling, "UIProcess: sent mouse event %s (queue size %zu)", webMouseEventTypeString(event.type()), m_mouseEventQueue.size()); 1921 1939 m_process->send(Messages::WebPage::MouseEvent(event), m_pageID); 1922 1940 } … … 4834 4852 } 4835 4853 4854 bool WebPageProxy::isProcessingMouseEvents() const 4855 { 4856 return !m_mouseEventQueue.isEmpty(); 4857 } 4858 4836 4859 NativeWebMouseEvent* WebPageProxy::currentlyProcessedMouseDownEvent() 4837 4860 { 4838 return m_currentlyProcessedMouseDownEvent.get(); 4861 // <https://bugs.webkit.org/show_bug.cgi?id=57904> We need to keep track of the mouse down event in the case where we 4862 // display a popup menu for select elements. When the user changes the selected item, we fake a mouseup event by 4863 // using this stored mousedown event and changing the event type. This trickery happens when WebProcess handles 4864 // a mousedown event that runs the default handler for HTMLSelectElement, so the triggering mousedown must be the first event. 4865 4866 if (m_mouseEventQueue.isEmpty()) 4867 return nullptr; 4868 4869 auto& event = m_mouseEventQueue.first(); 4870 if (event.type() != WebEvent::Type::MouseDown) 4871 return nullptr; 4872 4873 return &event; 4839 4874 } 4840 4875 … … 5244 5279 case WebEvent::NoType: 5245 5280 break; 5246 case WebEvent::MouseMove:5247 m_processingMouseMoveEvent = false;5248 if (m_nextMouseMoveEvent)5249 handleMouseEvent(*std::exchange(m_nextMouseMoveEvent, nullptr));5250 break;5251 case WebEvent::MouseDown:5252 break;5253 case WebEvent::MouseUp:5254 m_currentlyProcessedMouseDownEvent = nullptr;5255 break;5256 5281 case WebEvent::MouseForceChanged: 5257 5282 case WebEvent::MouseForceDown: 5258 5283 case WebEvent::MouseForceUp: 5284 case WebEvent::MouseMove: 5285 case WebEvent::MouseDown: 5286 case WebEvent::MouseUp: { 5287 LOG(MouseHandling, "WebPageProxy::didReceiveEvent: %s (queue size %zu)", webMouseEventTypeString(type), m_mouseEventQueue.size()); 5288 5289 // Retire the last sent event now that WebProcess is done handling it. 5290 MESSAGE_CHECK(!m_mouseEventQueue.isEmpty()); 5291 NativeWebMouseEvent event = m_mouseEventQueue.takeFirst(); 5292 MESSAGE_CHECK(type == event.type()); 5293 5294 if (!m_mouseEventQueue.isEmpty()) { 5295 LOG(MouseHandling, " UIProcess: handling a queued mouse event from didReceiveEvent"); 5296 processNextQueuedMouseEvent(); 5297 } else if (auto* automationSession = process().processPool().automationSession()) 5298 automationSession->mouseEventsFlushedForPage(*this); 5299 5259 5300 break; 5301 } 5260 5302 5261 5303 case WebEvent::Wheel: { … … 5286 5328 MESSAGE_CHECK(type == event.type()); 5287 5329 5288 if (!m_keyEventQueue.isEmpty()) { 5330 bool canProcessMoreKeyEvents = !m_keyEventQueue.isEmpty(); 5331 if (canProcessMoreKeyEvents) { 5289 5332 LOG(KeyHandling, " UI process: sent keyEvent from didReceiveEvent"); 5290 5333 m_process->send(Messages::WebPage::KeyEvent(m_keyEventQueue.first()), m_pageID); … … 5300 5343 5301 5344 // Notify the session after -[NSApp sendEvent:] has a crack at turning the event into an action. 5302 if ( m_keyEventQueue.isEmpty()) {5345 if (!canProcessMoreKeyEvents) { 5303 5346 if (auto* automationSession = process().processPool().automationSession()) 5304 5347 automationSession->keyboardEventsFlushedForPage(*this); … … 5904 5947 5905 5948 // Can't expect DidReceiveEvent notifications from a crashed web process. 5949 m_mouseEventQueue.clear(); 5906 5950 m_keyEventQueue.clear(); 5907 5951 m_wheelEventQueue.clear(); 5908 5952 m_currentlyProcessedWheelEvents.clear(); 5909 5910 m_nextMouseMoveEvent = nullptr;5911 m_currentlyProcessedMouseDownEvent = nullptr;5912 5913 m_processingMouseMoveEvent = false;5914 5915 5953 #if ENABLE(TOUCH_EVENTS) && !ENABLE(IOS_TOUCH_EVENTS) 5916 5954 m_touchEventQueue.clear(); -
trunk/Source/WebKit/UIProcess/WebPageProxy.h
r230812 r230817 687 687 #endif 688 688 689 bool isProcessingMouseEvents() const; 690 void processNextQueuedMouseEvent(); 689 691 void handleMouseEvent(const NativeWebMouseEvent&); 692 690 693 void handleWheelEvent(const NativeWebWheelEvent&); 691 694 void handleKeyboardEvent(const NativeWebKeyboardEvent&); … … 1940 1943 bool m_shouldSuppressAppLinksInNextNavigationPolicyDecision { false }; 1941 1944 1945 Deque<NativeWebMouseEvent> m_mouseEventQueue; 1942 1946 Deque<NativeWebKeyboardEvent> m_keyEventQueue; 1943 1947 Deque<NativeWebWheelEvent> m_wheelEventQueue; … … 1946 1950 Deque<NativeWebGestureEvent> m_gestureEventQueue; 1947 1951 #endif 1948 1949 bool m_processingMouseMoveEvent { false };1950 std::unique_ptr<NativeWebMouseEvent> m_nextMouseMoveEvent;1951 std::unique_ptr<NativeWebMouseEvent> m_currentlyProcessedMouseDownEvent;1952 1952 1953 1953 #if ENABLE(TOUCH_EVENTS)
Note:
See TracChangeset
for help on using the changeset viewer.