Changeset 230743 in webkit
- Timestamp:
- Apr 17, 2018 5:46:06 PM (6 years ago)
- Location:
- trunk/Source/WebKit
- Files:
-
- 7 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/Source/WebKit/ChangeLog
r230739 r230743 1 2018-04-11 Brian Burg <bburg@apple.com> 2 3 Web Automation: simulated mouse interactions should not be done until associated DOM events have been dispatched 4 https://bugs.webkit.org/show_bug.cgi?id=184462 5 <rdar://problem/39323336> 6 7 Reviewed by Carlos Garcia Campos and Tim Horton. 8 9 Covered by existing layout tests and actions endpoints in WebDriver test suite. 10 11 In preparation for implementing the W3C WebDriver command "Perform Actions", we need a way to 12 know when a simulated mouse event has been fully processed by WebProcess and it is okay to continue 13 to dispatch more simulated events. 14 15 This patch makes mouse events go through a queue as they are delivered to WebPageProxy. The approach 16 is very similar to how key events are handled. In the key event case, lots of WebEvents can come out 17 of typing one keystroke, so these need to be queued up and retired one by one when the WebProcess has 18 finished handling each event. In some mouse event cases---particularly fake mouse moves---there can 19 also be more than one mouse event waiting to be handled by WebProcess. 20 21 In the past, these queued mouse events were tracked with several member variables as different 22 use cases emerged. These are all replaced with ordinary deque operations, such as peeking or 23 checking the queue length. 24 25 * Platform/Logging.h: Add logging channel for mouse events. 26 * UIProcess/Automation/WebAutomationSession.cpp: 27 (WebKit::AutomationCommandError::toProtocolString): Add type-safe helper class for command errors. 28 In future patches we can hide knowledge of how this is sent over the protocol by relying more on 29 the convenience constructors and .toProtocolString() method. 30 31 (WebKit::WebAutomationSession::willShowJavaScriptDialog): 32 This section needs adjustments. Since performMouseInteraction now depends on key events being processed 33 prior to returning from the command, we need to abort any key event callbacks that are pending if an 34 alert pops up as a result of sending a mousedown event. Any mouse events that are still queued will 35 be handled when the alert is dismissed and the nested run loop exits. 36 37 (WebKit::WebAutomationSession::mouseEventsFlushedForPage): 38 (WebKit::WebAutomationSession::keyboardEventsFlushedForPage): 39 Modernize this a bit. Don't spread knowledge about how commands are sent back out into event handling code. 40 Our wrapper callbacks in performXXXInteraction handle the protocol-specific details of the response. 41 42 (WebKit::WebAutomationSession::performMouseInteraction): 43 Add code similar to performKeyboardInteractions so that the command doesn't finish until the mouse 44 event has been fully handled. Unlike keyboards, sometimes mouse interactions don't turn into WebEvents 45 so we also need to handle the case where there is nothing to be waited on because hit testing did 46 not return a target to deliver the event to. 47 48 (WebKit::WebAutomationSession::performKeyboardInteractions): 49 Modernize a little bit to use generic callbacks rather than protocol-generated callbacks in the 50 event waiting/handling code. Now it matches the types used for the mouse event case. 51 52 * UIProcess/Automation/WebAutomationSession.h: 53 (WebKit::AutomationCommandError::AutomationCommandError): 54 Add a helper struct to hold an enumerated error name and an optional free-form error message. 55 56 * UIProcess/WebPageProxy.h: 57 * UIProcess/WebPageProxy.cpp: 58 (WebKit::webMouseEventTypeString): 59 (WebKit::webKeyboardEventTypeString): 60 (WebKit::WebPageProxy::handleMouseEvent): 61 (WebKit::WebPageProxy::processNextQueuedMouseEvent): 62 Split the old method into handleMouseEvent (called by other code) and processNextQueuedMouseEvent. 63 The latter sends the next mouse event to WebProcess, and can be triggered in didReceiveEvent 64 if there are more mouse events to be sent to WebProcess. 65 66 (WebKit::WebPageProxy::isProcessingMouseEvents const): Added. 67 (WebKit::WebPageProxy::currentlyProcessedMouseDownEvent): Reimplemented on top of the deque. 68 (WebKit::WebPageProxy::didReceiveEvent): 69 Unify the code paths for different mouse event types to all use the deque. They also will 70 notify the automation session if there are no more mouse events to send (i.e., interaction is over). 71 72 (WebKit::WebPageProxy::resetStateAfterProcessExited): Add handling for new map. 73 1 74 2018-04-17 Adrian Perez de Castro <aperez@igalia.com> 2 75 -
trunk/Source/WebKit/Platform/Logging.h
r230640 r230743 52 52 M(Layers) \ 53 53 M(Loading) \ 54 M(MouseHandling) \ 54 55 M(Network) \ 55 56 M(NetworkCache) \ -
trunk/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp
r230367 r230743 54 54 namespace WebKit { 55 55 56 String AutomationCommandError::toProtocolString() const 57 { 58 String protocolErrorName = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(type); 59 if (!message.has_value()) 60 return protocolErrorName; 61 62 return makeString(protocolErrorName, errorNameAndDetailsSeparator, message.value()); 63 } 64 56 65 // §8. Sessions 57 66 // https://www.w3.org/TR/webdriver/#dfn-session-page-load-timeout … … 582 591 auto callback = m_evaluateJavaScriptFunctionCallbacks.take(key); 583 592 callback->sendFailure(unexpectedAlertOpenError); 593 } 594 } 595 596 if (!m_pendingMouseEventsFlushedCallbacksPerPage.isEmpty()) { 597 for (auto key : copyToVector(m_pendingMouseEventsFlushedCallbacksPerPage.keys())) { 598 auto callback = m_pendingMouseEventsFlushedCallbacksPerPage.take(key); 599 callback(std::nullopt); 600 } 601 } 602 603 if (!m_pendingKeyboardEventsFlushedCallbacksPerPage.isEmpty()) { 604 for (auto key : copyToVector(m_pendingKeyboardEventsFlushedCallbacksPerPage.keys())) { 605 auto callback = m_pendingKeyboardEventsFlushedCallbacksPerPage.take(key); 606 callback(std::nullopt); 584 607 } 585 608 } … … 699 722 } 700 723 724 void WebAutomationSession::mouseEventsFlushedForPage(const WebPageProxy& page) 725 { 726 if (auto callback = m_pendingMouseEventsFlushedCallbacksPerPage.take(page.pageID())) { 727 if (m_pendingMouseEventsFlushedCallbacksPerPage.isEmpty()) 728 m_simulatingUserInteraction = false; 729 730 callback(std::nullopt); 731 } 732 } 733 701 734 void WebAutomationSession::keyboardEventsFlushedForPage(const WebPageProxy& page) 702 735 { 703 736 if (auto callback = m_pendingKeyboardEventsFlushedCallbacksPerPage.take(page.pageID())) { 704 callback->sendSuccess(JSON::Object::create());705 706 737 if (m_pendingKeyboardEventsFlushedCallbacksPerPage.isEmpty()) 707 738 m_simulatingUserInteraction = false; 739 740 callback(std::nullopt); 708 741 } 709 742 } … … 1399 1432 ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InvalidParameter, "The parameter 'button' is invalid."); 1400 1433 1434 auto mouseEventsFlushedCallback = [protectedThis = WTFMove(protectedThis), callback = WTFMove(callback), page = page.copyRef(), x, y](std::optional<AutomationCommandError> error) { 1435 if (error) 1436 callback->sendFailure(error.value().toProtocolString()); 1437 else { 1438 callback->sendSuccess(Inspector::Protocol::Automation::Point::create() 1439 .setX(x) 1440 .setY(y - page->topContentInset()) 1441 .release()); 1442 } 1443 }; 1444 1445 auto& callbackInMap = m_pendingMouseEventsFlushedCallbacksPerPage.add(page->pageID(), nullptr).iterator->value; 1446 if (callbackInMap) 1447 callbackInMap(AUTOMATION_COMMAND_ERROR_WITH_NAME(Timeout)); 1448 callbackInMap = WTFMove(mouseEventsFlushedCallback); 1449 1450 // This is cleared when all mouse events are flushed. 1451 m_simulatingUserInteraction = true; 1452 1401 1453 platformSimulateMouseInteraction(page, viewPosition, parsedInteraction.value(), parsedButton.value(), keyModifiers); 1402 1403 callback->sendSuccess(Inspector::Protocol::Automation::Point::create() 1404 .setX(x) 1405 .setY(y - page->topContentInset()) 1406 .release()); 1454 1455 // If the event location was previously clipped and does not hit test anything in the window, then it will not be processed. 1456 // For compatibility with pre-W3C driver implementations, don't make this a hard error; just do nothing silently. 1457 // In W3C-only code paths, we can reject any pointer actions whose coordinates are outside the viewport rect. 1458 if (callbackInMap && !page->isProcessingMouseEvents()) { 1459 auto callbackToCancel = m_pendingMouseEventsFlushedCallbacksPerPage.take(page->pageID()); 1460 callbackToCancel(std::nullopt); 1461 } 1407 1462 }); 1408 1463 #endif // USE(APPKIT) || PLATFORM(GTK) … … 1474 1529 ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InternalError, "No actions to perform."); 1475 1530 1531 auto keyboardEventsFlushedCallback = [protectedThis = makeRef(*this), callback = WTFMove(callback), page = makeRef(*page)](std::optional<AutomationCommandError> error) { 1532 if (error) 1533 callback->sendFailure(error.value().toProtocolString()); 1534 else 1535 callback->sendSuccess(); 1536 }; 1537 1476 1538 auto& callbackInMap = m_pendingKeyboardEventsFlushedCallbacksPerPage.add(page->pageID(), nullptr).iterator->value; 1477 1539 if (callbackInMap) 1478 callbackInMap ->sendFailure(STRING_FOR_PREDEFINED_ERROR_NAME(Timeout));1479 callbackInMap = WTFMove( callback);1540 callbackInMap(AUTOMATION_COMMAND_ERROR_WITH_NAME(Timeout)); 1541 callbackInMap = WTFMove(keyboardEventsFlushedCallback); 1480 1542 1481 1543 // This is cleared when all keyboard events are flushed. -
trunk/Source/WebKit/UIProcess/Automation/WebAutomationSession.h
r229998 r230743 1 1 /* 2 * Copyright (C) 2016 , 2017Apple Inc. All rights reserved.2 * Copyright (C) 2016-2018 Apple Inc. All rights reserved. 3 3 * 4 4 * Redistribution and use in source and binary forms, with or without … … 77 77 class WebProcessPool; 78 78 79 struct AutomationCommandError { 80 public: 81 AutomationCommandError(Inspector::Protocol::Automation::ErrorMessage type) 82 : type(type) { } 83 84 AutomationCommandError(Inspector::Protocol::Automation::ErrorMessage type, const String& message) 85 : type(type) 86 , message(message) { } 87 88 String toProtocolString() const; 89 90 Inspector::Protocol::Automation::ErrorMessage type; 91 std::optional<String> message; 92 }; 93 79 94 class WebAutomationSession final : public API::ObjectImpl<API::Object::Type::AutomationSession>, public IPC::MessageReceiver 80 95 #if ENABLE(REMOTE_INSPECTOR) … … 99 114 void inspectorFrontendLoaded(const WebPageProxy&); 100 115 void keyboardEventsFlushedForPage(const WebPageProxy&); 116 void mouseEventsFlushedForPage(const WebPageProxy&); 101 117 void willClosePage(const WebPageProxy&); 102 118 void handleRunOpenPanel(const WebPageProxy&, const WebFrameProxy&, const API::OpenPanelParameters&, WebOpenPanelResultListenerProxy&); … … 236 252 HashMap<uint64_t, RefPtr<Inspector::BackendDispatcher::CallbackBase>> m_pendingEagerNavigationInBrowsingContextCallbacksPerFrame; 237 253 HashMap<uint64_t, RefPtr<Inspector::BackendDispatcher::CallbackBase>> m_pendingInspectorCallbacksPerPage; 238 HashMap<uint64_t, RefPtr<Inspector::BackendDispatcher::CallbackBase>> m_pendingKeyboardEventsFlushedCallbacksPerPage; 254 HashMap<uint64_t, Function<void(std::optional<AutomationCommandError>)>> m_pendingKeyboardEventsFlushedCallbacksPerPage; 255 HashMap<uint64_t, Function<void(std::optional<AutomationCommandError>)>> m_pendingMouseEventsFlushedCallbacksPerPage; 239 256 240 257 uint64_t m_nextEvaluateJavaScriptCallbackID { 1 }; -
trunk/Source/WebKit/UIProcess/Automation/WebAutomationSessionMacros.h
r229493 r230743 39 39 #define STRING_FOR_PREDEFINED_ERROR_MESSAGE_AND_DETAILS(errorMessage, detailsString) makeString(Inspector::Protocol::AutomationHelpers::getEnumConstantValue(VALIDATED_ERROR_MESSAGE(errorMessage)), errorNameAndDetailsSeparator, detailsString) 40 40 41 #define AUTOMATION_COMMAND_ERROR_WITH_NAME(errorName) AutomationCommandError(Inspector::Protocol::Automation::ErrorMessage::errorName) 42 41 43 // Convenience macros for filling in the error string of synchronous commands in bailout branches. 42 44 #define SYNC_FAIL_WITH_PREDEFINED_ERROR(errorName) \ -
trunk/Source/WebKit/UIProcess/WebPageProxy.cpp
r230721 r230743 292 292 293 293 #if !LOG_DISABLED 294 static const char* webMouseEventTypeString(WebEvent::Type type) 295 { 296 switch (type) { 297 case WebEvent::MouseDown: 298 return "MouseDown"; 299 case WebEvent::MouseUp: 300 return "MouseUp"; 301 case WebEvent::MouseMove: 302 return "MouseMove"; 303 default: 304 ASSERT_NOT_REACHED(); 305 return "<unknown>"; 306 } 307 } 308 294 309 static const char* webKeyboardEventTypeString(WebEvent::Type type) 295 310 { … … 297 312 case WebEvent::KeyDown: 298 313 return "KeyDown"; 299 300 314 case WebEvent::KeyUp: 301 315 return "KeyUp"; 302 303 316 case WebEvent::RawKeyDown: 304 317 return "RawKeyDown"; 305 306 318 case WebEvent::Char: 307 319 return "Char"; 308 309 320 default: 310 321 ASSERT_NOT_REACHED(); … … 1899 1910 return; 1900 1911 1912 m_mouseEventQueue.append(event); 1913 if (m_mouseEventQueue.size() == 1) // Otherwise, called from DidReceiveEvent message handler. 1914 processNextQueuedMouseEvent(); 1915 } 1916 1917 void WebPageProxy::processNextQueuedMouseEvent() 1918 { 1919 if (!isValid()) 1920 return; 1921 1922 ASSERT(!m_mouseEventQueue.isEmpty()); 1923 1924 const NativeWebMouseEvent& event = m_mouseEventQueue.first(); 1925 1901 1926 if (m_pageClient.windowIsFrontWindowUnderMouse(event)) 1902 1927 setToolTip(String()); … … 1905 1930 if (event.type() != WebEvent::MouseMove) 1906 1931 m_process->responsivenessTimer().start(); 1907 else { 1908 if (m_processingMouseMoveEvent) { 1909 m_nextMouseMoveEvent = std::make_unique<NativeWebMouseEvent>(event); 1910 return; 1911 } 1912 1913 m_processingMouseMoveEvent = true; 1914 } 1915 1916 // <https://bugs.webkit.org/show_bug.cgi?id=57904> We need to keep track of the mouse down event in the case where we 1917 // display a popup menu for select elements. When the user changes the selected item, 1918 // we fake a mouse up event by using this stored down event. This event gets cleared 1919 // when the mouse up message is received from WebProcess. 1920 if (event.type() == WebEvent::MouseDown) 1921 m_currentlyProcessedMouseDownEvent = std::make_unique<NativeWebMouseEvent>(event); 1922 1932 1933 LOG(MouseHandling, " UI process: sent mouseEvent from handleMouseEvent"); 1923 1934 m_process->send(Messages::WebPage::MouseEvent(event), m_pageID); 1924 1935 } … … 4805 4816 } 4806 4817 4818 bool WebPageProxy::isProcessingMouseEvents() const 4819 { 4820 return !m_mouseEventQueue.isEmpty(); 4821 } 4822 4807 4823 NativeWebMouseEvent* WebPageProxy::currentlyProcessedMouseDownEvent() 4808 4824 { 4809 return m_currentlyProcessedMouseDownEvent.get(); 4825 // <https://bugs.webkit.org/show_bug.cgi?id=57904> We need to keep track of the mouse down event in the case where we 4826 // display a popup menu for select elements. When the user changes the selected item, we fake a mouseup event by 4827 // using this stored mousedown event and changing the event type. This trickery happens when WebProcess handles 4828 // a mousedown event that runs the default handler for HTMLSelectElement, so the triggering mousedown must be the first event. 4829 4830 if (m_mouseEventQueue.isEmpty()) 4831 return nullptr; 4832 4833 auto& event = m_mouseEventQueue.first(); 4834 if (event.type() != WebEvent::Type::MouseDown) 4835 return nullptr; 4836 4837 return &event; 4810 4838 } 4811 4839 … … 5216 5244 break; 5217 5245 case WebEvent::MouseMove: 5218 m_processingMouseMoveEvent = false; 5219 if (m_nextMouseMoveEvent) 5220 handleMouseEvent(*std::exchange(m_nextMouseMoveEvent, nullptr)); 5246 case WebEvent::MouseDown: 5247 case WebEvent::MouseUp: { 5248 LOG(MouseHandling, "WebPageProxy::didReceiveEvent: %s (queue empty %d)", webMouseEventTypeString(type), m_mouseEventQueue.isEmpty()); 5249 5250 // Retire the last sent event now that WebProcess is done handling it. 5251 MESSAGE_CHECK(!m_mouseEventQueue.isEmpty()); 5252 NativeWebMouseEvent event = m_mouseEventQueue.takeFirst(); 5253 MESSAGE_CHECK(type == event.type()); 5254 5255 if (!m_mouseEventQueue.isEmpty()) { 5256 LOG(MouseHandling, " UI process: handling a queued mouse event from didReceiveEvent"); 5257 processNextQueuedMouseEvent(); 5258 } else if (auto* automationSession = process().processPool().automationSession()) 5259 automationSession->mouseEventsFlushedForPage(*this); 5260 5221 5261 break; 5222 case WebEvent::MouseDown: 5223 break; 5224 case WebEvent::MouseUp: 5225 m_currentlyProcessedMouseDownEvent = nullptr; 5226 break; 5262 } 5263 5227 5264 case WebEvent::MouseForceChanged: 5228 5265 case WebEvent::MouseForceDown: … … 5257 5294 MESSAGE_CHECK(type == event.type()); 5258 5295 5259 if (!m_keyEventQueue.isEmpty()) { 5296 bool canProcessMoreKeyEvents = !m_keyEventQueue.isEmpty(); 5297 if (canProcessMoreKeyEvents) { 5260 5298 LOG(KeyHandling, " UI process: sent keyEvent from didReceiveEvent"); 5261 5299 m_process->send(Messages::WebPage::KeyEvent(m_keyEventQueue.first()), m_pageID); … … 5271 5309 5272 5310 // Notify the session after -[NSApp sendEvent:] has a crack at turning the event into an action. 5273 if ( m_keyEventQueue.isEmpty()) {5311 if (!canProcessMoreKeyEvents) { 5274 5312 if (auto* automationSession = process().processPool().automationSession()) 5275 5313 automationSession->keyboardEventsFlushedForPage(*this); … … 5875 5913 5876 5914 // Can't expect DidReceiveEvent notifications from a crashed web process. 5915 m_mouseEventQueue.clear(); 5877 5916 m_keyEventQueue.clear(); 5878 5917 m_wheelEventQueue.clear(); 5879 5918 m_currentlyProcessedWheelEvents.clear(); 5880 5881 m_nextMouseMoveEvent = nullptr;5882 m_currentlyProcessedMouseDownEvent = nullptr;5883 5884 m_processingMouseMoveEvent = false;5885 5886 5919 #if ENABLE(TOUCH_EVENTS) && !ENABLE(IOS_TOUCH_EVENTS) 5887 5920 m_touchEventQueue.clear(); -
trunk/Source/WebKit/UIProcess/WebPageProxy.h
r230721 r230743 685 685 #endif 686 686 687 bool isProcessingMouseEvents() const; 688 void processNextQueuedMouseEvent(); 687 689 void handleMouseEvent(const NativeWebMouseEvent&); 690 688 691 void handleWheelEvent(const NativeWebWheelEvent&); 689 692 void handleKeyboardEvent(const NativeWebKeyboardEvent&); … … 1933 1936 bool m_shouldSuppressAppLinksInNextNavigationPolicyDecision { false }; 1934 1937 1938 Deque<NativeWebMouseEvent> m_mouseEventQueue; 1935 1939 Deque<NativeWebKeyboardEvent> m_keyEventQueue; 1936 1940 Deque<NativeWebWheelEvent> m_wheelEventQueue; … … 1939 1943 Deque<NativeWebGestureEvent> m_gestureEventQueue; 1940 1944 #endif 1941 1942 bool m_processingMouseMoveEvent { false };1943 std::unique_ptr<NativeWebMouseEvent> m_nextMouseMoveEvent;1944 std::unique_ptr<NativeWebMouseEvent> m_currentlyProcessedMouseDownEvent;1945 1945 1946 1946 #if ENABLE(TOUCH_EVENTS)
Note: See TracChangeset
for help on using the changeset viewer.