Changeset 270772 in webkit


Ignore:
Timestamp:
Dec 14, 2020 9:45:54 AM (19 months ago)
Author:
sihui_liu@apple.com
Message:

Implement recognizer for SpeechRecognition
​https://bugs.webkit.org/show_bug.cgi?id=219459
<rdar://problem/71914465>

Reviewed by Youenn Fablet.

Source/WebCore:

Add WebSpeechRecognizerTask, which connects to speech recognition service using Speech framework APIs.

Tests: fast/speechrecognition/ios/restart-recognition-after-stop.html

fast/speechrecognition/ios/start-recognition-then-stop.html

  • Modules/speech/SpeechRecognizer.cpp:

(WebCore::SpeechRecognizer::reset):
(WebCore::SpeechRecognizer::abort):
(WebCore::SpeechRecognizer::stop):
(WebCore::SpeechRecognizer::start):
(WebCore::SpeechRecognizer::startCapture):
(WebCore::SpeechRecognizer::stopCapture):
(WebCore::SpeechRecognizer::dataCaptured):
(WebCore::SpeechRecognizer::startRecognition):
(WebCore::SpeechRecognizer::abortRecognition):
(WebCore::SpeechRecognizer::stopRecognition):
(WebCore::SpeechRecognizer::resetRecognition):
(WebCore::SpeechRecognizer::setSource): Deleted.
(WebCore::SpeechRecognizer::stopInternal): Deleted.

  • Modules/speech/SpeechRecognizer.h:
  • Modules/speech/cocoa/SpeechRecognizerCocoa.mm: Added.

(WebCore::SpeechRecognizer::dataCaptured):
(WebCore::SpeechRecognizer::startRecognition):
(WebCore::SpeechRecognizer::stopRecognition):
(WebCore::SpeechRecognizer::abortRecognition):
(WebCore::SpeechRecognizer::resetRecognition):

  • Modules/speech/cocoa/WebSpeechRecognizerTask.h: Added.
  • Modules/speech/cocoa/WebSpeechRecognizerTask.mm: Added.

(-[WebSpeechRecognizerTaskImpl initWithIdentifier:locale:doMultipleRecognitions:reportInterimResults:maxAlternatives:delegateCallback:]):
(-[WebSpeechRecognizerTaskImpl callbackWithResult:isFinal:]):
(-[WebSpeechRecognizerTaskImpl audioSamplesAvailable:]):
(-[WebSpeechRecognizerTaskImpl abort]):
(-[WebSpeechRecognizerTaskImpl stop]):
(-[WebSpeechRecognizerTaskImpl sendSpeechStartIfNeeded]):
(-[WebSpeechRecognizerTaskImpl sendSpeechEndIfNeeded]):
(-[WebSpeechRecognizerTaskImpl sendEndIfNeeded]):
(-[WebSpeechRecognizerTaskImpl speechRecognizer:availabilityDidChange:]):
(-[WebSpeechRecognizerTaskImpl speechRecognitionTask:didHypothesizeTranscription:]):
(-[WebSpeechRecognizerTaskImpl speechRecognitionTask:didFinishRecognition:]):
(-[WebSpeechRecognizerTaskImpl speechRecognitionTaskWasCancelled:]):
(-[WebSpeechRecognizerTaskImpl speechRecognitionTask:didFinishSuccessfully:]):
(-[WebSpeechRecognizerTask initWithIdentifier:locale:doMultipleRecognitions:reportInterimResults:maxAlternatives:delegateCallback:]):
(-[WebSpeechRecognizerTask audioSamplesAvailable:]):
(-[WebSpeechRecognizerTask abort]):
(-[WebSpeechRecognizerTask stop]):

  • Modules/speech/cocoa/WebSpeechRecognizerTaskMock.h: Added.
  • Modules/speech/cocoa/WebSpeechRecognizerTaskMock.mm: Added.

(-[WebSpeechRecognizerTaskMock initWithIdentifier:locale:doMultipleRecognitions:reportInterimResults:maxAlternatives:delegateCallback:]):
(-[WebSpeechRecognizerTaskMock audioSamplesAvailable:]):
(-[WebSpeechRecognizerTaskMock abort]):
(-[WebSpeechRecognizerTaskMock stop]):

  • SourcesCocoa.txt:
  • WebCore.xcodeproj/project.pbxproj:

Source/WebCore/PAL:

Add soft linking to Speech framework and SPI.

  • PAL.xcodeproj/project.pbxproj:
  • pal/cocoa/SpeechSoftLink.h: Added.
  • pal/cocoa/SpeechSoftLink.mm: Added.
  • pal/spi/cocoa/SpeechSPI.h: Added.

Source/WebKit:

  • UIProcess/SpeechRecognitionServer.cpp:

(WebKit::SpeechRecognitionServer::SpeechRecognitionServer):
(WebKit::SpeechRecognitionServer::requestPermissionForRequest):
(WebKit::SpeechRecognitionServer::handleRequest):
(WebKit::SpeechRecognitionServer::abort):
(WebKit::SpeechRecognitionServer::invalidate):

  • UIProcess/SpeechRecognitionServer.h:
  • UIProcess/WebProcessProxy.cpp:

(WebKit::WebProcessProxy::createSpeechRecognitionServer):

LayoutTests:

  • fast/speechrecognition/ios/restart-recognition-after-stop-expected.txt: Added.
  • fast/speechrecognition/ios/restart-recognition-after-stop.html: Added.
  • fast/speechrecognition/ios/start-recognition-then-stop-expected.txt: Added.
  • fast/speechrecognition/ios/start-recognition-then-stop.html: Added.
Location:
trunk
Files:
8 added
12 edited
5 copied

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r270771 r270772  
     12020-12-14  Sihui Liu  <sihui_liu@apple.com>
     2
     3        Implement recognizer for SpeechRecognition
     4        https://bugs.webkit.org/show_bug.cgi?id=219459
     5        <rdar://problem/71914465>
     6
     7        Reviewed by Youenn Fablet.
     8
     9        * fast/speechrecognition/ios/restart-recognition-after-stop-expected.txt: Added.
     10        * fast/speechrecognition/ios/restart-recognition-after-stop.html: Added.
     11        * fast/speechrecognition/ios/start-recognition-then-stop-expected.txt: Added.
     12        * fast/speechrecognition/ios/start-recognition-then-stop.html: Added.
     13
    1142020-12-14  Andy Estes  <aestes@apple.com>
    215

  • trunk/Source/WebCore/ChangeLog

    r270768 r270772  
     12020-12-14  Sihui Liu  <sihui_liu@apple.com>
     2
     3        Implement recognizer for SpeechRecognition
     4        https://bugs.webkit.org/show_bug.cgi?id=219459
     5        <rdar://problem/71914465>
     6
     7        Reviewed by Youenn Fablet.
     8
     9        Add WebSpeechRecognizerTask, which connects to speech recognition service using Speech framework APIs.
     10
     11        Tests: fast/speechrecognition/ios/restart-recognition-after-stop.html
     12               fast/speechrecognition/ios/start-recognition-then-stop.html
     13
     14        * Modules/speech/SpeechRecognizer.cpp:
     15        (WebCore::SpeechRecognizer::reset):
     16        (WebCore::SpeechRecognizer::abort):
     17        (WebCore::SpeechRecognizer::stop):
     18        (WebCore::SpeechRecognizer::start):
     19        (WebCore::SpeechRecognizer::startCapture):
     20        (WebCore::SpeechRecognizer::stopCapture):
     21        (WebCore::SpeechRecognizer::dataCaptured):
     22        (WebCore::SpeechRecognizer::startRecognition):
     23        (WebCore::SpeechRecognizer::abortRecognition):
     24        (WebCore::SpeechRecognizer::stopRecognition):
     25        (WebCore::SpeechRecognizer::resetRecognition):
     26        (WebCore::SpeechRecognizer::setSource): Deleted.
     27        (WebCore::SpeechRecognizer::stopInternal): Deleted.
     28        * Modules/speech/SpeechRecognizer.h:
     29        * Modules/speech/cocoa/SpeechRecognizerCocoa.mm: Added.
     30        (WebCore::SpeechRecognizer::dataCaptured):
     31        (WebCore::SpeechRecognizer::startRecognition):
     32        (WebCore::SpeechRecognizer::stopRecognition):
     33        (WebCore::SpeechRecognizer::abortRecognition):
     34        (WebCore::SpeechRecognizer::resetRecognition):
     35        * Modules/speech/cocoa/WebSpeechRecognizerTask.h: Added.
     36        * Modules/speech/cocoa/WebSpeechRecognizerTask.mm: Added.
     37        (-[WebSpeechRecognizerTaskImpl initWithIdentifier:locale:doMultipleRecognitions:reportInterimResults:maxAlternatives:delegateCallback:]):
     38        (-[WebSpeechRecognizerTaskImpl callbackWithResult:isFinal:]):
     39        (-[WebSpeechRecognizerTaskImpl audioSamplesAvailable:]):
     40        (-[WebSpeechRecognizerTaskImpl abort]):
     41        (-[WebSpeechRecognizerTaskImpl stop]):
     42        (-[WebSpeechRecognizerTaskImpl sendSpeechStartIfNeeded]):
     43        (-[WebSpeechRecognizerTaskImpl sendSpeechEndIfNeeded]):
     44        (-[WebSpeechRecognizerTaskImpl sendEndIfNeeded]):
     45        (-[WebSpeechRecognizerTaskImpl speechRecognizer:availabilityDidChange:]):
     46        (-[WebSpeechRecognizerTaskImpl speechRecognitionTask:didHypothesizeTranscription:]):
     47        (-[WebSpeechRecognizerTaskImpl speechRecognitionTask:didFinishRecognition:]):
     48        (-[WebSpeechRecognizerTaskImpl speechRecognitionTaskWasCancelled:]):
     49        (-[WebSpeechRecognizerTaskImpl speechRecognitionTask:didFinishSuccessfully:]):
     50        (-[WebSpeechRecognizerTask initWithIdentifier:locale:doMultipleRecognitions:reportInterimResults:maxAlternatives:delegateCallback:]):
     51        (-[WebSpeechRecognizerTask audioSamplesAvailable:]):
     52        (-[WebSpeechRecognizerTask abort]):
     53        (-[WebSpeechRecognizerTask stop]):
     54        * Modules/speech/cocoa/WebSpeechRecognizerTaskMock.h: Added.
     55        * Modules/speech/cocoa/WebSpeechRecognizerTaskMock.mm: Added.
     56        (-[WebSpeechRecognizerTaskMock initWithIdentifier:locale:doMultipleRecognitions:reportInterimResults:maxAlternatives:delegateCallback:]):
     57        (-[WebSpeechRecognizerTaskMock audioSamplesAvailable:]):
     58        (-[WebSpeechRecognizerTaskMock abort]):
     59        (-[WebSpeechRecognizerTaskMock stop]):
     60        * SourcesCocoa.txt:
     61        * WebCore.xcodeproj/project.pbxproj:
     62
    1632020-12-14  Chris Dumez  <cdumez@apple.com>
    264

  • trunk/Source/WebCore/Modules/speech/SpeechRecognizer.cpp

    r270574 r270772  
    2828
    2929#include "SpeechRecognitionUpdate.h"
     30#include <wtf/MediaTime.h>
    3031
    3132#if PLATFORM(COCOA)
    3233#include "MediaUtilities.h"
    33 #include <pal/avfoundation/MediaTimeAVFoundation.h>
    3434#endif
    3535
    … …  
    4646        return;
    4747
    48     if (m_source)
    49         m_source = nullptr;
     48    stopCapture();
     49    resetRecognition();
     50    m_clientIdentifier = WTF::nullopt;
     51}
    5052
    51     auto error = SpeechRecognitionError { SpeechRecognitionErrorType::Aborted, "Another request is started" };
    52     m_delegateCallback(SpeechRecognitionUpdate::createError(*m_clientIdentifier, error));
     53void SpeechRecognizer::abort()
     54{
     55    ASSERT(m_clientIdentifier);
     56    stopCapture();
     57    abortRecognition();
     58}
    5359
    54     m_clientIdentifier = WTF::nullopt;
     60void SpeechRecognizer::stop()
     61{
     62    ASSERT(m_clientIdentifier);
     63    stopCapture();
     64    stopRecognition();
    5565}
    5666
    5767#if ENABLE(MEDIA_STREAM)
    5868
    59 void SpeechRecognizer::start(SpeechRecognitionConnectionClientIdentifier identifier, Ref<RealtimeMediaSource>&& source)
     69void SpeechRecognizer::start(SpeechRecognitionConnectionClientIdentifier clientIdentifier, Ref<RealtimeMediaSource>&& source, bool mockSpeechRecognitionEnabled, const String& localeIdentifier, bool continuous, bool interimResults, uint64_t maxAlternatives)
    6070{
    61     ASSERT(!m_source);
    62 
    63     m_clientIdentifier = identifier;
     71    ASSERT(!m_clientIdentifier);
     72    m_clientIdentifier = clientIdentifier;
    6473    m_delegateCallback(SpeechRecognitionUpdate::create(*m_clientIdentifier, SpeechRecognitionUpdateType::Start));
    6574
    66     setSource(WTFMove(source));
     75    if (!startRecognition(mockSpeechRecognitionEnabled, clientIdentifier, localeIdentifier, continuous, interimResults, maxAlternatives)) {
     76        auto error = WebCore::SpeechRecognitionError { WebCore::SpeechRecognitionErrorType::ServiceNotAllowed, "Failed to start recognition"_s };
     77        m_delegateCallback(WebCore::SpeechRecognitionUpdate::createError(clientIdentifier, WTFMove(error)));
     78        return;
     79    }
     80
     81    startCapture(WTFMove(source));
    6782}
    6883
    69 void SpeechRecognizer::setSource(Ref<RealtimeMediaSource>&& source)
     84void SpeechRecognizer::startCapture(Ref<RealtimeMediaSource>&& source)
    7085{
    7186    auto dataCallback = [weakThis = makeWeakPtr(this)](const auto& time, const auto& data, const auto& description, auto sampleCount) {
    72         if (!weakThis)
    73             return;
    74 
    75 #if PLATFORM(COCOA)
    76         auto buffer = createAudioSampleBuffer(data, description, PAL::toCMTime(time), sampleCount);
    77         UNUSED_PARAM(buffer);
    78 #else
    79         UNUSED_PARAM(time);
    80         UNUSED_PARAM(data);
    81         UNUSED_PARAM(description);
    82         UNUSED_PARAM(sampleCount);
    83 #endif
     87        if (weakThis)
     88            weakThis->dataCaptured(time, data, description, sampleCount);
    8489    };
    8590
    … …  
    9095        ASSERT(m_clientIdentifier && m_clientIdentifier.value() == update.clientIdentifier());
    9196        m_delegateCallback(update);
    92 
    93         if (update.type() == SpeechRecognitionUpdateType::Error)
    94             m_source = nullptr;
    9597    };
    9698
    … …  
    100102#endif
    101103
    102 void SpeechRecognizer::stop(ShouldGenerateFinalResult shouldGenerateFinalResult)
    103 {
    104     if (!m_clientIdentifier)
    105         return;
    106 
    107     stopInternal();
    108 
    109     if (shouldGenerateFinalResult == ShouldGenerateFinalResult::Yes) {
    110         // TODO: generate real result when speech recognition backend is implemented.
    111         Vector<SpeechRecognitionResultData> resultDatas;
    112         m_delegateCallback(SpeechRecognitionUpdate::createResult(*m_clientIdentifier, resultDatas));
    113     }
    114 
    115     m_delegateCallback(SpeechRecognitionUpdate::create(*m_clientIdentifier, SpeechRecognitionUpdateType::End));
    116     m_clientIdentifier = WTF::nullopt;
    117 }
    118 
    119 void SpeechRecognizer::stopInternal()
     104void SpeechRecognizer::stopCapture()
    120105{
    121106    if (!m_source)
    … …  
    126111}
    127112
     113#if !HAVE(SPEECHRECOGNIZER)
     114
     115void SpeechRecognizer::dataCaptured(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t)
     116{
     117}
     118
     119bool SpeechRecognizer::startRecognition(bool, SpeechRecognitionConnectionClientIdentifier, const String&, bool, bool, uint64_t)
     120{
     121    return true;
     122}
     123
     124void SpeechRecognizer::abortRecognition()
     125{
     126    m_delegateCallback(SpeechRecognitionUpdate::create(*m_clientIdentifier, SpeechRecognitionUpdateType::End));
     127}
     128
     129void SpeechRecognizer::stopRecognition()
     130{
     131    m_delegateCallback(SpeechRecognitionUpdate::create(*m_clientIdentifier, SpeechRecognitionUpdateType::End));
     132}
     133
     134void SpeechRecognizer::resetRecognition()
     135{
     136}
     137
     138#endif
     139
    128140} // namespace WebCore

  • trunk/Source/WebCore/Modules/speech/SpeechRecognizer.h

    r270574 r270772  
    2929#include "SpeechRecognitionConnectionClientIdentifier.h"
    3030
     31#if HAVE(SPEECHRECOGNIZER)
     32#include <wtf/RetainPtr.h>
     33OBJC_CLASS WebSpeechRecognizerTask;
     34#endif
     35
    3136namespace WebCore {
    3237
    … …  
    4146
    4247#if ENABLE(MEDIA_STREAM)
    43     WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier, Ref<RealtimeMediaSource>&&);
     48    WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier, Ref<RealtimeMediaSource>&&, bool mockSpeechRecognitionEnabled, const String& localeIdentifier, bool continuous, bool interimResults, uint64_t maxAlternatives);
    4449#endif
    4550    WEBCORE_EXPORT void reset();
    46     enum class ShouldGenerateFinalResult { No, Yes };
    47     WEBCORE_EXPORT void stop(ShouldGenerateFinalResult = ShouldGenerateFinalResult::Yes);
     51    WEBCORE_EXPORT void abort();
     52    WEBCORE_EXPORT void stop();
    4853
    4954    Optional<SpeechRecognitionConnectionClientIdentifier> currentClientIdentifier() const { return m_clientIdentifier; }
    … …  
    5358
    5459#if ENABLE(MEDIA_STREAM)
    55     void setSource(Ref<RealtimeMediaSource>&&);
     60    void startCapture(Ref<RealtimeMediaSource>&&);
    5661#endif
     62    void stopCapture();
     63    void dataCaptured(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t sampleCount);
     64    bool startRecognition(bool mockSpeechRecognitionEnabled, SpeechRecognitionConnectionClientIdentifier, const String& localeIdentifier, bool continuous, bool interimResults, uint64_t alternatives);
     65    void abortRecognition();
     66    void stopRecognition();
     67    void resetRecognition();
    5768
    5869    Optional<SpeechRecognitionConnectionClientIdentifier> m_clientIdentifier;
    5970    DelegateCallback m_delegateCallback;
    6071    std::unique_ptr<SpeechRecognitionCaptureSource> m_source;
     72
     73#if HAVE(SPEECHRECOGNIZER)
     74    RetainPtr<WebSpeechRecognizerTask> m_task;
     75#endif
    6176};
    6277

  • trunk/Source/WebCore/Modules/speech/cocoa/WebSpeechRecognizerTask.h

    r270771 r270772  
    2424 */
    2525
    26 #pragma once
     26#if HAVE(SPEECHRECOGNIZER)
    2727
    28 #include "SpeechRecognitionCaptureSource.h"
    29 #include "SpeechRecognitionConnectionClientIdentifier.h"
     28#import "SpeechRecognitionConnectionClientIdentifier.h"
     29#import "SpeechRecognitionUpdate.h"
    3030
    31 namespace WebCore {
     31NS_ASSUME_NONNULL_BEGIN
    3232
    33 class SpeechRecognitionUpdate;
     33typedef struct opaqueCMSampleBuffer *CMSampleBufferRef;
    3434
    35 class SpeechRecognizer : public CanMakeWeakPtr<SpeechRecognizer> {
    36     WTF_MAKE_FAST_ALLOCATED;
    37 public:
    38     using DelegateCallback = Function<void(const SpeechRecognitionUpdate&)>;
    39     WEBCORE_EXPORT explicit SpeechRecognizer(DelegateCallback&&);
    40     WEBCORE_EXPORT ~SpeechRecognizer() = default;
     35@class WebSpeechRecognizerTaskImpl;
    4136
    42 #if ENABLE(MEDIA_STREAM)
    43     WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier, Ref<RealtimeMediaSource>&&);
     37@interface WebSpeechRecognizerTask : NSObject {
     38@private
     39    RetainPtr<WebSpeechRecognizerTaskImpl> _impl;
     40}
     41
     42- (instancetype)initWithIdentifier:(WebCore::SpeechRecognitionConnectionClientIdentifier)identifier locale:(NSString*)localeIdentifier doMultipleRecognitions:(BOOL)continuous reportInterimResults:(BOOL)interimResults maxAlternatives:(unsigned long)alternatives delegateCallback:(void(^)(const WebCore::SpeechRecognitionUpdate&))callback;
     43- (void)audioSamplesAvailable:(CMSampleBufferRef)sampleBuffer;
     44- (void)abort;
     45- (void)stop;
     46
     47@end
     48
     49NS_ASSUME_NONNULL_END
     50
    4451#endif
    45     WEBCORE_EXPORT void reset();
    46     enum class ShouldGenerateFinalResult { No, Yes };
    47     WEBCORE_EXPORT void stop(ShouldGenerateFinalResult = ShouldGenerateFinalResult::Yes);
    48 
    49     Optional<SpeechRecognitionConnectionClientIdentifier> currentClientIdentifier() const { return m_clientIdentifier; }
    50 
    51 private:
    52     void stopInternal();
    53 
    54 #if ENABLE(MEDIA_STREAM)
    55     void setSource(Ref<RealtimeMediaSource>&&);
    56 #endif
    57 
    58     Optional<SpeechRecognitionConnectionClientIdentifier> m_clientIdentifier;
    59     DelegateCallback m_delegateCallback;
    60     std::unique_ptr<SpeechRecognitionCaptureSource> m_source;
    61 };
    62 
    63 } // namespace WebCore

  • trunk/Source/WebCore/Modules/speech/cocoa/WebSpeechRecognizerTaskMock.h

    r270771 r270772  
    2424 */
    2525
    26 #pragma once
     26#if HAVE(SPEECHRECOGNIZER)
    2727
    28 #include "SpeechRecognitionCaptureSource.h"
    29 #include "SpeechRecognitionConnectionClientIdentifier.h"
     28#import "WebSpeechRecognizerTask.h"
     29#import <wtf/BlockPtr.h>
    3030
    31 namespace WebCore {
     31NS_ASSUME_NONNULL_BEGIN
    3232
    33 class SpeechRecognitionUpdate;
     33@class WebSpeechRecognizerTaskMock;
    3434
    35 class SpeechRecognizer : public CanMakeWeakPtr<SpeechRecognizer> {
    36     WTF_MAKE_FAST_ALLOCATED;
    37 public:
    38     using DelegateCallback = Function<void(const SpeechRecognitionUpdate&)>;
    39     WEBCORE_EXPORT explicit SpeechRecognizer(DelegateCallback&&);
    40     WEBCORE_EXPORT ~SpeechRecognizer() = default;
     35@interface WebSpeechRecognizerTaskMock : WebSpeechRecognizerTask {
     36@private
     37    WebCore::SpeechRecognitionConnectionClientIdentifier _identifier;
     38    BlockPtr<void(const WebCore::SpeechRecognitionUpdate&)> _delegateCallback;
     39    bool _doMultipleRecognitions;
     40    bool _hasSentSpeechStart;
     41    bool _hasSentSpeechEnd;
     42    bool _completed;
     43}
    4144
    42 #if ENABLE(MEDIA_STREAM)
    43     WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier, Ref<RealtimeMediaSource>&&);
     45- (instancetype)initWithIdentifier:(WebCore::SpeechRecognitionConnectionClientIdentifier)identifier locale:(NSString*)localeIdentifier doMultipleRecognitions:(BOOL)continuous reportInterimResults:(BOOL)interimResults maxAlternatives:(unsigned long)alternatives delegateCallback:(void(^)(const WebCore::SpeechRecognitionUpdate&))callback;
     46- (void)audioSamplesAvailable:(CMSampleBufferRef)sampleBuffer;
     47- (void)abort;
     48- (void)stop;
     49
     50@end
     51
     52NS_ASSUME_NONNULL_END
     53
    4454#endif
    45     WEBCORE_EXPORT void reset();
    46     enum class ShouldGenerateFinalResult { No, Yes };
    47     WEBCORE_EXPORT void stop(ShouldGenerateFinalResult = ShouldGenerateFinalResult::Yes);
    48 
    49     Optional<SpeechRecognitionConnectionClientIdentifier> currentClientIdentifier() const { return m_clientIdentifier; }
    50 
    51 private:
    52     void stopInternal();
    53 
    54 #if ENABLE(MEDIA_STREAM)
    55     void setSource(Ref<RealtimeMediaSource>&&);
    56 #endif
    57 
    58     Optional<SpeechRecognitionConnectionClientIdentifier> m_clientIdentifier;
    59     DelegateCallback m_delegateCallback;
    60     std::unique_ptr<SpeechRecognitionCaptureSource> m_source;
    61 };
    62 
    63 } // namespace WebCore

  • trunk/Source/WebCore/PAL/ChangeLog

    r270758 r270772  
     12020-12-14  Sihui Liu  <sihui_liu@apple.com>
     2
     3        Implement recognizer for SpeechRecognition
     4        https://bugs.webkit.org/show_bug.cgi?id=219459
     5        <rdar://problem/71914465>
     6
     7        Reviewed by Youenn Fablet.
     8
     9        Add soft linking to Speech framework and SPI.
     10
     11        * PAL.xcodeproj/project.pbxproj:
     12        * pal/cocoa/SpeechSoftLink.h: Added.
     13        * pal/cocoa/SpeechSoftLink.mm: Added.
     14        * pal/spi/cocoa/SpeechSPI.h: Added.
     15
    1162020-12-13  Andy Estes  <aestes@apple.com>
    217

  • trunk/Source/WebCore/PAL/PAL.xcodeproj/project.pbxproj

    r269893 r270772  
    145145                7A36D0F9223AD9AB00B0522E /* CommonCryptoSPI.h in Headers */ = {isa = PBXBuildFile; fileRef = 7A36D0F8223AD9AB00B0522E /* CommonCryptoSPI.h */; };
    146146                7A3A6A8020CADB4700317AAE /* NSImageSPI.h in Headers */ = {isa = PBXBuildFile; fileRef = 7A3A6A7F20CADB4600317AAE /* NSImageSPI.h */; };
     147                93B38EBE25821CB600198E63 /* SpeechSoftLink.h in Headers */ = {isa = PBXBuildFile; fileRef = 93B38EBD25821CB600198E63 /* SpeechSoftLink.h */; };
     148                93B38EC025821CD800198E63 /* SpeechSoftLink.mm in Sources */ = {isa = PBXBuildFile; fileRef = 93B38EBF25821CD700198E63 /* SpeechSoftLink.mm */; };
     149                93B38EC225821D2200198E63 /* SpeechSPI.h in Headers */ = {isa = PBXBuildFile; fileRef = 93B38EC125821D2200198E63 /* SpeechSPI.h */; };
    147150                93DB7D3724626BCD004BD8A3 /* NSTextInputContextSPI.h in Headers */ = {isa = PBXBuildFile; fileRef = 93DB7D3624626BCC004BD8A3 /* NSTextInputContextSPI.h */; };
    148151                93DB7D3A24626F86004BD8A3 /* NSUndoManagerSPI.h in Headers */ = {isa = PBXBuildFile; fileRef = 93DB7D3924626F86004BD8A3 /* NSUndoManagerSPI.h */; };
    … …  
    341344                7A36D0F8223AD9AB00B0522E /* CommonCryptoSPI.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CommonCryptoSPI.h; sourceTree = "<group>"; };
    342345                7A3A6A7F20CADB4600317AAE /* NSImageSPI.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NSImageSPI.h; sourceTree = "<group>"; };
     346                93B38EBD25821CB600198E63 /* SpeechSoftLink.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SpeechSoftLink.h; sourceTree = "<group>"; };
     347                93B38EBF25821CD700198E63 /* SpeechSoftLink.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = SpeechSoftLink.mm; sourceTree = "<group>"; };
     348                93B38EC125821D2200198E63 /* SpeechSPI.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SpeechSPI.h; sourceTree = "<group>"; };
    343349                93DB7D3624626BCC004BD8A3 /* NSTextInputContextSPI.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NSTextInputContextSPI.h; sourceTree = "<group>"; };
    344350                93DB7D3924626F86004BD8A3 /* NSUndoManagerSPI.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NSUndoManagerSPI.h; sourceTree = "<group>"; };
    … …  
    512518                                570AB8F020AE2E8D00B8BE87 /* SecKeyProxySPI.h */,
    513519                                0C2DA13C1F3BEB4900DBC317 /* ServersSPI.h */,
     520                                93B38EC125821D2200198E63 /* SpeechSPI.h */,
    514521                                0C2DA12B1F3BEB4900DBC317 /* URLFormattingSPI.h */,
    515522                                0C2DA13D1F3BEB4900DBC317 /* WebFilterEvaluatorSPI.h */,
    … …  
    683690                                A1F63C9D21A4DBF7006FB43B /* PassKitSoftLink.h */,
    684691                                A1F63C9E21A4DBF7006FB43B /* PassKitSoftLink.mm */,
     692                                93B38EBD25821CB600198E63 /* SpeechSoftLink.h */,
     693                                93B38EBF25821CD700198E63 /* SpeechSoftLink.mm */,
    685694                                07611DB4243FA5BE00D80704 /* UsageTrackingSoftLink.h */,
    686695                                07611DB5243FA5BF00D80704 /* UsageTrackingSoftLink.mm */,
    … …  
    908917                                A3AB6E611F3D1E39009C14B1 /* SleepDisablerCocoa.h in Headers */,
    909918                                A3788E981F05B6CE00679425 /* Sound.h in Headers */,
     919                                93B38EBE25821CB600198E63 /* SpeechSoftLink.h in Headers */,
     920                                93B38EC225821D2200198E63 /* SpeechSPI.h in Headers */,
    910921                                A1175B491F6AFF8E00C4B9F0 /* SpeechSynthesisSPI.h in Headers */,
    911922                                0C5AF9211F43A4C7002EAC02 /* SQLite3SPI.h in Headers */,
    … …  
    10381049                                A3788E9C1F05B78200679425 /* Sound.cpp in Sources */,
    10391050                                A3788E9E1F05B78E00679425 /* SoundMac.mm in Sources */,
     1051                                93B38EC025821CD800198E63 /* SpeechSoftLink.mm in Sources */,
    10401052                                A3AB6E571F3D1DDB009C14B1 /* SystemSleepListener.cpp in Sources */,
    10411053                                A3AB6E651F3D217F009C14B1 /* SystemSleepListenerMac.mm in Sources */,

  • trunk/Source/WebCore/PAL/pal/cocoa/SpeechSoftLink.h

    r270771 r270772  
    2626#pragma once
    2727
    28 #include "SpeechRecognitionCaptureSource.h"
    29 #include "SpeechRecognitionConnectionClientIdentifier.h"
     28#if HAVE(SPEECHRECOGNIZER)
    3029
    31 namespace WebCore {
     30#import <Speech/Speech.h>
     31#import <wtf/SoftLinking.h>
    3232
    33 class SpeechRecognitionUpdate;
     33SOFT_LINK_FRAMEWORK_FOR_HEADER(PAL, Speech);
     34SOFT_LINK_CLASS_FOR_HEADER(PAL, SFSpeechRecognitionResult);
     35SOFT_LINK_CLASS_FOR_HEADER(PAL, SFSpeechRecognitionRequest);
     36SOFT_LINK_CLASS_FOR_HEADER(PAL, SFSpeechAudioBufferRecognitionRequest);
     37SOFT_LINK_CLASS_FOR_HEADER(PAL, SFSpeechRecognitionTask);
     38SOFT_LINK_CLASS_FOR_HEADER(PAL, SFSpeechRecognizer);
     39SOFT_LINK_CLASS_FOR_HEADER(PAL, SFTranscriptionSegment);
     40SOFT_LINK_CLASS_FOR_HEADER(PAL, SFTranscription);
    3441
    35 class SpeechRecognizer : public CanMakeWeakPtr<SpeechRecognizer> {
    36     WTF_MAKE_FAST_ALLOCATED;
    37 public:
    38     using DelegateCallback = Function<void(const SpeechRecognitionUpdate&)>;
    39     WEBCORE_EXPORT explicit SpeechRecognizer(DelegateCallback&&);
    40     WEBCORE_EXPORT ~SpeechRecognizer() = default;
    41 
    42 #if ENABLE(MEDIA_STREAM)
    43     WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier, Ref<RealtimeMediaSource>&&);
    4442#endif
    45     WEBCORE_EXPORT void reset();
    46     enum class ShouldGenerateFinalResult { No, Yes };
    47     WEBCORE_EXPORT void stop(ShouldGenerateFinalResult = ShouldGenerateFinalResult::Yes);
    48 
    49     Optional<SpeechRecognitionConnectionClientIdentifier> currentClientIdentifier() const { return m_clientIdentifier; }
    50 
    51 private:
    52     void stopInternal();
    53 
    54 #if ENABLE(MEDIA_STREAM)
    55     void setSource(Ref<RealtimeMediaSource>&&);
    56 #endif
    57 
    58     Optional<SpeechRecognitionConnectionClientIdentifier> m_clientIdentifier;
    59     DelegateCallback m_delegateCallback;
    60     std::unique_ptr<SpeechRecognitionCaptureSource> m_source;
    61 };
    62 
    63 } // namespace WebCore

  • trunk/Source/WebCore/PAL/pal/cocoa/SpeechSoftLink.mm

    r270771 r270772  
    2424 */
    2525
    26 #pragma once
     26#import "config.h"
    2727
    28 #include "SpeechRecognitionCaptureSource.h"
    29 #include "SpeechRecognitionConnectionClientIdentifier.h"
     28#if HAVE(SPEECHRECOGNIZER)
    3029
    31 namespace WebCore {
     30#import <Speech/Speech.h>
     31#import <wtf/SoftLinking.h>
    3232
    33 class SpeechRecognitionUpdate;
     33SOFT_LINK_FRAMEWORK_FOR_SOURCE(PAL, Speech)
    3434
    35 class SpeechRecognizer : public CanMakeWeakPtr<SpeechRecognizer> {
    36     WTF_MAKE_FAST_ALLOCATED;
    37 public:
    38     using DelegateCallback = Function<void(const SpeechRecognitionUpdate&)>;
    39     WEBCORE_EXPORT explicit SpeechRecognizer(DelegateCallback&&);
    40     WEBCORE_EXPORT ~SpeechRecognizer() = default;
     35SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, Speech, SFSpeechRecognitionResult, PAL_EXPORT);
     36SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, Speech, SFSpeechRecognitionRequest, PAL_EXPORT);
     37SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, Speech, SFSpeechAudioBufferRecognitionRequest, PAL_EXPORT);
     38SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, Speech, SFSpeechRecognitionTask, PAL_EXPORT);
     39SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, Speech, SFSpeechRecognizer, PAL_EXPORT);
     40SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, Speech, SFTranscriptionSegment, PAL_EXPORT);
     41SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, Speech, SFTranscription, PAL_EXPORT);
    4142
    42 #if ENABLE(MEDIA_STREAM)
    43     WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier, Ref<RealtimeMediaSource>&&);
    4443#endif
    45     WEBCORE_EXPORT void reset();
    46     enum class ShouldGenerateFinalResult { No, Yes };
    47     WEBCORE_EXPORT void stop(ShouldGenerateFinalResult = ShouldGenerateFinalResult::Yes);
    48 
    49     Optional<SpeechRecognitionConnectionClientIdentifier> currentClientIdentifier() const { return m_clientIdentifier; }
    50 
    51 private:
    52     void stopInternal();
    53 
    54 #if ENABLE(MEDIA_STREAM)
    55     void setSource(Ref<RealtimeMediaSource>&&);
    56 #endif
    57 
    58     Optional<SpeechRecognitionConnectionClientIdentifier> m_clientIdentifier;
    59     DelegateCallback m_delegateCallback;
    60     std::unique_ptr<SpeechRecognitionCaptureSource> m_source;
    61 };
    62 
    63 } // namespace WebCore

  • trunk/Source/WebCore/PAL/pal/spi/cocoa/SpeechSPI.h

    r270771 r270772  
    2424 */
    2525
    26 #pragma once
     26#if HAVE(SPEECHRECOGNIZER)
    2727
    28 #include "SpeechRecognitionCaptureSource.h"
    29 #include "SpeechRecognitionConnectionClientIdentifier.h"
     28#if USE(APPLE_INTERNAL_SDK)
    3029
    31 namespace WebCore {
     30#import <Speech/SFSpeechRecognitionRequest_Private.h>
    3231
    33 class SpeechRecognitionUpdate;
     32#else
    3433
    35 class SpeechRecognizer : public CanMakeWeakPtr<SpeechRecognizer> {
    36     WTF_MAKE_FAST_ALLOCATED;
    37 public:
    38     using DelegateCallback = Function<void(const SpeechRecognitionUpdate&)>;
    39     WEBCORE_EXPORT explicit SpeechRecognizer(DelegateCallback&&);
    40     WEBCORE_EXPORT ~SpeechRecognizer() = default;
     34NS_ASSUME_NONNULL_BEGIN
    4135
    42 #if ENABLE(MEDIA_STREAM)
    43     WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier, Ref<RealtimeMediaSource>&&);
    44 #endif
    45     WEBCORE_EXPORT void reset();
    46     enum class ShouldGenerateFinalResult { No, Yes };
    47     WEBCORE_EXPORT void stop(ShouldGenerateFinalResult = ShouldGenerateFinalResult::Yes);
     36@interface SFSpeechRecognitionRequest ()
     37@property (nonatomic) BOOL detectMultipleUtterances;
     38@property (nonatomic, getter=_maximumRecognitionDuration, setter=_setMaximumRecognitionDuration:) NSTimeInterval _maximumRecognitionDuration;
    4839
    49     Optional<SpeechRecognitionConnectionClientIdentifier> currentClientIdentifier() const { return m_clientIdentifier; }
     40@end
    5041
    51 private:
    52     void stopInternal();
     42NS_ASSUME_NONNULL_END
    5343
    54 #if ENABLE(MEDIA_STREAM)
    55     void setSource(Ref<RealtimeMediaSource>&&);
    56 #endif
     44#endif // USE(APPLE_INTERNAL_SDK)
    5745
    58     Optional<SpeechRecognitionConnectionClientIdentifier> m_clientIdentifier;
    59     DelegateCallback m_delegateCallback;
    60     std::unique_ptr<SpeechRecognitionCaptureSource> m_source;
    61 };
    62 
    63 } // namespace WebCore
     46#endif // HAVE(SPEECHRECOGNIZER)

  • trunk/Source/WebCore/SourcesCocoa.txt

    r270662 r270772  
    9393Modules/plugins/QuickTimePluginReplacement.mm
    9494Modules/plugins/YouTubePluginReplacement.cpp
     95Modules/speech/cocoa/SpeechRecognizerCocoa.mm
     96Modules/speech/cocoa/WebSpeechRecognizerTask.mm
     97Modules/speech/cocoa/WebSpeechRecognizerTaskMock.mm
    9598Modules/webdatabase/cocoa/DatabaseManagerCocoa.mm
    9699accessibility/ios/AXObjectCacheIOS.mm

  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r270748 r270772  
    28202820                93A806201E03B585008A1F26 /* JSLongRange.h in Headers */ = {isa = PBXBuildFile; fileRef = 93A8061C1E03B585008A1F26 /* JSLongRange.h */; };
    28212821                93B2D8160F9920D2006AE6B2 /* SuddenTermination.h in Headers */ = {isa = PBXBuildFile; fileRef = 93B2D8150F9920D2006AE6B2 /* SuddenTermination.h */; settings = {ATTRIBUTES = (Private, ); }; };
     2822                93B38EC325821DB400198E63 /* WebSpeechRecognizerTask.h in Headers */ = {isa = PBXBuildFile; fileRef = 93B38EBA2582193B00198E63 /* WebSpeechRecognizerTask.h */; };
     2823                93B38EC425821DB700198E63 /* WebSpeechRecognizerTaskMock.h in Headers */ = {isa = PBXBuildFile; fileRef = 93B38EBB2582193D00198E63 /* WebSpeechRecognizerTaskMock.h */; };
    28222824                93B6A0E60B0BCA5C00F5027A /* ContextMenu.h in Headers */ = {isa = PBXBuildFile; fileRef = 93B6A0E50B0BCA5C00F5027A /* ContextMenu.h */; settings = {ATTRIBUTES = (Private, ); }; };
    28232825                93B70D6409EB0C7C009D8468 /* JSDOMBinding.h in Headers */ = {isa = PBXBuildFile; fileRef = 93B70D4809EB0C7C009D8468 /* JSDOMBinding.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    1154211544                93B2D8150F9920D2006AE6B2 /* SuddenTermination.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SuddenTermination.h; sourceTree = "<group>"; };
    1154311545                93B2D8170F9920EE006AE6B2 /* SuddenTermination.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = SuddenTermination.mm; sourceTree = "<group>"; };
     11546                93B38EB82582189900198E63 /* SpeechRecognizerCocoa.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = SpeechRecognizerCocoa.mm; sourceTree = "<group>"; };
     11547                93B38EB92582193A00198E63 /* WebSpeechRecognizerTaskMock.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WebSpeechRecognizerTaskMock.mm; sourceTree = "<group>"; };
     11548                93B38EBA2582193B00198E63 /* WebSpeechRecognizerTask.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebSpeechRecognizerTask.h; sourceTree = "<group>"; };
     11549                93B38EBB2582193D00198E63 /* WebSpeechRecognizerTaskMock.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebSpeechRecognizerTaskMock.h; sourceTree = "<group>"; };
     11550                93B38EBC2582193E00198E63 /* WebSpeechRecognizerTask.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WebSpeechRecognizerTask.mm; sourceTree = "<group>"; };
    1154411551                93B6A0E50B0BCA5C00F5027A /* ContextMenu.h */ = {isa = PBXFileReference; fileEncoding = 30; lastKnownFileType = sourcecode.c.h; path = ContextMenu.h; sourceTree = "<group>"; };
    1154511552                93B70D4809EB0C7C009D8468 /* JSDOMBinding.h */ = {isa = PBXFileReference; fileEncoding = 30; lastKnownFileType = sourcecode.c.h; path = JSDOMBinding.h; sourceTree = "<group>"; };
    … …  
    2319223199                        sourceTree = "<group>";
    2319323200                };
     23201                93B38EB62582183300198E63 /* cocoa */ = {
     23202                        isa = PBXGroup;
     23203                        children = (
     23204                                93B38EB82582189900198E63 /* SpeechRecognizerCocoa.mm */,
     23205                                93B38EBA2582193B00198E63 /* WebSpeechRecognizerTask.h */,
     23206                                93B38EBC2582193E00198E63 /* WebSpeechRecognizerTask.mm */,
     23207                                93B38EBB2582193D00198E63 /* WebSpeechRecognizerTaskMock.h */,
     23208                                93B38EB92582193A00198E63 /* WebSpeechRecognizerTaskMock.mm */,
     23209                        );
     23210                        path = cocoa;
     23211                        sourceTree = "<group>";
     23212                };
    2319423213                93C09A820B064F05005ABD4D /* mac */ = {
    2319523214                        isa = PBXGroup;
    … …  
    2572425743                        isa = PBXGroup;
    2572525744                        children = (
     25745                                93B38EB62582183300198E63 /* cocoa */,
    2572625746                                AA2A5ABA16A485D500975A25 /* DOMWindow+SpeechSynthesis.idl */,
    2572725747                                AA2A5AB816A485D500975A25 /* DOMWindowSpeechSynthesis.cpp */,
    … …  
    3530435324                                97AABD2714FA09D5007457AE /* WebSocketHandshake.h in Headers */,
    3530535325                                41E12E9F24FE74E20093FFB4 /* WebSocketIdentifier.h in Headers */,
     35326                                93B38EC325821DB400198E63 /* WebSpeechRecognizerTask.h in Headers */,
     35327                                93B38EC425821DB700198E63 /* WebSpeechRecognizerTaskMock.h in Headers */,
    3530635328                                1F8756B21E22C3350042C40D /* WebSQLiteDatabaseTrackerClient.h in Headers */,
    3530735329                                31DEA4561B39F4D900F77178 /* WebSystemBackdropLayer.h in Headers */,

  • trunk/Source/WebKit/ChangeLog

    r270768 r270772  
     12020-12-14  Sihui Liu  <sihui_liu@apple.com>
     2
     3        Implement recognizer for SpeechRecognition
     4        https://bugs.webkit.org/show_bug.cgi?id=219459
     5        <rdar://problem/71914465>
     6
     7        Reviewed by Youenn Fablet.
     8
     9        * UIProcess/SpeechRecognitionServer.cpp:
     10        (WebKit::SpeechRecognitionServer::SpeechRecognitionServer):
     11        (WebKit::SpeechRecognitionServer::requestPermissionForRequest):
     12        (WebKit::SpeechRecognitionServer::handleRequest):
     13        (WebKit::SpeechRecognitionServer::abort):
     14        (WebKit::SpeechRecognitionServer::invalidate):
     15        * UIProcess/SpeechRecognitionServer.h:
     16        * UIProcess/WebProcessProxy.cpp:
     17        (WebKit::WebProcessProxy::createSpeechRecognitionServer):
     18
    1192020-12-14  Chris Dumez  <cdumez@apple.com>
    220

  • trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.cpp

    r270574 r270772  
    3737namespace WebKit {
    3838
     39
     40SpeechRecognitionServer::SpeechRecognitionServer(Ref<IPC::Connection>&& connection, SpeechRecognitionServerIdentifier identifier, SpeechRecognitionPermissionChecker&& permissionChecker, SpeechRecognitionCheckIfmockSpeechRecognitionEnabled&& checkIfEnabled
    3941#if ENABLE(MEDIA_STREAM)
    40 SpeechRecognitionServer::SpeechRecognitionServer(Ref<IPC::Connection>&& connection, SpeechRecognitionServerIdentifier identifier, SpeechRecognitionPermissionChecker&& permissionChecker, RealtimeMediaSourceCreateFunction&& function)
     42    , RealtimeMediaSourceCreateFunction&& function
     43#endif
     44    )
    4145    : m_connection(WTFMove(connection))
    4246    , m_identifier(identifier)
    4347    , m_permissionChecker(WTFMove(permissionChecker))
     48    , m_checkIfmockSpeechRecognitionEnabled(WTFMove(checkIfEnabled))
     49#if ENABLE(MEDIA_STREAM)
    4450    , m_realtimeMediaSourceCreateFunction(WTFMove(function))
     51#endif
    4552{
    4653}
    47 #else
    48 SpeechRecognitionServer::SpeechRecognitionServer(Ref<IPC::Connection>&& connection, SpeechRecognitionServerIdentifier identifier, SpeechRecognitionPermissionChecker&& permissionChecker)
    49     : m_connection(WTFMove(connection))
    50     , m_identifier(identifier)
    51     , m_permissionChecker(WTFMove(permissionChecker))
    52 {
    53 }
    54 #endif
    5554
    5655void SpeechRecognitionServer::start(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier, String&& lang, bool continuous, bool interimResults, uint64_t maxAlternatives, WebCore::ClientOrigin&& origin)
    … …  
    8180        }
    8281
    83         handleRequest(identifier);
     82        handleRequest(*weakRequest);
    8483    });
    8584}
    8685
    87 void SpeechRecognitionServer::handleRequest(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier)
     86void SpeechRecognitionServer::handleRequest(WebCore::SpeechRecognitionRequest& request)
    8887{
    8988    if (!m_recognizer) {
    … …  
    9695                return;
    9796
     97            sendUpdate(update);
     98
    9899            auto type = update.type();
    99             if (type == WebCore::SpeechRecognitionUpdateType::Error || type == WebCore::SpeechRecognitionUpdateType::End)
    100                 m_requests.remove(clientIdentifier);
     100            if (type != WebCore::SpeechRecognitionUpdateType::Error && type != WebCore::SpeechRecognitionUpdateType::End)
     101                return;
    101102
    102             sendUpdate(update);
     103            if (m_isResetting)
     104                return;
     105            m_isResetting = true;
     106
     107            m_recognizer->reset();
     108            m_requests.remove(clientIdentifier);
     109            m_isResetting = false;
    103110        });
    104111    }
    105    
    106     m_recognizer->reset();
    107112
     113    if (auto currentClientIdentifier = m_recognizer->currentClientIdentifier()) {
     114        auto error = WebCore::SpeechRecognitionError { WebCore::SpeechRecognitionErrorType::Aborted, "Another request is started"_s };
     115        sendUpdate(*currentClientIdentifier, WebCore::SpeechRecognitionUpdateType::Error, error);
     116        m_recognizer->reset();
     117    }
     118
     119    auto clientIdentifier = request.clientIdentifier();
    108120#if ENABLE(MEDIA_STREAM)
    109121    auto sourceOrError = m_realtimeMediaSourceCreateFunction();
    … …  
    113125        return;
    114126    }
    115     m_recognizer->start(clientIdentifier, sourceOrError.source());
     127
     128    bool mockDeviceCapturesEnabled = m_checkIfmockSpeechRecognitionEnabled();
     129    m_recognizer->start(clientIdentifier, sourceOrError.source(), mockDeviceCapturesEnabled, request.lang(), request.continuous(), request.interimResults(), request.maxAlternatives());
    116130#else
    117131    m_requests.remove(clientIdentifier);
    118     sendUpdate(clientIdentifier, WebCore::SpeechRecognitionUpdateType::Error, WebCore::SpeechRecognitionError { WebCore::SpeechRecognitionErrorType::AudioCapture, "Audio capture is not implemented"});
     132    sendUpdate(clientIdentifier, WebCore::SpeechRecognitionUpdateType::Error, WebCore::SpeechRecognitionError { WebCore::SpeechRecognitionErrorType::AudioCapture, "Audio capture is not implemented"_s });
    119133#endif
    120134}
    … …  
    136150    MESSAGE_CHECK(clientIdentifier);
    137151    if (m_recognizer && m_recognizer->currentClientIdentifier() == clientIdentifier) {
    138         m_recognizer->stop(WebCore::SpeechRecognizer::ShouldGenerateFinalResult::No);
     152        m_recognizer->abort();
    139153        return;
    140154    }
    … …  
    149163    if (m_requests.remove(clientIdentifier)) {
    150164        if (m_recognizer && m_recognizer->currentClientIdentifier() == clientIdentifier)
    151             m_recognizer->stop();
     165            m_recognizer->abort();
    152166    }
    153167}

  • trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.h

    r270574 r270772  
    4848using SpeechRecognitionServerIdentifier = WebCore::PageIdentifier;
    4949using SpeechRecognitionPermissionChecker = Function<void(const WebCore::ClientOrigin&, CompletionHandler<void(SpeechRecognitionPermissionDecision)>&&)>;
     50using SpeechRecognitionCheckIfmockSpeechRecognitionEnabled = Function<bool()>;
    5051
    5152class SpeechRecognitionServer : public CanMakeWeakPtr<SpeechRecognitionServer>, public IPC::MessageReceiver, private IPC::MessageSender {
    … …  
    5455#if ENABLE(MEDIA_STREAM)
    5556    using RealtimeMediaSourceCreateFunction = Function<WebCore::CaptureSourceOrError()>;
    56     SpeechRecognitionServer(Ref<IPC::Connection>&&, SpeechRecognitionServerIdentifier, SpeechRecognitionPermissionChecker&&, RealtimeMediaSourceCreateFunction&&);
     57    SpeechRecognitionServer(Ref<IPC::Connection>&&, SpeechRecognitionServerIdentifier, SpeechRecognitionPermissionChecker&&, SpeechRecognitionCheckIfmockSpeechRecognitionEnabled&&, RealtimeMediaSourceCreateFunction&&);
    5758#else
    58     SpeechRecognitionServer(Ref<IPC::Connection>&&, SpeechRecognitionServerIdentifier, SpeechRecognitionPermissionChecker&&);
     59    SpeechRecognitionServer(Ref<IPC::Connection>&&, SpeechRecognitionServerIdentifier, SpeechRecognitionPermissionChecker&&, SpeechRecognitionCheckIfmockSpeechRecognitionEnabled&&);
    5960#endif
    6061
    … …  
    6667private:
    6768    void requestPermissionForRequest(WebCore::SpeechRecognitionRequest&);
    68     void handleRequest(WebCore::SpeechRecognitionConnectionClientIdentifier);
     69    void handleRequest(WebCore::SpeechRecognitionRequest&);
    6970    void sendUpdate(WebCore::SpeechRecognitionConnectionClientIdentifier, WebCore::SpeechRecognitionUpdateType, Optional<WebCore::SpeechRecognitionError> = WTF::nullopt, Optional<Vector<WebCore::SpeechRecognitionResultData>> = WTF::nullopt);
    7071    void sendUpdate(const WebCore::SpeechRecognitionUpdate&);
    … …  
    8283    SpeechRecognitionPermissionChecker m_permissionChecker;
    8384    std::unique_ptr<WebCore::SpeechRecognizer> m_recognizer;
     85    SpeechRecognitionCheckIfmockSpeechRecognitionEnabled m_checkIfmockSpeechRecognitionEnabled;
     86    bool m_isResetting { false };
    8487
    8588#if ENABLE(MEDIA_STREAM)

  • trunk/Source/WebKit/UIProcess/WebProcessProxy.cpp

    r270646 r270772  
    17231723    ASSERT(!m_speechRecognitionServerMap.contains(identifier));
    17241724    auto& speechRecognitionServer = m_speechRecognitionServerMap.add(identifier, nullptr).iterator->value;
    1725     speechRecognitionServer = makeUnique<SpeechRecognitionServer>(makeRef(*connection()), identifier, [weakPage = makeWeakPtr(targetPage)](auto& origin, auto&& completionHandler) mutable {
     1725    auto permissionChecker = [weakPage = makeWeakPtr(targetPage)](auto& origin, auto&& completionHandler) mutable {
    17261726        if (!weakPage) {
    17271727            completionHandler(SpeechRecognitionPermissionDecision::Deny);
    … …  
    17301730
    17311731        weakPage->requestSpeechRecognitionPermission(origin, WTFMove(completionHandler));
    1732     }
     1732    };
     1733    auto checkIfMockCaptureDevicesEnabled = [weakPage = makeWeakPtr(targetPage)]() {
     1734        return weakPage && weakPage->preferences().mockCaptureDevicesEnabled();
     1735    };
     1736
    17331737#if ENABLE(MEDIA_STREAM)
    1734     , [weakPage = makeWeakPtr(targetPage)]() {
     1738    auto createRealtimeMediaSource = [weakPage = makeWeakPtr(targetPage)]() {
    17351739        return weakPage ? weakPage->createRealtimeMediaSourceForSpeechRecognition() : CaptureSourceOrError { "Page is invalid" };
    1736     }
    1737 #endif
    1738     );
     1740    };
     1741    speechRecognitionServer = makeUnique<SpeechRecognitionServer>(makeRef(*connection()), identifier, WTFMove(permissionChecker), WTFMove(checkIfMockCaptureDevicesEnabled), WTFMove(createRealtimeMediaSource));
     1742#else
     1743    speechRecognitionServer = makeUnique<SpeechRecognitionServer>(makeRef(*connection()), identifier, WTFMove(permissionChecker), WTFMove(checkIfMockCaptureDevicesEnabled));
     1744#endif
     1745
    17391746    addMessageReceiver(Messages::SpeechRecognitionServer::messageReceiverName(), identifier, *speechRecognitionServer);
    17401747}
Note: See TracChangeset for help on using the changeset viewer.