⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 284829 in webkit


Ignore:
Timestamp:
Oct 25, 2021, 2:50:54 PM (5 years ago)
Author:
Alan Coon
Message:

Cherry-pick r283592. rdar://problem/84625519

[iOS] Transcode videos selected from UIImagePickerController
​https://bugs.webkit.org/show_bug.cgi?id=230639
rdar://79665678

Reviewed by Tim Horton.

Source/WebCore:

  • en.lproj/Localizable.strings:

Add a localizable string for the message displayed while transcoding
video.

Source/WebCore/PAL:

Add AVFoundation API needed to transcode video.

  • pal/cocoa/AVFoundationSoftLink.h:
  • pal/cocoa/AVFoundationSoftLink.mm:

Source/WebKit:

File inputs on iOS allow users to choose images/videos from the system
photo picker, using UIImagePickerController. In single selection mode,
UIImagePickerController transcodes the selected video to H.264. However,
in multiple selection mode, video is not transcoded and is left in its
original format.

Today, videos on most iOS devices are encoded with HEVC by default.
However, some sites, such as Twitter, only accept H.264 encoded video.
Thus, the current video upload behavior is problematic, as users may be
unable to upload video.

Unfortunately, the photo picking functionality of UIImagePickerController
is deprecated. The best solution would be to adopt PHPickerViewController,
the replacement API, which performs transcoding when retrieving selected
items (regardless of single/multiple selection). However,
PHPickerViewController currently lacks other functionality that WebKit
requires, preventing adoption.

Consequently, the short term solution is to transcode the videos in
WebKit, ensuring H.264 encoded video is always provided to sites. See
below for implementation details.

  • Platform/spi/ios/PhotosUISPI.h: Added.
  • UIProcess/ios/forms/WKFileUploadPanel.mm: (-[_WKFileUploadItem setFileURL:]):

Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.

(-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):

Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
and the display of progress UI. Transcoding is performed serially
(one video at a time), but occurs off the main thread.

(-[WKFileUploadMediaTranscoder start]):

Begin transcoding. Run a timer to update the progress UI, as
AVAssetExportSession does not provide progress updates on its own.

The progress UI is implemented using PUActivityProgressController, to
match system Photos UI.

(-[WKFileUploadMediaTranscoder _processItemAtIndex:]):

Transcode a single video, using AVAssetExportSession. If transcoding
fails for any reason, the original video is used as a fallback.
Transcoding can also be cancelled using the progress UI, in which case
no more videos are processed.

(-[WKFileUploadMediaTranscoder _finishedProcessing]):
(-[WKFileUploadMediaTranscoder _dismissProgress]):
(-[WKFileUploadMediaTranscoder _updateProgress:]):
(-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
(-[WKFileUploadPanel _chooseMediaItems:]):

Refactor the common aspects of single/multiple media selection into a
single method.

(-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
(-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
(-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):

Remove redundant platform conditional.

(-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):

If any videos were selected, transcode them prior to uploading.

  • WebKit.xcodeproj/project.pbxproj:

git-svn-id: ​https://svn.webkit.org/repository/webkit/trunk@283592 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Location:
branches/safari-612-branch/Source
Files:
1 added
8 edited

Legend:

Unmodified
Added
Removed

  • branches/safari-612-branch/Source/WebCore/ChangeLog

    r284828 r284829  
     12021-10-25  Null  <null@apple.com>
     2
     3        Cherry-pick r283592. rdar://problem/84625519
     4
     5    [iOS] Transcode videos selected from UIImagePickerController
     6    https://bugs.webkit.org/show_bug.cgi?id=230639
     7    rdar://79665678
     8   
     9    Reviewed by Tim Horton.
     10   
     11    Source/WebCore:
     12   
     13    * en.lproj/Localizable.strings:
     14   
     15    Add a localizable string for the message displayed while transcoding
     16    video.
     17   
     18    Source/WebCore/PAL:
     19   
     20    Add AVFoundation API needed to transcode video.
     21   
     22    * pal/cocoa/AVFoundationSoftLink.h:
     23    * pal/cocoa/AVFoundationSoftLink.mm:
     24   
     25    Source/WebKit:
     26   
     27    File inputs on iOS allow users to choose images/videos from the system
     28    photo picker, using UIImagePickerController. In single selection mode,
     29    UIImagePickerController transcodes the selected video to H.264. However,
     30    in multiple selection mode, video is not transcoded and is left in its
     31    original format.
     32   
     33    Today, videos on most iOS devices are encoded with HEVC by default.
     34    However, some sites, such as Twitter, only accept H.264 encoded video.
     35    Thus, the current video upload behavior is problematic, as users may be
     36    unable to upload video.
     37   
     38    Unfortunately, the photo picking functionality of UIImagePickerController
     39    is deprecated. The best solution would be to adopt PHPickerViewController,
     40    the replacement API, which performs transcoding when retrieving selected
     41    items (regardless of single/multiple selection). However,
     42    PHPickerViewController currently lacks other functionality that WebKit
     43    requires, preventing adoption.
     44   
     45    Consequently, the short term solution is to transcode the videos in
     46    WebKit, ensuring H.264 encoded video is always provided to sites. See
     47    below for implementation details.
     48   
     49    * Platform/spi/ios/PhotosUISPI.h: Added.
     50    * UIProcess/ios/forms/WKFileUploadPanel.mm:
     51    (-[_WKFileUploadItem setFileURL:]):
     52   
     53    Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.
     54   
     55    (-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):
     56   
     57    Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
     58    and the display of progress UI. Transcoding is performed serially
     59    (one video at a time), but occurs off the main thread.
     60   
     61    (-[WKFileUploadMediaTranscoder start]):
     62   
     63    Begin transcoding. Run a timer to update the progress UI, as
     64    AVAssetExportSession does not provide progress updates on its own.
     65   
     66    The progress UI is implemented using PUActivityProgressController, to
     67    match system Photos UI.
     68   
     69    (-[WKFileUploadMediaTranscoder _processItemAtIndex:]):
     70   
     71    Transcode a single video, using AVAssetExportSession. If transcoding
     72    fails for any reason, the original video is used as a fallback.
     73    Transcoding can also be cancelled using the progress UI, in which case
     74    no more videos are processed.
     75   
     76    (-[WKFileUploadMediaTranscoder _finishedProcessing]):
     77    (-[WKFileUploadMediaTranscoder _dismissProgress]):
     78    (-[WKFileUploadMediaTranscoder _updateProgress:]):
     79    (-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
     80    (-[WKFileUploadPanel _chooseMediaItems:]):
     81   
     82    Refactor the common aspects of single/multiple media selection into a
     83    single method.
     84   
     85    (-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
     86    (-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
     87    (-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
     88    (-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
     89    (-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):
     90   
     91    Remove redundant platform conditional.
     92   
     93    (-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):
     94   
     95    If any videos were selected, transcode them prior to uploading.
     96   
     97    * WebKit.xcodeproj/project.pbxproj:
     98   
     99   
     100    git-svn-id: https://svn.webkit.org/repository/webkit/trunk@283592 268f45cc-cd09-0410-ab3c-d52691b4dbfc
     101
     102    2021-10-05  Aditya Keerthi  <akeerthi@apple.com>
     103
     104            [iOS] Transcode videos selected from UIImagePickerController
     105            https://bugs.webkit.org/show_bug.cgi?id=230639
     106            rdar://79665678
     107
     108            Reviewed by Tim Horton.
     109
     110            * en.lproj/Localizable.strings:
     111
     112            Add a localizable string for the message displayed while transcoding
     113            video.
     114
    11152021-10-25  Null  <null@apple.com>
    2116

  • branches/safari-612-branch/Source/WebCore/PAL/ChangeLog

    r282985 r284829  
     12021-10-25  Null  <null@apple.com>
     2
     3        Cherry-pick r283592. rdar://problem/84625519
     4
     5    [iOS] Transcode videos selected from UIImagePickerController
     6    https://bugs.webkit.org/show_bug.cgi?id=230639
     7    rdar://79665678
     8   
     9    Reviewed by Tim Horton.
     10   
     11    Source/WebCore:
     12   
     13    * en.lproj/Localizable.strings:
     14   
     15    Add a localizable string for the message displayed while transcoding
     16    video.
     17   
     18    Source/WebCore/PAL:
     19   
     20    Add AVFoundation API needed to transcode video.
     21   
     22    * pal/cocoa/AVFoundationSoftLink.h:
     23    * pal/cocoa/AVFoundationSoftLink.mm:
     24   
     25    Source/WebKit:
     26   
     27    File inputs on iOS allow users to choose images/videos from the system
     28    photo picker, using UIImagePickerController. In single selection mode,
     29    UIImagePickerController transcodes the selected video to H.264. However,
     30    in multiple selection mode, video is not transcoded and is left in its
     31    original format.
     32   
     33    Today, videos on most iOS devices are encoded with HEVC by default.
     34    However, some sites, such as Twitter, only accept H.264 encoded video.
     35    Thus, the current video upload behavior is problematic, as users may be
     36    unable to upload video.
     37   
     38    Unfortunately, the photo picking functionality of UIImagePickerController
     39    is deprecated. The best solution would be to adopt PHPickerViewController,
     40    the replacement API, which performs transcoding when retrieving selected
     41    items (regardless of single/multiple selection). However,
     42    PHPickerViewController currently lacks other functionality that WebKit
     43    requires, preventing adoption.
     44   
     45    Consequently, the short term solution is to transcode the videos in
     46    WebKit, ensuring H.264 encoded video is always provided to sites. See
     47    below for implementation details.
     48   
     49    * Platform/spi/ios/PhotosUISPI.h: Added.
     50    * UIProcess/ios/forms/WKFileUploadPanel.mm:
     51    (-[_WKFileUploadItem setFileURL:]):
     52   
     53    Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.
     54   
     55    (-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):
     56   
     57    Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
     58    and the display of progress UI. Transcoding is performed serially
     59    (one video at a time), but occurs off the main thread.
     60   
     61    (-[WKFileUploadMediaTranscoder start]):
     62   
     63    Begin transcoding. Run a timer to update the progress UI, as
     64    AVAssetExportSession does not provide progress updates on its own.
     65   
     66    The progress UI is implemented using PUActivityProgressController, to
     67    match system Photos UI.
     68   
     69    (-[WKFileUploadMediaTranscoder _processItemAtIndex:]):
     70   
     71    Transcode a single video, using AVAssetExportSession. If transcoding
     72    fails for any reason, the original video is used as a fallback.
     73    Transcoding can also be cancelled using the progress UI, in which case
     74    no more videos are processed.
     75   
     76    (-[WKFileUploadMediaTranscoder _finishedProcessing]):
     77    (-[WKFileUploadMediaTranscoder _dismissProgress]):
     78    (-[WKFileUploadMediaTranscoder _updateProgress:]):
     79    (-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
     80    (-[WKFileUploadPanel _chooseMediaItems:]):
     81   
     82    Refactor the common aspects of single/multiple media selection into a
     83    single method.
     84   
     85    (-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
     86    (-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
     87    (-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
     88    (-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
     89    (-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):
     90   
     91    Remove redundant platform conditional.
     92   
     93    (-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):
     94   
     95    If any videos were selected, transcode them prior to uploading.
     96   
     97    * WebKit.xcodeproj/project.pbxproj:
     98   
     99   
     100    git-svn-id: https://svn.webkit.org/repository/webkit/trunk@283592 268f45cc-cd09-0410-ab3c-d52691b4dbfc
     101
     102    2021-10-05  Aditya Keerthi  <akeerthi@apple.com>
     103
     104            [iOS] Transcode videos selected from UIImagePickerController
     105            https://bugs.webkit.org/show_bug.cgi?id=230639
     106            rdar://79665678
     107
     108            Reviewed by Tim Horton.
     109
     110            Add AVFoundation API needed to transcode video.
     111
     112            * pal/cocoa/AVFoundationSoftLink.h:
     113            * pal/cocoa/AVFoundationSoftLink.mm:
     114
    11152021-09-23  Alan Coon  <alancoon@apple.com>
    2116

  • branches/safari-612-branch/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.h

    r280416 r284829  
    3939SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCache)
    4040SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCollection)
     41SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetExportSession)
    4142SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetImageGenerator)
    4243SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetReader)
    … …  
    186187SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeMPEG4, NSString *)
    187188#define AVFileTypeMPEG4 PAL::get_AVFoundation_AVFileTypeMPEG4()
     189SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *)
     190#define AVFileTypeQuickTimeMovie PAL::get_AVFoundation_AVFileTypeQuickTimeMovie()
    188191SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVVideoCodecKey, NSString *)
    189192#define AVVideoCodecKey PAL::get_AVFoundation_AVVideoCodecKey()
    … …  
    332335#endif // PLATFORM(COCOA)
    333336
     337SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *)
     338#define AVAssetExportPresetHighestQuality PAL::get_AVFoundation_AVAssetExportPresetHighestQuality()
     339
    334340#endif // USE(AVFOUNDATION)

  • branches/safari-612-branch/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.mm

    r280416 r284829  
    6464
    6565SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetCache, PAL_EXPORT)
     66SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportSession, PAL_EXPORT)
    6667SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetImageGenerator, PAL_EXPORT)
    6768SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetReader, PAL_EXPORT)
    … …  
    134135SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVCaptureDeviceWasDisconnectedNotification, NSString *, PAL_EXPORT)
    135136SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeMPEG4, NSString *, PAL_EXPORT)
     137SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *, PAL_EXPORT)
    136138SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResize, NSString *, PAL_EXPORT)
    137139SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResizeAspect, NSString *, PAL_EXPORT)
    … …  
    252254#endif
    253255
     256SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *, PAL_EXPORT)
     257
    254258#endif // USE(AVFOUNDATION)

  • branches/safari-612-branch/Source/WebCore/en.lproj/Localizable.strings

    r279912 r284829  
    755755"PostScript" = "PostScript";
    756756
     757/* Title for file upload progress view */
     758"Preparing (file upload)" = "Preparing…";
     759
    757760/* Title for Quick Look action button */
    758761"Quick Look" = "Quick Look";

  • branches/safari-612-branch/Source/WebKit/ChangeLog

    r284824 r284829  
     12021-10-25  Null  <null@apple.com>
     2
     3        Cherry-pick r283592. rdar://problem/84625519
     4
     5    [iOS] Transcode videos selected from UIImagePickerController
     6    https://bugs.webkit.org/show_bug.cgi?id=230639
     7    rdar://79665678
     8   
     9    Reviewed by Tim Horton.
     10   
     11    Source/WebCore:
     12   
     13    * en.lproj/Localizable.strings:
     14   
     15    Add a localizable string for the message displayed while transcoding
     16    video.
     17   
     18    Source/WebCore/PAL:
     19   
     20    Add AVFoundation API needed to transcode video.
     21   
     22    * pal/cocoa/AVFoundationSoftLink.h:
     23    * pal/cocoa/AVFoundationSoftLink.mm:
     24   
     25    Source/WebKit:
     26   
     27    File inputs on iOS allow users to choose images/videos from the system
     28    photo picker, using UIImagePickerController. In single selection mode,
     29    UIImagePickerController transcodes the selected video to H.264. However,
     30    in multiple selection mode, video is not transcoded and is left in its
     31    original format.
     32   
     33    Today, videos on most iOS devices are encoded with HEVC by default.
     34    However, some sites, such as Twitter, only accept H.264 encoded video.
     35    Thus, the current video upload behavior is problematic, as users may be
     36    unable to upload video.
     37   
     38    Unfortunately, the photo picking functionality of UIImagePickerController
     39    is deprecated. The best solution would be to adopt PHPickerViewController,
     40    the replacement API, which performs transcoding when retrieving selected
     41    items (regardless of single/multiple selection). However,
     42    PHPickerViewController currently lacks other functionality that WebKit
     43    requires, preventing adoption.
     44   
     45    Consequently, the short term solution is to transcode the videos in
     46    WebKit, ensuring H.264 encoded video is always provided to sites. See
     47    below for implementation details.
     48   
     49    * Platform/spi/ios/PhotosUISPI.h: Added.
     50    * UIProcess/ios/forms/WKFileUploadPanel.mm:
     51    (-[_WKFileUploadItem setFileURL:]):
     52   
     53    Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.
     54   
     55    (-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):
     56   
     57    Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
     58    and the display of progress UI. Transcoding is performed serially
     59    (one video at a time), but occurs off the main thread.
     60   
     61    (-[WKFileUploadMediaTranscoder start]):
     62   
     63    Begin transcoding. Run a timer to update the progress UI, as
     64    AVAssetExportSession does not provide progress updates on its own.
     65   
     66    The progress UI is implemented using PUActivityProgressController, to
     67    match system Photos UI.
     68   
     69    (-[WKFileUploadMediaTranscoder _processItemAtIndex:]):
     70   
     71    Transcode a single video, using AVAssetExportSession. If transcoding
     72    fails for any reason, the original video is used as a fallback.
     73    Transcoding can also be cancelled using the progress UI, in which case
     74    no more videos are processed.
     75   
     76    (-[WKFileUploadMediaTranscoder _finishedProcessing]):
     77    (-[WKFileUploadMediaTranscoder _dismissProgress]):
     78    (-[WKFileUploadMediaTranscoder _updateProgress:]):
     79    (-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
     80    (-[WKFileUploadPanel _chooseMediaItems:]):
     81   
     82    Refactor the common aspects of single/multiple media selection into a
     83    single method.
     84   
     85    (-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
     86    (-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
     87    (-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
     88    (-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
     89    (-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):
     90   
     91    Remove redundant platform conditional.
     92   
     93    (-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):
     94   
     95    If any videos were selected, transcode them prior to uploading.
     96   
     97    * WebKit.xcodeproj/project.pbxproj:
     98   
     99   
     100    git-svn-id: https://svn.webkit.org/repository/webkit/trunk@283592 268f45cc-cd09-0410-ab3c-d52691b4dbfc
     101
     102    2021-10-05  Aditya Keerthi  <akeerthi@apple.com>
     103
     104            [iOS] Transcode videos selected from UIImagePickerController
     105            https://bugs.webkit.org/show_bug.cgi?id=230639
     106            rdar://79665678
     107
     108            Reviewed by Tim Horton.
     109
     110            File inputs on iOS allow users to choose images/videos from the system
     111            photo picker, using UIImagePickerController. In single selection mode,
     112            UIImagePickerController transcodes the selected video to H.264. However,
     113            in multiple selection mode, video is not transcoded and is left in its
     114            original format.
     115
     116            Today, videos on most iOS devices are encoded with HEVC by default.
     117            However, some sites, such as Twitter, only accept H.264 encoded video.
     118            Thus, the current video upload behavior is problematic, as users may be
     119            unable to upload video.
     120
     121            Unfortunately, the photo picking functionality of UIImagePickerController
     122            is deprecated. The best solution would be to adopt PHPickerViewController,
     123            the replacement API, which performs transcoding when retrieving selected
     124            items (regardless of single/multiple selection). However,
     125            PHPickerViewController currently lacks other functionality that WebKit
     126            requires, preventing adoption.
     127
     128            Consequently, the short term solution is to transcode the videos in
     129            WebKit, ensuring H.264 encoded video is always provided to sites. See
     130            below for implementation details.
     131
     132            * Platform/spi/ios/PhotosUISPI.h: Added.
     133            * UIProcess/ios/forms/WKFileUploadPanel.mm:
     134            (-[_WKFileUploadItem setFileURL:]):
     135
     136            Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.
     137
     138            (-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):
     139
     140            Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
     141            and the display of progress UI. Transcoding is performed serially
     142            (one video at a time), but occurs off the main thread.
     143
     144            (-[WKFileUploadMediaTranscoder start]):
     145
     146            Begin transcoding. Run a timer to update the progress UI, as
     147            AVAssetExportSession does not provide progress updates on its own.
     148
     149            The progress UI is implemented using PUActivityProgressController, to
     150            match system Photos UI.
     151
     152            (-[WKFileUploadMediaTranscoder _processItemAtIndex:]):
     153
     154            Transcode a single video, using AVAssetExportSession. If transcoding
     155            fails for any reason, the original video is used as a fallback.
     156            Transcoding can also be cancelled using the progress UI, in which case
     157            no more videos are processed.
     158
     159            (-[WKFileUploadMediaTranscoder _finishedProcessing]):
     160            (-[WKFileUploadMediaTranscoder _dismissProgress]):
     161            (-[WKFileUploadMediaTranscoder _updateProgress:]):
     162            (-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
     163            (-[WKFileUploadPanel _chooseMediaItems:]):
     164
     165            Refactor the common aspects of single/multiple media selection into a
     166            single method.
     167
     168            (-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
     169            (-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
     170            (-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
     171            (-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
     172            (-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):
     173
     174            Remove redundant platform conditional.
     175
     176            (-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):
     177
     178            If any videos were selected, transcode them prior to uploading.
     179
     180            * WebKit.xcodeproj/project.pbxproj:
     181
    11822021-10-25  Null  <null@apple.com>
    2183

  • branches/safari-612-branch/Source/WebKit/UIProcess/ios/forms/WKFileUploadPanel.mm

    r282994 r284829  
    3333#import "APIOpenPanelParameters.h"
    3434#import "APIString.h"
     35#import "PhotosUISPI.h"
    3536#import "UIKitSPI.h"
    3637#import "UserInterfaceIdiom.h"
    … …  
    4647#import <WebCore/LocalizedStrings.h>
    4748#import <WebCore/MIMETypeRegistry.h>
     49#import <wtf/MainThread.h>
    4850#import <wtf/OptionSet.h>
    4951#import <wtf/RetainPtr.h>
    … …  
    5153#import <wtf/text/StringView.h>
    5254
     55#import <pal/cocoa/AVFoundationSoftLink.h>
     56
     57SOFT_LINK_FRAMEWORK(PhotosUI)
     58SOFT_LINK_CLASS(PhotosUI, PUActivityProgressController)
     59
    5360using namespace WebKit;
    5461
    … …  
    8087- (instancetype)initWithFileURL:(NSURL *)fileURL;
    8188@property (nonatomic, readonly, getter=isVideo) BOOL video;
    82 @property (nonatomic, readonly) NSURL *fileURL;
    8389@property (nonatomic, readonly) RetainPtr<UIImage> displayImage;
    8490@end
    … …  
    110116}
    111117
     118- (void)setFileURL:(NSURL *)fileURL
     119{
     120    _fileURL = fileURL;
     121}
     122
    112123- (RetainPtr<UIImage>)displayImage
    113124{
    … …  
    154165@end
    155166
     167#pragma mark - WKFileUploadMediaTranscoder
     168
     169@interface WKFileUploadMediaTranscoder : NSObject
     170
     171- (instancetype)initWithItems:(NSArray *)items videoCount:(NSUInteger)videoCount completionHandler:(WTF::Function<void(NSArray<_WKFileUploadItem *> *)>&&)completionHandler;
     172
     173- (void)start;
     174
     175@end
     176
     177@implementation WKFileUploadMediaTranscoder {
     178    RetainPtr<NSTimer> _progressTimer;
     179    RetainPtr<PUActivityProgressController> _progressController;
     180    RetainPtr<AVAssetExportSession> _exportSession;
     181    RetainPtr<NSArray<_WKFileUploadItem *>> _items;
     182    RetainPtr<NSString> _temporaryDirectoryPath;
     183
     184    // Only called if the transcoding is not cancelled.
     185    WTF::Function<void(NSArray<_WKFileUploadItem *> *)> _completionHandler;
     186
     187    NSUInteger _videoCount;
     188    NSUInteger _processedVideoCount;
     189}
     190
     191- (instancetype)initWithItems:(NSArray<_WKFileUploadItem *> *)items videoCount:(NSUInteger)videoCount completionHandler:(WTF::Function<void(NSArray<_WKFileUploadItem *> *)>&&)completionHandler
     192{
     193    if (!(self = [super init]))
     194        return nil;
     195
     196    _items = items;
     197    _processedVideoCount = 0;
     198    _videoCount = videoCount;
     199
     200    _completionHandler = WTFMove(completionHandler);
     201
     202    return self;
     203}
     204
     205- (void)start
     206{
     207    _progressController = adoptNS([allocPUActivityProgressControllerInstance() init]);
     208    [_progressController setTitle:WEB_UI_STRING_KEY("Preparing…", "Preparing (file upload)", "Title for file upload progress view")];
     209    [_progressController showAnimated:YES allowDelay:YES];
     210
     211    [_progressController setCancellationHandler:makeBlockPtr([weakSelf = WeakObjCPtr<WKFileUploadMediaTranscoder>(self)] {
     212        auto strongSelf = weakSelf.get();
     213        if (!strongSelf)
     214            return;
     215
     216        [strongSelf->_exportSession cancelExport];
     217        [strongSelf _dismissProgress];
     218    }).get()];
     219
     220    _progressTimer = [NSTimer scheduledTimerWithTimeInterval:0.1f target:self selector:@selector(_updateProgress:) userInfo:nil repeats:YES];
     221
     222    [self _processItemAtIndex:0];
     223}
     224
     225- (void)_processItemAtIndex:(NSUInteger)index
     226{
     227    if ([_progressController isCancelled])
     228        return;
     229
     230    if (index >= [_items count]) {
     231        [self _finishedProcessing];
     232        return;
     233    }
     234
     235    _WKFileUploadItem *item = [_items objectAtIndex:index];
     236
     237    while (!item.isVideo) {
     238        index++;
     239
     240        if (index == [_items count]) {
     241            [self _finishedProcessing];
     242            return;
     243        }
     244
     245        item = [_items objectAtIndex:index];
     246    }
     247
     248    NSString *temporaryDirectory = [self _temporaryDirectoryCreateIfNecessary];
     249    if (!temporaryDirectory) {
     250        LOG_ERROR("WKFileUploadMediaTranscoder: Failed to make temporary directory");
     251        [self _finishedProcessing];
     252        return;
     253    }
     254
     255    NSString *fileName = [item.fileURL.lastPathComponent.stringByDeletingPathExtension stringByAppendingPathExtension:UTTypeQuickTimeMovie.preferredFilenameExtension.uppercaseString];
     256    NSString *filePath = [temporaryDirectory stringByAppendingPathComponent:fileName];
     257    NSURL *outputURL = [NSURL fileURLWithPath:filePath isDirectory:NO];
     258
     259    RetainPtr<AVURLAsset> asset = adoptNS([PAL::allocAVURLAssetInstance() initWithURL:item.fileURL options:nil]);
     260    _exportSession = adoptNS([PAL::allocAVAssetExportSessionInstance() initWithAsset:asset.get() presetName:AVAssetExportPresetHighestQuality]);
     261    [_exportSession setOutputURL:outputURL];
     262    [_exportSession setOutputFileType:AVFileTypeQuickTimeMovie];
     263
     264    [_exportSession exportAsynchronouslyWithCompletionHandler:makeBlockPtr([weakSelf = WeakObjCPtr<WKFileUploadMediaTranscoder>(self), index] () mutable {
     265        ensureOnMainRunLoop([weakSelf = WTFMove(weakSelf), index] {
     266            auto strongSelf = weakSelf.get();
     267            if (!strongSelf)
     268                return;
     269
     270            AVAssetExportSessionStatus status = [strongSelf->_exportSession status];
     271
     272            if (status == AVAssetExportSessionStatusCancelled)
     273                return;
     274
     275            if (status == AVAssetExportSessionStatusCompleted) {
     276                _WKFileUploadItem *item = [strongSelf->_items objectAtIndex:index];
     277                [item setFileURL:[strongSelf->_exportSession outputURL]];
     278            }
     279
     280            strongSelf->_exportSession = nil;
     281
     282            strongSelf->_processedVideoCount++;
     283            [strongSelf _processItemAtIndex:index + 1];
     284        });
     285    }).get()];
     286}
     287
     288- (void)_finishedProcessing
     289{
     290    [self _dismissProgress];
     291
     292    if (auto completionHandler = std::exchange(_completionHandler, nullptr))
     293        completionHandler(_items.get());
     294}
     295
     296- (void)_dismissProgress
     297{
     298    [_progressTimer invalidate];
     299    [_progressController hideAnimated:NO allowDelay:NO];
     300}
     301
     302- (void)_updateProgress:(NSTimer *)timer
     303{
     304    auto currentSessionProgress = [_exportSession progress];
     305    [_progressController setFractionCompleted:(currentSessionProgress + _processedVideoCount) / _videoCount];
     306}
     307
     308- (NSString *)_temporaryDirectoryCreateIfNecessary
     309{
     310    if (_temporaryDirectoryPath) {
     311        BOOL isDirectory = NO;
     312        BOOL exists = [[NSFileManager defaultManager] fileExistsAtPath:_temporaryDirectoryPath.get() isDirectory:&isDirectory];
     313
     314        if (exists && isDirectory)
     315            return _temporaryDirectoryPath.get();
     316    }
     317
     318    _temporaryDirectoryPath = FileSystem::createTemporaryDirectory(@"WKVideoUpload");
     319    return _temporaryDirectoryPath.get();
     320}
     321
     322@end
    156323
    157324#pragma mark - WKFileUploadPanel
    158 
    159325
    160326@interface WKFileUploadPanel () <UIPopoverControllerDelegate, UINavigationControllerDelegate, UIImagePickerControllerDelegate, UIDocumentPickerDelegate, UIAdaptivePresentationControllerDelegate
    … …  
    173339    BOOL _allowMultipleFiles;
    174340    BOOL _usingCamera;
     341    RetainPtr<WKFileUploadMediaTranscoder> _mediaTranscoder;
    175342    RetainPtr<UIImagePickerController> _imagePicker;
    176343    RetainPtr<UIViewController> _presentationViewController; // iPhone always. iPad for Fullscreen Camera.
    … …  
    219386   
    220387    [self _dispatchDidDismiss];
     388}
     389
     390- (void)_chooseMediaItems:(NSArray<_WKFileUploadItem *> *)mediaItems
     391{
     392    RetainPtr<UIImage> iconImage = nil;
     393    NSMutableArray *fileURLs = [NSMutableArray array];
     394    NSUInteger videoCount = 0;
     395
     396    for (_WKFileUploadItem *item in mediaItems) {
     397        [fileURLs addObject:item.fileURL];
     398
     399        if (!iconImage)
     400            iconImage = item.displayImage;
     401
     402        if (item.isVideo)
     403            videoCount++;
     404    }
     405
     406    NSUInteger imageCount = mediaItems.count - videoCount;
     407
     408    NSString *displayString = (imageCount || videoCount) ? [NSString localizedStringWithFormat:WEB_UI_NSSTRING(@"%lu photo(s) and %lu video(s)", "label next to file upload control; parameters are the number of photos and the number of videos"), (unsigned long)imageCount, (unsigned long)videoCount] : nil;
     409
     410    [self _dismissDisplayAnimated:YES];
     411    [self _chooseFiles:fileURLs displayString:displayString iconImage:iconImage.get()];
    221412}
    222413
    … …  
    728919        return;
    729920
    730     [self _dismissDisplayAnimated:YES];
    731 
    732921    [self _processMediaInfoDictionaries:@[info]
    733         successBlock:^(NSArray *processedResults, NSString *displayString) {
    734             ASSERT([processedResults count] == 1);
    735             _WKFileUploadItem *result = [processedResults objectAtIndex:0];
    736             RunLoop::main().dispatch([self, strongSelf = retainPtr(self), result = retainPtr(result), displayString = retainPtr(displayString)] {
    737                 [self _chooseFiles:@[result.get().fileURL] displayString:displayString.get() iconImage:result.get().displayImage.get()];
     922        successBlock:^(NSArray<_WKFileUploadItem *> *items) {
     923            ASSERT([items count] == 1);
     924            ensureOnMainRunLoop([self, strongSelf = retainPtr(self), items = retainPtr(items)] {
     925                [self _chooseMediaItems:items.get()];
    738926            });
    739927        }
    740928        failureBlock:^{
    741             RunLoop::main().dispatch([self, strongSelf = retainPtr(self)] {
     929            ensureOnMainRunLoop([self, strongSelf = retainPtr(self)] {
     930                [self _dismissDisplayAnimated:YES];
    742931                [self _cancel];
    743932            });
    … …  
    748937- (void)imagePickerController:(UIImagePickerController *)imagePicker didFinishPickingMultipleMediaWithInfo:(NSArray *)infos
    749938{
    750     [self _dismissDisplayAnimated:YES];
    751 
    752939    [self _processMediaInfoDictionaries:infos
    753         successBlock:^(NSArray *processedResults, NSString *displayString) {
    754             RetainPtr<UIImage> iconImage = nil;
    755             NSMutableArray *fileURLs = [NSMutableArray array];
    756             for (_WKFileUploadItem *result in processedResults) {
    757                 NSURL *fileURL = result.fileURL;
    758                 if (!fileURL)
    759                     continue;
    760                 [fileURLs addObject:result.fileURL];
    761                 if (!iconImage)
    762                     iconImage = result.displayImage;
    763             }
    764 
    765             RunLoop::main().dispatch([self, strongSelf = retainPtr(self), fileURLs = retainPtr(fileURLs), displayString = retainPtr(displayString), iconImage] {
    766                 [self _chooseFiles:fileURLs.get() displayString:displayString.get() iconImage:iconImage.get()];
    767             });
     940        successBlock:^(NSArray<_WKFileUploadItem *> *items) {
     941            [self _uploadMediaItemsTranscodingVideo:items];
    768942        }
    769943        failureBlock:^{
    770             RunLoop::main().dispatch([self, strongSelf = retainPtr(self)] {
     944            ensureOnMainRunLoop([self, strongSelf = retainPtr(self)] {
     945                [self _dismissDisplayAnimated:YES];
    771946                [self _cancel];
    772947            });
    … …  
    783958#pragma mark - Process UIImagePicker results
    784959
    785 - (void)_processMediaInfoDictionaries:(NSArray *)infos successBlock:(void (^)(NSArray *processedResults, NSString *displayString))successBlock failureBlock:(void (^)(void))failureBlock
    786 {
    787     [self _processMediaInfoDictionaries:infos atIndex:0 processedResults:[NSMutableArray array] processedImageCount:0 processedVideoCount:0 successBlock:successBlock failureBlock:failureBlock];
    788 }
    789 
    790 - (void)_processMediaInfoDictionaries:(NSArray *)infos atIndex:(NSUInteger)index processedResults:(NSMutableArray *)processedResults processedImageCount:(NSUInteger)processedImageCount processedVideoCount:(NSUInteger)processedVideoCount successBlock:(void (^)(NSArray *processedResults, NSString *displayString))successBlock failureBlock:(void (^)(void))failureBlock
     960- (void)_processMediaInfoDictionaries:(NSArray *)infos successBlock:(void (^)(NSArray<_WKFileUploadItem *> *processedResults))successBlock failureBlock:(void (^)(void))failureBlock
     961{
     962    [self _processMediaInfoDictionaries:infos atIndex:0 processedResults:[NSMutableArray array] successBlock:successBlock failureBlock:failureBlock];
     963}
     964
     965- (void)_processMediaInfoDictionaries:(NSArray *)infos atIndex:(NSUInteger)index processedResults:(NSMutableArray<_WKFileUploadItem *> *)processedResults successBlock:(void (^)(NSArray<_WKFileUploadItem *> *processedResults))successBlock failureBlock:(void (^)(void))failureBlock
    791966{
    792967    NSUInteger count = [infos count];
    793968    if (index == count) {
    794         NSString *displayString = (processedImageCount || processedVideoCount) ? [NSString localizedStringWithFormat:WEB_UI_NSSTRING(@"%lu photo(s) and %lu video(s)", "label next to file upload control; parameters are the number of photos and the number of videos"), (unsigned long)processedImageCount, (unsigned long)processedVideoCount] : nil;
    795         successBlock(processedResults, displayString);
     969        successBlock(processedResults);
    796970        return;
    797971    }
    … …  
    802976
    803977    auto uploadItemSuccessBlock = ^(_WKFileUploadItem *uploadItem) {
    804         NSUInteger newProcessedVideoCount = processedVideoCount + (uploadItem.isVideo ? 1 : 0);
    805         NSUInteger newProcessedImageCount = processedImageCount + (uploadItem.isVideo ? 0 : 1);
    806978        [processedResults addObject:uploadItem];
    807         [self _processMediaInfoDictionaries:infos atIndex:index processedResults:processedResults processedImageCount:newProcessedImageCount processedVideoCount:newProcessedVideoCount successBlock:successBlock failureBlock:failureBlock];
     979        [self _processMediaInfoDictionaries:infos atIndex:index processedResults:processedResults successBlock:successBlock failureBlock:failureBlock];
    808980    };
    809981
    … …  
    8891061    }
    8901062
    891 #if PLATFORM(IOS_FAMILY)
    8921063    if (NSURL *imageURL = info[UIImagePickerControllerImageURL]) {
    8931064        if (!imageURL.isFileURL) {
    … …  
    9011072        return;
    9021073    }
    903 #endif
    9041074
    9051075    UIImage *originalImage = [info objectForKey:UIImagePickerControllerOriginalImage];
    … …  
    9151085}
    9161086
     1087- (void)_uploadMediaItemsTranscodingVideo:(NSArray<_WKFileUploadItem *> *)items
     1088{
     1089    auto videoCount = [[items indexesOfObjectsPassingTest:^(_WKFileUploadItem *item, NSUInteger, BOOL*) {
     1090        return item.isVideo;
     1091    }] count];
     1092
     1093    ensureOnMainRunLoop([self, strongSelf = retainPtr(self), items = retainPtr(items), videoCount] {
     1094        if (!videoCount) {
     1095            [self _chooseMediaItems:items.get()];
     1096            return;
     1097        }
     1098
     1099        _mediaTranscoder = adoptNS([[WKFileUploadMediaTranscoder alloc] initWithItems:items.get() videoCount:videoCount completionHandler:[weakSelf = WeakObjCPtr<WKFileUploadPanel>(self)] (NSArray<_WKFileUploadItem *> *items) {
     1100            auto strongSelf = weakSelf.get();
     1101            if (!strongSelf)
     1102                return;
     1103
     1104            [strongSelf _chooseMediaItems:items];
     1105        }]);
     1106
     1107        [_mediaTranscoder start];
     1108    });
     1109}
     1110
    9171111- (BOOL)platformSupportsPickerViewController
    9181112{

  • branches/safari-612-branch/Source/WebKit/WebKit.xcodeproj/project.pbxproj

    r284424 r284829  
    20322032                E5BEF6822130C48000F31111 /* WebDataListSuggestionsDropdownIOS.h in Headers */ = {isa = PBXBuildFile; fileRef = E5BEF6802130C47F00F31111 /* WebDataListSuggestionsDropdownIOS.h */; };
    20332033                E5CB07DC20E1678F0022C183 /* WKFormColorControl.h in Headers */ = {isa = PBXBuildFile; fileRef = E5CB07DA20E1678F0022C183 /* WKFormColorControl.h */; };
     2034                E5DEFA6826F8F42600AB68DB /* PhotosUISPI.h in Headers */ = {isa = PBXBuildFile; fileRef = E5DEFA6726F8F42600AB68DB /* PhotosUISPI.h */; };
    20342035                ED82A7F2128C6FAF004477B3 /* WKBundlePageOverlay.h in Headers */ = {isa = PBXBuildFile; fileRef = 1A22F0FF1289FCD90085E74F /* WKBundlePageOverlay.h */; settings = {ATTRIBUTES = (Private, ); }; };
    20352036                F4094CBD2553053D003D73E3 /* DisplayListReaderHandle.h in Headers */ = {isa = PBXBuildFile; fileRef = F4094CBB255304AF003D73E3 /* DisplayListReaderHandle.h */; };
    … …  
    60436044                E5CB07DA20E1678F0022C183 /* WKFormColorControl.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = WKFormColorControl.h; path = ios/forms/WKFormColorControl.h; sourceTree = "<group>"; };
    60446045                E5CB07DB20E1678F0022C183 /* WKFormColorControl.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; name = WKFormColorControl.mm; path = ios/forms/WKFormColorControl.mm; sourceTree = "<group>"; };
     6046                E5DEFA6726F8F42600AB68DB /* PhotosUISPI.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = PhotosUISPI.h; sourceTree = "<group>"; };
    60456047                ECA680D31E6904B500731D20 /* ExtraPrivateSymbolsForTAPI.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ExtraPrivateSymbolsForTAPI.h; sourceTree = "<group>"; };
    60466048                ECBFC1DB1E6A4D66000300C7 /* ExtraPublicSymbolsForTAPI.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ExtraPublicSymbolsForTAPI.h; sourceTree = "<group>"; };
    … …  
    1170511707                                A13B3DA1207F39DE0090C58D /* MobileWiFiSPI.h */,
    1170611708                                3178AF9720E2A7F80074DE94 /* PDFKitSPI.h */,
     11709                                E5DEFA6726F8F42600AB68DB /* PhotosUISPI.h */,
    1170711710                                2D279E1826955768004B3EEB /* PrototypeToolsSPI.h */,
    1170811711                                46F38E8B2416E66D0059375A /* RunningBoardServicesSPI.h */,
    … …  
    1237012373                                832ED18C1E2FE157006BA64A /* PerActivityStateCPUUsageSampler.h in Headers */,
    1237112374                                7AFBD36F21E546F8005DBACB /* PersistencyUtils.h in Headers */,
     12375                            E5DEFA6826F8F42600AB68DB /* PhotosUISPI.h in Headers */,
    1237212376                                5CE85B201C88E64B0070BFCE /* PingLoad.h in Headers */,
    1237312377                                0F5E200418E77051003EC3E5 /* PlatformCAAnimationRemote.h in Headers */,
Note: See TracChangeset for help on using the changeset viewer.