Changeset 246759 in webkit


Ignore:
Timestamp:
Jun 24, 2019 12:57:46 PM (5 years ago)
Author:
jer.noble@apple.com
Message:

iOS 12.2 Drawing portrait video to canvas is sideways
https://bugs.webkit.org/show_bug.cgi?id=196772
<rdar://problem/49781802>

Reviewed by Eric Carlson.

Source/WebCore:

Test: media/video-orientation-canvas.html

Move rotation code into its own ImageRotationSessionVT class for re-use across
all existing classes with rotation operations. Should slightly increase performance
for painting rotated media files, as the rotation only occurs once per frame, rather
than once per drawing operation.

  • WebCore.xcodeproj/project.pbxproj:
  • platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h:

(WebCore::ImageDecoderAVFObjC::RotationProperties::isIdentity const): Deleted.

  • platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm:

(WebCore::ImageDecoderAVFObjC::readTrackMetadata):
(WebCore::ImageDecoderAVFObjC::storeSampleBuffer):
(WebCore::ImageDecoderAVFObjC::setTrack):
(WebCore::transformToRotationProperties): Deleted.

  • platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:

(WebCore::MediaPlayerPrivateAVFoundationObjC::tracksChanged):
(WebCore::MediaPlayerPrivateAVFoundationObjC::updateLastPixelBuffer):
(WebCore::MediaPlayerPrivateAVFoundationObjC::paintWithVideoOutput):

  • platform/graphics/cv/ImageRotationSessionVT.h: Added.

(WebCore::ImageRotationSessionVT::RotationProperties::isIdentity const):
(WebCore::ImageRotationSessionVT::rotationProperties const):
(WebCore::ImageRotationSessionVT::rotatedSize):

  • platform/graphics/cv/ImageRotationSessionVT.mm: Added.

(WebCore::transformToRotationProperties):
(WebCore::ImageRotationSessionVT::ImageRotationSessionVT):
(WebCore::ImageRotationSessionVT::rotate):

  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm:

(WebCore::rotationToAngle):
(WebCore::RealtimeOutgoingVideoSourceCocoa::rotatePixelBuffer):
(WebCore::computeRotatedWidthAndHeight): Deleted.

LayoutTests:

  • media/content/no-rotation.mp4:
  • media/media-source/only-bcp47-language-tags-accepted-as-valid-expected.txt:
  • media/video-orientation-canvas-expected.txt: Added.
  • media/video-orientation-canvas.html: Added.
  • media/video-test.js:

(waitFor):

Location:
trunk
Files:
4 added
13 edited

Legend:

Unmodified
Added
Removed
  • trunk/LayoutTests/ChangeLog

    r246757 r246759  
     12019-06-24  Jer Noble  <jer.noble@apple.com>
     2
     3        iOS 12.2 Drawing portrait video to canvas is sideways
     4        https://bugs.webkit.org/show_bug.cgi?id=196772
     5        <rdar://problem/49781802>
     6
     7        Reviewed by Eric Carlson.
     8
     9        * media/content/no-rotation.mp4:
     10        * media/media-source/only-bcp47-language-tags-accepted-as-valid-expected.txt:
     11        * media/video-orientation-canvas-expected.txt: Added.
     12        * media/video-orientation-canvas.html: Added.
     13        * media/video-test.js:
     14        (waitFor):
     15
    1162019-06-24  Daniel Bates  <dabates@apple.com>
    217
  • trunk/LayoutTests/media/media-source/only-bcp47-language-tags-accepted-as-valid-expected.txt

    r246699 r246759  
    1 CONSOLE MESSAGE: line 221: The language 'a' is not a valid BCP 47 language tag.
     1CONSOLE MESSAGE: line 222: The language 'a' is not a valid BCP 47 language tag.
    22CONSOLE MESSAGE: line 106: The language 'a' is not a valid BCP 47 language tag.
    3 CONSOLE MESSAGE: line 221: The language 'a' is not a valid BCP 47 language tag.
    4 CONSOLE MESSAGE: line 221: The language '1' is not a valid BCP 47 language tag.
     3CONSOLE MESSAGE: line 222: The language 'a' is not a valid BCP 47 language tag.
     4CONSOLE MESSAGE: line 222: The language '1' is not a valid BCP 47 language tag.
    55CONSOLE MESSAGE: line 106: The language '1' is not a valid BCP 47 language tag.
    6 CONSOLE MESSAGE: line 221: The language '1' is not a valid BCP 47 language tag.
    7 CONSOLE MESSAGE: line 221: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
     6CONSOLE MESSAGE: line 222: The language '1' is not a valid BCP 47 language tag.
     7CONSOLE MESSAGE: line 222: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
    88CONSOLE MESSAGE: line 106: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
    9 CONSOLE MESSAGE: line 221: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
    10 CONSOLE MESSAGE: line 221: The language '1a' is not a valid BCP 47 language tag.
     9CONSOLE MESSAGE: line 222: The language 'ab-abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij' is not a valid BCP 47 language tag.
     10CONSOLE MESSAGE: line 222: The language '1a' is not a valid BCP 47 language tag.
    1111CONSOLE MESSAGE: line 106: The language '1a' is not a valid BCP 47 language tag.
    12 CONSOLE MESSAGE: line 221: The language '1a' is not a valid BCP 47 language tag.
    13 CONSOLE MESSAGE: line 221: The language '-a' is not a valid BCP 47 language tag.
     12CONSOLE MESSAGE: line 222: The language '1a' is not a valid BCP 47 language tag.
     13CONSOLE MESSAGE: line 222: The language '-a' is not a valid BCP 47 language tag.
    1414CONSOLE MESSAGE: line 106: The language '-a' is not a valid BCP 47 language tag.
    15 CONSOLE MESSAGE: line 221: The language '-a' is not a valid BCP 47 language tag.
    16 CONSOLE MESSAGE: line 221: The language 'a-' is not a valid BCP 47 language tag.
     15CONSOLE MESSAGE: line 222: The language '-a' is not a valid BCP 47 language tag.
     16CONSOLE MESSAGE: line 222: The language 'a-' is not a valid BCP 47 language tag.
    1717CONSOLE MESSAGE: line 106: The language 'a-' is not a valid BCP 47 language tag.
    18 CONSOLE MESSAGE: line 221: The language 'a-' is not a valid BCP 47 language tag.
    19 CONSOLE MESSAGE: line 221: The language 'a1' is not a valid BCP 47 language tag.
     18CONSOLE MESSAGE: line 222: The language 'a-' is not a valid BCP 47 language tag.
     19CONSOLE MESSAGE: line 222: The language 'a1' is not a valid BCP 47 language tag.
    2020CONSOLE MESSAGE: line 106: The language 'a1' is not a valid BCP 47 language tag.
    21 CONSOLE MESSAGE: line 221: The language 'a1' is not a valid BCP 47 language tag.
    22 CONSOLE MESSAGE: line 221: The language 'aa1' is not a valid BCP 47 language tag.
     21CONSOLE MESSAGE: line 222: The language 'a1' is not a valid BCP 47 language tag.
     22CONSOLE MESSAGE: line 222: The language 'aa1' is not a valid BCP 47 language tag.
    2323CONSOLE MESSAGE: line 106: The language 'aa1' is not a valid BCP 47 language tag.
    24 CONSOLE MESSAGE: line 221: The language 'aa1' is not a valid BCP 47 language tag.
    25 CONSOLE MESSAGE: line 221: The language 'aaaa' is not a valid BCP 47 language tag.
     24CONSOLE MESSAGE: line 222: The language 'aa1' is not a valid BCP 47 language tag.
     25CONSOLE MESSAGE: line 222: The language 'aaaa' is not a valid BCP 47 language tag.
    2626CONSOLE MESSAGE: line 106: The language 'aaaa' is not a valid BCP 47 language tag.
    27 CONSOLE MESSAGE: line 221: The language 'aaaa' is not a valid BCP 47 language tag.
    28 CONSOLE MESSAGE: line 221: The language 'aaa1' is not a valid BCP 47 language tag.
     27CONSOLE MESSAGE: line 222: The language 'aaaa' is not a valid BCP 47 language tag.
     28CONSOLE MESSAGE: line 222: The language 'aaa1' is not a valid BCP 47 language tag.
    2929CONSOLE MESSAGE: line 106: The language 'aaa1' is not a valid BCP 47 language tag.
    30 CONSOLE MESSAGE: line 221: The language 'aaa1' is not a valid BCP 47 language tag.
    31 CONSOLE MESSAGE: line 221: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
     30CONSOLE MESSAGE: line 222: The language 'aaa1' is not a valid BCP 47 language tag.
     31CONSOLE MESSAGE: line 222: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
    3232CONSOLE MESSAGE: line 106: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
    33 CONSOLE MESSAGE: line 221: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
    34 CONSOLE MESSAGE: line 221: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
     33CONSOLE MESSAGE: line 222: The language 'inv-alid-char space' is not a valid BCP 47 language tag.
     34CONSOLE MESSAGE: line 222: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
    3535CONSOLE MESSAGE: line 106: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
    36 CONSOLE MESSAGE: line 221: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
    37 CONSOLE MESSAGE: line 221: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
     36CONSOLE MESSAGE: line 222: The language 'inv-alid-char–longDash' is not a valid BCP 47 language tag.
     37CONSOLE MESSAGE: line 222: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
    3838CONSOLE MESSAGE: line 106: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
    39 CONSOLE MESSAGE: line 221: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
    40 CONSOLE MESSAGE: line 221: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
     39CONSOLE MESSAGE: line 222: The language 'inv-alid-char-PÃ¥lska' is not a valid BCP 47 language tag.
     40CONSOLE MESSAGE: line 222: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
    4141CONSOLE MESSAGE: line 106: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
    42 CONSOLE MESSAGE: line 221: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
    43 CONSOLE MESSAGE: line 221: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
     42CONSOLE MESSAGE: line 222: The language 'inv-alid-char-*' is not a valid BCP 47 language tag.
     43CONSOLE MESSAGE: line 222: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
    4444CONSOLE MESSAGE: line 106: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
    45 CONSOLE MESSAGE: line 221: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
     45CONSOLE MESSAGE: line 222: The language 'inv-alid-char-' is not a valid BCP 47 language tag.
    4646Test that only BCP47 language tags are accepted as valid but still reflected.
    4747
  • trunk/LayoutTests/media/video-test.js

    r246699 r246759  
    187187}
    188188
    189 function waitFor(element, type) {
     189function waitFor(element, type, silent) {
    190190    return new Promise(resolve => {
    191191        element.addEventListener(type, event => {
    192             consoleWrite(`EVENT(${event.type})`);
     192            if (!silent)
     193                consoleWrite(`EVENT(${event.type})`);
    193194            resolve(event);
    194195        }, { once: true });
  • trunk/Source/WebCore/ChangeLog

    r246755 r246759  
     12019-06-24  Jer Noble  <jer.noble@apple.com>
     2
     3        iOS 12.2 Drawing portrait video to canvas is sideways
     4        https://bugs.webkit.org/show_bug.cgi?id=196772
     5        <rdar://problem/49781802>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Test: media/video-orientation-canvas.html
     10
     11        Move rotation code into its own ImageRotationSessionVT class for re-use across
     12        all existing classes with rotation operations. Should slightly increase performance
     13        for painting rotated media files, as the rotation only occurs once per frame, rather
     14        than once per drawing operation.
     15
     16        * WebCore.xcodeproj/project.pbxproj:
     17        * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h:
     18        (WebCore::ImageDecoderAVFObjC::RotationProperties::isIdentity const): Deleted.
     19        * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm:
     20        (WebCore::ImageDecoderAVFObjC::readTrackMetadata):
     21        (WebCore::ImageDecoderAVFObjC::storeSampleBuffer):
     22        (WebCore::ImageDecoderAVFObjC::setTrack):
     23        (WebCore::transformToRotationProperties): Deleted.
     24        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
     25        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
     26        (WebCore::MediaPlayerPrivateAVFoundationObjC::tracksChanged):
     27        (WebCore::MediaPlayerPrivateAVFoundationObjC::updateLastPixelBuffer):
     28        (WebCore::MediaPlayerPrivateAVFoundationObjC::paintWithVideoOutput):
     29        * platform/graphics/cv/ImageRotationSessionVT.h: Added.
     30        (WebCore::ImageRotationSessionVT::RotationProperties::isIdentity const):
     31        (WebCore::ImageRotationSessionVT::rotationProperties const):
     32        (WebCore::ImageRotationSessionVT::rotatedSize):
     33        * platform/graphics/cv/ImageRotationSessionVT.mm: Added.
     34        (WebCore::transformToRotationProperties):
     35        (WebCore::ImageRotationSessionVT::ImageRotationSessionVT):
     36        (WebCore::ImageRotationSessionVT::rotate):
     37        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
     38        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
     39        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm:
     40        (WebCore::rotationToAngle):
     41        (WebCore::RealtimeOutgoingVideoSourceCocoa::rotatePixelBuffer):
     42        (WebCore::computeRotatedWidthAndHeight): Deleted.
     43
    1442019-06-24  Zalan Bujtas  <zalan@apple.com>
    245
  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r246725 r246759  
    40314031                CD225C0B1C46FBF400140761 /* WebCoreNSURLSession.mm in Sources */ = {isa = PBXBuildFile; fileRef = CD225C091C46FBF400140761 /* WebCoreNSURLSession.mm */; };
    40324032                CD225C0C1C46FBF400140761 /* WebCoreNSURLSession.h in Headers */ = {isa = PBXBuildFile; fileRef = CD225C0A1C46FBF400140761 /* WebCoreNSURLSession.h */; settings = {ATTRIBUTES = (Private, ); }; };
     4033                CD27AE5022A9868700947FF9 /* ImageRotationSessionVT.h in Headers */ = {isa = PBXBuildFile; fileRef = CD27AE4E22A9868700947FF9 /* ImageRotationSessionVT.h */; };
     4034                CD27AE5122A9868700947FF9 /* ImageRotationSessionVT.mm in Sources */ = {isa = PBXBuildFile; fileRef = CD27AE4F22A9868700947FF9 /* ImageRotationSessionVT.mm */; };
    40334035                CD2F4A2418D89F700063746D /* AudioHardwareListener.h in Headers */ = {isa = PBXBuildFile; fileRef = CD2F4A2218D89F700063746D /* AudioHardwareListener.h */; settings = {ATTRIBUTES = (Private, ); }; };
    40344036                CD2F4A2818D8A3490063746D /* AudioHardwareListenerMac.h in Headers */ = {isa = PBXBuildFile; fileRef = CD2F4A2618D8A3490063746D /* AudioHardwareListenerMac.h */; };
     
    1352613528                CD225C091C46FBF400140761 /* WebCoreNSURLSession.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WebCoreNSURLSession.mm; sourceTree = "<group>"; };
    1352713529                CD225C0A1C46FBF400140761 /* WebCoreNSURLSession.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebCoreNSURLSession.h; sourceTree = "<group>"; };
     13530                CD27AE4E22A9868700947FF9 /* ImageRotationSessionVT.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ImageRotationSessionVT.h; sourceTree = "<group>"; };
     13531                CD27AE4F22A9868700947FF9 /* ImageRotationSessionVT.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = ImageRotationSessionVT.mm; sourceTree = "<group>"; };
    1352813532                CD27F6E014575C1B0078207D /* MediaController.idl */ = {isa = PBXFileReference; lastKnownFileType = text; path = MediaController.idl; sourceTree = "<group>"; };
    1352913533                CD27F6E2145767580078207D /* JSMediaController.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSMediaController.cpp; sourceTree = "<group>"; };
     
    2569225696                        isa = PBXGroup;
    2569325697                        children = (
     25698                                CD27AE4E22A9868700947FF9 /* ImageRotationSessionVT.h */,
     25699                                CD27AE4F22A9868700947FF9 /* ImageRotationSessionVT.mm */,
    2569425700                                0746D30C2146EA38003DDF84 /* ImageTransferSessionVT.h */,
    2569525701                                0746D30A2146EA37003DDF84 /* ImageTransferSessionVT.mm */,
     
    2953329539                                B51A2F3F17D7D3AE0072517A /* ImageQualityController.h in Headers */,
    2953429540                                49291E4B134172C800E753DE /* ImageRenderingMode.h in Headers */,
     29541                                CD27AE5022A9868700947FF9 /* ImageRotationSessionVT.h in Headers */,
    2953529542                                7C193BC11F5E0EED0088F3E6 /* ImageSmoothingQuality.h in Headers */,
    2953629543                                5546757B1FD212A9003B10B0 /* ImageSource.h in Headers */,
     
    3279132798                                1AC900C31943C0FC008625B5 /* HTTPHeaderNames.cpp in Sources */,
    3279232799                                CD19FEAF1F574B6D000C42FB /* ImageDecoderAVFObjC.mm in Sources */,
     32800                                CD27AE5122A9868700947FF9 /* ImageRotationSessionVT.mm in Sources */,
    3279332801                                BE961C5418AD338500D07DC5 /* InbandDataTextTrack.cpp in Sources */,
    3279432802                                BE16C59217CFE17200852C04 /* InbandGenericTextTrack.cpp in Sources */,
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h

    r246699 r246759  
    3939OBJC_CLASS WebCoreSharedBufferResourceLoaderDelegate;
    4040typedef struct opaqueCMSampleBuffer* CMSampleBufferRef;
    41 typedef struct OpaqueVTImageRotationSession* VTImageRotationSessionRef;
    42 typedef struct __CVPixelBufferPool* CVPixelBufferPoolRef;
    4341
    4442namespace WTF {
     
    5048class ContentType;
    5149class ImageDecoderAVFObjCSample;
     50class ImageRotationSessionVT;
    5251class PixelBufferConformerCV;
    5352class WebCoreDecompressionSession;
     
    9190    void clearFrameBufferCache(size_t) final;
    9291
    93     struct RotationProperties {
    94         bool flipX { false };
    95         bool flipY { false };
    96         unsigned angle { 0 };
    97 
    98         bool isIdentity() const { return !flipX && !flipY && !angle; }
    99     };
    100 
    10192private:
    10293    ImageDecoderAVFObjC(SharedBuffer&, const String& mimeType, AlphaOption, GammaAndColorProfileOption);
     
    117108    RetainPtr<AVAssetTrack> m_track;
    118109    RetainPtr<WebCoreSharedBufferResourceLoaderDelegate> m_loader;
    119     RetainPtr<VTImageRotationSessionRef> m_rotationSession;
    120     RetainPtr<CVPixelBufferPoolRef> m_rotationPool;
     110    std::unique_ptr<ImageRotationSessionVT> m_imageRotationSession;
    121111    Ref<WebCoreDecompressionSession> m_decompressionSession;
    122112    WTF::Function<void(EncodedDataStatus)> m_encodedDataStatusChangedCallback;
     
    127117    bool m_isAllDataReceived { false };
    128118    Optional<IntSize> m_size;
    129     Optional<RotationProperties> m_rotation;
    130119};
    131120
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm

    r246699 r246759  
    3535#import "FloatRect.h"
    3636#import "FloatSize.h"
     37#import "ImageRotationSessionVT.h"
    3738#import "Logging.h"
    3839#import "MIMETypeRegistry.h"
     
    234235}
    235236
    236 static ImageDecoderAVFObjC::RotationProperties transformToRotationProperties(AffineTransform inTransform)
    237 {
    238     ImageDecoderAVFObjC::RotationProperties rotation;
    239     if (inTransform.isIdentity())
    240         return rotation;
    241 
    242     AffineTransform::DecomposedType decomposed { };
    243     if (!inTransform.decompose(decomposed))
    244         return rotation;
    245 
    246     rotation.flipY = WTF::areEssentiallyEqual(decomposed.scaleX, -1.);
    247     rotation.flipX = WTF::areEssentiallyEqual(decomposed.scaleY, -1.);
    248     auto degrees = rad2deg(decomposed.angle);
    249     while (degrees < 0)
    250         degrees += 360;
    251 
    252     // Only support rotation in multiples of 90º:
    253     if (WTF::areEssentiallyEqual(fmod(degrees, 90.), 0.))
    254         rotation.angle = clampToUnsigned(degrees);
    255 
    256     return rotation;
    257 }
    258 
    259237class ImageDecoderAVFObjCSample : public MediaSampleAVFObjC {
    260238public:
     
    423401void ImageDecoderAVFObjC::readTrackMetadata()
    424402{
    425     if (!m_rotation)
    426         m_rotation = transformToRotationProperties(CGAffineTransformConcat(m_asset.get().preferredTransform, m_track.get().preferredTransform));
    427 
    428     if (!m_size) {
    429         auto size = FloatSize(m_track.get().naturalSize);
    430         auto angle = m_rotation.value().angle;
    431         if (angle == 90 || angle == 270)
    432             size = size.transposedSize();
    433 
    434         m_size = expandedIntSize(size);
    435     }
     403    AffineTransform finalTransform = CGAffineTransformConcat(m_asset.get().preferredTransform, m_track.get().preferredTransform);
     404    auto size = expandedIntSize(FloatSize(m_track.get().naturalSize));
     405    if (finalTransform.isIdentity()) {
     406        m_size = size;
     407        m_imageRotationSession = nullptr;
     408        return;
     409    }
     410
     411    if (!m_imageRotationSession
     412        || !m_imageRotationSession->transform()
     413        || m_imageRotationSession->transform().value() != finalTransform
     414        || m_imageRotationSession->size() != size)
     415        m_imageRotationSession = std::make_unique<ImageRotationSessionVT>(WTFMove(finalTransform), size, kCVPixelFormatType_32BGRA, ImageRotationSessionVT::IsCGImageCompatible::Yes);
     416
     417    m_size = expandedIntSize(m_imageRotationSession->rotatedSize());
    436418}
    437419
     
    447429    auto iter = m_sampleData.presentationOrder().findSampleWithPresentationTime(presentationTime);
    448430
    449     if (m_rotation && !m_rotation.value().isIdentity()) {
    450         auto& rotation = m_rotation.value();
    451         if (!m_rotationSession) {
    452             VTImageRotationSessionRef rawRotationSession = nullptr;
    453             VTImageRotationSessionCreate(kCFAllocatorDefault, rotation.angle, &rawRotationSession);
    454             m_rotationSession = rawRotationSession;
    455             VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_EnableHighSpeedTransfer, kCFBooleanTrue);
    456 
    457             if (rotation.flipY)
    458                 VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipVerticalOrientation, kCFBooleanTrue);
    459             if (rotation.flipX)
    460                 VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipHorizontalOrientation, kCFBooleanTrue);
    461         }
    462 
    463         if (!m_rotationPool) {
    464             auto pixelAttributes = @{
    465                 (__bridge NSString *)kCVPixelBufferWidthKey: @(m_size.value().width()),
    466                 (__bridge NSString *)kCVPixelBufferHeightKey: @(m_size.value().height()),
    467                 (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA),
    468                 (__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: @YES,
    469             };
    470             CVPixelBufferPoolRef rawPool = nullptr;
    471             CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &rawPool);
    472             m_rotationPool = adoptCF(rawPool);
    473         }
    474 
    475         CVPixelBufferRef rawRotatedBuffer = nullptr;
    476         CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, m_rotationPool.get(), &rawRotatedBuffer);
    477         auto status = VTImageRotationSessionTransferImage(m_rotationSession.get(), pixelBuffer.get(), rawRotatedBuffer);
    478         if (status == noErr)
    479             pixelBuffer = adoptCF(rawRotatedBuffer);
    480     }
     431    if (m_imageRotationSession)
     432        pixelBuffer = m_imageRotationSession->rotate(pixelBuffer.get());
    481433
    482434    CGImageRef rawImage = nullptr;
     
    507459    m_sampleData.clear();
    508460    m_size.reset();
    509     m_rotation.reset();
    510461    m_cursor = m_sampleData.decodeOrder().end();
    511     m_rotationSession = nullptr;
     462    m_imageRotationSession = nullptr;
    512463
    513464    [track loadValuesAsynchronouslyForKeys:@[@"naturalSize", @"preferredTransform"] completionHandler:[protectedThis = makeRefPtr(this)] () mutable {
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h

    r246699 r246759  
    5959class CDMInstanceFairPlayStreamingAVFObjC;
    6060class CDMSessionAVFoundationObjC;
     61class ImageRotationSessionVT;
    6162class InbandMetadataTextTrackPrivateAVF;
    6263class MediaSelectionGroupAVFObjC;
     
    326327
    327328    void setShouldDisableSleep(bool) override;
     329    void updateRotationSession();
    328330
    329331    Optional<VideoPlaybackQualityMetrics> videoPlaybackQualityMetrics() final;
     
    361363    RetainPtr<CGImageRef> m_lastImage;
    362364    BinarySemaphore m_videoOutputSemaphore;
     365    std::unique_ptr<ImageRotationSessionVT> m_imageRotationSession;
    363366    std::unique_ptr<VideoTextureCopierCV> m_videoTextureCopier;
    364367#endif
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm

    r246699 r246759  
    4444#import "GraphicsContext3D.h"
    4545#import "GraphicsContextCG.h"
     46#import "ImageRotationSessionVT.h"
    4647#import "InbandMetadataTextTrackPrivateAVF.h"
    4748#import "InbandTextTrackPrivateAVFObjC.h"
     
    18641865}
    18651866
     1867void MediaPlayerPrivateAVFoundationObjC::updateRotationSession()
     1868{
     1869    AffineTransform finalTransform = m_avAsset.get().preferredTransform;
     1870    FloatSize naturalSize;
     1871    if (auto* firstEnabledVideoTrack = firstEnabledTrack([m_avAsset.get() tracksWithMediaCharacteristic:AVMediaCharacteristicVisual])) {
     1872        naturalSize = FloatSize(firstEnabledVideoTrack.naturalSize);
     1873        finalTransform *= firstEnabledVideoTrack.preferredTransform;
     1874    }
     1875
     1876    if (finalTransform.isIdentity()) {
     1877        m_imageRotationSession = nullptr;
     1878        return;
     1879    }
     1880
     1881    if (m_imageRotationSession
     1882        && m_imageRotationSession->transform()
     1883        && m_imageRotationSession->transform().value() == finalTransform
     1884        && m_imageRotationSession->size() == naturalSize)
     1885        return;
     1886
     1887    m_imageRotationSession = std::make_unique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, kCVPixelFormatType_32BGRA, ImageRotationSessionVT::IsCGImageCompatible::Yes);
     1888}
     1889
    18661890#if ENABLE(VIDEO_TRACK)
    18671891
     
    20542078        return;
    20552079
     2080    updateRotationSession();
    20562081    setNaturalSize(m_cachedPresentationSize);
    20572082}
     
    21412166
    21422167    m_lastPixelBuffer = adoptCF([m_videoOutput.get() copyPixelBufferForItemTime:currentTime itemTimeForDisplay:nil]);
     2168
     2169    if (m_imageRotationSession)
     2170        m_lastPixelBuffer = m_imageRotationSession->rotate(m_lastPixelBuffer.get());
     2171
    21432172    m_lastImage = nullptr;
    21442173    return true;
     
    22072236    INFO_LOG(LOGIDENTIFIER);
    22082237
    2209     GraphicsContextStateSaver stateSaver(context);
    22102238    FloatRect imageRect(0, 0, CGImageGetWidth(m_lastImage.get()), CGImageGetHeight(m_lastImage.get()));
    2211     AffineTransform videoTransform = [firstEnabledVideoTrack preferredTransform];
    2212     FloatRect transformedOutputRect = videoTransform.inverse().valueOr(AffineTransform()).mapRect(outputRect);
    2213 
    2214     context.concatCTM(videoTransform);
    2215     context.drawNativeImage(m_lastImage.get(), imageRect.size(), transformedOutputRect, imageRect);
     2239    context.drawNativeImage(m_lastImage.get(), imageRect.size(), outputRect, imageRect);
    22162240
    22172241    // If we have created an AVAssetImageGenerator in the past due to m_videoOutput not having an available
  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp

    r246699 r246759  
    2929#if USE(LIBWEBRTC)
    3030
     31#include "ImageRotationSessionVT.h"
    3132#include "Logging.h"
    3233#include "RealtimeIncomingVideoSourceCocoa.h"
  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h

    r246699 r246759  
    3232#include <webrtc/api/video/video_rotation.h>
    3333
    34 typedef struct OpaqueVTImageRotationSession* VTImageRotationSessionRef;
    35 typedef struct __CVPixelBufferPool* CVPixelBufferPoolRef;
     34namespace WebCore {
    3635
    37 namespace WebCore {
     36class ImageRotationSessionVT;
    3837
    3938class RealtimeOutgoingVideoSourceCocoa final : public RealtimeOutgoingVideoSource {
     
    5352
    5453    std::unique_ptr<PixelBufferConformerCV> m_pixelBufferConformer;
    55     RetainPtr<VTImageRotationSessionRef> m_rotationSession;
    56     RetainPtr<CVPixelBufferPoolRef> m_rotationPool;
     54    std::unique_ptr<ImageRotationSessionVT> m_rotationSession;
    5755    webrtc::VideoRotation m_currentRotationSessionAngle { webrtc::kVideoRotation_0 };
    5856    size_t m_rotatedWidth { 0 };
  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm

    r246699 r246759  
    2929#if USE(LIBWEBRTC)
    3030
     31#import "AffineTransform.h"
     32#import "ImageRotationSessionVT.h"
    3133#import "Logging.h"
    3234#import "MediaSample.h"
     
    5052}
    5153
    52 static inline void computeRotatedWidthAndHeight(CVPixelBufferRef pixelBuffer, webrtc::VideoRotation rotation, size_t& width, size_t& height)
     54static inline unsigned rotationToAngle(webrtc::VideoRotation rotation)
    5355{
    5456    switch (rotation) {
    5557    case webrtc::kVideoRotation_0:
     58        return 0;
     59    case webrtc::kVideoRotation_90:
     60        return 90;
    5661    case webrtc::kVideoRotation_180:
    57         width = CVPixelBufferGetWidth(pixelBuffer);
    58         height = CVPixelBufferGetHeight(pixelBuffer);
    59         return;
    60     case webrtc::kVideoRotation_90:
     62        return 180;
    6163    case webrtc::kVideoRotation_270:
    62         width = CVPixelBufferGetHeight(pixelBuffer);
    63         height = CVPixelBufferGetWidth(pixelBuffer);
    64         return;
     64        return 270;
    6565    }
    6666}
     
    7373
    7474    if (!m_rotationSession || rotation != m_currentRotationSessionAngle) {
    75         VTImageRotationSessionRef rawRotationSession = nullptr;
    76         auto status = VTImageRotationSessionCreate(kCFAllocatorDefault, rotation, &rawRotationSession);
    77         if (status != noErr) {
    78             ERROR_LOG(LOGIDENTIFIER, "Failed creating a rotation session with error ", status);
    79             return nullptr;
    80         }
    81 
    82         m_rotationSession = adoptCF(rawRotationSession);
    83         m_currentRotationSessionAngle = rotation;
    84 
    85         VTImageRotationSessionSetProperty(rawRotationSession, kVTImageRotationPropertyKey_EnableHighSpeedTransfer, kCFBooleanTrue);
     75        IntSize size = { (int)CVPixelBufferGetWidth(pixelBuffer) , (int)CVPixelBufferGetHeight(pixelBuffer) };
     76        AffineTransform transform;
     77        transform.rotate(rotationToAngle(rotation));
     78        m_rotationSession = std::make_unique<ImageRotationSessionVT>(WTFMove(transform), size, CVPixelBufferGetPixelFormatType(pixelBuffer), ImageRotationSessionVT::IsCGImageCompatible::No);
    8679    }
    8780
    88     size_t rotatedWidth, rotatedHeight;
    89     computeRotatedWidthAndHeight(pixelBuffer, rotation, rotatedWidth, rotatedHeight);
    90     auto format = CVPixelBufferGetPixelFormatType(pixelBuffer);
    91     if (!m_rotationPool || rotatedWidth != m_rotatedWidth || rotatedHeight != m_rotatedHeight || format != m_rotatedFormat) {
    92         auto pixelAttributes = @{
    93             (__bridge NSString *)kCVPixelBufferWidthKey: @(rotatedWidth),
    94             (__bridge NSString *)kCVPixelBufferHeightKey: @(rotatedHeight),
    95             (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(format),
    96             (__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: @NO,
    97         };
    98 
    99         CVPixelBufferPoolRef pool = nullptr;
    100         auto status = CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &pool);
    101 
    102         if (status != kCVReturnSuccess) {
    103             ERROR_LOG(LOGIDENTIFIER, "Failed creating a pixel buffer pool with error ", status);
    104             return nullptr;
    105         }
    106         m_rotationPool = adoptCF(pool);
    107 
    108         m_rotatedWidth = rotatedWidth;
    109         m_rotatedHeight = rotatedHeight;
    110         m_rotatedFormat = format;
    111     }
    112 
    113     CVPixelBufferRef rawRotatedBuffer = nullptr;
    114     auto status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, m_rotationPool.get(), &rawRotatedBuffer);
    115 
    116     if (status != kCVReturnSuccess) {
    117         ERROR_LOG(LOGIDENTIFIER, "Failed creating a pixel buffer with error ", status);
    118         return nullptr;
    119     }
    120     RetainPtr<CVPixelBufferRef> rotatedBuffer = adoptCF(rawRotatedBuffer);
    121 
    122     status = VTImageRotationSessionTransferImage(m_rotationSession.get(), pixelBuffer, rotatedBuffer.get());
    123 
    124     if (status != noErr) {
    125         ERROR_LOG(LOGIDENTIFIER, "Failed rotating with error ", status, " for rotation ", m_currentRotation);
    126         return nullptr;
    127     }
    128     return rotatedBuffer;
     81    return m_rotationSession->rotate(pixelBuffer);
    12982}
    13083
Note: See TracChangeset for help on using the changeset viewer.