Changes between Initial Version and Version 1 of QtWebKitMediaElementSupport


Ignore:
Timestamp:
Dec 7, 2009 12:46:27 AM (11 years ago)
Author:
vestbo@webkit.org
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • QtWebKitMediaElementSupport

    v1 v1  
     1---+ Details of the QtWebKit HTML5 media element implementation
     2
     3<a href="http://chaos.troll.no/~tavestbo/webkit/mediaelement/"><img src="http://chaos.troll.no/~tavestbo/webkit/mediaelement/screenshot.png" width=500></a>
     4
     5---++ Structure
     6
     7DOM elements:
     8
     9   * =WebCore/html/HTMLMediaElement.cpp= and =WebCore/html/HTMLMediaElement.h=
     10   * =WebCore/html/HTMLVideoElement.cpp= and =WebCore/html/HTMLVideoElement.h=
     11   * =WebCore/html/HTMLAudioElement.cpp= and =WebCore/html/HTMLAudioElement.h=
     12
     13Renderers:
     14
     15   * =WebCore/rendering/RenderMedia.cpp= and =WebCore/rendering/RenderMedia.h=
     16   * =WebCore/rendering/RenderVideo.cpp= and =WebCore/rendering/RenderVideo.h=
     17
     18=MediaPlayerPrivate= implementation:
     19
     20   * =WebCore/platform/graphics/qt/MediaPlayerPrivatePhonon.cpp= and =WebCore/platform/graphics/qt/MediaPlayerPrivatePhonon.h=
     21
     22---++ Limitations
     23
     24The current implementation has some limitations, partly due to lack of support in the Phonon API and backends.
     25
     26---+++ Software rendering
     27
     28The implementation is based on a =[[http://doc.trolltech.com/4.4/phonon-videowidget.html][Phonon::VideoWidget]]=, which is rendered in =MediaPlayerPrivate::paint()= by a call to =[[http://doc.trolltech.com/4.4/qwidget.html#render][QWidget::render()]]=, passing the active webkit =[[http://doc.trolltech.com/4.4/qwidget.html][QPainter]]=.
     29
     30For this technique to work we have to make sure the Phonon back-end either uses a software rendering path, using the QPainter to draw the video, or that any accelerated drawing does not change the context state of the =QPainter= and allows WebKit to continue drawing on top after the video has been rendered. In other words, no XVideo or ad hock accelerated drawing.
     31
     32We set the window flag =WA_DontShowOnScreen= on the video widget, as a hint to the back-end to use this software path. The reason for the software path requirement is that we want HTML content overlapping with the video to not be clipped by the video.
     33
     34This works fine on X11 and Mac (a little slow), and the Windows bugs will get ironed out soon.
     35
     36See =$QTDIR/src/3rdparty/kdebase/runtime/phonon/gstreamer/videowidget.cpp=, =$QTDIR/src/3rdparty/kdebase/runtime/phonon/ds9/videowidget.cpp=, and =$QTDIR/src/3rdparty/kdebase/runtime/phonon/qt7/videowidget.cpp= for examples.
     37
     38---+++ Media loading
     39
     40Media loading is currently handled by Phonon, since we call =[[http://doc.trolltech.com/4.4/phonon-mediaobject.html#setCurrentSource][MediaObject::setCurrentSource()]]= passing in a =Phonon::MediaSource=
     41constructed from a URL. The alternative would be to handle loading using WebKit's normal resource loading mechanisms, and then use a =MediaSource= constructed from a =[[http://api.kde.org/4.0-api/kdelibs-apidocs/phonon/html/classPhonon_1_1AbstractMediaStream.html][Phonon::AbstractMediaStream]]= to feed the loaded data from WebKit to Phonon.
     42
     43The problem is that on Mac OSX there is no way to continuously feed QuickTime the data on the back-end side (only in one big chunk, which is hardly optimal for streaming data), so a solution based on =AbstractMediaStream= would not be cross platform.
     44Another problem is that none of the Phonon back-ends support the =AbstractMediaStream= push model, where we feed data to Phonon as the data arrives in the WebKit resource loader. The model they do support
     45(again, only on Linux on Windows) is the =AbstractMediaStream= pull model, where Phonon asks us to provide more data, and we are expected to at least give it something. That leaves us with having to do a busy wait
     46using =QApplication::processEvents()= to ensure that we have loaded enough data to feed back to Phonon when returning, which is pretty ugly. So in,  conclusion, =AbstractMediaStream= does not have the back-end
     47support yet to be a basis for this implementation. (I do have an experimental branch where i tried this out though, available [[http://code.staikos.net/cgi-bin/gitweb.cgi?p=webkit;a=shortlog;h=torarne/mediaelement/medialoader][here]]).
     48
     49So what are the downsides of letting Phonon handle media loading? For one, the Phonon API does not have a way for the client to query how much of the media has been loaded. This is not only useful for displaying
     50a visualization to the user of how much of the file has been loaded, but also required by the platform independent WebKit media element code to know whether or not it can seek to a given position. Confusingly enough
     51Phonon does provide a =[[http://api.kde.org/4.0-api/kdelibs-apidocs/phonon/html/classPhonon_1_1MediaObject.html#0f5ce603d4a061e694ee3f531955d5f7][MediaObject::bufferStatus()]]= signal, but that
     52signal refers to how much of the internal _pre-buffering_ buffer has been filled (for example when you seek to a new position, Phonon goes into a buffering state where it fills this buffer, and then goes back to playing state), not
     53to how much of the media files has been loaded. In effect, there is no way to provide a seek slider without lying to the WebKit platform independent code about how much data we've loaded (by claiming that we've got all the data, even though we don't know that).
     54
     55Another problem is that user credentials are not forwarded to Phonon. Normally when you are logged into a site, for example a news site, the browser knows you are authorized, so when you requests more resources (images, videos, documents) it uses that authorization for those resources. In the case of Phonon, there is no way to tell it that you are authorized, so if you for example log into a site, and want to watch a video, Phonon will try to load that video but will get permission denied. That would not happen if we loaded resources though WebKit, because then the credentials would be passed along the request.