---+ Details of the QtWebKit HTML5 media element implementation

<a href="http://chaos.troll.no/~tavestbo/webkit/mediaelement/"><img src="http://chaos.troll.no/~tavestbo/webkit/mediaelement/screenshot.png" width=500></a>

---++ Structure

DOM elements:

   * =WebCore/html/HTMLMediaElement.cpp= and =WebCore/html/HTMLMediaElement.h=
   * =WebCore/html/HTMLVideoElement.cpp= and =WebCore/html/HTMLVideoElement.h=
   * =WebCore/html/HTMLAudioElement.cpp= and =WebCore/html/HTMLAudioElement.h=

Renderers:

   * =WebCore/rendering/RenderMedia.cpp= and =WebCore/rendering/RenderMedia.h=
   * =WebCore/rendering/RenderVideo.cpp= and =WebCore/rendering/RenderVideo.h=

=MediaPlayerPrivate= implementation:

   * =WebCore/platform/graphics/qt/MediaPlayerPrivatePhonon.cpp= and =WebCore/platform/graphics/qt/MediaPlayerPrivatePhonon.h=

---++ Limitations

The current implementation has some limitations, partly due to lack of support in the Phonon API and backends.

---+++ Software rendering

The implementation is based on a =[[http://doc.trolltech.com/4.4/phonon-videowidget.html][Phonon::VideoWidget]]=, which is rendered in =MediaPlayerPrivate::paint()= by a call to =[[http://doc.trolltech.com/4.4/qwidget.html#render][QWidget::render()]]=, passing the active webkit =[[http://doc.trolltech.com/4.4/qwidget.html][QPainter]]=.

For this technique to work we have to make sure the Phonon back-end either uses a software rendering path, using the QPainter to draw the video, or that any accelerated drawing does not change the context state of the =QPainter= and allows WebKit to continue drawing on top after the video has been rendered. In other words, no XVideo or ad hock accelerated drawing.

We set the window flag =WA_DontShowOnScreen= on the video widget, as a hint to the back-end to use this software path. The reason for the software path requirement is that we want HTML content overlapping with the video to not be clipped by the video. 

This works fine on X11 and Mac (a little slow), and the Windows bugs will get ironed out soon.

See =$QTDIR/src/3rdparty/kdebase/runtime/phonon/gstreamer/videowidget.cpp=, =$QTDIR/src/3rdparty/kdebase/runtime/phonon/ds9/videowidget.cpp=, and =$QTDIR/src/3rdparty/kdebase/runtime/phonon/qt7/videowidget.cpp= for examples.

---+++ Media loading

Media loading is currently handled by Phonon, since we call =[[http://doc.trolltech.com/4.4/phonon-mediaobject.html#setCurrentSource][MediaObject::setCurrentSource()]]= passing in a =Phonon::MediaSource=
constructed from a URL. The alternative would be to handle loading using WebKit's normal resource loading mechanisms, and then use a =MediaSource= constructed from a =[[http://api.kde.org/4.0-api/kdelibs-apidocs/phonon/html/classPhonon_1_1AbstractMediaStream.html][Phonon::AbstractMediaStream]]= to feed the loaded data from WebKit to Phonon.

The problem is that on Mac OSX there is no way to continuously feed QuickTime the data on the back-end side (only in one big chunk, which is hardly optimal for streaming data), so a solution based on =AbstractMediaStream= would not be cross platform.
Another problem is that none of the Phonon back-ends support the =AbstractMediaStream= push model, where we feed data to Phonon as the data arrives in the WebKit resource loader. The model they do support
(again, only on Linux on Windows) is the =AbstractMediaStream= pull model, where Phonon asks us to provide more data, and we are expected to at least give it something. That leaves us with having to do a busy wait
using =QApplication::processEvents()= to ensure that we have loaded enough data to feed back to Phonon when returning, which is pretty ugly. So in,  conclusion, =AbstractMediaStream= does not have the back-end
support yet to be a basis for this implementation. (I do have an experimental branch where i tried this out though, available [[http://code.staikos.net/cgi-bin/gitweb.cgi?p=webkit;a=shortlog;h=torarne/mediaelement/medialoader][here]]).

So what are the downsides of letting Phonon handle media loading? For one, the Phonon API does not have a way for the client to query how much of the media has been loaded. This is not only useful for displaying
a visualization to the user of how much of the file has been loaded, but also required by the platform independent WebKit media element code to know whether or not it can seek to a given position. Confusingly enough
Phonon does provide a =[[http://api.kde.org/4.0-api/kdelibs-apidocs/phonon/html/classPhonon_1_1MediaObject.html#0f5ce603d4a061e694ee3f531955d5f7][MediaObject::bufferStatus()]]= signal, but that
signal refers to how much of the internal _pre-buffering_ buffer has been filled (for example when you seek to a new position, Phonon goes into a buffering state where it fills this buffer, and then goes back to playing state), not
to how much of the media files has been loaded. In effect, there is no way to provide a seek slider without lying to the WebKit platform independent code about how much data we've loaded (by claiming that we've got all the data, even though we don't know that).

Another problem is that user credentials are not forwarded to Phonon. Normally when you are logged into a site, for example a news site, the browser knows you are authorized, so when you requests more resources (images, videos, documents) it uses that authorization for those resources. In the case of Phonon, there is no way to tell it that you are authorized, so if you for example log into a site, and want to watch a video, Phonon will try to load that video but will get permission denied. That would not happen if we loaded resources though WebKit, because then the credentials would be passed along the request.