Changeset 142497 in webkit

Timestamp:

Feb 11, 2013 12:41:49 PM (11 years ago)

Author:

eric@webkit.org

Message:

Make WebVTTTokenizer stop inheriting from MarkupTokenizerBase
​https://bugs.webkit.org/show_bug.cgi?id=109411

Reviewed by Adam Barth.

Moved InputStreamPreprocessor into its own header file so it can be
used by both WebVTTTokenizer and HTMLTokenizer.

Also split out kEndOfFileMarker from InputStreamPreprocessor<T> so that
it can be used w/o a specific instantiation of the template class.
This also made it possible to fix three old fixmes about wanting to share
that constant.

Again, separating WebVTT code from Markup* base classes made it simpler
at the cost of a little copy/paste code. WebVTT tokenization is remarkably
simple compared to HTML.

This will make it immediately possible to pull MarkupTokenizerBase up into
HTMLTokenizer and further simplify the code.

• GNUmakefile.list.am:
• Target.pri:
• WebCore.gypi:
• WebCore.vcproj/WebCore.vcproj:
• WebCore.vcxproj/WebCore.vcxproj:
• WebCore.vcxproj/WebCore.vcxproj.filters:
• WebCore.xcodeproj/project.pbxproj:
• html/parser/BackgroundHTMLParser.cpp:

(WebCore::BackgroundHTMLParser::markEndOfFile):

• html/parser/HTMLInputStream.h:

(WebCore::HTMLInputStream::markEndOfFile):

• html/parser/HTMLTokenizer.cpp:

(WebCore::HTMLTokenizer::nextToken):

• html/parser/InputStreamPreprocessor.h: Added.

(WebCore):
(InputStreamPreprocessor):
(WebCore::InputStreamPreprocessor::InputStreamPreprocessor):
(WebCore::InputStreamPreprocessor::nextInputCharacter):
(WebCore::InputStreamPreprocessor::peek):
(WebCore::InputStreamPreprocessor::advance):
(WebCore::InputStreamPreprocessor::skipNextNewLine):
(WebCore::InputStreamPreprocessor::reset):
(WebCore::InputStreamPreprocessor::shouldTreatNullAsEndOfFileMarker):

• html/track/WebVTTTokenizer.cpp:

(WebCore::WebVTTTokenizer::WebVTTTokenizer):
(WebCore::WebVTTTokenizer::nextToken):

• html/track/WebVTTTokenizer.h:

(WebVTTTokenizer):
(WebCore::WebVTTTokenizer::haveBufferedCharacterToken):
(WebCore::WebVTTTokenizer::bufferCharacter):
(WebCore::WebVTTTokenizer::emitAndResumeIn):
(WebCore::WebVTTTokenizer::emitEndOfFile):
(WebCore::WebVTTTokenizer::shouldSkipNullCharacters):

• xml/parser/MarkupTokenizerBase.h:

(MarkupTokenizerBase):
(WebCore::MarkupTokenizerBase::bufferCharacter):

Location:

trunk/Source/WebCore

Files:

• ChangeLog (modified) (1 diff)
• GNUmakefile.list.am (modified) (1 diff)
• Target.pri (modified) (1 diff)
• WebCore.gypi (modified) (1 diff)
• WebCore.vcproj/WebCore.vcproj (modified) (1 diff)
• WebCore.vcxproj/WebCore.vcxproj (modified) (1 diff)
• WebCore.vcxproj/WebCore.vcxproj.filters (modified) (1 diff)
• WebCore.xcodeproj/project.pbxproj (modified) (18 diffs)
• html/parser/BackgroundHTMLParser.cpp (modified) (1 diff)
• html/parser/HTMLInputStream.h (modified) (2 diffs)
• html/parser/HTMLTokenizer.cpp (modified) (46 diffs)
• html/parser/InputStreamPreprocessor.h (added)
• html/track/WebVTTTokenizer.cpp (modified) (10 diffs)
• html/track/WebVTTTokenizer.h (modified) (3 diffs)
• xml/parser/MarkupTokenizerBase.h (modified) (3 diffs)

trunk/Source/WebCore/ChangeLog

@@ +1 @@
+2013-02-11  Eric Seidel  <eric@webkit.org>
+
+        Make WebVTTTokenizer stop inheriting from MarkupTokenizerBase
+        https://bugs.webkit.org/show_bug.cgi?id=109411
+
+        Reviewed by Adam Barth.
+
+        Moved InputStreamPreprocessor into its own header file so it can be
+        used by both WebVTTTokenizer and HTMLTokenizer.
+
+        Also split out kEndOfFileMarker from InputStreamPreprocessor<T> so that
+        it can be used w/o a specific instantiation of the template class.
+        This also made it possible to fix three old fixmes about wanting to share
+        that constant.
+
+        Again, separating WebVTT code from Markup* base classes made it simpler
+        at the cost of a little copy/paste code.  WebVTT tokenization is remarkably
+        simple compared to HTML.
+
+        This will make it immediately possible to pull MarkupTokenizerBase up into
+        HTMLTokenizer and further simplify the code.
+
+        * GNUmakefile.list.am:
+        * Target.pri:
+        * WebCore.gypi:
+        * WebCore.vcproj/WebCore.vcproj:
+        * WebCore.vcxproj/WebCore.vcxproj:
+        * WebCore.vcxproj/WebCore.vcxproj.filters:
+        * WebCore.xcodeproj/project.pbxproj:
+        * html/parser/BackgroundHTMLParser.cpp:
+        (WebCore::BackgroundHTMLParser::markEndOfFile):
+        * html/parser/HTMLInputStream.h:
+        (WebCore::HTMLInputStream::markEndOfFile):
+        * html/parser/HTMLTokenizer.cpp:
+        (WebCore::HTMLTokenizer::nextToken):
+        * html/parser/InputStreamPreprocessor.h: Added.
+        (WebCore):
+        (InputStreamPreprocessor):
+        (WebCore::InputStreamPreprocessor::InputStreamPreprocessor):
+        (WebCore::InputStreamPreprocessor::nextInputCharacter):
+        (WebCore::InputStreamPreprocessor::peek):
+        (WebCore::InputStreamPreprocessor::advance):
+        (WebCore::InputStreamPreprocessor::skipNextNewLine):
+        (WebCore::InputStreamPreprocessor::reset):
+        (WebCore::InputStreamPreprocessor::shouldTreatNullAsEndOfFileMarker):
+        * html/track/WebVTTTokenizer.cpp:
+        (WebCore::WebVTTTokenizer::WebVTTTokenizer):
+        (WebCore::WebVTTTokenizer::nextToken):
+        * html/track/WebVTTTokenizer.h:
+        (WebVTTTokenizer):
+        (WebCore::WebVTTTokenizer::haveBufferedCharacterToken):
+        (WebCore::WebVTTTokenizer::bufferCharacter):
+        (WebCore::WebVTTTokenizer::emitAndResumeIn):
+        (WebCore::WebVTTTokenizer::emitEndOfFile):
+        (WebCore::WebVTTTokenizer::shouldSkipNullCharacters):
+        * xml/parser/MarkupTokenizerBase.h:
+        (MarkupTokenizerBase):
+        (WebCore::MarkupTokenizerBase::bufferCharacter):
+
 2013-02-11  Adam Barth  <abarth@webkit.org>
 

trunk/Source/WebCore/GNUmakefile.list.am

@@ -3619 +3619 @@
         Source/WebCore/html/parser/HTMLViewSourceParser.cpp \
         Source/WebCore/html/parser/HTMLViewSourceParser.h \
+        Source/WebCore/html/parser/InputStreamPreprocessor.h \
         Source/WebCore/html/parser/NestingLevelIncrementer.h \
         Source/WebCore/html/parser/TextDocumentParser.cpp \

trunk/Source/WebCore/Target.pri

@@ -1936 +1936 @@
     html/parser/HTMLTreeBuilder.h \
     html/parser/HTMLViewSourceParser.h \
+    html/parser/InputStreamPreprocessor.h \
     html/parser/XSSAuditor.h \
     html/parser/XSSAuditorDelegate.h \

trunk/Source/WebCore/WebCore.gypi

@@ -3428 +3428 @@
             'html/parser/HTMLViewSourceParser.cpp',
             'html/parser/HTMLViewSourceParser.h',
+            'html/parser/InputStreamPreprocessor.h',
             'html/parser/NestingLevelIncrementer.h',
             'html/parser/TextDocumentParser.cpp',

trunk/Source/WebCore/WebCore.vcproj/WebCore.vcproj

@@ -65011 +65011 @@
                                 </File>
                                 <File
+                                        RelativePath="..\html\parser\InputStreamPreprocessor.h"
+                                        >
+                                </File>
+                                <File
                                         RelativePath="..\html\parser\NestingLevelIncrementer.h"
                                         >

trunk/Source/WebCore/WebCore.vcxproj/WebCore.vcxproj

@@ -7954 +7954 @@
     <ClInclude Include="..\html\parser\HTMLTreeBuilder.h" />
     <ClInclude Include="..\html\parser\HTMLViewSourceParser.h" />
+    <ClInclude Include="..\html\parser\InputStreamPreprocessor.h" />
     <ClInclude Include="..\html\parser\NestingLevelIncrementer.h" />
     <ClInclude Include="..\html\parser\TextDocumentParser.h" />

trunk/Source/WebCore/WebCore.vcxproj/WebCore.vcxproj.filters

@@ -12613 +12613 @@
       <Filter>html\parser</Filter>
     </ClInclude>
+    <ClInclude Include="..\html\parser\InputStreamPreprocessor.h">
+      <Filter>html\parser</Filter>
+    </ClInclude>
     <ClInclude Include="..\html\parser\NestingLevelIncrementer.h">
       <Filter>html\parser</Filter>

trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

@@ -1562 +1562 @@
                 510D4A37103165EE0049EA54 /* SocketStreamHandleBase.h in Headers */ = {isa = PBXBuildFile; fileRef = 510D4A31103165EE0049EA54 /* SocketStreamHandleBase.h */; };
                 510D4A38103165EE0049EA54 /* SocketStreamHandleClient.h in Headers */ = {isa = PBXBuildFile; fileRef = 510D4A32103165EE0049EA54 /* SocketStreamHandleClient.h */; };
+                5112935F3D54B4B52FAF973F /* InspectorHeapProfilerAgent.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 511293613D6DB4B52FAF973F /* InspectorHeapProfilerAgent.cpp */; };
+                511293603D60B4B52FAF973F /* InspectorHeapProfilerAgent.h in Headers */ = {isa = PBXBuildFile; fileRef = 511293623D85B4B52FAF973F /* InspectorHeapProfilerAgent.h */; };
                 511F23170DC160DA004F0032 /* StorageTask.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 511F23130DC160DA004F0032 /* StorageTask.cpp */; };
                 511F23180DC160DA004F0032 /* StorageTask.h in Headers */ = {isa = PBXBuildFile; fileRef = 511F23140DC160DA004F0032 /* StorageTask.h */; };
@@ -3816 +3818 @@
                 9F0D6B2E121BFEBA006C0288 /* InspectorProfilerAgent.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 9F0D6B2C121BFEBA006C0288 /* InspectorProfilerAgent.cpp */; };
                 9F0D6B2F121BFEBA006C0288 /* InspectorProfilerAgent.h in Headers */ = {isa = PBXBuildFile; fileRef = 9F0D6B2D121BFEBA006C0288 /* InspectorProfilerAgent.h */; };
-                5112935F3D54B4B52FAF973F /* InspectorHeapProfilerAgent.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 511293613D6DB4B52FAF973F /* InspectorHeapProfilerAgent.cpp */; };
-                511293603D60B4B52FAF973F /* InspectorHeapProfilerAgent.h in Headers */ = {isa = PBXBuildFile; fileRef = 511293623D85B4B52FAF973F /* InspectorHeapProfilerAgent.h */; };
                 9F3B947E12241758005304E7 /* ScriptHeapSnapshot.h in Headers */ = {isa = PBXBuildFile; fileRef = 9F3B947D12241758005304E7 /* ScriptHeapSnapshot.h */; };
                 9F6FC1961122E82A00E80196 /* ScriptDebugServer.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 9F6FC1941122E82A00E80196 /* ScriptDebugServer.cpp */; };
@@ -8977 +8977 @@
                 510D4A31103165EE0049EA54 /* SocketStreamHandleBase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SocketStreamHandleBase.h; sourceTree = "<group>"; };
                 510D4A32103165EE0049EA54 /* SocketStreamHandleClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SocketStreamHandleClient.h; sourceTree = "<group>"; };
+                511293613D6DB4B52FAF973F /* InspectorHeapProfilerAgent.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InspectorHeapProfilerAgent.cpp; sourceTree = "<group>"; };
+                511293623D85B4B52FAF973F /* InspectorHeapProfilerAgent.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InspectorHeapProfilerAgent.h; sourceTree = "<group>"; };
                 511F23130DC160DA004F0032 /* StorageTask.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StorageTask.cpp; sourceTree = "<group>"; };
                 511F23140DC160DA004F0032 /* StorageTask.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = StorageTask.h; sourceTree = "<group>"; };
@@ -11283 +11285 @@
                 9F0D6B2C121BFEBA006C0288 /* InspectorProfilerAgent.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InspectorProfilerAgent.cpp; sourceTree = "<group>"; };
                 9F0D6B2D121BFEBA006C0288 /* InspectorProfilerAgent.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InspectorProfilerAgent.h; sourceTree = "<group>"; };
-                511293613D6DB4B52FAF973F /* InspectorHeapProfilerAgent.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InspectorHeapProfilerAgent.cpp; sourceTree = "<group>"; };
-                511293623D85B4B52FAF973F /* InspectorHeapProfilerAgent.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InspectorHeapProfilerAgent.h; sourceTree = "<group>"; };
                 9F3B947D12241758005304E7 /* ScriptHeapSnapshot.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ScriptHeapSnapshot.h; sourceTree = "<group>"; };
                 9F6FC1941122E82A00E80196 /* ScriptDebugServer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ScriptDebugServer.cpp; sourceTree = "<group>"; };
@@ -11718 +11718 @@
                 A8748D6612CC3763001FBA41 /* ImageOrientation.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ImageOrientation.h; sourceTree = "<group>"; };
                 A8748D7412CC3F89001FBA41 /* ImageOrientation.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ImageOrientation.cpp; sourceTree = "<group>"; };
+                A8805AF116C96AEA000E9D98 /* InputStreamPreprocessor.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = InputStreamPreprocessor.h; path = parser/InputStreamPreprocessor.h; sourceTree = "<group>"; };
                 A882DA1F1593846A000115ED /* CSSToStyleMap.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CSSToStyleMap.cpp; sourceTree = "<group>"; };
                 A882DA221593848D000115ED /* CSSToStyleMap.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CSSToStyleMap.h; sourceTree = "<group>"; };
@@ -14865 +14866 @@
                         isa = PBXGroup;
                         children = (
-                                CDA98DCD1601515C00FEA3B1 /* EncryptedMedia */,
                                 A83B79100CCB001B000B0825 /* Core */,
                                 A83B790E0CCAFF97000B0825 /* CSS */,
+                                CDA98DCD1601515C00FEA3B1 /* EncryptedMedia */,
                                 A83B79120CCB003F000B0825 /* Events */,
                                 89878576122CA1DA003AABDA /* FileAPI */,
@@ -15570 +15571 @@
                                 A7BE7EDD14C9175A0014489D /* MallocStatistics.h */,
                                 A7BE7EDE14C9175A0014489D /* MallocStatistics.idl */,
+                                CDC26B3C160A62B00026757B /* MockCDM.cpp */,
+                                CDC26B3D160A62B00026757B /* MockCDM.h */,
                                 EB081CD81696084400553730 /* TypeConversions.h */,
                                 EB081CD91696084400553730 /* TypeConversions.idl */,
-                                CDC26B3C160A62B00026757B /* MockCDM.cpp */,
-                                CDC26B3D160A62B00026757B /* MockCDM.h */,
                         );
                         path = testing;
@@ -18749 +18750 @@
                                 977B3860122883E900B81FF8 /* HTMLViewSourceParser.cpp */,
                                 977B3861122883E900B81FF8 /* HTMLViewSourceParser.h */,
+                                A8805AF116C96AEA000E9D98 /* InputStreamPreprocessor.h */,
                                 8A309C9E123950BE00CB9204 /* NestingLevelIncrementer.h */,
                                 97BC84811236FD93000C6161 /* TextDocumentParser.cpp */,
@@ -21539 +21541 @@
                                 C5BAC16F14E30E4700008837 /* PlatformPasteboard.h */,
                                 BCEC01D60C274EB4009F4EC9 /* PlatformScreen.h */,
+                                2527CC9516BF95DD009DDAC0 /* PlatformSpeechSynthesisUtterance.cpp */,
+                                2527CC9116BF8BA1009DDAC0 /* PlatformSpeechSynthesisUtterance.h */,
+                                2527CC9316BF92EC009DDAC0 /* PlatformSpeechSynthesisVoice.cpp */,
+                                2527CC9216BF90B4009DDAC0 /* PlatformSpeechSynthesisVoice.h */,
                                 297BE3D916C043D8003316BD /* PlatformSpeechSynthesizer.cpp */,
                                 29E4D8DF16B0940F00C84704 /* PlatformSpeechSynthesizer.h */,
-                                2527CC9316BF92EC009DDAC0 /* PlatformSpeechSynthesisVoice.cpp */,
-                                2527CC9216BF90B4009DDAC0 /* PlatformSpeechSynthesisVoice.h */,
-                                2527CC9516BF95DD009DDAC0 /* PlatformSpeechSynthesisUtterance.cpp */,
-                                2527CC9116BF8BA1009DDAC0 /* PlatformSpeechSynthesisUtterance.h */,
                                 1AD8F81A11CAB9E900E93E54 /* PlatformStrategies.cpp */,
                                 1AD8F81911CAB9E900E93E54 /* PlatformStrategies.h */,
@@ -21634 +21636 @@
                         isa = PBXGroup;
                         children = (
+                                CDA98E091603CD5900FEA3B1 /* CDM.cpp */,
+                                CDA98E0A1603CD5900FEA3B1 /* CDM.h */,
                                 CDA98DD516025BED00FEA3B1 /* MediaKeyMessageEvent.cpp */,
                                 CDA98DD616025BED00FEA3B1 /* MediaKeyMessageEvent.h */,
@@ -21646 +21650 @@
                                 CDA98DC816014F2C00FEA3B1 /* MediaKeySession.h */,
                                 CDA98DC916014F4000FEA3B1 /* MediaKeySession.idl */,
-                                CDA98E091603CD5900FEA3B1 /* CDM.cpp */,
-                                CDA98E0A1603CD5900FEA3B1 /* CDM.h */,
                                 CDA98E0C1603CF3C00FEA3B1 /* Modules/encryptedmedia/CDMPrivate.h */,
                         );
@@ -23088 +23090 @@
                                 A740B59514C935AB00A77FA4 /* JSMallocStatistics.h in Headers */,
                                 EBF5121D1696496C0056BD25 /* JSTypeConversions.h in Headers */,
+                                CDC26B41160A8CCE0026757B /* MockCDM.h in Headers */,
                                 41815C1F138319830057AAA4 /* WebCoreTestSupport.h in Headers */,
-                                CDC26B41160A8CCE0026757B /* MockCDM.h in Headers */,
                         );
                         runOnlyForDeploymentPostprocessing = 0;
@@ -26823 +26825 @@
                                 A740B59714C935AF00A77FA4 /* JSMallocStatistics.cpp in Sources */,
                                 EBF5121C1696496C0056BD25 /* JSTypeConversions.cpp in Sources */,
+                                CDC26B40160A8CC60026757B /* MockCDM.cpp in Sources */,
                                 41815C1E138319830057AAA4 /* WebCoreTestSupport.cpp in Sources */,
-                                CDC26B40160A8CC60026757B /* MockCDM.cpp in Sources */,
                         );
                         runOnlyForDeploymentPostprocessing = 0;
@@ -26981 +26983 @@
                                 079D086C162F21F900DB8658 /* CaptionUserPreferencesMac.mm in Sources */,
                                 6550B69D099DF0270090D781 /* CDATASection.cpp in Sources */,
+                                CDA98E0B1603CD6000FEA3B1 /* CDM.cpp in Sources */,
                                 97BC69DC1505F076001B74AC /* ChangeVersionWrapper.cpp in Sources */,
                                 FD315FFE12B0267600C1A359 /* ChannelMergerNode.cpp in Sources */,
@@ -28292 +28295 @@
                                 FD23A12513F5FA5900F67001 /* JSMediaElementAudioSourceNode.cpp in Sources */,
                                 E44614180CD6826900FADA75 /* JSMediaError.cpp in Sources */,
+                                CDA98DA31601464100FEA3B1 /* JSMediaKeyError.cpp in Sources */,
                                 CDB859FA160D494900E5B07F /* JSMediaKeyEvent.cpp in Sources */,
+                                CDA98DDF16026A3700FEA3B1 /* JSMediaKeyMessageEvent.cpp in Sources */,
+                                CDA98DE016026A3700FEA3B1 /* JSMediaKeyNeededEvent.cpp in Sources */,
+                                CDA98E0416039E1A00FEA3B1 /* JSMediaKeys.cpp in Sources */,
+                                CDA98E0616039E1F00FEA3B1 /* JSMediaKeySession.cpp in Sources */,
                                 BC3C39B60C0D3D8D005F4D7A /* JSMediaList.cpp in Sources */,
                                 1464E06C135EC10600FDB00A /* JSMediaListCustom.cpp in Sources */,
@@ -28711 +28719 @@
                                 07A6D1EB1491137700051D0C /* MediaFragmentURIParser.cpp in Sources */,
                                 CDB859F7160D48A400E5B07F /* MediaKeyEvent.cpp in Sources */,
+                                CDA98DD816025BEF00FEA3B1 /* MediaKeyMessageEvent.cpp in Sources */,
+                                CD1B4A65160786AE00282DF9 /* MediaKeyNeededEvent.cpp in Sources */,
+                                CDA98E0D1603FE4A00FEA3B1 /* MediaKeys.cpp in Sources */,
+                                CDA98E0E1603FE5800FEA3B1 /* MediaKeySession.cpp in Sources */,
                                 A8EA80090A19516E00A8EF5F /* MediaList.cpp in Sources */,
                                 E44613E30CD6819F00FADA75 /* MediaPlayer.cpp in Sources */,
@@ -28856 +28868 @@
                                 C598902E14E9B0F800E8D18B /* PlatformPasteboardMac.mm in Sources */,
                                 BC94D1080C274F88006BC617 /* PlatformScreenMac.mm in Sources */,
+                                2934940B16C02C0A00901318 /* PlatformSpeechSynthesisUtterance.cpp in Sources */,
                                 2934940A16C02C0700901318 /* PlatformSpeechSynthesisVoice.cpp in Sources */,
-                                2934940B16C02C0A00901318 /* PlatformSpeechSynthesisUtterance.cpp in Sources */,
+                                297BE3DA16C043D8003316BD /* PlatformSpeechSynthesizer.cpp in Sources */,
                                 297BE3D816C03CCE003316BD /* PlatformSpeechSynthesizerMac.mm in Sources */,
-                                297BE3DA16C043D8003316BD /* PlatformSpeechSynthesizer.cpp in Sources */,
                                 1AD8F81C11CAB9E900E93E54 /* PlatformStrategies.cpp in Sources */,
                                 A9C6E4F30D745E48006442E9 /* PluginData.cpp in Sources */,
@@ -29672 +29684 @@
                                 977E2E0E12F0FC9C00C13380 /* XSSAuditorDelegate.cpp in Sources */,
                                 FD537352137B651800008DCE /* ZeroPole.cpp in Sources */,
-                                CDA98DA31601464100FEA3B1 /* JSMediaKeyError.cpp in Sources */,
-                                CDA98DD816025BEF00FEA3B1 /* MediaKeyMessageEvent.cpp in Sources */,
-                                CDA98DDF16026A3700FEA3B1 /* JSMediaKeyMessageEvent.cpp in Sources */,
-                                CDA98DE016026A3700FEA3B1 /* JSMediaKeyNeededEvent.cpp in Sources */,
-                                CDA98E0416039E1A00FEA3B1 /* JSMediaKeys.cpp in Sources */,
-                                CDA98E0616039E1F00FEA3B1 /* JSMediaKeySession.cpp in Sources */,
-                                CDA98E0B1603CD6000FEA3B1 /* CDM.cpp in Sources */,
-                                CDA98E0D1603FE4A00FEA3B1 /* MediaKeys.cpp in Sources */,
-                                CDA98E0E1603FE5800FEA3B1 /* MediaKeySession.cpp in Sources */,
-                                CD1B4A65160786AE00282DF9 /* MediaKeyNeededEvent.cpp in Sources */,
                         );
                         runOnlyForDeploymentPostprocessing = 0;

trunk/Source/WebCore/html/parser/BackgroundHTMLParser.cpp

@@ -107 +107 @@
 void BackgroundHTMLParser::markEndOfFile()
 {
-    // FIXME: This should use InputStreamPreprocessor::endOfFileMarker
-    // once InputStreamPreprocessor is split off into its own header.
-    const LChar endOfFileMarker = 0;
-
     ASSERT(!m_input.current().isClosed());
-    m_input.append(String(&endOfFileMarker, 1));
+    m_input.append(String(&kEndOfFileMarker, 1));
     m_input.close();
 }

trunk/Source/WebCore/html/parser/HTMLInputStream.h

@@ -27 +27 @@
 #define HTMLInputStream_h
 
+#include "InputStreamPreprocessor.h"
 #include "SegmentedString.h"
 
@@ -73 +74 @@
     void markEndOfFile()
     {
-        // FIXME: This should use InputStreamPreprocessor::endOfFileMarker
-        // once InputStreamPreprocessor is split off into its own header.
-        static const LChar endOfFileMarker = 0;
-        m_last->append(SegmentedString(String(&endOfFileMarker, 1)));
+        m_last->append(SegmentedString(String(&kEndOfFileMarker, 1)));
         m_last->close();
     }

trunk/Source/WebCore/html/parser/HTMLTokenizer.cpp

@@ -254 +254 @@
             }
             HTML_ADVANCE_TO(TagOpenState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        } else if (cc == kEndOfFileMarker)
             return emitEndOfFile(source);
         else {
@@ -275 +275 @@
         else if (cc == '<')
             HTML_ADVANCE_TO(RCDATALessThanSignState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        else if (cc == kEndOfFileMarker)
             return emitEndOfFile(source);
         else {
@@ -294 +294 @@
         if (cc == '<')
             HTML_ADVANCE_TO(RAWTEXTLessThanSignState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        else if (cc == kEndOfFileMarker)
             return emitEndOfFile(source);
         else {
@@ -306 +306 @@
         if (cc == '<')
             HTML_ADVANCE_TO(ScriptDataLessThanSignState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        else if (cc == kEndOfFileMarker)
             return emitEndOfFile(source);
         else {
@@ -316 +316 @@
 
     HTML_BEGIN_STATE(PLAINTEXTState) {
-        if (cc == InputStreamPreprocessor::endOfFileMarker)
+        if (cc == kEndOfFileMarker)
             return emitEndOfFile(source);
-        else
-            bufferCharacter(cc);
+        bufferCharacter(cc);
         HTML_ADVANCE_TO(PLAINTEXTState);
     }
@@ -361 +360 @@
             parseError();
             HTML_ADVANCE_TO(DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             bufferCharacter('<');
@@ -385 +384 @@
             m_token->appendToName(toLowerCase(cc));
             HTML_ADVANCE_TO(TagNameState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -618 +617 @@
         } else if (cc == '<')
             HTML_ADVANCE_TO(ScriptDataEscapedLessThanSignState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -634 +633 @@
         } else if (cc == '<')
             HTML_ADVANCE_TO(ScriptDataEscapedLessThanSignState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -653 +652 @@
             bufferCharacter(cc);
             HTML_ADVANCE_TO(ScriptDataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -767 +766 @@
             bufferCharacter(cc);
             HTML_ADVANCE_TO(ScriptDataDoubleEscapedLessThanSignState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -784 +783 @@
             bufferCharacter(cc);
             HTML_ADVANCE_TO(ScriptDataDoubleEscapedLessThanSignState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -804 +803 @@
             bufferCharacter(cc);
             HTML_ADVANCE_TO(ScriptDataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -858 +857 @@
             m_token->appendToAttributeName(toLowerCase(cc));
             HTML_ADVANCE_TO(AttributeNameState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -891 +890 @@
             m_token->appendToAttributeName(toLowerCase(cc));
             HTML_ADVANCE_TO(AttributeNameState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->endAttributeName(source.numberOfCharactersConsumed());
@@ -920 +919 @@
             m_token->appendToAttributeName(toLowerCase(cc));
             HTML_ADVANCE_TO(AttributeNameState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -949 +948 @@
             parseError();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -969 +968 @@
             m_additionalAllowedCharacter = '"';
             HTML_ADVANCE_TO(CharacterReferenceInAttributeValueState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->endAttributeValue(source.numberOfCharactersConsumed());
@@ -987 +986 @@
             m_additionalAllowedCharacter = '\'';
             HTML_ADVANCE_TO(CharacterReferenceInAttributeValueState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->endAttributeValue(source.numberOfCharactersConsumed());
@@ -1008 +1007 @@
             m_token->endAttributeValue(source.numberOfCharactersConsumed());
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->endAttributeValue(source.numberOfCharactersConsumed());
@@ -1058 +1057 @@
         else if (m_options.usePreHTML5ParserQuirks && cc == '<')
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -1072 +1071 @@
             m_token->setSelfClosing();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             HTML_RECONSUME_IN(DataState);
@@ -1091 +1090 @@
         if (cc == '>')
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        else if (cc == kEndOfFileMarker)
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
         else {
@@ -1139 +1138 @@
             parseError();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
@@ -1155 +1154 @@
             parseError();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
@@ -1169 +1168 @@
         if (cc == '-')
             HTML_ADVANCE_TO(CommentEndDashState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
@@ -1182 +1181 @@
         if (cc == '-')
             HTML_ADVANCE_TO(CommentEndState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
@@ -1203 +1202 @@
             m_token->appendToComment('-');
             HTML_ADVANCE_TO(CommentEndState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
@@ -1224 +1223 @@
         } else if (cc == '>')
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
@@ -1240 +1239 @@
         if (isTokenizerWhitespace(cc))
             HTML_ADVANCE_TO(BeforeDOCTYPENameState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->beginDOCTYPE();
@@ -1263 +1262 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->beginDOCTYPE();
@@ -1283 +1282 @@
             m_token->appendToName(toLowerCase(cc));
             HTML_ADVANCE_TO(DOCTYPENameState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1299 +1298 @@
         if (cc == '>')
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1343 +1342 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1368 +1367 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1387 +1386 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1405 +1404 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1429 +1428 @@
             m_token->setSystemIdentifierToEmptyString();
             HTML_ADVANCE_TO(DOCTYPESystemIdentifierSingleQuotedState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1452 +1451 @@
             m_token->setSystemIdentifierToEmptyString();
             HTML_ADVANCE_TO(DOCTYPESystemIdentifierSingleQuotedState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1479 +1478 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1504 +1503 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1523 +1522 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1541 +1540 @@
             m_token->setForceQuirks();
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1557 +1556 @@
         else if (cc == '>')
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        else if (cc == kEndOfFileMarker) {
             parseError();
             m_token->setForceQuirks();
@@ -1571 +1570 @@
         if (cc == '>')
             return emitAndResumeIn(source, HTMLTokenizerState::DataState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        else if (cc == kEndOfFileMarker)
             return emitAndReconsumeIn(source, HTMLTokenizerState::DataState);
         HTML_ADVANCE_TO(BogusDOCTYPEState);
@@ -1580 +1579 @@
         if (cc == ']')
             HTML_ADVANCE_TO(CDATASectionRightSquareBracketState);
-        else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        else if (cc == kEndOfFileMarker)
             HTML_RECONSUME_IN(DataState);
         else {

trunk/Source/WebCore/html/track/WebVTTTokenizer.cpp

@@ -43 +43 @@
 
 WebVTTTokenizer::WebVTTTokenizer()
+    : m_inputStreamPreprocessor(this)
 {
     reset();
-}
-
-template<>
-inline bool MarkupTokenizerBase<WebVTTToken, WebVTTTokenizerState>::shouldSkipNullCharacters() const
-{
-    return true;
 }
 
@@ -96 +91 @@
             else
                 return emitAndResumeIn(source, WebVTTTokenizerState::TagState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker)
+        } else if (cc == kEndOfFileMarker)
             return emitEndOfFile(source);
         else {
@@ -122 +117 @@
             m_buffer.append(static_cast<LChar>(cc));
             WEBVTT_ADVANCE_TO(EscapeState);
-        } else if (cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == kEndOfFileMarker) {
             m_token->appendToCharacter(m_buffer);
             return emitEndOfFile(source);
@@ -146 +141 @@
             m_token->beginTimestampTag(cc);
             WEBVTT_ADVANCE_TO(TimestampTagState);
-        } else if (cc == '>' || cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == '>' || cc == kEndOfFileMarker) {
             m_token->beginEmptyStartTag();
             return emitAndResumeIn(source, WebVTTTokenizerState::DataState);
@@ -161 +156 @@
         else if (cc == '.')
             WEBVTT_ADVANCE_TO(StartTagClassState);
-        else if (cc == '>' || cc == InputStreamPreprocessor::endOfFileMarker)
+        else if (cc == '>' || cc == kEndOfFileMarker)
             return emitAndResumeIn(source, WebVTTTokenizerState::DataState);
         else {
@@ -177 +172 @@
             m_token->addNewClass();
             WEBVTT_ADVANCE_TO(StartTagClassState);
-        } else if (cc == '>' || cc == InputStreamPreprocessor::endOfFileMarker) {
+        } else if (cc == '>' || cc == kEndOfFileMarker) {
             m_token->addNewClass();
             return emitAndResumeIn(source, WebVTTTokenizerState::DataState);
@@ -189 +184 @@
 
     WEBVTT_BEGIN_STATE(StartTagAnnotationState) {
-        if (cc == '>' || cc == InputStreamPreprocessor::endOfFileMarker) {
+        if (cc == '>' || cc == kEndOfFileMarker) {
             m_token->addNewAnnotation();
             return emitAndResumeIn(source, WebVTTTokenizerState::DataState);
@@ -199 +194 @@
     
     WEBVTT_BEGIN_STATE(EndTagOpenState) {
-        if (cc == '>' || cc == InputStreamPreprocessor::endOfFileMarker) {
+        if (cc == '>' || cc == kEndOfFileMarker) {
             m_token->beginEndTag('\0');
             return emitAndResumeIn(source, WebVTTTokenizerState::DataState);
@@ -209 +204 @@
 
     WEBVTT_BEGIN_STATE(EndTagState) {
-        if (cc == '>' || cc == InputStreamPreprocessor::endOfFileMarker)
+        if (cc == '>' || cc == kEndOfFileMarker)
             return emitAndResumeIn(source, WebVTTTokenizerState::DataState);
         m_token->appendToName(cc);
@@ -217 +212 @@
 
     WEBVTT_BEGIN_STATE(TimestampTagState) {
-        if (cc == '>' || cc == InputStreamPreprocessor::endOfFileMarker)
+        if (cc == '>' || cc == kEndOfFileMarker)
             return emitAndResumeIn(source, WebVTTTokenizerState::DataState);
         m_token->appendToTimestamp(cc);

trunk/Source/WebCore/html/track/WebVTTTokenizer.h

@@ -34 +34 @@
 #if ENABLE(VIDEO_TRACK)
 
-#include "MarkupTokenizerBase.h"
-#include "SegmentedString.h"
+#include "InputStreamPreprocessor.h"
 #include "WebVTTToken.h"
+#include <wtf/PassOwnPtr.h>
 
 namespace WebCore {
@@ -55 +55 @@
 };
 
-class WebVTTTokenizer : MarkupTokenizerBase<WebVTTToken, WebVTTTokenizerState> {
+class WebVTTTokenizer {
     WTF_MAKE_NONCOPYABLE(WebVTTTokenizer);
     WTF_MAKE_FAST_ALLOCATED;
@@ -61 +61 @@
     static PassOwnPtr<WebVTTTokenizer> create() { return adoptPtr(new WebVTTTokenizer); }
 
+    typedef WebVTTTokenizerState State;
+
     void reset();
     
     bool nextToken(SegmentedString&, WebVTTToken&);
 
+    inline bool haveBufferedCharacterToken()
+    {
+        return m_token->type() == WebVTTToken::Type::Character;
+    }
+
+    inline void bufferCharacter(UChar character)
+    {
+        ASSERT(character != kEndOfFileMarker);
+        m_token->ensureIsCharacterToken();
+        m_token->appendToCharacter(character);
+    }
+
+    inline bool emitAndResumeIn(SegmentedString& source, State::State state)
+    {
+        m_state = state;
+        source.advanceAndUpdateLineNumber();
+        return true;
+    }
+
+    inline bool emitEndOfFile(SegmentedString& source)
+    {
+        if (haveBufferedCharacterToken())
+            return true;
+        m_state = State::DataState;
+        source.advanceAndUpdateLineNumber();
+        m_token->clear();
+        m_token->makeEndOfFile();
+        return true;
+    }
+
+    bool shouldSkipNullCharacters() const { return true; }
+
 private:
     WebVTTTokenizer();
-    
+
+    // m_token is owned by the caller. If nextToken is not on the stack,
+    // this member might be pointing to unallocated memory.
+    WebVTTToken* m_token;
+    WebVTTTokenizerState::State m_state;
+
     Vector<LChar, 32> m_buffer;
+
+    // ://www.whatwg.org/specs/web-apps/current-work/#preprocessing-the-input-stream
+    InputStreamPreprocessor<WebVTTTokenizer> m_inputStreamPreprocessor;
 };
 

trunk/Source/WebCore/xml/parser/MarkupTokenizerBase.h

@@ -29 +29 @@
 #define MarkupTokenizerBase_h
 
-#include "SegmentedString.h"
-#include <wtf/Noncopyable.h>
+#include "InputStreamPreprocessor.h"
 #include <wtf/PassOwnPtr.h>
 #include <wtf/Vector.h>
@@ -56 +55 @@
 
 protected:
-    // http://www.whatwg.org/specs/web-apps/current-work/#preprocessing-the-input-stream
-    class InputStreamPreprocessor {
-        WTF_MAKE_NONCOPYABLE(InputStreamPreprocessor);
-    public:
-        InputStreamPreprocessor(MarkupTokenizerBase<Token, State>* tokenizer)
-            : m_tokenizer(tokenizer)
-        {
-            reset();
-        }
-
-        ALWAYS_INLINE UChar nextInputCharacter() const { return m_nextInputCharacter; }
-
-        // Returns whether we succeeded in peeking at the next character.
-        // The only way we can fail to peek is if there are no more
-        // characters in |source| (after collapsing \r\n, etc).
-        ALWAYS_INLINE bool peek(SegmentedString& source)
-        {
-        PeekAgain:
-            m_nextInputCharacter = source.currentChar();
-
-            // Every branch in this function is expensive, so we have a
-            // fast-reject branch for characters that don't require special
-            // handling. Please run the parser benchmark whenever you touch
-            // this function. It's very hot.
-            static const UChar specialCharacterMask = '\n' | '\r' | '\0';
-            if (m_nextInputCharacter & ~specialCharacterMask) {
-                m_skipNextNewLine = false;
-                return true;
-            }
-
-            if (m_nextInputCharacter == '\n' && m_skipNextNewLine) {
-                m_skipNextNewLine = false;
-                source.advancePastNewlineAndUpdateLineNumber();
-                if (source.isEmpty())
-                    return false;
-                m_nextInputCharacter = source.currentChar();
-            }
-            if (m_nextInputCharacter == '\r') {
-                m_nextInputCharacter = '\n';
-                m_skipNextNewLine = true;
-            } else {
-                m_skipNextNewLine = false;
-                // FIXME: The spec indicates that the surrogate pair range as well as
-                // a number of specific character values are parse errors and should be replaced
-                // by the replacement character. We suspect this is a problem with the spec as doing
-                // that filtering breaks surrogate pair handling and causes us not to match Minefield.
-                if (m_nextInputCharacter == '\0' && !shouldTreatNullAsEndOfFileMarker(source)) {
-                    if (m_tokenizer->shouldSkipNullCharacters()) {
-                        source.advancePastNonNewline();
-                        if (source.isEmpty())
-                            return false;
-                        goto PeekAgain;
-                    }
-                    m_nextInputCharacter = 0xFFFD;
-                }
-            }
-            return true;
-        }
-
-        // Returns whether there are more characters in |source| after advancing.
-        ALWAYS_INLINE bool advance(SegmentedString& source)
-        {
-            source.advanceAndUpdateLineNumber();
-            if (source.isEmpty())
-                return false;
-            return peek(source);
-        }
-
-        bool skipNextNewLine() const { return m_skipNextNewLine; }
-
-        void reset(bool skipNextNewLine = false)
-        {
-            m_nextInputCharacter = '\0';
-            m_skipNextNewLine = skipNextNewLine;
-        }
-
-        static const UChar endOfFileMarker = 0;
-
-    private:
-        bool shouldTreatNullAsEndOfFileMarker(SegmentedString& source) const
-        {
-            return source.isClosed() && source.length() == 1;
-        }
-
-        MarkupTokenizerBase<Token, State>* m_tokenizer;
-
-        // http://www.whatwg.org/specs/web-apps/current-work/#next-input-character
-        UChar m_nextInputCharacter;
-        bool m_skipNextNewLine;
-    };
-
     MarkupTokenizerBase() : m_inputStreamPreprocessor(this) { reset(); }
 
     inline void bufferCharacter(UChar character)
     {
-        ASSERT(character != InputStreamPreprocessor::endOfFileMarker);
+        ASSERT(character != kEndOfFileMarker);
         m_token->ensureIsCharacterToken();
         m_token->appendToCharacter(character);
@@ -207 +115 @@
 
     // http://www.whatwg.org/specs/web-apps/current-work/#preprocessing-the-input-stream
-    InputStreamPreprocessor m_inputStreamPreprocessor;
+    InputStreamPreprocessor<MarkupTokenizerBase<Token, State> > m_inputStreamPreprocessor;
 };