Changeset 206869 in webkit


Ignore:
Timestamp:
Oct 6, 2016 10:40:12 AM (8 years ago)
Author:
wilander@apple.com
Message:

Update Resource Load Statistics
​https://bugs.webkit.org/show_bug.cgi?id=162811

Reviewed by Alex Christensen.

Source/WebCore:

No new tests. The counting is based on top privately owned domains
which currently is not supported by layout tests nor API tests.

  • Modules/websockets/WebSocket.cpp:

(WebCore::WebSocket::connect):

Now captures statistics for web sockets too.

  • loader/FrameLoader.cpp:

(WebCore::FrameLoader::loadResourceSynchronously):

  • loader/ResourceLoadObserver.cpp:

Now captures statistics for synchronous XHR too.

(WebCore::is3xxRedirect):

Convenience function.

(WebCore::ResourceLoadObserver::shouldLog):

Convenience function.

(WebCore::ResourceLoadObserver::logFrameNavigation):

Updated to make use of new convenience functions.

(WebCore::ResourceLoadObserver::logSubresourceLoading):

Updated to make use of new convenience functions.

(WebCore::ResourceLoadObserver::logWebSocketLoading):

Added.

(WebCore::ResourceLoadObserver::logUserInteraction):

Updated to make use of new convenience functions.

(WebCore::ResourceLoadObserver::primaryDomain):

Now makes use of the Public Suffix list.
Removed old custom parsing of primary domain.

  • loader/ResourceLoadObserver.h:
  • loader/ResourceLoadStatisticsStore.cpp:

(WebCore::ResourceLoadStatisticsStore::prevalentResourceDomainsWithoutUserInteraction):

Convenience function.

(WebCore::ResourceLoadStatisticsStore::processStatistics): Deleted.

  • loader/ResourceLoadStatisticsStore.h:
  • loader/SubresourceLoader.cpp:

(WebCore::SubresourceLoader::willSendRequestInternal):

Moved logging call higher up and added a check for whether we
are loading the main resource. The reason for moving it up is
to capture the request before some data may be cleared out in
redirect handling. We also want to capture failed CORS requests
since they are sent and then cancelled on the way back.

Source/WebKit2:

  • UIProcess/WebResourceLoadStatisticsStore.cpp:

(WebKit::WebResourceLoadStatisticsStore::hasPrevalentResourceCharacteristics):

Switched to vector-based classification.

(WebKit::WebResourceLoadStatisticsStore::classifyResource):

Simplified logic and moved the split between has and has
no user interaction into ResourceLoadStatisticsStore.

(WebKit::WebResourceLoadStatisticsStore::clearDataRecords):

Added.

(WebKit::WebResourceLoadStatisticsStore::resourceLoadStatisticsUpdated):

Updated to make use of the new functions.

(WebKit::WebResourceLoadStatisticsStore::persistentStoragePath):

Removed stray whitespace.

(WebKit::WebResourceLoadStatisticsStore::writeEncoderToDisk):

Removed stray whitespace.

(WebKit::WebResourceLoadStatisticsStore::createDecoderFromDisk):

Removed stray whitespace.

(WebKit::hasPrevalentResourceCharacteristics): Deleted.
(WebKit::classifyPrevalentResources): Deleted.

  • UIProcess/WebResourceLoadStatisticsStore.h:

Added member variables for clearing of data records.

Tools:

  • TestWebKitAPI/Tests/mac/PublicSuffix.mm:

Change from USE(PUBLIC_SUFFIX_LIST) to ENABLE(PUBLIC_SUFFIX_LIST)

Location:
trunk
Files:
13 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r206868 r206869  
     12016-10-06  John Wilander  <wilander@apple.com>
     2
     3        Update Resource Load Statistics
     4        https://bugs.webkit.org/show_bug.cgi?id=162811
     5
     6        Reviewed by Alex Christensen.
     7
     8        No new tests. The counting is based on top privately owned domains
     9        which currently is not supported by layout tests nor API tests.
     10
     11        * Modules/websockets/WebSocket.cpp:
     12        (WebCore::WebSocket::connect):
     13            Now captures statistics for web sockets too.
     14        * loader/FrameLoader.cpp:
     15        (WebCore::FrameLoader::loadResourceSynchronously):
     16        * loader/ResourceLoadObserver.cpp:
     17            Now captures statistics for synchronous XHR too.
     18        (WebCore::is3xxRedirect):
     19            Convenience function.
     20        (WebCore::ResourceLoadObserver::shouldLog):
     21            Convenience function.
     22        (WebCore::ResourceLoadObserver::logFrameNavigation):
     23            Updated to make use of new convenience functions.
     24        (WebCore::ResourceLoadObserver::logSubresourceLoading):
     25            Updated to make use of new convenience functions.
     26        (WebCore::ResourceLoadObserver::logWebSocketLoading):
     27            Added.
     28        (WebCore::ResourceLoadObserver::logUserInteraction):
     29            Updated to make use of new convenience functions.
     30        (WebCore::ResourceLoadObserver::primaryDomain):
     31            Now makes use of the Public Suffix list.
     32            Removed old custom parsing of primary domain.
     33        * loader/ResourceLoadObserver.h:
     34        * loader/ResourceLoadStatisticsStore.cpp:
     35        (WebCore::ResourceLoadStatisticsStore::prevalentResourceDomainsWithoutUserInteraction):
     36            Convenience function.
     37        (WebCore::ResourceLoadStatisticsStore::processStatistics): Deleted.
     38        * loader/ResourceLoadStatisticsStore.h:
     39        * loader/SubresourceLoader.cpp:
     40        (WebCore::SubresourceLoader::willSendRequestInternal):
     41            Moved logging call higher up and added a check for whether we
     42            are loading the main resource. The reason for moving it up is
     43            to capture the request before some data may be cleared out in
     44            redirect handling. We also want to capture failed CORS requests
     45            since they are sent and then cancelled on the way back.
     46
    1472016-10-06  Adam Bergkvist  <adam.bergkvist@ericsson.com>
    248

  • trunk/Source/WebCore/Modules/websockets/WebSocket.cpp

    r206254 r206869  
    4848#include "Logging.h"
    4949#include "MessageEvent.h"
     50#include "ResourceLoadObserver.h"
    5051#include "ScriptController.h"
    5152#include "ScriptExecutionContext.h"
    … …  
    319320#endif
    320321            return;
    321         }
     322        } else
     323            ResourceLoadObserver::sharedObserver().logWebSocketLoading(document.frame(), m_url);
    322324    }
    323325

  • trunk/Source/WebCore/loader/FrameLoader.cpp

    r206254 r206869  
    9898#include "ResourceHandle.h"
    9999#include "ResourceLoadInfo.h"
     100#include "ResourceLoadObserver.h"
    100101#include "ResourceRequest.h"
    101102#include "SVGDocument.h"
    … …  
    27752776            data = SharedBuffer::adoptVector(buffer);
    27762777            documentLoader()->applicationCacheHost()->maybeLoadFallbackSynchronously(newRequest, error, response, data);
     2778            ResourceLoadObserver::sharedObserver().logSubresourceLoading(&m_frame, newRequest, response);
    27772779        }
    27782780    }

  • trunk/Source/WebCore/loader/ResourceLoadObserver.cpp

    r204733 r206869  
    3434#include "Page.h"
    3535#include "PlatformStrategies.h"
     36#include "PublicSuffix.h"
    3637#include "ResourceLoadStatistics.h"
    3738#include "ResourceLoadStatisticsStore.h"
    … …  
    5859}
    5960
     61static inline bool is3xxRedirect(const ResourceResponse& response)
     62{
     63    return response.httpStatusCode() >= 300 && response.httpStatusCode() <= 399;
     64}
     65
     66bool ResourceLoadObserver::shouldLog(Page* page)
     67{
     68    // FIXME: Err on the safe side until we have sorted out what to do in worker contexts
     69    if (!page)
     70        return false;
     71    return Settings::resourceLoadStatisticsEnabled()
     72        && !page->usesEphemeralSession()
     73        && m_store;
     74}
     75
    6076void ResourceLoadObserver::logFrameNavigation(const Frame& frame, const Frame& topFrame, const ResourceRequest& newRequest, const ResourceResponse& redirectResponse)
    6177{
    62     if (!Settings::resourceLoadStatisticsEnabled())
    63         return;
    64 
    65     if (!m_store)
    66         return;
    67 
    6878    ASSERT(frame.document());
    6979    ASSERT(topFrame.document());
    7080    ASSERT(topFrame.page());
    71 
    72     bool needPrivacy = topFrame.page() ? topFrame.page()->usesEphemeralSession() : false;
    73     if (needPrivacy)
    74         return;
    75 
    76     bool isRedirect = !redirectResponse.isNull();
     81   
     82    if (!shouldLog(topFrame.page()))
     83        return;
     84
     85    bool isRedirect = is3xxRedirect(redirectResponse);
    7786    bool isMainFrame = frame.isMainFrame();
    7887    const URL& sourceURL = frame.document()->url();
    … …  
    149158void ResourceLoadObserver::logSubresourceLoading(const Frame* frame, const ResourceRequest& newRequest, const ResourceResponse& redirectResponse)
    150159{
    151     if (!Settings::resourceLoadStatisticsEnabled())
    152         return;
    153 
    154     if (!m_store)
    155         return;
    156 
    157     bool needPrivacy = (frame && frame->page()) ? frame->page()->usesEphemeralSession() : false;
    158     if (needPrivacy)
    159         return;
    160    
    161     bool isRedirect = !redirectResponse.isNull();
     160    ASSERT(frame->page());
     161
     162    if (!shouldLog(frame->page()))
     163        return;
     164
     165    bool isRedirect = is3xxRedirect(redirectResponse);
    162166    const URL& sourceURL = redirectResponse.url();
    163167    const URL& targetURL = newRequest.url();
    … …  
    167171    auto mainFrameHost = mainFrameURL.host();
    168172
    169     if (targetHost.isEmpty() || mainFrameHost.isEmpty() || targetHost == mainFrameHost || targetHost == sourceURL.host())
     173    if (targetHost.isEmpty()
     174        || mainFrameHost.isEmpty()
     175        || targetHost == mainFrameHost
     176        || (isRedirect && targetHost == sourceURL.host()))
    170177        return;
    171178
    … …  
    174181    auto sourcePrimaryDomain = primaryDomain(sourceURL);
    175182   
    176     if (targetPrimaryDomain == mainFramePrimaryDomain || targetPrimaryDomain == sourcePrimaryDomain)
     183    if (targetPrimaryDomain == mainFramePrimaryDomain || (isRedirect && targetPrimaryDomain == sourcePrimaryDomain))
    177184        return;
    178185
    … …  
    211218    m_store->fireDataModificationHandler();
    212219}
    213    
     220
     221void ResourceLoadObserver::logWebSocketLoading(const Frame* frame, const URL& targetURL)
     222{
     223    // FIXME: Web sockets can run in detached frames. Decide how to count such connections.
     224    // See LayoutTests/http/tests/websocket/construct-in-detached-frame.html
     225    if (!frame)
     226        return;
     227
     228    if (!shouldLog(frame->page()))
     229        return;
     230
     231    const URL& mainFrameURL = frame->mainFrame().document()->url();
     232
     233    auto targetHost = targetURL.host();
     234    auto mainFrameHost = mainFrameURL.host();
     235   
     236    if (targetHost.isEmpty()
     237        || mainFrameHost.isEmpty()
     238        || targetHost == mainFrameHost)
     239        return;
     240   
     241    auto targetPrimaryDomain = primaryDomain(targetURL);
     242    auto mainFramePrimaryDomain = primaryDomain(mainFrameURL);
     243   
     244    if (targetPrimaryDomain == mainFramePrimaryDomain)
     245        return;
     246
     247    auto& targetStatistics = m_store->ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain);
     248   
     249    auto mainFrameOrigin = SecurityOrigin::create(mainFrameURL);
     250    targetStatistics.subresourceUnderTopFrameOrigins.add(mainFramePrimaryDomain);
     251   
     252    ++targetStatistics.subresourceHasBeenSubresourceCount;
     253   
     254    auto totalVisited = std::max(m_originsVisitedMap.size(), 1U);
     255   
     256    targetStatistics.subresourceHasBeenSubresourceCountDividedByTotalNumberOfOriginsVisited = static_cast<double>(targetStatistics.subresourceHasBeenSubresourceCount) / totalVisited;
     257
     258    m_store->fireDataModificationHandler();
     259}
     260
    214261void ResourceLoadObserver::logUserInteraction(const Document& document)
    215262{
    216     if (!Settings::resourceLoadStatisticsEnabled())
    217         return;
    218 
    219     if (!m_store)
    220         return;
    221 
    222     bool needPrivacy = document.page() ? document.page()->usesEphemeralSession() : false;
    223     if (needPrivacy)
    224         return;
    225 
    226     auto& statistics = m_store->ensureResourceStatisticsForPrimaryDomain(primaryDomain(document.url()));
     263    ASSERT(document.page());
     264
     265    if (!shouldLog(document.page()))
     266        return;
     267
     268    auto& url = document.url();
     269
     270    if (url.isBlankURL() || url.isEmpty())
     271        return;
     272
     273    auto& statistics = m_store->ensureResourceStatisticsForPrimaryDomain(primaryDomain(url));
    227274    statistics.hadUserInteraction = true;
    228275    m_store->fireDataModificationHandler();
    … …  
    231278String ResourceLoadObserver::primaryDomain(const URL& url)
    232279{
     280    String primaryDomain;
    233281    String host = url.host();
    234     Vector<String> hostSplitOnDot;
    235    
    236     host.split('.', false, hostSplitOnDot);
    237 
    238     String primaryDomain;
    239     if (host.isNull())
     282    if (host.isNull() || host.isEmpty())
    240283        primaryDomain = "nullOrigin";
    241     else if (hostSplitOnDot.size() < 3)
     284#if ENABLE(PUBLIC_SUFFIX_LIST)
     285    else {
     286        primaryDomain = topPrivatelyControlledDomain(host);
     287        // We will have an empty string here if there is no TLD.
     288        // Use the host in such case.
     289        if (primaryDomain.isEmpty())
     290            primaryDomain = host;
     291    }
     292#else
     293    else
    242294        primaryDomain = host;
    243     else {
    244         // Skip TLD and then up to two domains smaller than 4 characters
    245         int primaryDomainCutOffIndex = hostSplitOnDot.size() - 2;
    246 
    247         // Start with TLD as a given part
    248         size_t numberOfParts = 1;
    249         for (; primaryDomainCutOffIndex >= 0; --primaryDomainCutOffIndex) {
    250             ++numberOfParts;
    251 
    252             // We have either a domain part that's 4 chars or longer, or 3 domain parts including TLD
    253             if (hostSplitOnDot.at(primaryDomainCutOffIndex).length() >= 4 || numberOfParts >= 3)
    254                 break;
    255         }
    256 
    257         if (primaryDomainCutOffIndex < 0)
    258             primaryDomain = host;
    259         else {
    260             StringBuilder builder;
    261             builder.append(hostSplitOnDot.at(primaryDomainCutOffIndex));
    262             for (size_t j = primaryDomainCutOffIndex + 1; j < hostSplitOnDot.size(); ++j) {
    263                 builder.append('.');
    264                 builder.append(hostSplitOnDot[j]);
    265             }
    266             primaryDomain = builder.toString();
    267         }
    268     }
     295#endif
    269296
    270297    return primaryDomain;

  • trunk/Source/WebCore/loader/ResourceLoadObserver.h

    r197608 r206869  
    3535class Document;
    3636class Frame;
     37class Page;
    3738class ResourceRequest;
    3839class ResourceResponse;
    … …  
    4849    void logFrameNavigation(const Frame& frame, const Frame& topFrame, const ResourceRequest& newRequest, const ResourceResponse& redirectResponse);
    4950    void logSubresourceLoading(const Frame*, const ResourceRequest& newRequest, const ResourceResponse& redirectResponse);
     51    void logWebSocketLoading(const Frame*, const URL&);
    5052
    5153    void logUserInteraction(const Document&);
    … …  
    5658
    5759private:
     60    bool shouldLog(Page*);
    5861    static String primaryDomain(const URL&);
    5962

  • trunk/Source/WebCore/loader/ResourceLoadStatisticsStore.cpp

    r204466 r206869  
    155155        processFunction(resourceStatistic);
    156156}
     157
     158Vector<String> ResourceLoadStatisticsStore::prevalentResourceDomainsWithoutUserInteraction()
     159{
     160    Vector<String> prevalentResources;
     161    for (auto& resourceStatistic : m_resourceStatisticsMap.values()) {
     162        if (resourceStatistic.isPrevalentResource && !resourceStatistic.hadUserInteraction)
     163            prevalentResources.append(resourceStatistic.highLevelDomain);
     164    }
     165    return prevalentResources;
    157166}
     167}

  • trunk/Source/WebCore/loader/ResourceLoadStatisticsStore.h

    r200511 r206869  
    6464    WEBCORE_EXPORT bool hasEnoughDataForStatisticsProcessing();
    6565    WEBCORE_EXPORT void processStatistics(std::function<void(ResourceLoadStatistics&)>&&);
     66
     67    WEBCORE_EXPORT Vector<String> prevalentResourceDomainsWithoutUserInteraction();
    6668private:
    6769    ResourceLoadStatisticsStore() = default;

  • trunk/Source/WebCore/loader/SubresourceLoader.cpp

    r206858 r206869  
    174174    }
    175175
     176    if (newRequest.requester() != ResourceRequestBase::Requester::Main)
     177        ResourceLoadObserver::sharedObserver().logSubresourceLoading(m_frame.get(), newRequest, redirectResponse);
     178
    176179    ASSERT(!newRequest.isNull());
    177180    if (!redirectResponse.isNull()) {
    … …  
    229232    if (newRequest.isNull())
    230233        cancel();
    231 
    232     ResourceLoadObserver::sharedObserver().logSubresourceLoading(m_frame.get(), newRequest, redirectResponse);
    233234}
    234235

  • trunk/Source/WebKit2/ChangeLog

    r206855 r206869  
     12016-10-06  John Wilander  <wilander@apple.com>
     2
     3        Update Resource Load Statistics
     4        https://bugs.webkit.org/show_bug.cgi?id=162811
     5
     6        Reviewed by Alex Christensen.
     7
     8        * UIProcess/WebResourceLoadStatisticsStore.cpp:
     9        (WebKit::WebResourceLoadStatisticsStore::hasPrevalentResourceCharacteristics):
     10            Switched to vector-based classification.
     11        (WebKit::WebResourceLoadStatisticsStore::classifyResource):
     12            Simplified logic and moved the split between has and has
     13            no user interaction into ResourceLoadStatisticsStore.
     14        (WebKit::WebResourceLoadStatisticsStore::clearDataRecords):
     15            Added.
     16        (WebKit::WebResourceLoadStatisticsStore::resourceLoadStatisticsUpdated):
     17            Updated to make use of the new functions.
     18        (WebKit::WebResourceLoadStatisticsStore::persistentStoragePath):
     19            Removed stray whitespace.
     20        (WebKit::WebResourceLoadStatisticsStore::writeEncoderToDisk):
     21            Removed stray whitespace.
     22        (WebKit::WebResourceLoadStatisticsStore::createDecoderFromDisk):
     23            Removed stray whitespace.
     24        (WebKit::hasPrevalentResourceCharacteristics): Deleted.
     25        (WebKit::classifyPrevalentResources): Deleted.
     26        * UIProcess/WebResourceLoadStatisticsStore.h:
     27            Added member variables for clearing of data records.
     28
    1292016-10-06  Youenn Fablet  <youenn@apple.com>
    230

  • trunk/Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp

    r203303 r206869  
    2727#include "WebResourceLoadStatisticsStore.h"
    2828
     29#include "APIWebsiteDataStore.h"
    2930#include "WebProcessMessages.h"
    3031#include "WebProcessPool.h"
    3132#include "WebResourceLoadStatisticsStoreMessages.h"
     33#include "WebsiteDataFetchOption.h"
     34#include "WebsiteDataType.h"
    3235#include <WebCore/KeyedCoding.h>
    3336#include <WebCore/ResourceLoadStatistics.h>
     37#include <wtf/CurrentTime.h>
     38#include <wtf/MainThread.h>
     39#include <wtf/MathExtras.h>
     40#include <wtf/RunLoop.h>
    3441#include <wtf/threads/BinarySemaphore.h>
    3542
    … …  
    3845namespace WebKit {
    3946
    40 // Sub frame classification thresholds
    41 static const unsigned subframeUnderTopFrameOriginsThreshold = 3;
    42    
    43 // Subresource classification thresholds
    44 static const unsigned subresourceUnderTopFrameOriginsThreshold = 5;
    45 static const unsigned subresourceHasBeenRedirectedFromToUniqueDomainsThreshold = 3;
    46 static const unsigned redirectedToOtherPrevalentResourceOriginsThreshold = 2;
     47static const auto numberOfSecondsBetweenClearingDataRecords = 600;
     48static const auto featureVectorLengthThreshold = 3;
    4749
    4850Ref<WebResourceLoadStatisticsStore> WebResourceLoadStatisticsStore::create(const String& resourceLoadStatisticsDirectory)
    … …  
    5254
    5355WebResourceLoadStatisticsStore::WebResourceLoadStatisticsStore(const String& resourceLoadStatisticsDirectory)
    54     : m_resourceStatisticsStore(WebCore::ResourceLoadStatisticsStore::create())
     56    : m_resourceStatisticsStore(ResourceLoadStatisticsStore::create())
    5557    , m_statisticsQueue(WorkQueue::create("WebResourceLoadStatisticsStore Process Data Queue"))
    5658    , m_storagePath(resourceLoadStatisticsDirectory)
    … …  
    6264}
    6365
    64 static inline bool hasPrevalentResourceCharacteristics(const ResourceLoadStatistics& resourceStatistic)
    65 {
    66     return resourceStatistic.subframeUnderTopFrameOrigins.size() > subframeUnderTopFrameOriginsThreshold
    67         || resourceStatistic.subresourceUnderTopFrameOrigins.size() > subresourceUnderTopFrameOriginsThreshold
    68         || resourceStatistic.subresourceUniqueRedirectsTo.size() > subresourceHasBeenRedirectedFromToUniqueDomainsThreshold
    69         || resourceStatistic.redirectedToOtherPrevalentResourceOrigins.size() > redirectedToOtherPrevalentResourceOriginsThreshold;
    70 }
    71    
    72 static inline void classifyPrevalentResources(ResourceLoadStatistics& resourceStatistic, Vector<String>& prevalentResources, Vector<String>& prevalentResourcesWithUserInteraction)
    73 {
    74     if (resourceStatistic.isPrevalentResource || hasPrevalentResourceCharacteristics(resourceStatistic)) {
     66bool WebResourceLoadStatisticsStore::hasPrevalentResourceCharacteristics(const ResourceLoadStatistics& resourceStatistic)
     67{
     68    auto subresourceUnderTopFrameOriginsCount = resourceStatistic.subresourceUnderTopFrameOrigins.size();
     69    auto subresourceUniqueRedirectsToCount = resourceStatistic.subresourceUniqueRedirectsTo.size();
     70    auto subframeUnderTopFrameOriginsCount = resourceStatistic.subframeUnderTopFrameOrigins.size();
     71   
     72    if (!subresourceUnderTopFrameOriginsCount
     73        && !subresourceUniqueRedirectsToCount
     74        && !subframeUnderTopFrameOriginsCount)
     75        return false;
     76
     77    if (subresourceUnderTopFrameOriginsCount > featureVectorLengthThreshold
     78        || subresourceUniqueRedirectsToCount > featureVectorLengthThreshold
     79        || subframeUnderTopFrameOriginsCount > featureVectorLengthThreshold)
     80        return true;
     81
     82    // The resource is considered prevalent if the feature vector
     83    // is longer than the threshold.
     84    // Vector length for n dimensions is sqrt(a^2 + (...) + n^2).
     85    double vectorLength = 0;
     86    vectorLength += subresourceUnderTopFrameOriginsCount * subresourceUnderTopFrameOriginsCount;
     87    vectorLength += subresourceUniqueRedirectsToCount * subresourceUniqueRedirectsToCount;
     88    vectorLength += subframeUnderTopFrameOriginsCount * subframeUnderTopFrameOriginsCount;
     89
     90    ASSERT(vectorLength > 0);
     91
     92    return sqrt(vectorLength) > featureVectorLengthThreshold;
     93}
     94   
     95void WebResourceLoadStatisticsStore::classifyResource(ResourceLoadStatistics& resourceStatistic)
     96{
     97    if (!resourceStatistic.isPrevalentResource && hasPrevalentResourceCharacteristics(resourceStatistic)) {
    7598        resourceStatistic.isPrevalentResource = true;
    76         if (resourceStatistic.hadUserInteraction)
    77             prevalentResourcesWithUserInteraction.append(resourceStatistic.highLevelDomain);
    78         else
    79             prevalentResources.append(resourceStatistic.highLevelDomain);
    8099    }
    81100}
    82101
     102void WebResourceLoadStatisticsStore::clearDataRecords()
     103{
     104    if (m_dataStoreClearPending)
     105        return;
     106
     107    Vector<String> prevalentResourceDomains = coreStore().prevalentResourceDomainsWithoutUserInteraction();
     108    if (!prevalentResourceDomains.size())
     109        return;
     110
     111    double now = currentTime();
     112    if (!m_lastTimeDataRecordsWereCleared) {
     113        m_lastTimeDataRecordsWereCleared = now;
     114        return;
     115    }
     116
     117    if (now < (m_lastTimeDataRecordsWereCleared + numberOfSecondsBetweenClearingDataRecords))
     118        return;
     119
     120    m_dataStoreClearPending = true;
     121    m_lastTimeDataRecordsWereCleared = now;
     122
     123    // Switch to the main thread to get the default website data store
     124    RunLoop::main().dispatch([prevalentResourceDomains = WTFMove(prevalentResourceDomains), this] () mutable {
     125        auto& websiteDataStore = API::WebsiteDataStore::defaultDataStore()->websiteDataStore();
     126
     127        websiteDataStore.fetchData(WebsiteDataType::Cookies, { }, [prevalentResourceDomains = WTFMove(prevalentResourceDomains), this](auto websiteDataRecords) {
     128            Vector<WebsiteDataRecord> dataRecords;
     129            for (auto& websiteDataRecord : websiteDataRecords) {
     130                for (auto& prevalentResourceDomain : prevalentResourceDomains) {
     131                    if (websiteDataRecord.displayName.endsWithIgnoringASCIICase(prevalentResourceDomain)) {
     132                        auto suffixStart = websiteDataRecord.displayName.length() - prevalentResourceDomain.length();
     133                        if (!suffixStart || websiteDataRecord.displayName[suffixStart - 1] == '.')
     134                            dataRecords.append(websiteDataRecord);
     135                    }
     136                }
     137            }
     138
     139            if (!dataRecords.size()) {
     140                m_dataStoreClearPending = false;
     141                return;
     142            }
     143
     144            auto& websiteDataStore = API::WebsiteDataStore::defaultDataStore()->websiteDataStore();
     145            websiteDataStore.removeData(WebsiteDataType::Cookies, { WTFMove(dataRecords) }, [this] {
     146                m_dataStoreClearPending = false;
     147            });
     148        });
     149    });
     150}
     151
    83152void WebResourceLoadStatisticsStore::resourceLoadStatisticsUpdated(const Vector<WebCore::ResourceLoadStatistics>& origins)
    84153{
    85154    coreStore().mergeStatistics(origins);
    86155
    87     Vector<String> prevalentResources, prevalentResourcesWithUserInteraction;
    88156    if (coreStore().hasEnoughDataForStatisticsProcessing()) {
    89         coreStore().processStatistics([this, &prevalentResources, &prevalentResourcesWithUserInteraction] (ResourceLoadStatistics& resourceStatistic) {
    90             classifyPrevalentResources(resourceStatistic, prevalentResources, prevalentResourcesWithUserInteraction);
     157        coreStore().processStatistics([this] (ResourceLoadStatistics& resourceStatistic) {
     158            classifyResource(resourceStatistic);
     159            clearDataRecords();
    91160        });
    92161    }
    93162
    94     // FIXME: Notify individual WebProcesses of prevalent domains using the two vectors populated by the classifier. <rdar://problem/24703099>
    95163    auto encoder = coreStore().createEncoderFromData();
    96164   
    … …  
    153221    if (m_storagePath.isEmpty())
    154222        return emptyString();
    155    
     223
    156224    // TODO Decide what to call this file
    157225    return pathByAppendingComponent(m_storagePath, label + "_resourceLog.plist");
    … …  
    163231    if (!rawData)
    164232        return;
    165    
     233
    166234    String resourceLog = persistentStoragePath(label);
    167235    if (resourceLog.isEmpty())
    168236        return;
    169    
     237
    170238    if (!m_storagePath.isEmpty())
    171239        makeAllDirectories(m_storagePath);
    172    
     240
    173241    auto handle = openFile(resourceLog, OpenForWrite);
    174242    if (!handle)
    … …  
    177245    int64_t writtenBytes = writeToFile(handle, rawData->data(), rawData->size());
    178246    closeFile(handle);
    179    
     247
    180248    if (writtenBytes != static_cast<int64_t>(rawData->size()))
    181249        WTFLogAlways("WebResourceLoadStatisticsStore: We only wrote %d out of %d bytes to disk", static_cast<unsigned>(writtenBytes), rawData->size());
    … …  
    187255    if (resourceLog.isEmpty())
    188256        return nullptr;
    189    
     257
    190258    RefPtr<SharedBuffer> rawData = SharedBuffer::createWithContentsOfFile(resourceLog);
    191259    if (!rawData)
    192260        return nullptr;
    193    
     261
    194262    return KeyedDecoder::decoder(reinterpret_cast<const uint8_t*>(rawData->data()), rawData->size());
    195263}

  • trunk/Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.h

    r204668 r206869  
    2929#include "APIObject.h"
    3030#include "Connection.h"
     31#include "WebsiteDataRecord.h"
    3132#include <WebCore/ResourceLoadStatisticsStore.h>
    3233#include <wtf/Vector.h>
    … …  
    7273    explicit WebResourceLoadStatisticsStore(const String&);
    7374
     75    bool hasPrevalentResourceCharacteristics(const WebCore::ResourceLoadStatistics&);
     76    void classifyResource(WebCore::ResourceLoadStatistics&);
     77    void clearDataRecords();
     78
    7479    String persistentStoragePath(const String& label) const;
    7580
    … …  
    8489    String m_storagePath;
    8590    bool m_resourceLoadStatisticsEnabled { false };
     91
     92    double m_lastTimeDataRecordsWereCleared { 0 };
     93    bool m_dataStoreClearPending { false };
    8694};
    8795

  • trunk/Tools/ChangeLog

    r206866 r206869  
     12016-10-06  John Wilander  <wilander@apple.com>
     2
     3        Update Resource Load Statistics
     4        https://bugs.webkit.org/show_bug.cgi?id=162811
     5
     6        Reviewed by Alex Christensen.
     7
     8        * TestWebKitAPI/Tests/mac/PublicSuffix.mm:
     9            Change from USE(PUBLIC_SUFFIX_LIST) to ENABLE(PUBLIC_SUFFIX_LIST)
     10
    1112016-10-05  Philippe Normand  <pnormand@igalia.com>
    212

  • trunk/Tools/TestWebKitAPI/Tests/mac/PublicSuffix.mm

    r143986 r206869  
    2626#include "config.h"
    2727
    28 #if USE(PUBLIC_SUFFIX_LIST)
     28#if ENABLE(PUBLIC_SUFFIX_LIST)
    2929
    3030#include "WTFStringUtilities.h"
Note: See TracChangeset for help on using the changeset viewer.