Changeset 64398 in webkit


Ignore:
Timestamp:
Jul 30, 2010 7:22:16 PM (14 years ago)
Author:
Joseph Pecoraro
Message:

2010-07-30 Joseph Pecoraro <Joseph Pecoraro>

Reviewed by David Kilzer.

Limit ApplicationCache Total and Per-Origin Storage Capacity (Quotas)
​https://bugs.webkit.org/show_bug.cgi?id=40627

Part 2 - Update Schema and enforce Per-Origin Quotas

Added an "Origins" table to the application cache databases.
This, like the Database's Origins table, is a list of origin
and quota pairs. Origins records are added as soon as they are
needed, and deleted only when the ApplicationCacheStorage is
emptied. This means Origins records persist even after all
caches for that origin may be deleted. The "CacheGroups" table
now has a foreign key column "origin" which relates to the
"Origins" table.

To enforce the quotas, remaining quota space is checked at
the start of update as an estimate and at the end before
inserting. Currently, reaching the quota limit will simply
cause an update error. A later part will provide a
notification to the client to allow an action, and refactor
the final quota limit check into a transaction.

Respect the quota during the update process. And cause
the update process to fail when the quota is reached.

  • loader/appcache/ApplicationCacheGroup.cpp: added loading counter, counts bytes as they load (WebCore::ApplicationCacheGroup::ApplicationCacheGroup): (WebCore::ApplicationCacheGroup::didReceiveData): (WebCore::ApplicationCacheGroup::didFinishLoading): (WebCore::ApplicationCacheGroup::checkIfLoadIsComplete):
  • loader/appcache/ApplicationCacheGroup.h: added security origin, based on the manifest URL (WebCore::ApplicationCacheGroup::origin): accessor

Updates the schema of the database tables as described
above. Handle other SQL operations such as checking the
remaining space and inserting and deleting Origins records.

  • loader/appcache/ApplicationCacheStorage.cpp: (WebCore::ApplicationCacheStorage::quotaForOrigin): query for the quota of an origin, may return the default origin quota if it didn't exist. (WebCore::ApplicationCacheStorage::remainingSizeForOriginExcludingCache): calculate the remaining size in a quota for an origin, possibly excluding a cache. (WebCore::ApplicationCacheStorage::storeUpdatedQuotaForOrigin): persistent update. (WebCore::ApplicationCacheStorage::openDatabase): updated schema for CachesGroups, added new table Origins. (WebCore::ApplicationCacheStorage::empty): wipe Origins table as well. (WebCore::ApplicationCacheStorage::unknownQuota): constant to mean unknown quota
Location:
trunk/WebCore
Files:
5 edited

Legend:

Unmodified
Added
Removed

  • trunk/WebCore/ChangeLog

    r64397 r64398  
     12010-07-30  Joseph Pecoraro  <joepeck@webkit.org>
     2
     3        Reviewed by David Kilzer.
     4
     5        Limit ApplicationCache Total and Per-Origin Storage Capacity (Quotas)
     6        https://bugs.webkit.org/show_bug.cgi?id=40627
     7
     8        Part 2 - Update Schema and enforce Per-Origin Quotas
     9
     10        Added an "Origins" table to the application cache databases.
     11        This, like the Database's Origins table, is a list of origin
     12        and quota pairs. Origins records are added as soon as they are
     13        needed, and deleted only when the ApplicationCacheStorage is
     14        emptied. This means Origins records persist even after all
     15        caches for that origin may be deleted. The "CacheGroups" table
     16        now has a foreign key column "origin" which relates to the
     17        "Origins" table.
     18
     19        To enforce the quotas, remaining quota space is checked at
     20        the start of update as an estimate and at the end before
     21        inserting. Currently, reaching the quota limit will simply
     22        cause an update error. A later part will provide a
     23        notification to the client to allow an action, and refactor
     24        the final quota limit check into a transaction.
     25
     26          Respect the quota during the update process. And cause
     27          the update process to fail when the quota is reached.
     28
     29        * loader/appcache/ApplicationCacheGroup.cpp: added loading counter, counts bytes as they load
     30        (WebCore::ApplicationCacheGroup::ApplicationCacheGroup):
     31        (WebCore::ApplicationCacheGroup::didReceiveData):
     32        (WebCore::ApplicationCacheGroup::didFinishLoading):
     33        (WebCore::ApplicationCacheGroup::checkIfLoadIsComplete):
     34        * loader/appcache/ApplicationCacheGroup.h: added security origin, based on the manifest URL
     35        (WebCore::ApplicationCacheGroup::origin): accessor
     36
     37          Updates the schema of the database tables as described
     38          above. Handle other SQL operations such as checking the
     39          remaining space and inserting and deleting Origins records.
     40
     41        * loader/appcache/ApplicationCacheStorage.cpp:
     42        (WebCore::ApplicationCacheStorage::quotaForOrigin): query for the quota of an origin, may return the default origin quota if it didn't exist.
     43        (WebCore::ApplicationCacheStorage::remainingSizeForOriginExcludingCache): calculate the remaining size in a quota for an origin, possibly excluding a cache.
     44        (WebCore::ApplicationCacheStorage::storeUpdatedQuotaForOrigin): persistent update.
     45        (WebCore::ApplicationCacheStorage::openDatabase): updated schema for CachesGroups, added new table Origins.
     46        (WebCore::ApplicationCacheStorage::empty): wipe Origins table as well.
     47        (WebCore::ApplicationCacheStorage::unknownQuota): constant to mean unknown quota
     48
    1492010-07-30  Joseph Pecoraro  <joepeck@webkit.org>
    250

  • trunk/WebCore/loader/appcache/ApplicationCacheGroup.cpp

    r63753 r64398  
    4343#include "ManifestParser.h"
    4444#include "Page.h"
     45#include "SecurityOrigin.h"
    4546#include "Settings.h"
    4647#include <wtf/HashMap.h>
    … …  
    5859ApplicationCacheGroup::ApplicationCacheGroup(const KURL& manifestURL, bool isCopy)
    5960    : m_manifestURL(manifestURL)
     61    , m_origin(SecurityOrigin::create(manifestURL))
    6062    , m_updateStatus(Idle)
    6163    , m_downloadingPendingMasterResourceLoadersCount(0)
    … …  
    6870    , m_isCopy(isCopy)
    6971    , m_calledReachedMaxAppCacheSize(false)
     72    , m_loadedSize(0)
     73    , m_availableSpaceInQuota(ApplicationCacheStorage::unknownQuota())
    7074{
    7175}
    … …  
    593597    ASSERT(m_currentResource);
    594598    m_currentResource->data()->append(data, length);
     599
     600    m_loadedSize += length;
    595601}
    596602
    … …  
    606612        return;
    607613    }
    608  
     614
     615    // After finishing the loading of any resource, we check if it will
     616    // fit in our last known quota limit.
     617    if (m_availableSpaceInQuota == ApplicationCacheStorage::unknownQuota()) {
     618        // Failed to determine what is left in the quota. Fallback to allowing anything.
     619        if (!cacheStorage().remainingSizeForOriginExcludingCache(m_origin.get(), m_newestCache.get(), m_availableSpaceInQuota))
     620            m_availableSpaceInQuota = ApplicationCacheStorage::noQuota();
     621    }
     622
     623    // Check each resource, as it loads, to see if it would fit in our
     624    // idea of the available quota space.
     625    if (m_availableSpaceInQuota < m_loadedSize) {
     626        m_currentResource = 0;
     627        cacheUpdateFailed();
     628        return;
     629    }
     630
    609631    ASSERT(m_currentHandle == handle);
    610632    ASSERT(m_pendingEntries.contains(handle->firstRequest().url()));
    … …  
    860882
    861883        RefPtr<ApplicationCache> oldNewestCache = (m_newestCache == m_cacheBeingUpdated) ? 0 : m_newestCache;
     884
     885        // Check one more time, before committing to the new cache, if the cache will fit in the quota.
     886        int64_t remainingSpaceInOrigin;
     887        if (cacheStorage().remainingSizeForOriginExcludingCache(m_origin.get(), oldNewestCache.get(), remainingSpaceInOrigin)) {
     888            if (m_cacheBeingUpdated->estimatedSizeInStorage() > remainingSpaceInOrigin) {
     889                cacheUpdateFailed();
     890                break;
     891            }
     892        }
    862893
    863894        setNewestCache(m_cacheBeingUpdated.release());
    … …  
    923954    setUpdateStatus(Idle);
    924955    m_frame = 0;
     956    m_loadedSize = 0;
     957    m_availableSpaceInQuota = ApplicationCacheStorage::unknownQuota();
    925958    m_calledReachedMaxAppCacheSize = false;
    926959}

  • trunk/WebCore/loader/appcache/ApplicationCacheGroup.h

    r62957 r64398  
    11/*
    2  * Copyright (C) 2008, 2009 Apple Inc. All Rights Reserved.
     2 * Copyright (C) 2008, 2009, 2010 Apple Inc. All Rights Reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2929#if ENABLE(OFFLINE_WEB_APPLICATIONS)
    3030
    31 #include <wtf/Noncopyable.h>
    32 #include <wtf/HashMap.h>
    33 #include <wtf/HashSet.h>
    34 
    3531#include "DOMApplicationCache.h"
    3632#include "KURL.h"
    … …  
    4036#include "SharedBuffer.h"
    4137
     38#include <wtf/Noncopyable.h>
     39#include <wtf/HashMap.h>
     40#include <wtf/HashSet.h>
     41
    4242namespace WebCore {
    4343
    … …  
    4747class DocumentLoader;
    4848class Frame;
     49class SecurityOrigin;
    4950
    5051enum ApplicationCacheUpdateOption {
    … …  
    6768   
    6869    const KURL& manifestURL() const { return m_manifestURL; }
     70    const SecurityOrigin* origin() const { return m_origin.get(); }
    6971    UpdateStatus updateStatus() const { return m_updateStatus; }
    7072    void setUpdateStatus(UpdateStatus status);
    … …  
    132134   
    133135    KURL m_manifestURL;
     136    RefPtr<SecurityOrigin> m_origin;
    134137    UpdateStatus m_updateStatus;
    135138   
    … …  
    195198    RefPtr<ResourceHandle> m_manifestHandle;
    196199
     200    int64_t m_loadedSize;
     201    int64_t m_availableSpaceInQuota;
     202
    197203    friend class ChromeClientCallbackTimer;
    198204};

  • trunk/WebCore/loader/appcache/ApplicationCacheStorage.cpp

    r64397 r64398  
    11/*
    2  * Copyright (C) 2008, 2009 Apple Inc. All Rights Reserved.
     2 * Copyright (C) 2008, 2009, 2010 Apple Inc. All Rights Reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3030
    3131#include "ApplicationCache.h"
     32#include "ApplicationCacheGroup.h"
    3233#include "ApplicationCacheHost.h"
    33 #include "ApplicationCacheGroup.h"
    3434#include "ApplicationCacheResource.h"
    3535#include "FileSystem.h"
    … …  
    3737#include "SQLiteStatement.h"
    3838#include "SQLiteTransaction.h"
     39#include "SecurityOrigin.h"
    3940#include <wtf/text/CString.h>
    4041#include <wtf/StdLibExtras.h>
    … …  
    420421}
    421422
     423bool ApplicationCacheStorage::quotaForOrigin(const SecurityOrigin* origin, int64_t& quota)
     424{
     425    // If an Origin record doesn't exist, then the COUNT will be 0 and quota will be 0.
     426    // Using the count to determine if a record existed or not is a safe way to determine
     427    // if a quota of 0 is real, from the record, or from null.
     428    SQLiteStatement statement(m_database, "SELECT COUNT(quota), quota FROM Origins WHERE origin=?");
     429    if (statement.prepare() != SQLResultOk)
     430        return false;
     431
     432    statement.bindText(1, origin->databaseIdentifier());
     433    int result = statement.step();
     434
     435    // Return the quota, or if it was null the default.
     436    if (result == SQLResultRow) {
     437        bool wasNoRecord = statement.getColumnInt64(0) == 0;
     438        quota = wasNoRecord ? m_defaultOriginQuota : statement.getColumnInt64(1);
     439        return true;
     440    }
     441
     442    LOG_ERROR("Could not get the quota of an origin, error \"%s\"", m_database.lastErrorMsg());
     443    return false;
     444}
     445
     446bool ApplicationCacheStorage::remainingSizeForOriginExcludingCache(const SecurityOrigin* origin, ApplicationCache* cache, int64_t& remainingSize)
     447{
     448    openDatabase(false);
     449    if (!m_database.isOpen())
     450        return false;
     451
     452    // Remaining size = total origin quota - size of all caches with origin excluding the provided cache.
     453    // Keep track of the number of caches so we can tell if the result was a calculation or not.
     454    const char* query;
     455    int64_t excludingCacheIdentifier = cache ? cache->storageID() : 0;
     456    if (excludingCacheIdentifier != 0) {
     457        query = "SELECT COUNT(Caches.size), Origins.quota - SUM(Caches.size)"
     458                "  FROM CacheGroups"
     459                " INNER JOIN Origins ON CacheGroups.origin = Origins.origin"
     460                " INNER JOIN Caches ON CacheGroups.id = Caches.cacheGroup"
     461                " WHERE Origins.origin=?"
     462                "   AND Caches.id!=?";
     463    } else {
     464        query = "SELECT COUNT(Caches.size), Origins.quota - SUM(Caches.size)"
     465                "  FROM CacheGroups"
     466                " INNER JOIN Origins ON CacheGroups.origin = Origins.origin"
     467                " INNER JOIN Caches ON CacheGroups.id = Caches.cacheGroup"
     468                " WHERE Origins.origin=?";
     469    }
     470
     471    SQLiteStatement statement(m_database, query);
     472    if (statement.prepare() != SQLResultOk)
     473        return false;
     474
     475    statement.bindText(1, origin->databaseIdentifier());
     476    if (excludingCacheIdentifier != 0)
     477        statement.bindInt64(2, excludingCacheIdentifier);
     478    int result = statement.step();
     479
     480    // If the count was 0 that then we have to query the origin table directly
     481    // for its quota. Otherwise we can use the calculated value.
     482    if (result == SQLResultRow) {
     483        int64_t numberOfCaches = statement.getColumnInt64(0);
     484        if (numberOfCaches == 0)
     485            quotaForOrigin(origin, remainingSize);
     486        else
     487            remainingSize = statement.getColumnInt64(1);
     488        return true;
     489    }
     490
     491    LOG_ERROR("Could not get the remaining size of an origin's quota, error \"%s\"", m_database.lastErrorMsg());
     492    return false;
     493}
     494
     495bool ApplicationCacheStorage::storeUpdatedQuotaForOrigin(const SecurityOrigin* origin, int64_t quota)
     496{
     497    openDatabase(false);
     498    if (!m_database.isOpen())
     499        return false;
     500
     501    SQLiteStatement updateStatement(m_database, "UPDATE Origins SET quota=? WHERE origin=?");
     502    if (updateStatement.prepare() != SQLResultOk)
     503        return false;
     504
     505    updateStatement.bindInt64(1, quota);
     506    updateStatement.bindText(2, origin->databaseIdentifier());
     507
     508    return executeStatement(updateStatement);
     509}
     510
    422511bool ApplicationCacheStorage::executeSQLCommand(const String& sql)
    423512{
    … …  
    432521}
    433522
    434 static const int schemaVersion = 5;
     523// Update the schemaVersion when the schema of any the Application Cache
     524// SQLite tables changes. This allows the database to be rebuilt when
     525// a new, incompatible change has been introduced to the database schema.
     526static const int schemaVersion = 6;
    435527   
    436528void ApplicationCacheStorage::verifySchemaVersion()
    … …  
    481573    // Create tables
    482574    executeSQLCommand("CREATE TABLE IF NOT EXISTS CacheGroups (id INTEGER PRIMARY KEY AUTOINCREMENT, "
    483                       "manifestHostHash INTEGER NOT NULL ON CONFLICT FAIL, manifestURL TEXT UNIQUE ON CONFLICT FAIL, newestCache INTEGER)");
     575                      "manifestHostHash INTEGER NOT NULL ON CONFLICT FAIL, manifestURL TEXT UNIQUE ON CONFLICT FAIL, newestCache INTEGER, origin TEXT)");
    484576    executeSQLCommand("CREATE TABLE IF NOT EXISTS Caches (id INTEGER PRIMARY KEY AUTOINCREMENT, cacheGroup INTEGER, size INTEGER)");
    485577    executeSQLCommand("CREATE TABLE IF NOT EXISTS CacheWhitelistURLs (url TEXT NOT NULL ON CONFLICT FAIL, cache INTEGER NOT NULL ON CONFLICT FAIL)");
    … …  
    491583                      "statusCode INTEGER NOT NULL, responseURL TEXT NOT NULL, mimeType TEXT, textEncodingName TEXT, headers TEXT, data INTEGER NOT NULL ON CONFLICT FAIL)");
    492584    executeSQLCommand("CREATE TABLE IF NOT EXISTS CacheResourceData (id INTEGER PRIMARY KEY AUTOINCREMENT, data BLOB)");
     585    executeSQLCommand("CREATE TABLE IF NOT EXISTS Origins (origin TEXT UNIQUE ON CONFLICT IGNORE, quota INTEGER NOT NULL ON CONFLICT FAIL)");
    493586
    494587    // When a cache is deleted, all its entries and its whitelist should be deleted.
    … …  
    529622    ASSERT(journal);
    530623
    531     SQLiteStatement statement(m_database, "INSERT INTO CacheGroups (manifestHostHash, manifestURL) VALUES (?, ?)");
     624    SQLiteStatement statement(m_database, "INSERT INTO CacheGroups (manifestHostHash, manifestURL, origin) VALUES (?, ?, ?)");
    532625    if (statement.prepare() != SQLResultOk)
    533626        return false;
     627
     628    String originIdentifier = group->origin()->databaseIdentifier();
    534629
    535630    statement.bindInt64(1, urlHostHash(group->manifestURL()));
    536631    statement.bindText(2, group->manifestURL());
     632    statement.bindText(3, originIdentifier);
    537633
    538634    if (!executeStatement(statement))
    539635        return false;
     636
     637    // Create Origin if needed.
     638    {
     639        SQLiteStatement insertOriginStatement(m_database, "INSERT INTO Origins (origin, quota) VALUES (?, ?)");
     640        if (insertOriginStatement.prepare() != SQLResultOk)
     641            return false;
     642
     643        insertOriginStatement.bindText(1, originIdentifier);
     644        insertOriginStatement.bindInt64(2, m_defaultOriginQuota);
     645        if (!executeStatement(insertOriginStatement))
     646            return false;
     647    }
    540648
    541649    group->setStorageID(static_cast<unsigned>(m_database.lastInsertRowID()));
    … …  
    9701078        return;
    9711079   
    972     // Clear cache groups, caches and cache resources.
     1080    // Clear cache groups, caches, cache resources, and origins.
    9731081    executeSQLCommand("DELETE FROM CacheGroups");
    9741082    executeSQLCommand("DELETE FROM Caches");
     1083    executeSQLCommand("DELETE FROM Origins");
    9751084   
    9761085    // Clear the storage IDs for the caches in memory.

  • trunk/WebCore/loader/appcache/ApplicationCacheStorage.h

    r64397 r64398  
    11/*
    2  * Copyright (C) 2008 Apple Inc. All Rights Reserved.
     2 * Copyright (C) 2008, 2010 Apple Inc. All Rights Reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3838
    3939class ApplicationCache;
     40class ApplicationCacheGroup;
    4041class ApplicationCacheHost;
    41 class ApplicationCacheGroup;
    4242class ApplicationCacheResource;
    4343class KURL;
    4444template <class T>
    4545class StorageIDJournal;
     46class SecurityOrigin;
    4647
    4748class ApplicationCacheStorage : public Noncopyable {
    … …  
    5758    int64_t defaultOriginQuota() const { return m_defaultOriginQuota; }
    5859    void setDefaultOriginQuota(int64_t quota);
     60    bool quotaForOrigin(const SecurityOrigin*, int64_t& quota);
     61    bool remainingSizeForOriginExcludingCache(const SecurityOrigin*, ApplicationCache*, int64_t& remainingSize);
     62    bool storeUpdatedQuotaForOrigin(const SecurityOrigin*, int64_t quota);
    5963
    6064    ApplicationCacheGroup* cacheGroupForURL(const KURL&); // Cache to load a main resource from.
    … …  
    8185    void vacuumDatabaseFile();
    8286
     87    static int64_t unknownQuota() { return -1; }
    8388    static int64_t noQuota() { return std::numeric_limits<int64_t>::max(); }
    8489private:
Note: See TracChangeset for help on using the changeset viewer.