Changeset 29983 in webkit


Ignore:
Timestamp:
Feb 4, 2008, 4:55:47 PM (17 years ago)
Author:
beidson@apple.com
Message:

Reviewed by Darin

Fix for <rdar://problem/5628468> - Quotas need to be implemented per-origin, and not per-database

To accomplish this, we need to track the sizes of all databases in an origin to constantly keep an up to date
count of the origin's total disk usage. I've introduced the OriginQuotaManager and OriginUsageRecord classes
to accomplish this.

Whenever a transaction is known to mutate the size of a database (tracked by the DatabaseAuthorizer), it marks
that database as unknown in the OriginQuotaManager. When a transaction later comes along to ask the
OriginQuotaManager the usage for that origin, it stat's all of the unknown databases in the origin and returns
the result.

Since the OriginQuotaManager is interesting from both the main thread and a DatabaseThread, all accessors it
provides require it to be locked first. ASSERTs help guarantee this is always the case.

Layout test will involve adding functionality to DRT on multiple platforms and will be coming up shortly

  • WebCore.xcodeproj/project.pbxproj:
  • platform/SecurityOrigin.h: Changed to be ThreadSafeShared instead of RefCounted
  • storage/Database.cpp: (WebCore::Database::databaseSize): Return the current filesize of this database on disk (WebCore::Database::maximumSize): Added - calculates maximum size of this database based on quota and usage
  • storage/Database.h: Add databaseSize() accessor, and get rid of unused declared methods that *were* going to be the solution for this bug.
  • storage/DatabaseTracker.cpp: (WebCore::DatabaseTracker::originQuotaManager): Accessor to the OriginQuotaManager which is lazily created (WebCore::DatabaseTracker::canEstablishDatabase): Fetch the usage for this database slightly earlier, which will ensure that the OriginQuotaManager is primed to track this origin (WebCore::DatabaseTracker::fullPathForDatabase): Ditto (WebCore::DatabaseTracker::populateOrigins): Create the OriginQuotaManager here. (WebCore::DatabaseTracker::usageForOrigin): Use the OriginQuotaManager instead of looping through each database in the origin (WebCore::DatabaseTracker::deleteOrigin): Remove this origin from the OriginQuotaManager as it is no longer interesting (WebCore::DatabaseTracker::deleteDatabase): Remove this database from the OriginQuotaManager as it is no longer interesting
  • storage/DatabaseTracker.h:
  • storage/OriginQuotaManager.cpp: Added. (WebCore::OriginQuotaManager::OriginQuotaManager): (WebCore::OriginQuotaManager::lock): (WebCore::OriginQuotaManager::unlock): (WebCore::OriginQuotaManager::trackOrigin): Add an origin to be tracked. Useful for when the very first database in a new origin is still in the process of being created (WebCore::OriginQuotaManager::tracksOrigin): (WebCore::OriginQuotaManager::addDatabase): (WebCore::OriginQuotaManager::removeDatabase): (WebCore::OriginQuotaManager::removeOrigin): Removes all records in a certain origin from being tracked (WebCore::OriginQuotaManager::markDatabase): Mark a specific database as having an unknown size - called when the DatabaseAuthorizer in a SQLTransaction knows the file size might change. (WebCore::OriginQuotaManager::diskUsage): Returns the disk usage for the given origin
  • storage/OriginQuotaManager.h: Added.
  • storage/OriginUsageRecord.cpp: Added. (WebCore::OriginUsageRecord::OriginUsageRecord): (WebCore::OriginUsageRecord::addDatabase): Adds an entry for the database in this origin's record (WebCore::OriginUsageRecord::removeDatabase): Removes that entry (WebCore::OriginUsageRecord::markDatabase): Marks the database as of unknown size (WebCore::OriginUsageRecord::diskUsage): Returns the cached disk usage value, or recalculates it if any databases are marked
  • storage/OriginUsageRecord.h: Added.
  • storage/SQLTransaction.cpp: (WebCore::SQLTransaction::openTransactionAndPreflight): Use Database::maximumSize() instead of attributing the entire quota to each database. (WebCore::SQLTransaction::runStatements): Use Database::maximumSize() instead of attributing the entire quota to each database. (WebCore::SQLTransaction::runCurrentStatement): Mark this databases's size as unknown in the OriginQuotaManager if this statement will change the size of the database
  • storage/SQLTransaction.h:
Location:
trunk/WebCore
Files:
4 added
8 edited

Legend:

Unmodified
Added
Removed

  • trunk/WebCore/ChangeLog

    r29982 r29983  
     12008-02-04  Brady Eidson  <beidson@apple.com>
     2
     3        Reviewed by Darin
     4
     5        Fix for <rdar://problem/5628468> - Quotas need to be implemented per-origin, and not per-database
     6
     7        To accomplish this, we need to track the sizes of all databases in an origin to constantly keep an up to date
     8        count of the origin's total disk usage.  I've introduced the OriginQuotaManager and OriginUsageRecord classes
     9        to accomplish this.
     10
     11        Whenever a transaction is known to mutate the size of a database (tracked by the DatabaseAuthorizer), it marks
     12        that database as unknown in the OriginQuotaManager.  When a transaction later comes along to ask the
     13        OriginQuotaManager the usage for that origin, it stat's all of the unknown databases in the origin and returns
     14        the result.
     15
     16        Since the OriginQuotaManager is interesting from both the main thread and a DatabaseThread, all accessors it
     17        provides require it to be locked first.  ASSERTs help guarantee this is always the case.
     18
     19        Layout test will involve adding functionality to DRT on multiple platforms and will be coming up shortly
     20
     21        * WebCore.xcodeproj/project.pbxproj:
     22
     23        * platform/SecurityOrigin.h: Changed to be ThreadSafeShared instead of RefCounted
     24
     25        * storage/Database.cpp:
     26        (WebCore::Database::databaseSize): Return the current filesize of this database on disk
     27        (WebCore::Database::maximumSize): Added - calculates maximum size of this database based on quota and usage
     28        * storage/Database.h: Add databaseSize() accessor, and get rid of unused declared methods that *were* going
     29          to be the solution for this bug.
     30
     31        * storage/DatabaseTracker.cpp:
     32        (WebCore::DatabaseTracker::originQuotaManager): Accessor to the OriginQuotaManager which is lazily created
     33        (WebCore::DatabaseTracker::canEstablishDatabase): Fetch the usage for this database slightly earlier, which
     34          will ensure that the OriginQuotaManager is primed to track this origin
     35        (WebCore::DatabaseTracker::fullPathForDatabase): Ditto
     36        (WebCore::DatabaseTracker::populateOrigins): Create the OriginQuotaManager here.
     37        (WebCore::DatabaseTracker::usageForOrigin): Use the OriginQuotaManager instead of looping through each database
     38          in the origin
     39        (WebCore::DatabaseTracker::deleteOrigin): Remove this origin from the OriginQuotaManager as it is no longer
     40          interesting
     41        (WebCore::DatabaseTracker::deleteDatabase): Remove this database from the OriginQuotaManager as it is no longer
     42          interesting
     43        * storage/DatabaseTracker.h:
     44
     45        * storage/OriginQuotaManager.cpp: Added.
     46        (WebCore::OriginQuotaManager::OriginQuotaManager):
     47        (WebCore::OriginQuotaManager::lock):
     48        (WebCore::OriginQuotaManager::unlock):
     49        (WebCore::OriginQuotaManager::trackOrigin): Add an origin to be tracked.  Useful for when the very first database
     50          in a new origin is still in the process of being created
     51        (WebCore::OriginQuotaManager::tracksOrigin):
     52        (WebCore::OriginQuotaManager::addDatabase):
     53        (WebCore::OriginQuotaManager::removeDatabase):
     54        (WebCore::OriginQuotaManager::removeOrigin): Removes all records in a certain origin from being tracked
     55        (WebCore::OriginQuotaManager::markDatabase): Mark a specific database as having an unknown size - called when the
     56          DatabaseAuthorizer in a SQLTransaction knows the file size might change.
     57        (WebCore::OriginQuotaManager::diskUsage): Returns the disk usage for the given origin
     58        * storage/OriginQuotaManager.h: Added.
     59
     60        * storage/OriginUsageRecord.cpp: Added.
     61        (WebCore::OriginUsageRecord::OriginUsageRecord):
     62        (WebCore::OriginUsageRecord::addDatabase): Adds an entry for the database in this origin's record
     63        (WebCore::OriginUsageRecord::removeDatabase): Removes that entry
     64        (WebCore::OriginUsageRecord::markDatabase): Marks the database as of unknown size
     65        (WebCore::OriginUsageRecord::diskUsage): Returns the cached disk usage value, or recalculates it if any databases
     66          are marked
     67        * storage/OriginUsageRecord.h: Added.
     68
     69        * storage/SQLTransaction.cpp:
     70        (WebCore::SQLTransaction::openTransactionAndPreflight): Use Database::maximumSize() instead of attributing the
     71          entire quota to each database.
     72        (WebCore::SQLTransaction::runStatements): Use Database::maximumSize() instead of attributing the entire quota
     73          to each database.
     74        (WebCore::SQLTransaction::runCurrentStatement): Mark this databases's size as unknown in the
     75          OriginQuotaManager if this statement will change the size of the database
     76        * storage/SQLTransaction.h:
     77
    1782008-02-04   David Harrison  <harrison@apple.com>
    279

  • trunk/WebCore/WebCore.xcodeproj/project.pbxproj

    r29929 r29983  
    418418                51A45B560CAD7FD7000D2BE9 /* DatabaseAuthorizer.h in Headers */ = {isa = PBXBuildFile; fileRef = 51A45B540CAD7FD7000D2BE9 /* DatabaseAuthorizer.h */; };
    419419                51A45B570CAD7FD7000D2BE9 /* DatabaseAuthorizer.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 51A45B550CAD7FD7000D2BE9 /* DatabaseAuthorizer.cpp */; };
     420                51A9267C0D53F0570063ECC2 /* OriginQuotaManager.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 51A926780D53F0570063ECC2 /* OriginQuotaManager.cpp */; };
     421                51A9267D0D53F0570063ECC2 /* OriginQuotaManager.h in Headers */ = {isa = PBXBuildFile; fileRef = 51A926790D53F0570063ECC2 /* OriginQuotaManager.h */; };
     422                51A9267E0D53F0570063ECC2 /* OriginUsageRecord.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 51A9267A0D53F0570063ECC2 /* OriginUsageRecord.cpp */; };
     423                51A9267F0D53F0570063ECC2 /* OriginUsageRecord.h in Headers */ = {isa = PBXBuildFile; fileRef = 51A9267B0D53F0570063ECC2 /* OriginUsageRecord.h */; };
    420424                51AA3F6F0BD5AA9E00892971 /* ResourceLoaderMac.mm in Sources */ = {isa = PBXBuildFile; fileRef = 51AA3F6E0BD5AA9E00892971 /* ResourceLoaderMac.mm */; };
    421425                51C81B890C4422F70019ECE3 /* FTPDirectoryParser.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 51C81B870C4422F70019ECE3 /* FTPDirectoryParser.cpp */; };
    … …  
    46744678                51A45B540CAD7FD7000D2BE9 /* DatabaseAuthorizer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DatabaseAuthorizer.h; sourceTree = "<group>"; };
    46754679                51A45B550CAD7FD7000D2BE9 /* DatabaseAuthorizer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = DatabaseAuthorizer.cpp; sourceTree = "<group>"; };
     4680                51A926780D53F0570063ECC2 /* OriginQuotaManager.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = OriginQuotaManager.cpp; sourceTree = "<group>"; };
     4681                51A926790D53F0570063ECC2 /* OriginQuotaManager.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = OriginQuotaManager.h; sourceTree = "<group>"; };
     4682                51A9267A0D53F0570063ECC2 /* OriginUsageRecord.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = OriginUsageRecord.cpp; sourceTree = "<group>"; };
     4683                51A9267B0D53F0570063ECC2 /* OriginUsageRecord.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = OriginUsageRecord.h; sourceTree = "<group>"; };
    46764684                51AA3F6E0BD5AA9E00892971 /* ResourceLoaderMac.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = ResourceLoaderMac.mm; sourceTree = "<group>"; };
    46774685                51C81B870C4422F70019ECE3 /* FTPDirectoryParser.cpp */ = {isa = PBXFileReference; fileEncoding = 30; lastKnownFileType = sourcecode.cpp.cpp; path = FTPDirectoryParser.cpp; sourceTree = "<group>"; };
    … …  
    80908098                                1AD51A120CB59CD300953D11 /* DatabaseTracker.h */,
    80918099                                51FAFE330CECBF2D00BB3F24 /* DatabaseTrackerClient.h */,
     8100                                51A926780D53F0570063ECC2 /* OriginQuotaManager.cpp */,
     8101                                51A926790D53F0570063ECC2 /* OriginQuotaManager.h */,
     8102                                51A9267A0D53F0570063ECC2 /* OriginUsageRecord.cpp */,
     8103                                51A9267B0D53F0570063ECC2 /* OriginUsageRecord.h */,
    80928104                                51EC92570CE90DB400F90308 /* SQLError.h */,
    80938105                                51EC92580CE90DB400F90308 /* SQLError.idl */,
    … …  
    1420614218                                515378030D52D0630063A78B /* Locker.h in Headers */,
    1420714219                                514075C60D53ECFE00BC2D85 /* SecurityOriginHash.h in Headers */,
     14220                                51A9267D0D53F0570063ECC2 /* OriginQuotaManager.h in Headers */,
     14221                                51A9267F0D53F0570063ECC2 /* OriginUsageRecord.h in Headers */,
    1420814222                        );
    1420914223                        runOnlyForDeploymentPostprocessing = 0;
    … …  
    1579715811                                E4EEFFC80D34550C00469A58 /* JSAudioConstructor.cpp in Sources */,
    1579815812                                BCE7B1930D4E86960075A539 /* JSHistoryCustom.cpp in Sources */,
     15813                                51A9267C0D53F0570063ECC2 /* OriginQuotaManager.cpp in Sources */,
     15814                                51A9267E0D53F0570063ECC2 /* OriginUsageRecord.cpp in Sources */,
    1579915815                        );
    1580015816                        runOnlyForDeploymentPostprocessing = 0;

  • trunk/WebCore/platform/SecurityOrigin.h

    r29663 r29983  
    3434
    3535#include "PlatformString.h"
     36#include "Threading.h"
    3637
    3738namespace WebCore {
    … …  
    4041    class KURL;
    4142   
    42     class SecurityOrigin : public RefCounted<SecurityOrigin> {
     43    class SecurityOrigin : public ThreadSafeShared<SecurityOrigin> {
    4344    public:
    4445        static PassRefPtr<SecurityOrigin> createForFrame(Frame*);

  • trunk/WebCore/storage/Database.cpp

    r29964 r29983  
    4444#include "NotImplemented.h"
    4545#include "Page.h"
     46#include "OriginQuotaManager.h"
    4647#include "SQLiteDatabase.h"
    4748#include "SQLiteStatement.h"
    … …  
    291292   
    292293    return true;
     294}
     295
     296unsigned long long Database::databaseSize() const
     297{
     298    long long size;
     299    if (!fileSize(m_filename, size))
     300        size = 0;
     301    return size;
     302}
     303
     304unsigned long long Database::maximumSize() const
     305{
     306    // The maximum size for this database is the full quota for this origin, minus the current usage within this origin,
     307    // except for the current usage of this database
     308   
     309    OriginQuotaManager& manager(DatabaseTracker::tracker().originQuotaManager());
     310    Locker<OriginQuotaManager> locker(manager);
     311   
     312    return DatabaseTracker::tracker().quotaForOrigin(m_securityOrigin.get()) - manager.diskUsage(m_securityOrigin.get()) + databaseSize();
    293313}
    294314

  • trunk/WebCore/storage/Database.h

    r29922 r29983  
    9191    bool versionMatchesExpected() const;
    9292   
     93    unsigned long long databaseSize() const;
     94    unsigned long long maximumSize() const;
     95
    9396// Called from DatabaseThread, must be prepared to work on the background thread
    9497    void resetAuthorizer();
    … …  
    113116    Deque<RefPtr<SQLTransaction> > m_transactionQueue;
    114117    RefPtr<SQLTransaction> m_currentTransaction;
    115    
    116     static void scheduleFileSizeTimerOnMainThread(Database*);
    117     static void performScheduleFileSizeTimers();
    118     void scheduleFileSizeTimer();
    119     void sizeTimerFired(Timer<Database>*);
    120     OwnPtr<Timer<Database> > m_sizeTimer;
    121118
    122119    static void deliverAllPendingCallbacks(void*);

  • trunk/WebCore/storage/DatabaseTracker.cpp

    r29929 r29983  
    3535#include "Document.h"
    3636#include "FileSystem.h"
     37#include "OriginQuotaManager.h"
    3738#include "Page.h"
    3839#include "SecurityOrigin.h"
    … …  
    4344
    4445namespace WebCore {
     46
     47OriginQuotaManager& DatabaseTracker::originQuotaManager()
     48{
     49    populateOrigins();
     50    ASSERT(m_quotaManager);
     51    return *m_quotaManager;
     52}
    4553
    4654DatabaseTracker& DatabaseTracker::tracker()
    … …  
    119127    populateOrigins();
    120128
     129    SecurityOrigin* origin = document->securityOrigin();
     130
     131    // Since we're imminently opening a database within this Document's origin, make sure this origin is being tracked by the QuotaTracker
     132    // by fetching it's current usage now
     133    unsigned long long usage = usageForOrigin(origin);
     134   
    121135    // If a database already exists, ignore the passed-in estimated size and say it's OK.
    122     SecurityOrigin* origin = document->securityOrigin();
    123136    if (hasEntryForDatabase(origin, name))
    124137        return true;
    125138
    126139    // If the database will fit, allow its creation.
    127     unsigned long long usage = usageForOrigin(origin);
    128140    unsigned long long requirement = usage + max(1UL, estimatedSize);
    129141    if (requirement < usage)
    … …  
    242254        return String();
    243255
     256    // If this origin's quota is being tracked (open handle to a database in this origin), add this new database
     257    // to the quota manager now
     258    {
     259        Locker<OriginQuotaManager> locker(originQuotaManager());
     260        if (originQuotaManager().tracksOrigin(origin))
     261            originQuotaManager().addDatabase(origin, name, filename);
     262    }
     263   
    244264    return filename;
    245265}
    … …  
    253273
    254274    m_quotaMap.set(new QuotaMap);
     275    m_quotaManager.set(new OriginQuotaManager);
    255276
    256277    openTrackerDatabase(false);
    … …  
    403424{
    404425    ASSERT(currentThread() == m_thread);
     426    Locker<OriginQuotaManager> locker(originQuotaManager());
     427
     428    // Use the OriginQuotaManager mechanism to calculate the usage
     429    if (originQuotaManager().tracksOrigin(origin))
     430        return originQuotaManager().diskUsage(origin);
     431   
     432    // If the OriginQuotaManager doesn't track this origin already, prime it to do so
     433    originQuotaManager().trackOrigin(origin);
     434   
    405435    Vector<String> names;
    406436    databaseNamesForOrigin(origin, names);
    407    
    408     unsigned long long result = 0;
     437
    409438    for (unsigned i = 0; i < names.size(); ++i)
    410         result += usageForDatabase(names[i], origin);
    411        
    412     return result;
     439        originQuotaManager().addDatabase(origin, names[i], fullPathForDatabase(origin, names[i], false));
     440   
     441    if (!originQuotaManager().tracksOrigin(origin))
     442        return 0;
     443    return originQuotaManager().diskUsage(origin);
    413444}
    414445
    … …  
    557588        MutexLocker lockQuotaMap(m_quotaMapGuard);
    558589        m_quotaMap->remove(origin);
     590        Locker<OriginQuotaManager> quotaManagerLocker(originQuotaManager());
     591        originQuotaManager().removeOrigin(origin);
    559592    }
    560593
    … …  
    600633    }
    601634   
     635    {
     636        Locker<OriginQuotaManager> quotaManagerLocker(originQuotaManager());
     637        originQuotaManager().removeDatabase(origin, name);
     638    }
     639   
    602640    if (m_client) {
    603641        m_client->dispatchDidModifyOrigin(origin);

  • trunk/WebCore/storage/DatabaseTracker.h

    r29922 r29983  
    4141class DatabaseTrackerClient;
    4242class Document;
     43class OriginQuotaManager;
    4344class SecurityOrigin;
    4445
    … …  
    7475    void scheduleNotifyDatabaseChanged(SecurityOrigin*, const String& name);
    7576   
     77    OriginQuotaManager& originQuotaManager();
     78   
    7679    static DatabaseTracker& tracker();
    7780
    … …  
    97100    Mutex m_quotaMapGuard;
    98101    mutable OwnPtr<QuotaMap> m_quotaMap;
     102   
     103    OwnPtr<OriginQuotaManager> m_quotaManager;
    99104
    100105    String m_databaseDirectoryPath;

  • trunk/WebCore/storage/SQLTransaction.cpp

    r29964 r29983  
    3838#include "ExceptionCode.h"
    3939#include "Logging.h"
     40#include "OriginQuotaManager.h"
    4041#include "Page.h"
    4142#include "PlatformString.h"
    … …  
    120121    LOG(StorageAPI, "Opening and preflighting transaction %p", this);
    121122   
    122     // FIXME: This is a glaring bug that gives each database in an origin the full quota of that origin
    123     // An efficient way to track the size of individual databases in an origin will need to be developed
    124     // before we can know
    125     // <rdar://problem/5628468> tracks this task
    126     m_database->m_sqliteDatabase.setMaximumSize(DatabaseTracker::tracker().quotaForOrigin(m_database->securityOriginCopy().get()));
     123    // Set the maximum usage for this transaction
     124    m_database->m_sqliteDatabase.setMaximumSize(m_database->maximumSize());
    127125   
    128126    ASSERT(!m_sqliteTransaction);
    … …  
    196194           
    197195            // Reset the maximum size here, as it was increased to allow us to retry this statement
    198             m_database->m_sqliteDatabase.setMaximumSize(DatabaseTracker::tracker().quotaForOrigin(m_database->securityOriginCopy().get()));
     196            m_database->m_sqliteDatabase.setMaximumSize(m_database->maximumSize());
    199197        } else {
    200198            // If the current statement has already been run, failed due to quota constraints, and we're not retrying it,
    … …  
    236234   
    237235    if (m_currentStatement->execute(m_database)) {
    238         // Flag this transaction as having changed the database for later delegate notification
    239         if (m_database->m_databaseAuthorizer->lastActionChangedDatabase())
     236        if (m_database->m_databaseAuthorizer->lastActionChangedDatabase()) {
     237            // Flag this transaction as having changed the database for later delegate notification
    240238            m_modifiedDatabase = true;
     239            // Also dirty the size of this database file for calculating quota usage
     240            OriginQuotaManager& manager(DatabaseTracker::tracker().originQuotaManager());
     241            Locker<OriginQuotaManager> locker(manager);
     242           
     243            manager.markDatabase(m_database);
     244        }
    241245           
    242246        if (m_currentStatement->hasStatementCallback()) {
Note: See TracChangeset for help on using the changeset viewer.