Changeset 58366 in webkit

Timestamp:

Apr 27, 2010 6:24:12 PM (14 years ago)

Author:

dumi@chromium.org

Message:

WebCore: Turn on AUTO_VACUUM = INCREMENTAL for all HTML5 databases.
​https://bugs.webkit.org/show_bug.cgi?id=38191

Reviewed by David Levin.

Vacuum all databases when the number of free pages is at least 10%
of the number of total pages. Also, add a guard against a bug that
was fixed in SQLite only starting with version 3.6.16.

• platform/sql/SQLiteDatabase.cpp:

(WebCore::SQLiteDatabase::maximumSize):
(WebCore::SQLiteDatabase::freeSpaceSize):
(WebCore::SQLiteDatabase::totalSize):
(WebCore::SQLiteDatabase::runIncrementalVacuumCommand):
(WebCore::SQLiteDatabase::turnOnIncrementalAutoVacuum):

• platform/sql/SQLiteDatabase.h:

(WebCore::SQLiteDatabase::):

• platform/sql/SQLiteStatement.cpp:

(WebCore::SQLiteStatement::prepare):
(WebCore::SQLiteStatement::step):

• storage/Database.cpp:

(WebCore::Database::performOpenAndVerify):
(WebCore::Database::incrementalVacuumIfNeeded):

• storage/Database.h:
• storage/SQLTransaction.cpp:

(WebCore::SQLTransaction::postflightAndCommit):

LayoutTests: Adjusting the expected amount of space used by quota-tracking.html.
​https://bugs.webkit.org/show_bug.cgi?id=38191

Reviewed by David Levin.

The expectations changed because of AUTO_VACUUM's overhead.

• platform/chromium/test_expectations.txt:
• storage/quota-tracking-expected.txt:
• storage/quota-tracking.html:

Location:

trunk

Files:

• LayoutTests/ChangeLog (modified) (1 diff)
• LayoutTests/platform/chromium/test_expectations.txt (modified) (1 diff)
• LayoutTests/storage/quota-tracking-expected.txt (modified) (1 diff)
• LayoutTests/storage/quota-tracking.html (modified) (2 diffs)
• WebCore/ChangeLog (modified) (1 diff)
• WebCore/platform/sql/SQLiteDatabase.cpp (modified) (4 diffs)
• WebCore/platform/sql/SQLiteDatabase.h (modified) (3 diffs)
• WebCore/platform/sql/SQLiteStatement.cpp (modified) (2 diffs)
• WebCore/storage/Database.cpp (modified) (2 diffs)
• WebCore/storage/Database.h (modified) (1 diff)
• WebCore/storage/SQLTransaction.cpp (modified) (1 diff)

trunk/LayoutTests/ChangeLog

@@ +1 @@
+2010-04-27  Dumitru Daniliuc  <dumi@chromium.org>
+
+        Reviewed by David Levin.
+
+        Adjusting the expected amount of space used by quota-tracking.html.
+        https://bugs.webkit.org/show_bug.cgi?id=38191
+
+        The expectations changed because of AUTO_VACUUM's overhead.
+
+        * platform/chromium/test_expectations.txt:
+        * storage/quota-tracking-expected.txt:
+        * storage/quota-tracking.html:
+
 2010-04-27  Evan Martin  <evan@chromium.org>
 

trunk/LayoutTests/platform/chromium/test_expectations.txt

@@ -2878 +2878 @@
 // Started failing at 58212
 BUGWK38108 MAC : fast/backgrounds/size/contain-and-cover.html = IMAGE
-
-// Started failing after revert.
-BUGWK38191 : storage/quota-tracking.html = TIMEOUT TEXT

trunk/LayoutTests/storage/quota-tracking-expected.txt

@@ -1 +1 @@
 UI DELEGATE DATABASE CALLBACK: exceededDatabaseQuotaForSecurityOrigin:{file, , 0} database:QuotaManagementDatabase2
 This test checks to make sure that quotas are enforced per-origin instead of per-database, as they were prior to http://trac.webkit.org/projects/webkit/changeset/29983.
-The test clears all databases, sets the quota for the origin to 32k, then tries to insert 17k of data into two databases. If things go as planned, the UI Delegate will be informed of the exceeded quota and should increase the quota for this origin. Inserting 17k of data the third time should succeed again.
+The test clears all databases, sets the quota for the origin to 40k, then tries to insert 17k of data into two databases. If things go as planned, the second insert should fail, the UI Delegate should be informed of the exceeded quota and should increase the quota for this origin. Inserting 17k of data the third time should succeed again.
 Adding a table
 Inserting some data

trunk/LayoutTests/storage/quota-tracking.html

@@ -92 +92 @@
         layoutTestController.clearAllDatabases();
         layoutTestController.dumpDatabaseCallbacks();
-        layoutTestController.setDatabaseQuota(32768);
+        layoutTestController.setDatabaseQuota(40960);
         layoutTestController.dumpAsText();
         layoutTestController.waitUntilDone();
@@ -106 +106 @@
 <body onload="runTest()">
 This test checks to make sure that quotas are enforced per-origin instead of per-database, as they were prior to http://trac.webkit.org/projects/webkit/changeset/29983.<br>
-The test clears all databases, sets the quota for the origin to 32k, then tries to insert 17k of data into two databases. If things go as planned, the UI Delegate will be informed of the exceeded quota and should increase the quota for this origin. Inserting 17k of data the third time should succeed again.
+The test clears all databases, sets the quota for the origin to 40k, then tries to insert 17k of data into two databases. If things go as planned, the second insert should fail, the UI Delegate should be informed of the exceeded quota and should increase the quota for this origin. Inserting 17k of data the third time should succeed again.
 <pre id="console">
 </pre>

trunk/WebCore/ChangeLog

@@ +1 @@
+2010-04-27  Dumitru Daniliuc  <dumi@chromium.org>
+
+        Reviewed by David Levin.
+
+        Turn on AUTO_VACUUM = INCREMENTAL for all HTML5 databases.
+        https://bugs.webkit.org/show_bug.cgi?id=38191
+
+        Vacuum all databases when the number of free pages is at least 10%
+        of the number of total pages. Also, add a guard against a bug that
+        was fixed in SQLite only starting with version 3.6.16.
+
+        * platform/sql/SQLiteDatabase.cpp:
+        (WebCore::SQLiteDatabase::maximumSize):
+        (WebCore::SQLiteDatabase::freeSpaceSize):
+        (WebCore::SQLiteDatabase::totalSize):
+        (WebCore::SQLiteDatabase::runIncrementalVacuumCommand):
+        (WebCore::SQLiteDatabase::turnOnIncrementalAutoVacuum):
+        * platform/sql/SQLiteDatabase.h:
+        (WebCore::SQLiteDatabase::):
+        * platform/sql/SQLiteStatement.cpp:
+        (WebCore::SQLiteStatement::prepare):
+        (WebCore::SQLiteStatement::step):
+        * storage/Database.cpp:
+        (WebCore::Database::performOpenAndVerify):
+        (WebCore::Database::incrementalVacuumIfNeeded):
+        * storage/Database.h:
+        * storage/SQLTransaction.cpp:
+        (WebCore::SQLTransaction::postflightAndCommit):
+
 2010-04-27  Garret Kelly  <gdk@chromium.org>
 

trunk/WebCore/platform/sql/SQLiteDatabase.cpp

@@ -103 +103 @@
 int64_t SQLiteDatabase::maximumSize()
 {
-    MutexLocker locker(m_authorizerLock);
-    enableAuthorizer(false);
-    
-    SQLiteStatement statement(*this, "PRAGMA max_page_count");
-    int64_t size = statement.getColumnInt64(0) * pageSize();
-    
-    enableAuthorizer(true);
-    return size;
+    int64_t maxPageCount = 0;
+
+    {
+        MutexLocker locker(m_authorizerLock);
+        enableAuthorizer(false);
+        SQLiteStatement statement(*this, "PRAGMA max_page_count");
+        maxPageCount = statement.getColumnInt64(0);
+        enableAuthorizer(true);
+    }
+
+    return maxPageCount * pageSize();
 }
 
@@ -154 +157 @@
 int64_t SQLiteDatabase::freeSpaceSize()
 {
-    MutexLocker locker(m_authorizerLock);
-    enableAuthorizer(false);
-    // Note: freelist_count was added in SQLite 3.4.1.
-    SQLiteStatement statement(*this, "PRAGMA freelist_count");
-    int64_t size = statement.getColumnInt64(0) * pageSize();
-
-    enableAuthorizer(true);
-    return size;
+    int64_t freelistCount = 0;
+
+    {
+        MutexLocker locker(m_authorizerLock);
+        enableAuthorizer(false);
+        // Note: freelist_count was added in SQLite 3.4.1.
+        SQLiteStatement statement(*this, "PRAGMA freelist_count");
+        freelistCount = statement.getColumnInt64(0);
+        enableAuthorizer(true);
+    }
+
+    return freelistCount * pageSize();
+}
+
+int64_t SQLiteDatabase::totalSize()
+{
+    int64_t pageCount = 0;
+
+    {
+        MutexLocker locker(m_authorizerLock);
+        enableAuthorizer(false);
+        SQLiteStatement statement(*this, "PRAGMA page_count");
+        pageCount = statement.getColumnInt64(0);
+        enableAuthorizer(true);
+    }
+
+    return pageCount * pageSize();
 }
 
@@ -228 +250 @@
     if (!executeCommand("VACUUM;"))
         LOG(SQLDatabase, "Unable to vacuum database - %s", lastErrorMsg());
+}
+
+void SQLiteDatabase::runIncrementalVacuumCommand()
+{
+    MutexLocker locker(m_authorizerLock);
+    enableAuthorizer(false);
+
+    if (!executeCommand("PRAGMA incremental_vacuum"))
+        LOG(SQLDatabase, "Unable to run incremental vacuum - %s", lastErrorMsg());
+
+    enableAuthorizer(true);
 }
 
@@ -380 +413 @@
 }
 
+bool SQLiteDatabase::turnOnIncrementalAutoVacuum()
+{
+    SQLiteStatement statement(*this, "PRAGMA auto_vacuum");
+    int autoVacuumMode = statement.getColumnInt(0);
+    int error = lastError();
+
+    // Check if we got an error while trying to get the value of the auto_vacuum flag.
+    // If we got a SQLITE_BUSY error, then there's probably another transaction in
+    // progress on this database. In this case, keep the current value of the
+    // auto_vacuum flag and try to set it to INCREMENTAL the next time we open this
+    // database. If the error is not SQLITE_BUSY, then we probably ran into a more
+    // serious problem and should return false (to log an error message).
+    if (error != SQLITE_ROW)
+        return false;
+
+    switch (autoVacuumMode) {
+    case AutoVacuumIncremental:
+        return true;
+    case AutoVacuumFull:
+        return executeCommand("PRAGMA auto_vacuum = 2");
+    case AutoVacuumNone:
+    default:
+        if (!executeCommand("PRAGMA auto_vacuum = 2"))
+            return false;
+        runVacuumCommand();
+        error = lastError();
+        return (error == SQLITE_OK);
+    }
+}
+
 } // namespace WebCore

trunk/WebCore/platform/sql/SQLiteDatabase.h

@@ -66 +66 @@
     void clearAllTables();
     void runVacuumCommand();
+    void runIncrementalVacuumCommand();
     
     bool transactionInProgress() const { return m_transactionInProgress; }
@@ -86 +87 @@
     // Gets the number of unused bytes in the database file.
     int64_t freeSpaceSize();
+    int64_t totalSize();
 
     // The SQLite SYNCHRONOUS pragma can be either FULL, NORMAL, or OFF
@@ -108 +110 @@
     void unlock();
     bool isAutoCommitOn() const;
+
+    // The SQLite AUTO_VACUUM pragma can be either NONE, FULL, or INCREMENTAL.
+    // NONE - SQLite does not do any vacuuming
+    // FULL - SQLite moves all empty pages to the end of the DB file and truncates
+    //        the file to remove those pages after every transaction. This option
+    //        requires SQLite to store additional information about each page in
+    //        the database file.
+    // INCREMENTAL - SQLite stores extra information for each page in the database
+    //               file, but removes the empty pages only when PRAGMA INCREMANTAL_VACUUM
+    //               is called.
+    enum AutoVacuumPragma { AutoVacuumNone = 0, AutoVacuumFull = 1, AutoVacuumIncremental = 2 };
+    bool turnOnIncrementalAutoVacuum();
 
     // Set this flag to allow access from multiple threads.  Not all multi-threaded accesses are safe!

trunk/WebCore/platform/sql/SQLiteStatement.cpp

@@ -66 +66 @@
     String strippedQuery = m_query.stripWhiteSpace();
     int error = sqlite3_prepare16_v2(m_database.sqlite3Handle(), strippedQuery.charactersWithNullTermination(), -1, &m_statement, &tail);
+
+    // Starting with version 3.6.16, sqlite has a patch (http://www.sqlite.org/src/ci/256ec3c6af)
+    // that should make sure sqlite3_prepare16_v2 doesn't return a SQLITE_SCHEMA error.
+    // If we're using an older sqlite version, try to emulate the patch.
+    if (error == SQLITE_SCHEMA) {
+      sqlite3_finalize(m_statement);
+      error = sqlite3_prepare16_v2(m_database.sqlite3Handle(), m_query.charactersWithNullTermination(), -1, &m_statement, &tail);
+    }
+
     if (error != SQLITE_OK)
         LOG(SQLDatabase, "sqlite3_prepare16 failed (%i)\n%s\n%s", error, m_query.ascii().data(), sqlite3_errmsg(m_database.sqlite3Handle()));
@@ -88 +97 @@
             error, m_query.ascii().data(), sqlite3_errmsg(m_database.sqlite3Handle()));
     }
+
     return error;
 }

trunk/WebCore/storage/Database.cpp

@@ -544 +544 @@
         return false;
     }
+    if (!m_sqliteDatabase.turnOnIncrementalAutoVacuum())
+        LOG_ERROR("Unable to turn on incremental auto-vacuum for database %s", m_filename.ascii().data());
 
     ASSERT(m_databaseAuthorizer);
@@ -809 +811 @@
 }
 
+void Database::incrementalVacuumIfNeeded()
+{
+    int64_t freeSpaceSize = m_sqliteDatabase.freeSpaceSize();
+    int64_t totalSize = m_sqliteDatabase.totalSize();
+    if (totalSize <= 10 * freeSpaceSize)
+        m_sqliteDatabase.runIncrementalVacuumCommand();
+}
+
 #endif // ENABLE(DATABASE)
 

trunk/WebCore/storage/Database.h

@@ -134 +134 @@
     SQLTransactionCoordinator* transactionCoordinator() const;
 
+    void incrementalVacuumIfNeeded();
+
 private:
     Database(ScriptExecutionContext* context, const String& name,

trunk/WebCore/storage/SQLTransaction.cpp

@@ -467 +467 @@
     }
 
+    // The commit was successful, so vacuum the database if needed
+    m_database->incrementalVacuumIfNeeded();
+
     // The commit was successful, notify the delegates if the transaction modified this database
     if (m_modifiedDatabase)