Changeset 260841 in webkit


Ignore:
Timestamp:
Apr 28, 2020 12:40:49 PM (4 years ago)
Author:
Kate Cheney
Message:

Create a mechanism to add missing ITP Database tables when the schema is updated
​https://bugs.webkit.org/show_bug.cgi?id=211004
<rdar://problem/62261187>

Reviewed by John Wilander.

Source/WebKit:

This patch updates the ITP database to better handle added tables to
the schema. Currently the database store deletes and re-creates the
database when any schema change is encountered. This is simple but
would result in ITP data loss during schema updates, so this patch
searches the schema for missing tables and adds any it finds.

  • NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:

(WebKit::createTableQueries):
Convert the tables array to a mapping of table names to their
respective CREATE TABLE queries so we can add missing tables on the
fly.
(WebKit::ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore):
(WebKit::ResourceLoadStatisticsDatabaseStore::openITPDatabase):
Remove call to createSchema() from the constructor. It makes more
sense to base this on whether the file is new as opposed to whether
one table exists.
(WebKit::ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema):
(WebKit::ResourceLoadStatisticsDatabaseStore::migrateDataToNewTablesIfNecessary):
(WebKit::ResourceLoadStatisticsDatabaseStore::addMissingTablesIfNecessary):
Checks for missing tables and adds them using a best-effort approach.
Call createUniqueIndices() to call all CREATE UNIQUE INDEX queries,
even though we may not need them all. It is much simpler than to
associate each query with its table, and it doesn't hurt to call
again.

(WebKit::ResourceLoadStatisticsDatabaseStore::openAndUpdateSchemaIfNecessary):
Add missing tables instead of deleting the old database file. Changed
the function name to reflect this. Deleted a FIXME which this patch
addresses but added a new FIXME for migrating data when adding new
columns, which this patch does not address.
(WebKit::ResourceLoadStatisticsDatabaseStore::isCorrectTableSchema): Deleted.
(WebKit::ResourceLoadStatisticsDatabaseStore::openAndDropOldDatabaseIfNecessary): Deleted.

  • NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h:
  • NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp:

(WebKit::WebResourceLoadStatisticsStore::statisticsDatabaseHasAllTables):

  • NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h:
  • NetworkProcess/NetworkProcess.cpp:

(WebKit::NetworkProcess::statisticsDatabaseHasAllTables):

  • NetworkProcess/NetworkProcess.h:
  • NetworkProcess/NetworkProcess.messages.in:
  • UIProcess/API/Cocoa/WKWebsiteDataStore.mm:

(-[WKWebsiteDataStore _statisticsDatabaseHasAllTables:]):

  • UIProcess/API/Cocoa/WKWebsiteDataStorePrivate.h:
  • UIProcess/Network/NetworkProcessProxy.cpp:

(WebKit::NetworkProcessProxy::statisticsDatabaseHasAllTables):

  • UIProcess/Network/NetworkProcessProxy.h:
  • UIProcess/WebsiteData/WebsiteDataStore.cpp:

(WebKit::WebsiteDataStore::statisticsDatabaseHasAllTables):

  • UIProcess/WebsiteData/WebsiteDataStore.h:

Added an SPI call to test that the database schema includes all
tables. This tests the database is not dropped when a new
table is added upon opening the database.

Tools:

Add test case which copies a database schema with a missing table into
the ITP database file, then ensures the pre-seeded data is
migrated over and that the schema now has all tables (including the
previously missing one).

  • TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
  • TestWebKitAPI/Tests/WebKitCocoa/ResourceLoadStatistics.mm:

(TEST):

  • TestWebKitAPI/Tests/WebKitCocoa/missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db: Added.
  • TestWebKitAPI/Tests/WebKitCocoa/missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db-shm: Added.
  • TestWebKitAPI/Tests/WebKitCocoa/missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db-wal: Added.
Location:
trunk
Files:
3 added
17 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebKit/ChangeLog

    r260840 r260841  
     12020-04-28  Kate Cheney  <katherine_cheney@apple.com>
     2
     3        Create a mechanism to add missing ITP Database tables when the schema is updated
     4        https://bugs.webkit.org/show_bug.cgi?id=211004
     5        <rdar://problem/62261187>
     6
     7        Reviewed by John Wilander.
     8
     9        This patch updates the ITP database to better handle added tables to
     10        the schema. Currently the database store deletes and re-creates the
     11        database when any schema change is encountered. This is simple but
     12        would result in ITP data loss during schema updates, so this patch
     13        searches the schema for missing tables and adds any it finds.
     14
     15        * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:
     16        (WebKit::createTableQueries):
     17        Convert the tables array to a mapping of table names to their
     18        respective CREATE TABLE queries so we can add missing tables on the
     19        fly.
     20        (WebKit::ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore):
     21        (WebKit::ResourceLoadStatisticsDatabaseStore::openITPDatabase):
     22        Remove call to createSchema() from the constructor. It makes more
     23        sense to base this on whether the file is new as opposed to whether
     24        one table exists.
     25        (WebKit::ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema):
     26        (WebKit::ResourceLoadStatisticsDatabaseStore::migrateDataToNewTablesIfNecessary):
     27        (WebKit::ResourceLoadStatisticsDatabaseStore::addMissingTablesIfNecessary):
     28        Checks for missing tables and adds them using a best-effort approach.
     29        Call createUniqueIndices() to call all CREATE UNIQUE INDEX queries,
     30        even though we may not need them all. It is much simpler than to
     31        associate each query with its table, and it doesn't hurt to call
     32        again.
     33
     34        (WebKit::ResourceLoadStatisticsDatabaseStore::openAndUpdateSchemaIfNecessary):
     35        Add missing tables instead of deleting the old database file. Changed
     36        the function name to reflect this. Deleted a FIXME which this patch
     37        addresses but added a new FIXME for migrating data when adding new
     38        columns, which this patch does not address.
     39        (WebKit::ResourceLoadStatisticsDatabaseStore::isCorrectTableSchema): Deleted.
     40        (WebKit::ResourceLoadStatisticsDatabaseStore::openAndDropOldDatabaseIfNecessary): Deleted.
     41        * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h:
     42        * NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp:
     43        (WebKit::WebResourceLoadStatisticsStore::statisticsDatabaseHasAllTables):
     44        * NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h:
     45        * NetworkProcess/NetworkProcess.cpp:
     46        (WebKit::NetworkProcess::statisticsDatabaseHasAllTables):
     47        * NetworkProcess/NetworkProcess.h:
     48        * NetworkProcess/NetworkProcess.messages.in:
     49        * UIProcess/API/Cocoa/WKWebsiteDataStore.mm:
     50        (-[WKWebsiteDataStore _statisticsDatabaseHasAllTables:]):
     51        * UIProcess/API/Cocoa/WKWebsiteDataStorePrivate.h:
     52        * UIProcess/Network/NetworkProcessProxy.cpp:
     53        (WebKit::NetworkProcessProxy::statisticsDatabaseHasAllTables):
     54        * UIProcess/Network/NetworkProcessProxy.h:
     55        * UIProcess/WebsiteData/WebsiteDataStore.cpp:
     56        (WebKit::WebsiteDataStore::statisticsDatabaseHasAllTables):
     57        * UIProcess/WebsiteData/WebsiteDataStore.h:
     58        Added an SPI call to test that the database schema includes all
     59        tables. This tests the database is not dropped when a new
     60        table is added upon opening the database.
     61
    1622020-04-28  Per Arne Vollan  <pvollan@apple.com>
    263

  • trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp

    r260668 r260841  
    147147constexpr auto removeAllDataQuery = "DELETE FROM ObservedDomains WHERE domainID = ?"_s;
    148148
    149 const char* tables[] = {
    150     "ObservedDomains",
    151     "TopLevelDomains",
    152     "StorageAccessUnderTopFrameDomains",
    153     "TopFrameUniqueRedirectsTo",
    154     "TopFrameUniqueRedirectsToSinceSameSiteStrictEnforcement",
    155     "TopFrameUniqueRedirectsFrom",
    156     "TopFrameLinkDecorationsFrom",
    157     "TopFrameLoadedThirdPartyScripts",
    158     "SubframeUnderTopFrameDomains",
    159     "SubresourceUnderTopFrameDomains",
    160     "SubresourceUniqueRedirectsTo",
    161     "SubresourceUniqueRedirectsFrom"
    162 };
    163 
    164 // CREATE TABLE Queries
    165149constexpr auto createObservedDomain = "CREATE TABLE ObservedDomains ("
    166150    "domainID INTEGER PRIMARY KEY, registrableDomain TEXT NOT NULL UNIQUE ON CONFLICT FAIL, lastSeen REAL NOT NULL, "
    … …  
    267251    return schema.contains("REFERENCES TopLevelDomains");
    268252}
     253
     254static const HashMap<String, String>& createTableQueries()
     255{
     256    static auto createTableQueries = makeNeverDestroyed(HashMap<String, String> {
     257        { "ObservedDomains"_s, createObservedDomain},
     258        { "TopLevelDomains"_s, createTopLevelDomains},
     259        { "StorageAccessUnderTopFrameDomains"_s, createStorageAccessUnderTopFrameDomains},
     260        { "TopFrameUniqueRedirectsTo"_s, createTopFrameUniqueRedirectsTo},
     261        { "TopFrameUniqueRedirectsToSinceSameSiteStrictEnforcement"_s, createTopFrameUniqueRedirectsToSinceSameSiteStrictEnforcement},
     262        { "TopFrameUniqueRedirectsFrom"_s, createTopFrameUniqueRedirectsFrom},
     263        { "TopFrameLinkDecorationsFrom"_s, createTopFrameLinkDecorationsFrom},
     264        { "TopFrameLoadedThirdPartyScripts"_s, createTopFrameLoadedThirdPartyScripts},
     265        { "SubframeUnderTopFrameDomains"_s, createSubframeUnderTopFrameDomains},
     266        { "SubresourceUnderTopFrameDomains"_s, createSubresourceUnderTopFrameDomains},
     267        { "SubresourceUniqueRedirectsTo"_s, createSubresourceUniqueRedirectsTo},
     268        { "SubresourceUniqueRedirectsFrom"_s, createSubresourceUniqueRedirectsFrom}
     269    });
     270   
     271    return createTableQueries;
     272}
     273
    269274
    270275ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore(WebResourceLoadStatisticsStore& store, WorkQueue& workQueue, ShouldIncludeLocalhost shouldIncludeLocalhost, const String& storageDirectoryPath, PAL::SessionID sessionID)
    … …  
    313318    ASSERT(!RunLoop::isMain());
    314319
    315     openAndDropOldDatabaseIfNecessary();
     320    openAndUpdateSchemaIfNecessary();
    316321    enableForeignKeys();
    317322
    318323    // Since we are using a workerQueue, the sequential dispatch blocks may be called by different threads.
    319324    m_database.disableThreadingChecks();
    320 
    321     if (!m_database.tableExists("ObservedDomains"_s)) {
     325   
     326    if (!m_database.turnOnIncrementalAutoVacuum())
     327        RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::turnOnIncrementalAutoVacuum failed, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
     328
     329    if (!prepareStatements()) {
     330        RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::prepareStatements failed, error message: %" PUBLIC_LOG_STRING ", database path: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg(), m_storageDirectoryPath.utf8().data());
     331        ASSERT_NOT_REACHED();
     332        return;
     333    }
     334
     335    workQueue.dispatchAfter(5_s, [weakThis = makeWeakPtr(*this)] {
     336        if (weakThis)
     337            weakThis->calculateAndSubmitTelemetry();
     338    });
     339}
     340
     341void ResourceLoadStatisticsDatabaseStore::openITPDatabase()
     342{
     343    if (!FileSystem::fileExists(m_storageDirectoryPath))
     344        m_isNewResourceLoadStatisticsDatabaseFile = true;
     345    else
     346        m_isNewResourceLoadStatisticsDatabaseFile = false;
     347
     348    if (!m_database.open(m_storageDirectoryPath)) {
     349        RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::open failed, error message: %" PUBLIC_LOG_STRING ", database path: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg(), m_storageDirectoryPath.utf8().data());
     350        ASSERT_NOT_REACHED();
     351    }
     352   
     353    if (m_isNewResourceLoadStatisticsDatabaseFile) {
    322354        if (!createSchema()) {
    323355            RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::createSchema failed, error message: %" PUBLIC_LOG_STRING ", database path: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg(), m_storageDirectoryPath.utf8().data());
    … …  
    326358        }
    327359    }
    328    
    329     if (!m_database.turnOnIncrementalAutoVacuum())
    330         RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::turnOnIncrementalAutoVacuum failed, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
    331 
    332     if (!prepareStatements()) {
    333         RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::prepareStatements failed, error message: %" PUBLIC_LOG_STRING ", database path: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg(), m_storageDirectoryPath.utf8().data());
    334         ASSERT_NOT_REACHED();
    335         return;
    336     }
    337 
    338     workQueue.dispatchAfter(5_s, [weakThis = makeWeakPtr(*this)] {
    339         if (weakThis)
    340             weakThis->calculateAndSubmitTelemetry();
    341     });
    342 }
    343 
    344 void ResourceLoadStatisticsDatabaseStore::openITPDatabase()
    345 {
    346     if (!FileSystem::fileExists(m_storageDirectoryPath))
    347         m_isNewResourceLoadStatisticsDatabaseFile = true;
    348     else
    349         m_isNewResourceLoadStatisticsDatabaseFile = false;
    350 
    351     if (!m_database.open(m_storageDirectoryPath)) {
    352         RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::open failed, error message: %" PUBLIC_LOG_STRING ", database path: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg(), m_storageDirectoryPath.utf8().data());
    353         ASSERT_NOT_REACHED();
    354     }
    355360}
    356361
    … …  
    361366}
    362367
    363 bool ResourceLoadStatisticsDatabaseStore::isCorrectTableSchema()
    364 {
     368Optional<Vector<String>> ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema()
     369{
     370    Vector<String> missingTables;
    365371    SQLiteStatement statement(m_database, "SELECT 1 from sqlite_master WHERE type='table' and tbl_name=?");
    366372    if (statement.prepare() != SQLITE_OK) {
    367         RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::isCorrectTableSchema failed to prepare, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
    368         return false;
    369     }
    370 
    371     bool hasAllTables = true;
    372     for (auto table : tables) {
     373        RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema failed to prepare, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
     374        return WTF::nullopt;
     375    }
     376
     377    for (auto& table : createTableQueries().keys()) {
    373378        if (statement.bindText(1, table) != SQLITE_OK) {
    374             RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::isCorrectTableSchema failed to bind, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
    375             return false;
     379            RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema failed to bind, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
     380            return WTF::nullopt;
    376381        }
    377382        if (statement.step() != SQLITE_ROW) {
    378             RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::isCorrectTableSchema schema is missing table: %s", this, table);
    379             hasAllTables = false;
     383            RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::checkForMissingTablesInSchema schema is missing table: %s", this, table.ascii().data());
     384            missingTables.append(String(table));
    380385        }
    381386        resetStatement(statement);
    382387    }
    383     return hasAllTables;
     388    if (missingTables.isEmpty())
     389        return WTF::nullopt;
     390
     391    return missingTables;
    384392}
    385393
    … …  
    425433    transaction.begin();
    426434
    427     for (auto& table : tables) {
     435    for (auto& table : createTableQueries().keys()) {
    428436        auto query = makeString("ALTER TABLE ", table, " RENAME TO _", table);
    429437        SQLiteStatement alterTable(m_database, query);
    … …  
    441449    }
    442450
    443     for (auto& table : tables) {
     451    for (auto& table : createTableQueries().keys()) {
    444452        auto query = makeString("INSERT INTO ", table, " SELECT * FROM _", table);
    445453        SQLiteStatement migrateTableData(m_database, query);
    … …  
    468476}
    469477
    470 void ResourceLoadStatisticsDatabaseStore::openAndDropOldDatabaseIfNecessary()
     478void ResourceLoadStatisticsDatabaseStore::addMissingTablesIfNecessary()
     479{
     480    auto missingTables = checkForMissingTablesInSchema();
     481    if (!missingTables)
     482        return;
     483
     484    for (auto& table : *missingTables) {
     485        auto createTableQuery = createTableQueries().get(table);
     486        if (!m_database.executeCommand(createTableQuery))
     487            RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::addMissingTables failed to execute, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg());
     488    }
     489
     490    if (!createUniqueIndices()) {
     491        RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::addMissingTables failed to create unique indices, error message: %{private}s", this, m_database.lastErrorMsg());
     492        ASSERT_NOT_REACHED();
     493        return;
     494    }
     495}
     496
     497void ResourceLoadStatisticsDatabaseStore::openAndUpdateSchemaIfNecessary()
    471498{
    472499    openITPDatabase();
    473 
    474     if (!isCorrectTableSchema()) {
    475         m_database.close();
    476         // FIXME: Migrate existing data to new database file instead of deleting it (204482).
    477         FileSystem::deleteFile(m_storageDirectoryPath);
    478         openITPDatabase();
    479         return;
    480     }
     500    addMissingTablesIfNecessary();
    481501
    482502    String currentSchema;
    … …  
    504524    ASSERT(!currentSchema.isEmpty());
    505525
    506     // If the schema in the ResourceLoadStatistics directory is not the current schema, delete the database file.
     526    // If the ObservedDomains schema in the ResourceLoadStatistics directory is not the current schema, delete the database file.
     527    // FIXME: Migrate old ObservedDomains data to new table schema.
    507528    if (currentSchema != ObservedDomainsTableSchemaV1() && currentSchema != ObservedDomainsTableSchemaV1Alternate()) {
    508529        m_database.close();

  • trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h

    r260668 r260841  
    141141    void removeDataForDomain(const RegistrableDomain&) override;
    142142    bool domainIDExistsInDatabase(int);
     143    Optional<Vector<String>> checkForMissingTablesInSchema();
    143144
    144145private:
    145146    void openITPDatabase();
    146     bool isCorrectTableSchema();
     147    void addMissingTablesIfNecessary();
    147148    void enableForeignKeys();
    148149    bool isMigrationNecessary();
    … …  
    150151    bool hasStorageAccess(const TopFrameDomain&, const SubFrameDomain&) const;
    151152    Vector<WebResourceLoadStatisticsStore::ThirdPartyDataForSpecificFirstParty> getThirdPartyDataForSpecificFirstPartyDomains(unsigned, const RegistrableDomain&) const;
    152     void openAndDropOldDatabaseIfNecessary();
     153    void openAndUpdateSchemaIfNecessary();
    153154    String getDomainStringFromDomainID(unsigned) const;
    154155    String getSubStatisticStatement(const String&) const;

  • trunk/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp

    r260791 r260841  
    323323}
    324324
     325void WebResourceLoadStatisticsStore::statisticsDatabaseHasAllTables(CompletionHandler<void(bool)>&& completionHandler)
     326{
     327    ASSERT(RunLoop::isMain());
     328   
     329    postTask([this, completionHandler = WTFMove(completionHandler)]() mutable {
     330        if (!m_statisticsStore || !is<ResourceLoadStatisticsDatabaseStore>(*m_statisticsStore)) {
     331            completionHandler(false);
     332            ASSERT_NOT_REACHED();
     333            return;
     334        }
     335        auto missingTables = downcast<ResourceLoadStatisticsDatabaseStore>(*m_statisticsStore).checkForMissingTablesInSchema();
     336        postTaskReply([hasAllTables = missingTables ? false : true, completionHandler = WTFMove(completionHandler)] () mutable {
     337            completionHandler(hasAllTables);
     338        });
     339    });
     340}
     341
    325342void WebResourceLoadStatisticsStore::resourceLoadStatisticsUpdated(Vector<ResourceLoadStatistics>&& statistics)
    326343{

  • trunk/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h

    r260791 r260841  
    250250    void scheduleCookieBlockingUpdateForDomains(const Vector<RegistrableDomain>&, CompletionHandler<void()>&&);
    251251    void scheduleStatisticsAndDataRecordsProcessing(CompletionHandler<void()>&&);
     252    void statisticsDatabaseHasAllTables(CompletionHandler<void(bool)>&&);
    252253    void submitTelemetry(CompletionHandler<void()>&&);
    253254    void scheduleClearInMemoryAndPersistent(ShouldGrandfatherStatistics, CompletionHandler<void()>&&);

  • trunk/Source/WebKit/NetworkProcess/NetworkProcess.cpp

    r260791 r260841  
    829829}
    830830
     831void NetworkProcess::statisticsDatabaseHasAllTables(PAL::SessionID sessionID, CompletionHandler<void(bool)>&& completionHandler)
     832{
     833    if (auto* networkSession = this->networkSession(sessionID)) {
     834        if (auto* resourceLoadStatistics = networkSession->resourceLoadStatistics())
     835            resourceLoadStatistics->statisticsDatabaseHasAllTables(WTFMove(completionHandler));
     836        else
     837            completionHandler(false);
     838    } else {
     839        ASSERT_NOT_REACHED();
     840        completionHandler(false);
     841    }
     842}
     843
    831844void NetworkProcess::setNotifyPagesWhenDataRecordsWereScanned(PAL::SessionID sessionID, bool value, CompletionHandler<void()>&& completionHandler)
    832845{

  • trunk/Source/WebKit/NetworkProcess/NetworkProcess.h

    r260668 r260841  
    238238    void scheduleCookieBlockingUpdate(PAL::SessionID, CompletionHandler<void()>&&);
    239239    void scheduleStatisticsAndDataRecordsProcessing(PAL::SessionID, CompletionHandler<void()>&&);
     240    void statisticsDatabaseHasAllTables(PAL::SessionID, CompletionHandler<void(bool)>&&);
    240241    void submitTelemetry(PAL::SessionID, CompletionHandler<void()>&&);
    241242    void setCacheMaxAgeCapForPrevalentResources(PAL::SessionID, Seconds, CompletionHandler<void()>&&);

  • trunk/Source/WebKit/NetworkProcess/NetworkProcess.messages.in

    r260668 r260841  
    112112    ScheduleCookieBlockingUpdate(PAL::SessionID sessionID) -> () Async
    113113    ScheduleStatisticsAndDataRecordsProcessing(PAL::SessionID sessionID) -> () Async
     114    StatisticsDatabaseHasAllTables(PAL::SessionID sessionID) -> (bool hasAllTables) Async
    114115    SubmitTelemetry(PAL::SessionID sessionID) -> () Async
    115116    SetCacheMaxAgeCapForPrevalentResources(PAL::SessionID sessionID, Seconds seconds) -> () Async

  • trunk/Source/WebKit/UIProcess/API/Cocoa/WKWebsiteDataStore.mm

    r260485 r260841  
    563563}
    564564
     565- (void)_statisticsDatabaseHasAllTables:(void (^)(BOOL))completionHandler
     566{
     567#if ENABLE(RESOURCE_LOAD_STATISTICS)
     568    _websiteDataStore->statisticsDatabaseHasAllTables([completionHandler = makeBlockPtr(completionHandler)](bool hasAllTables) {
     569        completionHandler(hasAllTables);
     570    });
     571#else
     572    completionHandler(NO);
     573#endif
     574}
     575
    565576- (void)_processStatisticsAndDataRecords:(void (^)(void))completionHandler
    566577{

  • trunk/Source/WebKit/UIProcess/API/Cocoa/WKWebsiteDataStorePrivate.h

    r260334 r260841  
    7878- (void)_isRegisteredAsSubresourceUnderFirstParty:(NSURL *)firstPartyURL thirdParty:(NSURL *)thirdPartyURL completionHandler:(void (^)(BOOL))completionHandler WK_API_AVAILABLE(macos(WK_MAC_TBA), ios(WK_IOS_TBA));
    7979- (void)_setThirdPartyCookieBlockingMode:(BOOL)enabled onlyOnSitesWithoutUserInteraction:(BOOL)onlyOnSitesWithoutUserInteraction completionHandler:(void (^)(void))completionHandler WK_API_AVAILABLE(macos(WK_MAC_TBA), ios(WK_IOS_TBA));
     80- (void)_statisticsDatabaseHasAllTables:(void (^)(BOOL))completionHandler WK_API_AVAILABLE(macos(WK_MAC_TBA), ios(WK_IOS_TBA));
    8081- (void)_processStatisticsAndDataRecords:(void (^)(void))completionHandler WK_API_AVAILABLE(macos(10.15), ios(13.0));
    8182- (void)_appBoundDomains:(void (^)(NSArray<NSString *> *))completionHandler WK_API_AVAILABLE(macos(WK_MAC_TBA), ios(WK_IOS_TBA));

  • trunk/Source/WebKit/UIProcess/Network/NetworkProcessProxy.cpp

    r260668 r260841  
    630630}
    631631
     632void NetworkProcessProxy::statisticsDatabaseHasAllTables(PAL::SessionID sessionID, CompletionHandler<void(bool)>&& completionHandler)
     633{
     634    if (!canSendMessage()) {
     635        completionHandler(false);
     636        return;
     637    }
     638   
     639    sendWithAsyncReply(Messages::NetworkProcess::StatisticsDatabaseHasAllTables(sessionID), WTFMove(completionHandler));
     640}
     641
    632642void NetworkProcessProxy::logUserInteraction(PAL::SessionID sessionID, const RegistrableDomain& resourceDomain, CompletionHandler<void()>&& completionHandler)
    633643{

  • trunk/Source/WebKit/UIProcess/Network/NetworkProcessProxy.h

    r260668 r260841  
    132132    void setLastSeen(PAL::SessionID, const RegistrableDomain&, Seconds, CompletionHandler<void()>&&);
    133133    void domainIDExistsInDatabase(PAL::SessionID, int domainID, CompletionHandler<void(bool)>&&);
     134    void statisticsDatabaseHasAllTables(PAL::SessionID, CompletionHandler<void(bool)>&&);
    134135    void mergeStatisticForTesting(PAL::SessionID, const RegistrableDomain&, const TopFrameDomain& topFrameDomain1, const TopFrameDomain& topFrameDomain2, Seconds lastSeen, bool hadUserInteraction, Seconds mostRecentUserInteraction, bool isGrandfathered, bool isPrevalent, bool isVeryPrevalent, unsigned dataRecordsRemoved, CompletionHandler<void()>&&);
    135136    void setAgeCapForClientSideCookies(PAL::SessionID, Optional<Seconds>, CompletionHandler<void()>&&);

  • trunk/Source/WebKit/UIProcess/WebsiteData/WebsiteDataStore.cpp

    r260668 r260841  
    15841584}
    15851585
     1586void WebsiteDataStore::statisticsDatabaseHasAllTables(CompletionHandler<void(bool)>&& completionHandler)
     1587{
     1588    ASSERT(RunLoop::isMain());
     1589
     1590    for (auto& processPool : processPools()) {
     1591        if (auto* process = processPool->networkProcess()) {
     1592            process->statisticsDatabaseHasAllTables(m_sessionID, WTFMove(completionHandler));
     1593            return;
     1594        }
     1595    }
     1596
     1597    completionHandler(false);
     1598}
     1599
    15861600void WebsiteDataStore::setLastSeen(const URL& url, Seconds seconds, CompletionHandler<void()>&& completionHandler)
    15871601{

  • trunk/Source/WebKit/UIProcess/WebsiteData/WebsiteDataStore.h

    r260668 r260841  
    169169    void setLastSeen(const URL&, Seconds, CompletionHandler<void()>&&);
    170170    void domainIDExistsInDatabase(int domainID, CompletionHandler<void(bool)>&&);
     171    void statisticsDatabaseHasAllTables(CompletionHandler<void(bool)>&&);
    171172    void mergeStatisticForTesting(const URL&, const URL& topFrameUrl1, const URL& topFrameUrl2, Seconds lastSeen, bool hadUserInteraction, Seconds mostRecentUserInteraction, bool isGrandfathered, bool isPrevalent, bool isVeryPrevalent, unsigned dataRecordsRemoved, CompletionHandler<void()>&&);
    172173    void setNotifyPagesWhenDataRecordsWereScanned(bool, CompletionHandler<void()>&&);

  • trunk/Tools/ChangeLog

    r260840 r260841  
     12020-04-28  Kate Cheney  <katherine_cheney@apple.com>
     2
     3        Create a mechanism to add missing ITP Database tables when the schema is updated
     4        https://bugs.webkit.org/show_bug.cgi?id=211004
     5        <rdar://problem/62261187>
     6
     7        Reviewed by John Wilander.
     8
     9        Add test case which copies a database schema with a missing table into
     10        the ITP database file, then ensures the pre-seeded data is
     11        migrated over and that the schema now has all tables (including the
     12        previously missing one).
     13
     14        * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
     15        * TestWebKitAPI/Tests/WebKitCocoa/ResourceLoadStatistics.mm:
     16        (TEST):
     17        * TestWebKitAPI/Tests/WebKitCocoa/missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db: Added.
     18        * TestWebKitAPI/Tests/WebKitCocoa/missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db-shm: Added.
     19        * TestWebKitAPI/Tests/WebKitCocoa/missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db-wal: Added.
     20
    1212020-04-28  Per Arne Vollan  <pvollan@apple.com>
    222

  • trunk/Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj

    r260769 r260841  
    257257                4909EE3A2D09480C88982D56 /* Markable.cpp in Sources */ = {isa = PBXBuildFile; fileRef = EC79F168BE454E579E417B05 /* Markable.cpp */; };
    258258                4971B1182451F29A0096994D /* incorrectCreateTableSchema.db in Copy Resources */ = {isa = PBXBuildFile; fileRef = 4971B1172451F2780096994D /* incorrectCreateTableSchema.db */; };
     259                4971B1202453A88C0096994D /* missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db in Copy Resources */ = {isa = PBXBuildFile; fileRef = 4971B11F2453A87F0096994D /* missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db */; };
    259260                49897D6C241FE9E400ECF153 /* in-app-browser-privacy-local-file.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = 49D7FBA7241FDDDA00AB67FA /* in-app-browser-privacy-local-file.html */; };
    260261                49AEEF6D2407359D00C87E4C /* InAppBrowserPrivacy.mm in Sources */ = {isa = PBXBuildFile; fileRef = 49AEEF6B2407358600C87E4C /* InAppBrowserPrivacy.mm */; };
    … …  
    14491450                                E1220DCA155B28AA0013E2FC /* MemoryCacheDisableWithinResourceLoadDelegate.html in Copy Resources */,
    14501451                                517E7E04151119C100D0B008 /* MemoryCachePruneWithinResourceLoadDelegate.html in Copy Resources */,
     1452                                4971B1202453A88C0096994D /* missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db in Copy Resources */,
    14511453                                51CD1C721B38D48400142CA5 /* modal-alerts-in-new-about-blank-window.html in Copy Resources */,
    14521454                                7A1458FC1AD5C07000E06772 /* mouse-button-listener.html in Copy Resources */,
    … …  
    18761878                46FACF7323E883EE00A9EBC6 /* beforeunload-slow.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "beforeunload-slow.html"; sourceTree = "<group>"; };
    18771879                4971B1172451F2780096994D /* incorrectCreateTableSchema.db */ = {isa = PBXFileReference; lastKnownFileType = file; path = incorrectCreateTableSchema.db; sourceTree = "<group>"; };
     1880                4971B11F2453A87F0096994D /* missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db */ = {isa = PBXFileReference; lastKnownFileType = file; name = missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db; path = Tests/WebKitCocoa/missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db; sourceTree = SOURCE_ROOT; };
    18781881                49AEEF682407276F00C87E4C /* Info.plist */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = "<group>"; };
    18791882                49AEEF6B2407358600C87E4C /* InAppBrowserPrivacy.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = InAppBrowserPrivacy.mm; sourceTree = "<group>"; };
    … …  
    40764079                                CD8394DE232AF15E00149495 /* media-loading.html */,
    40774080                                CDC9442B1EF1FBD20059C3C4 /* mediastreamtrack-detached.html */,
     4081                                4971B11F2453A87F0096994D /* missingTopFrameUniqueRedirectSameSiteStrictTableSchema.db */,
    40784082                                51CD1C711B38D48400142CA5 /* modal-alerts-in-new-about-blank-window.html */,
    40794083                                7A1458FB1AD5C03500E06772 /* mouse-button-listener.html */,

  • trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/ResourceLoadStatistics.mm

    r260594 r260841  
    14961496    TestWebKitAPI::Util::run(&doneFlag);
    14971497}
     1498
     1499TEST(ResourceLoadStatistics, MigrateDataFromMissingTopFrameUniqueRedirectSameSiteStrictTableSchema)
     1500{
     1501    auto *sharedProcessPool = [WKProcessPool _sharedProcessPool];
     1502
     1503    auto defaultFileManager = [NSFileManager defaultManager];
     1504    auto *dataStore = [WKWebsiteDataStore defaultDataStore];
     1505    NSURL *itpRootURL = [[dataStore _configuration] _resourceLoadStatisticsDirectory];
     1506    NSURL *fileURL = [itpRootURL URLByAppendingPathComponent:@"observations.db"];
     1507    [defaultFileManager removeItemAtPath:itpRootURL.path error:nil];
     1508    EXPECT_FALSE([defaultFileManager fileExistsAtPath:itpRootURL.path]);
     1509
     1510    // Load an incorrect database schema with pre-seeded ITP data.
     1511    [defaultFileManager createDirectoryAtURL:itpRootURL withIntermediateDirectories:YES attributes:nil error:nil];
     1512    NSURL *newFileURL = [[NSBundle mainBundle] URLForResource:@"missingTopFrameUniqueRedirectSameSiteStrictTableSchema" withExtension:@"db" subdirectory:@"TestWebKitAPI.resources"];
     1513    EXPECT_TRUE([defaultFileManager fileExistsAtPath:newFileURL.path]);
     1514    [defaultFileManager copyItemAtPath:newFileURL.path toPath:fileURL.path error:nil];
     1515    EXPECT_TRUE([defaultFileManager fileExistsAtPath:fileURL.path]);
     1516
     1517    auto configuration = adoptNS([[WKWebViewConfiguration alloc] init]);
     1518    [configuration setProcessPool: sharedProcessPool];
     1519    configuration.get().websiteDataStore = dataStore;
     1520
     1521    // We need an active NetworkProcess to perform ResourceLoadStatistics operations.
     1522    auto webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]);
     1523    [dataStore _setResourceLoadStatisticsEnabled:YES];
     1524    [webView loadRequest:[NSURLRequest requestWithURL:[NSURL URLWithString:@"about:blank"]]];
     1525
     1526    static bool doneFlag;
     1527    [dataStore _setUseITPDatabase:true completionHandler: ^(void) {
     1528        doneFlag = true;
     1529    }];
     1530
     1531    TestWebKitAPI::Util::run(&doneFlag);
     1532
     1533    // Since the database should not be deleted, the pre-seeded data should
     1534    // still be there after initializing ITP.
     1535    doneFlag = false;
     1536    [dataStore _isRegisteredAsSubresourceUnderFirstParty:[NSURL URLWithString:@"http://apple.com"] thirdParty:[NSURL URLWithString:@"http://webkit.org"] completionHandler: ^(BOOL isRegistered) {
     1537        EXPECT_TRUE(isRegistered);
     1538        doneFlag = true;
     1539    }];
     1540
     1541    TestWebKitAPI::Util::run(&doneFlag);
     1542   
     1543    // Check to make sure all tables are accounted for.
     1544    doneFlag = false;
     1545    [dataStore _statisticsDatabaseHasAllTables:^(BOOL hasAllTables) {
     1546        EXPECT_TRUE(hasAllTables);
     1547        doneFlag = true;
     1548    }];
     1549
     1550    TestWebKitAPI::Util::run(&doneFlag);
     1551}
Note: See TracChangeset for help on using the changeset viewer.