Changeset 268458 in webkit
- Timestamp:
- Oct 14, 2020 8:28:56 AM (4 years ago)
- Location:
- trunk
- Files:
-
- 4 deleted
- 41 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/LayoutTests/ChangeLog
r268454 r268458 1 2020-10-14 Kate Cheney <katherine_cheney@apple.com> 2 3 Remove ResourceLoadStatistics telemetry 4 https://bugs.webkit.org/show_bug.cgi?id=217665 5 <rdar://problem/69748988> 6 7 Reviewed by Sam Weinig. 8 9 Remove layout tests. 10 11 * http/tests/resourceLoadStatistics/telemetry-generation-expected.txt: Removed. 12 * http/tests/resourceLoadStatistics/telemetry-generation.html: Removed. 13 * platform/wk2/TestExpectations: 14 1 15 2020-10-14 Andres Gonzalez <andresg_22@apple.com> 2 16 -
trunk/LayoutTests/platform/wk2/TestExpectations
r268321 r268458 724 724 http/tests/resourceLoadStatistics/clear-in-memory-and-persistent-store.html [ Pass ] 725 725 http/tests/resourceLoadStatistics/grandfathering.html [ Pass ] 726 webkit.org/b/180703 http/tests/resourceLoadStatistics/telemetry-generation.html [ Pass Failure ]727 726 http/tests/resourceLoadStatistics/prune-statistics.html [ Pass ] 728 727 http/tests/resourceLoadStatistics [ Pass ] -
trunk/Source/WebKit/ChangeLog
r268457 r268458 1 2020-10-14 Kate Cheney <katherine_cheney@apple.com> 2 3 Remove ResourceLoadStatistics telemetry 4 https://bugs.webkit.org/show_bug.cgi?id=217665 5 <rdar://problem/69748988> 6 7 Reviewed by Sam Weinig. 8 9 Telemetry calculation in ITP is unused, very complicated, a potential 10 privacy risk, and it causes flaky tests. We should remove it. 11 12 No new tests. Existing ITP tests will check for no behavior change. 13 14 * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp: 15 (WebKit::ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore): 16 (WebKit::makeMedianWithUIQuery): Deleted. 17 (WebKit::buildQueryStartAndEnd): Deleted. 18 (WebKit::makeMedianWithoutUIQuery): Deleted. 19 (WebKit::getMedianOfPrevalentResourcesWithUserInteraction): Deleted. 20 (WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResources const): Deleted. 21 (WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithUI const): Deleted. 22 (WebKit::ResourceLoadStatisticsDatabaseStore::getTopPrevelentResourceDaysSinceUI const): Deleted. 23 (WebKit::getMedianOfPrevalentResourceWithoutUserInteraction): Deleted. 24 (WebKit::getNumberOfPrevalentResourcesInTopResources): Deleted. 25 (WebKit::makeStatisticQuery): Deleted. 26 (WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithoutUI const): Deleted. 27 (WebKit::ResourceLoadStatisticsDatabaseStore::calculateTelemetryData const): Deleted. 28 (WebKit::ResourceLoadStatisticsDatabaseStore::calculateAndSubmitTelemetry const): Deleted. 29 * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h: 30 * NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp: 31 (WebKit::ResourceLoadStatisticsMemoryStore::ResourceLoadStatisticsMemoryStore): 32 (WebKit::ResourceLoadStatisticsMemoryStore::calculateAndSubmitTelemetry const): Deleted. 33 * NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h: 34 * NetworkProcess/Classifier/ResourceLoadStatisticsStore.cpp: 35 (WebKit::ResourceLoadStatisticsStore::setShouldSubmitTelemetry): Deleted. 36 * NetworkProcess/Classifier/ResourceLoadStatisticsStore.h: 37 * NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp: 38 (WebKit::WebResourceLoadStatisticsStore::performDailyTasks): 39 (WebKit::WebResourceLoadStatisticsStore::setShouldSubmitTelemetry): Deleted. 40 (WebKit::WebResourceLoadStatisticsStore::submitTelemetry): Deleted. 41 (WebKit::WebResourceLoadStatisticsStore::notifyPageStatisticsTelemetryFinished const): Deleted. 42 * NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h: 43 * NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp: Removed. 44 * NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.h: Removed. 45 * NetworkProcess/NetworkProcess.cpp: 46 (WebKit::NetworkProcess::submitTelemetry): Deleted. 47 * NetworkProcess/NetworkProcess.h: 48 * NetworkProcess/NetworkProcess.messages.in: 49 * NetworkProcess/NetworkSession.cpp: 50 (WebKit::NetworkSession::notifyPageStatisticsTelemetryFinished): Deleted. 51 * NetworkProcess/NetworkSession.h: 52 * SourcesCocoa.txt: 53 * UIProcess/API/C/WKWebsiteDataStoreRef.cpp: 54 (WKWebsiteDataStoreStatisticsSubmitTelemetry): Deleted. 55 * UIProcess/API/C/WKWebsiteDataStoreRef.h: 56 * UIProcess/API/Cocoa/WKWebsiteDataStore.mm: 57 * UIProcess/Network/NetworkProcessProxy.cpp: 58 (WebKit::NetworkProcessProxy::submitTelemetry): Deleted. 59 (WebKit::NetworkProcessProxy::notifyResourceLoadStatisticsTelemetryFinished): Deleted. 60 * UIProcess/Network/NetworkProcessProxy.h: 61 * UIProcess/Network/NetworkProcessProxy.messages.in: 62 * UIProcess/WebProcessProxy.cpp: 63 (WebKit::WebProcessProxy::notifyPageStatisticsTelemetryFinished): Deleted. 64 * UIProcess/WebProcessProxy.h: 65 * UIProcess/WebsiteData/WebsiteDataStore.cpp: 66 (WebKit::WebsiteDataStore::submitTelemetry): Deleted. 67 * UIProcess/WebsiteData/WebsiteDataStore.h: 68 * WebKit.xcodeproj/project.pbxproj: 69 * PlatformFTW.cmake: 70 * SourcesGTK.txt: 71 * PlatformPlayStation.cmake: 72 * PlatformWin.cmake: 73 * SourcesWPE.txt: 74 1 75 2020-10-14 Kate Cheney <katherine_cheney@apple.com> 2 76 -
trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp
r266237 r268458 36 36 #include "StorageAccessStatus.h" 37 37 #include "WebProcessProxy.h" 38 #include "WebResourceLoadStatisticsTelemetry.h"39 38 #include "WebsiteDataStore.h" 40 39 #include <JavaScriptCore/ConsoleTypes.h> … … 235 234 constexpr auto createUniqueIndexOperatingDates = "CREATE UNIQUE INDEX IF NOT EXISTS OperatingDates_year_month_monthDay on OperatingDates ( year, month, monthDay );"_s; 236 235 237 const unsigned minimumPrevalentResourcesForTelemetry = 3;238 239 236 static const String ObservedDomainsTableSchemaV1() 240 237 { … … 296 293 if (!m_database.turnOnIncrementalAutoVacuum()) 297 294 RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::turnOnIncrementalAutoVacuum failed, error message: %" PUBLIC_LOG_STRING, this, m_database.lastErrorMsg()); 298 299 workQueue.dispatchAfter(5_s, [weakThis = makeWeakPtr(*this)] {300 if (weakThis)301 weakThis->calculateAndSubmitTelemetry();302 });303 295 304 296 includeTodayAsOperatingDateIfNecessary(); … … 943 935 } 944 936 945 static SQLiteStatement makeMedianWithUIQuery(SQLiteDatabase& database)946 {947 return SQLiteStatement(database, makeString("SELECT mostRecentUserInteractionTime FROM ObservedDomains INNER JOIN (SELECT ", joinSubStatisticsForSorting(), ") as q ON ObservedDomains.domainID = q.domainID LIMIT 1 OFFSET ?"));948 }949 950 937 Vector<WebResourceLoadStatisticsStore::ThirdPartyDataForSpecificFirstParty> ResourceLoadStatisticsDatabaseStore::getThirdPartyDataForSpecificFirstPartyDomains(unsigned thirdPartyDomainID, const RegistrableDomain& thirdPartyDomain) const 951 938 { … … 993 980 } 994 981 return thirdPartyDataList; 995 }996 997 static std::pair<StringView, StringView> buildQueryStartAndEnd(PrevalentResourceDatabaseTelemetry::Statistic statistic)998 {999 switch (statistic) {1000 case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubFrameWithoutUI:1001 return std::make_pair("SELECT countSubFrameUnderTopFrame FROM ObservedDomains o INNER JOIN(SELECT countSubFrameUnderTopFrame, ", ") as q ON o.domainID = q.domainID LIMIT 1 OFFSET ?");1002 case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubResourceWithoutUI:1003 return std::make_pair("SELECT countSubResourceUnderTopFrame FROM ObservedDomains o INNER JOIN(SELECT countSubResourceUnderTopFrame, ", ") as q ON o.domainID = q.domainID LIMIT 1 OFFSET ?");1004 case PrevalentResourceDatabaseTelemetry::Statistic::MedianUniqueRedirectsWithoutUI:1005 return std::make_pair("SELECT countUniqueRedirectTo FROM ObservedDomains o INNER JOIN(SELECT countUniqueRedirectTo, ", ") as q ON o.domainID = q.domainID LIMIT 1 OFFSET ?");1006 case PrevalentResourceDatabaseTelemetry::Statistic::MedianDataRecordsRemovedWithoutUI:1007 return std::make_pair("SELECT dataRecordsRemoved FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", ") as q ON o.domainID = q.domainID) LIMIT 1 OFFSET ?");1008 case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToUserInteractionWithoutUI:1009 return std::make_pair("SELECT timesAccessedAsFirstPartyDueToUserInteraction FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", ") as q ON o.domainID = q.domainID) LIMIT 1 OFFSET ?");1010 case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToStorageAccessAPIWithoutUI:1011 return std::make_pair("SELECT timesAccessedAsFirstPartyDueToStorageAccessAPI FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", ") as q ON o.domainID = q.domainID) LIMIT 1 OFFSET ?");1012 case PrevalentResourceDatabaseTelemetry::Statistic::NumberOfPrevalentResourcesWithUI:1013 LOG_ERROR("ResourceLoadStatisticsDatabaseStore::makeMedianWithoutUIQuery was called for an incorrect statistic, undetermined query behavior will result.");1014 RELEASE_ASSERT_NOT_REACHED();1015 }1016 return { };1017 }1018 1019 static SQLiteStatement makeMedianWithoutUIQuery(SQLiteDatabase& database, PrevalentResourceDatabaseTelemetry::Statistic statistic)1020 {1021 auto[queryStart, queryEnd] = buildQueryStartAndEnd(statistic);1022 1023 return SQLiteStatement(database, makeString(queryStart, joinSubStatisticsForSorting(), queryEnd));1024 }1025 1026 static unsigned getMedianOfPrevalentResourcesWithUserInteraction(SQLiteDatabase& database, unsigned prevalentResourcesWithUserInteractionCount)1027 {1028 SQLiteStatement medianDaysSinceUIStatement = makeMedianWithUIQuery(database);1029 1030 // Prepare1031 if (medianDaysSinceUIStatement.prepare() != SQLITE_OK) {1032 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %" PUBLIC_LOG_STRING, database.lastErrorMsg());1033 ASSERT_NOT_REACHED();1034 return 0;1035 }1036 1037 // Bind1038 if (medianDaysSinceUIStatement.bindInt(1, 1) != SQLITE_OK || medianDaysSinceUIStatement.bindInt(2, 1) != SQLITE_OK || medianDaysSinceUIStatement.bindInt(3, (prevalentResourcesWithUserInteractionCount / 2) != SQLITE_OK)) {1039 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %" PUBLIC_LOG_STRING, database.lastErrorMsg());1040 ASSERT_NOT_REACHED();1041 return 0;1042 }1043 1044 // Step1045 if (medianDaysSinceUIStatement.step() != SQLITE_ROW)1046 return 0;1047 1048 double rawSeconds = medianDaysSinceUIStatement.getColumnDouble(0);1049 WallTime wallTime = WallTime::fromRawSeconds(rawSeconds);1050 unsigned median = wallTime <= WallTime() ? 0 : std::floor((WallTime::now() - wallTime) / 24_h);1051 1052 if (prevalentResourcesWithUserInteractionCount & 1)1053 return median;1054 1055 SQLiteStatement lowerMedianDaysSinceUIStatement = makeMedianWithUIQuery(database);1056 1057 // Prepare1058 if (lowerMedianDaysSinceUIStatement.prepare() != SQLITE_OK) {1059 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %" PUBLIC_LOG_STRING, database.lastErrorMsg());1060 ASSERT_NOT_REACHED();1061 return 0;1062 }1063 1064 // Bind1065 if (lowerMedianDaysSinceUIStatement.bindInt(1, 1) != SQLITE_OK || lowerMedianDaysSinceUIStatement.bindInt(2, 1) != SQLITE_OK || lowerMedianDaysSinceUIStatement.bindInt(3, ((prevalentResourcesWithUserInteractionCount - 1) / 2)) != SQLITE_OK) {1066 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %" PUBLIC_LOG_STRING, database.lastErrorMsg());1067 ASSERT_NOT_REACHED();1068 return 0;1069 }1070 1071 // Step1072 if (lowerMedianDaysSinceUIStatement.step() == SQLITE_ROW) {1073 double rawSecondsLower = lowerMedianDaysSinceUIStatement.getColumnDouble(0);1074 WallTime wallTimeLower = WallTime::fromRawSeconds(rawSecondsLower);1075 return ((wallTimeLower <= WallTime() ? 0 : std::floor((WallTime::now() - wallTimeLower) / 24_h)) + median) / 2;1076 }1077 return 0;1078 }1079 1080 unsigned ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResources() const1081 {1082 auto scopedStatement = this->scopedStatement(m_countPrevalentResourcesStatement, countPrevalentResourcesQuery, "getNumberOfPrevalentResources"_s);1083 if (!scopedStatement)1084 return 0;1085 1086 auto stepValue = scopedStatement->step();1087 if (stepValue != SQLITE_ROW && stepValue != SQLITE_DONE) {1088 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResources failed to step, error message: %" PUBLIC_LOG_STRING, m_database.lastErrorMsg());1089 ASSERT_NOT_REACHED();1090 return 0;1091 }1092 if (stepValue == SQLITE_ROW) {1093 unsigned prevalentResourceCount = scopedStatement->getColumnInt(0);1094 if (prevalentResourceCount >= minimumPrevalentResourcesForTelemetry)1095 return prevalentResourceCount;1096 }1097 return 0;1098 }1099 1100 unsigned ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithUI() const1101 {1102 auto scopedStatement = this->scopedStatement(m_countPrevalentResourcesWithUserInteractionStatement, countPrevalentResourcesWithUserInteractionQuery, "getNumberOfPrevalentResourcesWithUI"_s);1103 if (scopedStatement && scopedStatement->step() == SQLITE_ROW) {1104 int count = scopedStatement->getColumnInt(0);1105 return count;1106 }1107 return 0;1108 }1109 1110 unsigned ResourceLoadStatisticsDatabaseStore::getTopPrevelentResourceDaysSinceUI() const1111 {1112 SQLiteStatement topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement(m_database, makeString("SELECT mostRecentUserInteractionTime FROM ObservedDomains INNER JOIN (SELECT ", joinSubStatisticsForSorting(), " LIMIT 1) as q ON ObservedDomains.domainID = q.domainID;"));1113 1114 // Prepare1115 if (topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.prepare() != SQLITE_OK) {1116 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement query failed to prepare, error message: %" PUBLIC_LOG_STRING, m_database.lastErrorMsg());1117 ASSERT_NOT_REACHED();1118 return 0;1119 }1120 1121 // Bind1122 if (topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.bindInt(1, 1) != SQLITE_OK1123 || topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.bindInt(2, 1) != SQLITE_OK) {1124 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement query failed to bind, error message: %" PUBLIC_LOG_STRING, m_database.lastErrorMsg());1125 ASSERT_NOT_REACHED();1126 return 0;1127 }1128 1129 // Step1130 if (topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.step() == SQLITE_ROW) {1131 double rawSeconds = topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.getColumnDouble(0);1132 WallTime wallTime = WallTime::fromRawSeconds(rawSeconds);1133 return wallTime <= WallTime() ? 0 : std::floor((WallTime::now() - wallTime) / 24_h);1134 }1135 return 0;1136 }1137 1138 static unsigned getMedianOfPrevalentResourceWithoutUserInteraction(SQLiteDatabase& database, unsigned bucketSize, PrevalentResourceDatabaseTelemetry::Statistic statistic, unsigned numberOfPrevalentResourcesWithoutUI)1139 {1140 if (numberOfPrevalentResourcesWithoutUI < bucketSize)1141 return 0;1142 1143 unsigned median = 0;1144 SQLiteStatement getMedianStatistic = makeMedianWithoutUIQuery(database, statistic);1145 1146 if (getMedianStatistic.prepare() == SQLITE_OK) {1147 if (getMedianStatistic.bindInt(1, 1) != SQLITE_OK1148 || getMedianStatistic.bindInt(2, 0) != SQLITE_OK1149 || getMedianStatistic.bindInt(3, (bucketSize / 2)) != SQLITE_OK) {1150 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::makeMedianWithoutUIQuery, error message: %" PUBLIC_LOG_STRING, database.lastErrorMsg());1151 ASSERT_NOT_REACHED();1152 return 0;1153 }1154 if (getMedianStatistic.step() == SQLITE_ROW)1155 median = getMedianStatistic.getColumnDouble(0);1156 }1157 1158 if (bucketSize & 1)1159 return median;1160 1161 SQLiteStatement getLowerMedianStatistic = makeMedianWithoutUIQuery(database, statistic);1162 1163 if (getLowerMedianStatistic.prepare() == SQLITE_OK) {1164 if (getLowerMedianStatistic.bindInt(1, 1) != SQLITE_OK1165 || getLowerMedianStatistic.bindInt(2, 0) != SQLITE_OK1166 || getLowerMedianStatistic.bindInt(2, ((bucketSize-1) / 2)) != SQLITE_OK) {1167 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::makeMedianWithoutUIQuery, error message: %" PUBLIC_LOG_STRING, database.lastErrorMsg());1168 ASSERT_NOT_REACHED();1169 return 0;1170 }1171 if (getLowerMedianStatistic.step() == SQLITE_ROW)1172 return (getLowerMedianStatistic.getColumnDouble(0) + median) / 2;1173 }1174 1175 return 0;1176 }1177 1178 static unsigned getNumberOfPrevalentResourcesInTopResources(SQLiteDatabase& database, unsigned bucketSize)1179 {1180 SQLiteStatement prevalentResourceCountInTop(database, makeString("SELECT COUNT(*) FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", joinSubStatisticsForSorting(), ") as q on q.domainID = o.domainID LIMIT ?) as p WHERE p.hadUserInteraction = 1;"));1181 1182 if (prevalentResourceCountInTop.prepare() == SQLITE_OK) {1183 if (prevalentResourceCountInTop.bindInt(1, 1) != SQLITE_OK1184 || prevalentResourceCountInTop.bindText(2, "%") != SQLITE_OK1185 || prevalentResourceCountInTop.bindInt(3, bucketSize) != SQLITE_OK) {1186 RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesInTopResources, error message: %" PUBLIC_LOG_STRING, database.lastErrorMsg());1187 ASSERT_NOT_REACHED();1188 return 0;1189 }1190 1191 if (prevalentResourceCountInTop.step() == SQLITE_ROW)1192 return prevalentResourceCountInTop.getColumnInt(0);1193 }1194 1195 return 0;1196 }1197 1198 static unsigned makeStatisticQuery(SQLiteDatabase& database, PrevalentResourceDatabaseTelemetry::Statistic statistic, int bucketSize, unsigned totalWithUI, unsigned totalWithoutUI)1199 {1200 switch (statistic) {1201 case PrevalentResourceDatabaseTelemetry::Statistic::NumberOfPrevalentResourcesWithUI:1202 return getNumberOfPrevalentResourcesInTopResources(database, bucketSize);1203 case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubFrameWithoutUI:1204 case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubResourceWithoutUI:1205 case PrevalentResourceDatabaseTelemetry::Statistic::MedianUniqueRedirectsWithoutUI:1206 case PrevalentResourceDatabaseTelemetry::Statistic::MedianDataRecordsRemovedWithoutUI:1207 case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToUserInteractionWithoutUI:1208 case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToStorageAccessAPIWithoutUI:1209 return getMedianOfPrevalentResourceWithoutUserInteraction(database, bucketSize, statistic, totalWithoutUI);1210 }1211 ASSERT_NOT_REACHED();1212 return 0;1213 }1214 1215 unsigned ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithoutUI() const1216 {1217 auto scopedStatement = this->scopedStatement(m_countPrevalentResourcesWithoutUserInteractionStatement, countPrevalentResourcesWithoutUserInteractionQuery, "getNumberOfPrevalentResourcesWithoutUI"_s);1218 if (scopedStatement && scopedStatement->step() == SQLITE_ROW) {1219 int count = m_countPrevalentResourcesWithoutUserInteractionStatement->getColumnInt(0);1220 return count;1221 }1222 return 0;1223 }1224 1225 void ResourceLoadStatisticsDatabaseStore::calculateTelemetryData(PrevalentResourceDatabaseTelemetry& data) const1226 {1227 data.numberOfPrevalentResources = getNumberOfPrevalentResources();1228 data.numberOfPrevalentResourcesWithUserInteraction = getNumberOfPrevalentResourcesWithUI();1229 data.numberOfPrevalentResourcesWithoutUserInteraction = getNumberOfPrevalentResourcesWithoutUI();1230 data.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction = getTopPrevelentResourceDaysSinceUI();1231 data.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction = getMedianOfPrevalentResourcesWithUserInteraction(m_database, data.numberOfPrevalentResourcesWithUserInteraction);1232 1233 for (unsigned bucketIndex = 0; bucketIndex < bucketSizes.size(); bucketIndex++) {1234 unsigned bucketSize = bucketSizes[bucketIndex];1235 1236 if (data.numberOfPrevalentResourcesWithoutUserInteraction < bucketSize)1237 return;1238 1239 for (unsigned statisticIndex = 0; statisticIndex < numberOfStatistics; statisticIndex++) {1240 auto statistic = static_cast<PrevalentResourceDatabaseTelemetry::Statistic>(statisticIndex);1241 data.statistics[statisticIndex][bucketIndex] = makeStatisticQuery(m_database, statistic, bucketSize, data.numberOfPrevalentResourcesWithUserInteraction, data.numberOfPrevalentResourcesWithoutUserInteraction);1242 }1243 }1244 }1245 1246 void ResourceLoadStatisticsDatabaseStore::calculateAndSubmitTelemetry(NotifyPagesForTesting shouldNotifyPagesForTesting) const1247 {1248 ASSERT(!RunLoop::isMain());1249 1250 if (parameters().shouldSubmitTelemetry) {1251 PrevalentResourceDatabaseTelemetry prevalentResourceDatabaseTelemetry;1252 calculateTelemetryData(prevalentResourceDatabaseTelemetry);1253 WebResourceLoadStatisticsTelemetry::submitTelemetry(*this, prevalentResourceDatabaseTelemetry, shouldNotifyPagesForTesting);1254 }1255 982 } 1256 983 -
trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h
r267750 r268458 53 53 static constexpr std::array<unsigned, numberOfBucketsPerStatistic> bucketSizes {{ 1, 3, 10, 50, 100 }}; 54 54 55 struct PrevalentResourceDatabaseTelemetry {56 using Buckets = std::array<unsigned, numberOfBucketsPerStatistic>;57 58 enum class Statistic {59 NumberOfPrevalentResourcesWithUI,60 MedianSubFrameWithoutUI,61 MedianSubResourceWithoutUI,62 MedianUniqueRedirectsWithoutUI,63 MedianDataRecordsRemovedWithoutUI,64 MedianTimesAccessedDueToUserInteractionWithoutUI,65 MedianTimesAccessedDueToStorageAccessAPIWithoutUI66 };67 68 unsigned numberOfPrevalentResources;69 unsigned numberOfPrevalentResourcesWithUserInteraction;70 unsigned numberOfPrevalentResourcesWithoutUserInteraction;71 unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction;72 unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction;73 74 std::array<Buckets, numberOfStatistics> statistics;75 };76 77 55 class ResourceLoadStatisticsMemoryStore; 78 56 … … 121 99 void setTopFrameUniqueRedirectTo(const TopFrameDomain&, const RedirectDomain&) override; 122 100 void setTopFrameUniqueRedirectFrom(const TopFrameDomain&, const RedirectDomain&) override; 123 124 void calculateAndSubmitTelemetry(NotifyPagesForTesting = NotifyPagesForTesting::No) const override;125 101 126 102 void hasStorageAccess(const SubFrameDomain&, const TopFrameDomain&, Optional<WebCore::FrameIdentifier>, WebCore::PageIdentifier, CompletionHandler<void(bool)>&&) override; … … 172 148 void merge(WebCore::SQLiteStatement*, const ResourceLoadStatistics&); 173 149 void clearDatabaseContents(); 174 unsigned getNumberOfPrevalentResources() const;175 unsigned getNumberOfPrevalentResourcesWithUI() const;176 unsigned getNumberOfPrevalentResourcesWithoutUI() const;177 unsigned getTopPrevelentResourceDaysSinceUI() const;178 void resetTelemetryPreparedStatements() const;179 void resetTelemetryStatements() const;180 void calculateTelemetryData(PrevalentResourceDatabaseTelemetry&) const;181 150 bool insertObservedDomain(const ResourceLoadStatistics&) WARN_UNUSED_RETURN; 182 151 void insertDomainRelationships(const ResourceLoadStatistics&); -
trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp
r267750 r268458 35 35 #include "StorageAccessStatus.h" 36 36 #include "WebProcessProxy.h" 37 #include "WebResourceLoadStatisticsTelemetry.h"38 37 #include "WebsiteDataStore.h" 39 38 #include <JavaScriptCore/ConsoleTypes.h> … … 75 74 { 76 75 RELEASE_ASSERT(!RunLoop::isMain()); 77 78 workQueue.dispatchAfter(5_s, [weakThis = makeWeakPtr(*this)] {79 if (weakThis)80 weakThis->calculateAndSubmitTelemetry();81 });82 83 76 includeTodayAsOperatingDateIfNecessary(); 84 77 } … … 89 82 90 83 return m_resourceStatisticsMap.isEmpty(); 91 }92 93 void ResourceLoadStatisticsMemoryStore::calculateAndSubmitTelemetry(NotifyPagesForTesting shouldNotifyPagesForTesting) const94 {95 ASSERT(!RunLoop::isMain());96 97 if (parameters().shouldSubmitTelemetry)98 WebResourceLoadStatisticsTelemetry::calculateAndSubmit(*this, shouldNotifyPagesForTesting);99 84 } 100 85 -
trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h
r267750 r268458 91 91 void setTopFrameUniqueRedirectFrom(const TopFrameDomain&, const RedirectDomain&) override; 92 92 93 void calculateAndSubmitTelemetry(NotifyPagesForTesting = NotifyPagesForTesting::No) const override;94 95 93 bool areAllThirdPartyCookiesBlockedUnder(const TopFrameDomain&) override; 96 94 CookieAccess cookieAccess(const ResourceLoadStatistics&, const TopFrameDomain&); -
trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.cpp
r267750 r268458 36 36 #include "StorageAccessStatus.h" 37 37 #include "WebProcessProxy.h" 38 #include "WebResourceLoadStatisticsTelemetry.h"39 38 #include "WebsiteDataStore.h" 40 39 #include <WebCore/CookieJar.h> … … 172 171 } 173 172 174 void ResourceLoadStatisticsStore::setShouldSubmitTelemetry(bool value)175 {176 ASSERT(!RunLoop::isMain());177 m_parameters.shouldSubmitTelemetry = value;178 }179 180 173 void ResourceLoadStatisticsStore::removeDataRecords(CompletionHandler<void()>&& completionHandler) 181 174 { -
trunk/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.h
r267750 r268458 30 30 #include "ResourceLoadStatisticsClassifier.h" 31 31 #include "WebResourceLoadStatisticsStore.h" 32 #include "WebResourceLoadStatisticsTelemetry.h"33 32 #include <JavaScriptCore/ConsoleTypes.h> 34 33 #include <WebCore/FrameIdentifier.h> … … 154 153 void resetParametersToDefaultValues(); 155 154 156 virtual void calculateAndSubmitTelemetry(NotifyPagesForTesting = NotifyPagesForTesting::No) const = 0;157 158 155 void setNotifyPagesWhenDataRecordsWereScanned(bool); 159 156 void setIsRunningTest(bool); 160 157 bool shouldSkip(const RegistrableDomain&) const; 161 158 void setShouldClassifyResourcesBeforeDataRecordsRemoval(bool); 162 void setShouldSubmitTelemetry(bool);163 159 void setTimeToLiveUserInteraction(Seconds); 164 160 void setMinimumTimeBetweenDataRecordsRemoval(Seconds); … … 241 237 bool shouldClassifyResourcesBeforeDataRecordsRemoval { true }; 242 238 size_t minimumTopFrameRedirectsForSameSiteStrictEnforcement { 10 }; 243 bool shouldSubmitTelemetry { true };244 239 bool isRunningTest { false }; 245 240 }; -
trunk/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp
r267750 r268458 43 43 #include "WebProcessPool.h" 44 44 #include "WebProcessProxy.h" 45 #include "WebResourceLoadStatisticsTelemetry.h"46 45 #include "WebsiteDataFetchOption.h" 47 46 #include <WebCore/CookieJar.h> … … 149 148 } 150 149 151 void WebResourceLoadStatisticsStore::setShouldSubmitTelemetry(bool value)152 {153 ASSERT(RunLoop::isMain());154 155 postTask([this, value] {156 if (m_statisticsStore)157 m_statisticsStore->setShouldSubmitTelemetry(value);158 });159 }160 161 150 static Ref<WorkQueue> sharedStatisticsQueue() 162 151 { … … 729 718 if (m_statisticsStore) { 730 719 m_statisticsStore->includeTodayAsOperatingDateIfNecessary(); 731 m_statisticsStore->calculateAndSubmitTelemetry();732 720 if (is<ResourceLoadStatisticsDatabaseStore>(*m_statisticsStore)) 733 721 downcast<ResourceLoadStatisticsDatabaseStore>(*m_statisticsStore).runIncrementalVacuumCommand(); 734 722 } 735 });736 }737 738 void WebResourceLoadStatisticsStore::submitTelemetry(CompletionHandler<void()>&& completionHandler)739 {740 ASSERT(RunLoop::isMain());741 742 postTask([this, completionHandler = WTFMove(completionHandler)]() mutable {743 if (!m_statisticsStore)744 return;745 m_statisticsStore->calculateAndSubmitTelemetry(NotifyPagesForTesting::Yes);746 postTaskReply(WTFMove(completionHandler));747 723 }); 748 724 } … … 1415 1391 } 1416 1392 1417 void WebResourceLoadStatisticsStore::notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI) const1418 {1419 ASSERT(RunLoop::isMain());1420 if (m_networkSession)1421 const_cast<WebResourceLoadStatisticsStore*>(this)->networkSession()->notifyPageStatisticsTelemetryFinished(numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI);1422 }1423 1424 1393 void WebResourceLoadStatisticsStore::aggregatedThirdPartyData(CompletionHandler<void(Vector<WebResourceLoadStatisticsStore::ThirdPartyData>&&)>&& completionHandler) 1425 1394 { -
trunk/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h
r267750 r268458 208 208 void setNotifyPagesWhenDataRecordsWereScanned(bool); 209 209 void setShouldClassifyResourcesBeforeDataRecordsRemoval(bool, CompletionHandler<void()>&&); 210 void setShouldSubmitTelemetry(bool);211 210 212 211 void grantStorageAccess(const SubFrameDomain&, const TopFrameDomain&, WebCore::FrameIdentifier, WebCore::PageIdentifier, StorageAccessPromptWasShown, StorageAccessScope, CompletionHandler<void(RequestStorageAccessResult)>&&); … … 251 250 void scheduleStatisticsAndDataRecordsProcessing(CompletionHandler<void()>&&); 252 251 void statisticsDatabaseHasAllTables(CompletionHandler<void(bool)>&&); 253 void submitTelemetry(CompletionHandler<void()>&&);254 252 void scheduleClearInMemoryAndPersistent(ShouldGrandfatherStatistics, CompletionHandler<void()>&&); 255 253 void scheduleClearInMemoryAndPersistent(WallTime modifiedSince, ShouldGrandfatherStatistics, CompletionHandler<void()>&&); … … 292 290 293 291 void sendDiagnosticMessageWithValue(const String& message, const String& description, unsigned value, unsigned sigDigits, WebCore::ShouldSample) const; 294 void notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI) const;295 292 296 293 void resourceLoadStatisticsUpdated(Vector<WebCore::ResourceLoadStatistics>&&, CompletionHandler<void()>&&); -
trunk/Source/WebKit/NetworkProcess/NetworkProcess.cpp
r268269 r268458 684 684 } 685 685 686 void NetworkProcess::submitTelemetry(PAL::SessionID sessionID, CompletionHandler<void()>&& completionHandler)687 {688 if (auto* networkSession = this->networkSession(sessionID)) {689 if (auto* resourceLoadStatistics = networkSession->resourceLoadStatistics())690 resourceLoadStatistics->submitTelemetry(WTFMove(completionHandler));691 else692 completionHandler();693 } else {694 ASSERT_NOT_REACHED();695 completionHandler();696 }697 }698 699 686 void NetworkProcess::scheduleCookieBlockingUpdate(PAL::SessionID sessionID, CompletionHandler<void()>&& completionHandler) 700 687 { -
trunk/Source/WebKit/NetworkProcess/NetworkProcess.h
r268261 r268458 238 238 void scheduleStatisticsAndDataRecordsProcessing(PAL::SessionID, CompletionHandler<void()>&&); 239 239 void statisticsDatabaseHasAllTables(PAL::SessionID, CompletionHandler<void(bool)>&&); 240 void submitTelemetry(PAL::SessionID, CompletionHandler<void()>&&);241 240 void setCacheMaxAgeCapForPrevalentResources(PAL::SessionID, Seconds, CompletionHandler<void()>&&); 242 241 void setGrandfatheringTime(PAL::SessionID, Seconds, CompletionHandler<void()>&&); -
trunk/Source/WebKit/NetworkProcess/NetworkProcess.messages.in
r268261 r268458 114 114 ScheduleStatisticsAndDataRecordsProcessing(PAL::SessionID sessionID) -> () Async 115 115 StatisticsDatabaseHasAllTables(PAL::SessionID sessionID) -> (bool hasAllTables) Async 116 SubmitTelemetry(PAL::SessionID sessionID) -> () Async117 116 SetCacheMaxAgeCapForPrevalentResources(PAL::SessionID sessionID, Seconds seconds) -> () Async 118 117 SetGrandfathered(PAL::SessionID sessionID, WebCore::RegistrableDomain resourceDomain, bool isGrandfathered) -> () Async -
trunk/Source/WebKit/NetworkProcess/NetworkSession.cpp
r267750 r268458 227 227 } 228 228 229 void NetworkSession::notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)230 {231 m_networkProcess->parentProcessConnection()->send(Messages::NetworkProcessProxy::NotifyResourceLoadStatisticsTelemetryFinished(numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI), 0);232 }233 234 229 void NetworkSession::deleteAndRestrictWebsiteDataForRegistrableDomains(OptionSet<WebsiteDataType> dataTypes, RegistrableDomainsToDeleteOrRestrictWebsiteDataFor&& domains, bool shouldNotifyPage, CompletionHandler<void(const HashSet<RegistrableDomain>&)>&& completionHandler) 235 230 { -
trunk/Source/WebKit/NetworkProcess/NetworkSession.h
r267750 r268458 97 97 void registrableDomainsWithWebsiteData(OptionSet<WebsiteDataType>, bool shouldNotifyPage, CompletionHandler<void(HashSet<WebCore::RegistrableDomain>&&)>&&); 98 98 void logDiagnosticMessageWithValue(const String& message, const String& description, unsigned value, unsigned significantFigures, WebCore::ShouldSample); 99 void notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI);100 99 bool enableResourceLoadStatisticsLogTestingEvent() const { return m_enableResourceLoadStatisticsLogTestingEvent; } 101 100 void setResourceLoadStatisticsLogTestingEvent(bool log) { m_enableResourceLoadStatisticsLogTestingEvent = log; } -
trunk/Source/WebKit/PlatformFTW.cmake
r260844 r268458 11 11 list(APPEND WebKit_SOURCES 12 12 NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp 13 NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp14 13 15 14 NetworkProcess/WebStorage/StorageManager.cpp -
trunk/Source/WebKit/PlatformPlayStation.cmake
r262022 r268458 24 24 25 25 NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp 26 NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp27 26 28 27 NetworkProcess/Cookies/curl/WebCookieManagerCurl.cpp -
trunk/Source/WebKit/PlatformWin.cmake
r267034 r268458 11 11 list(APPEND WebKit_SOURCES 12 12 NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp 13 NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp14 13 15 14 NetworkProcess/WebStorage/StorageManager.cpp -
trunk/Source/WebKit/SourcesCocoa.txt
r268431 r268458 31 31 32 32 NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp 33 NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp34 33 35 34 NetworkProcess/Cookies/mac/WebCookieManagerMac.mm -
trunk/Source/WebKit/SourcesGTK.txt
r266718 r268458 30 30 31 31 NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp 32 NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp33 32 34 33 NetworkProcess/Cookies/soup/WebCookieManagerSoup.cpp -
trunk/Source/WebKit/SourcesWPE.txt
r266718 r268458 30 30 31 31 NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp 32 NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp33 32 34 33 NetworkProcess/Cookies/soup/WebCookieManagerSoup.cpp -
trunk/Source/WebKit/UIProcess/API/C/WKWebsiteDataStoreRef.cpp
r267885 r268458 472 472 } 473 473 474 void WKWebsiteDataStoreStatisticsSubmitTelemetry(WKWebsiteDataStoreRef dataStoreRef)475 {476 #if ENABLE(RESOURCE_LOAD_STATISTICS)477 WebKit::toImpl(dataStoreRef)->submitTelemetry();478 #endif479 }480 481 474 void WKWebsiteDataStoreSetStatisticsNotifyPagesWhenDataRecordsWereScanned(WKWebsiteDataStoreRef dataStoreRef, bool value) 482 475 { -
trunk/Source/WebKit/UIProcess/API/C/WKWebsiteDataStoreRef.h
r267763 r268458 115 115 typedef void (*WKWebsiteDataStoreStatisticsUpdateCookieBlockingFunction)(void* functionContext); 116 116 WK_EXPORT void WKWebsiteDataStoreStatisticsUpdateCookieBlocking(WKWebsiteDataStoreRef dataStoreRef, void* context, WKWebsiteDataStoreStatisticsUpdateCookieBlockingFunction completionHandler); 117 WK_EXPORT void WKWebsiteDataStoreStatisticsSubmitTelemetry(WKWebsiteDataStoreRef dataStoreRef);118 117 WK_EXPORT void WKWebsiteDataStoreSetStatisticsNotifyPagesWhenDataRecordsWereScanned(WKWebsiteDataStoreRef dataStoreRef, bool value); 119 118 typedef void (*WKWebsiteDataStoreSetStatisticsIsRunningTestFunction)(void* functionContext); -
trunk/Source/WebKit/UIProcess/API/Cocoa/WKWebsiteDataStore.mm
r267966 r268458 38 38 #import "WebPageProxy.h" 39 39 #import "WebResourceLoadStatisticsStore.h" 40 #import "WebResourceLoadStatisticsTelemetry.h"41 40 #import "WebsiteDataFetchOption.h" 42 41 #import "_WKResourceLoadStatisticsThirdPartyInternal.h" -
trunk/Source/WebKit/UIProcess/Network/NetworkProcessProxy.cpp
r268273 r268458 1089 1089 } 1090 1090 1091 void NetworkProcessProxy::submitTelemetry(PAL::SessionID sessionID, CompletionHandler<void()>&& completionHandler)1092 {1093 if (!canSendMessage()) {1094 completionHandler();1095 return;1096 }1097 1098 sendWithAsyncReply(Messages::NetworkProcess::SubmitTelemetry(sessionID), WTFMove(completionHandler));1099 }1100 1101 1091 void NetworkProcessProxy::scheduleClearInMemoryAndPersistent(PAL::SessionID sessionID, ShouldGrandfatherStatistics shouldGrandfather, CompletionHandler<void()>&& completionHandler) 1102 1092 { … … 1130 1120 { 1131 1121 WebProcessProxy::notifyWebsiteDataScanForRegistrableDomainsFinished(); 1132 }1133 1134 void NetworkProcessProxy::notifyResourceLoadStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)1135 {1136 API::Dictionary::MapType messageBody;1137 messageBody.set("NumberOfPrevalentResources"_s, API::UInt64::create(numberOfPrevalentResources));1138 messageBody.set("NumberOfPrevalentResourcesWithUserInteraction"_s, API::UInt64::create(numberOfPrevalentResourcesWithUserInteraction));1139 messageBody.set("NumberOfPrevalentResourcesWithoutUserInteraction"_s, API::UInt64::create(numberOfPrevalentResourcesWithoutUserInteraction));1140 messageBody.set("TopPrevalentResourceWithUserInteractionDaysSinceUserInteraction"_s, API::UInt64::create(topPrevalentResourceWithUserInteractionDaysSinceUserInteraction));1141 messageBody.set("MedianDaysSinceUserInteractionPrevalentResourceWithUserInteraction"_s, API::UInt64::create(medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction));1142 messageBody.set("Top3NumberOfPrevalentResourcesWithUI"_s, API::UInt64::create(top3NumberOfPrevalentResourcesWithUI));1143 messageBody.set("Top3MedianSubFrameWithoutUI"_s, API::UInt64::create(top3MedianSubFrameWithoutUI));1144 messageBody.set("Top3MedianSubResourceWithoutUI"_s, API::UInt64::create(top3MedianSubResourceWithoutUI));1145 messageBody.set("Top3MedianUniqueRedirectsWithoutUI"_s, API::UInt64::create(top3MedianUniqueRedirectsWithoutUI));1146 messageBody.set("Top3MedianDataRecordsRemovedWithoutUI"_s, API::UInt64::create(top3MedianDataRecordsRemovedWithoutUI));1147 1148 WebProcessProxy::notifyPageStatisticsTelemetryFinished(API::Dictionary::create(messageBody).ptr());1149 1122 } 1150 1123 -
trunk/Source/WebKit/UIProcess/Network/NetworkProcessProxy.h
r267763 r268458 174 174 void scheduleClearInMemoryAndPersistent(PAL::SessionID, Optional<WallTime> modifiedSince, ShouldGrandfatherStatistics, CompletionHandler<void()>&&); 175 175 void scheduleCookieBlockingUpdate(PAL::SessionID, CompletionHandler<void()>&&); 176 void submitTelemetry(PAL::SessionID, CompletionHandler<void()>&&);177 176 void setCacheMaxAgeCapForPrevalentResources(PAL::SessionID, Seconds, CompletionHandler<void()>&&); 178 177 void setGrandfatheringTime(PAL::SessionID, Seconds, CompletionHandler<void()>&&); … … 305 304 void notifyWebsiteDataDeletionForRegistrableDomainsFinished(); 306 305 void notifyWebsiteDataScanForRegistrableDomainsFinished(); 307 void notifyResourceLoadStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI);308 306 #endif 309 307 void retrieveCacheStorageParameters(PAL::SessionID); -
trunk/Source/WebKit/UIProcess/Network/NetworkProcessProxy.messages.in
r267042 r268458 45 45 NotifyWebsiteDataDeletionForRegistrableDomainsFinished() 46 46 NotifyWebsiteDataScanForRegistrableDomainsFinished() 47 NotifyResourceLoadStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)48 47 RequestStorageAccessConfirm(WebKit::WebPageProxyIdentifier pageID, WebCore::FrameIdentifier frameID, WebCore::RegistrableDomain subFrameDomain, WebCore::RegistrableDomain topFrameDomain) -> (bool userDidGrantAccess) Async 49 48 DeleteWebsiteDataInUIProcessForRegistrableDomains(PAL::SessionID sessionID, OptionSet<WebKit::WebsiteDataType> dataTypes, OptionSet<WebKit::WebsiteDataFetchOption> fetchOptions, Vector<WebCore::RegistrableDomain> domains) -> (HashSet<WebCore::RegistrableDomain> domainsWithMatchingDataRecords) Async -
trunk/Source/WebKit/UIProcess/WebProcessProxy.cpp
r268443 r268458 485 485 } 486 486 487 void WebProcessProxy::notifyPageStatisticsTelemetryFinished(API::Object* messageBody)488 {489 for (auto& page : globalPageMap())490 page.value->postMessageToInjectedBundle("ResourceLoadStatisticsTelemetryFinished", messageBody);491 }492 493 487 void WebProcessProxy::setThirdPartyCookieBlockingMode(ThirdPartyCookieBlockingMode thirdPartyCookieBlockingMode, CompletionHandler<void()>&& completionHandler) 494 488 { -
trunk/Source/WebKit/UIProcess/WebProcessProxy.h
r268384 r268458 224 224 #if ENABLE(RESOURCE_LOAD_STATISTICS) 225 225 static void notifyPageStatisticsAndDataRecordsProcessed(); 226 static void notifyPageStatisticsTelemetryFinished(API::Object* messageBody);227 226 228 227 static void notifyWebsiteDataDeletionForRegistrableDomainsFinished(); -
trunk/Source/WebKit/UIProcess/WebsiteData/WebsiteDataStore.cpp
r267763 r268458 1446 1446 } 1447 1447 1448 void WebsiteDataStore::submitTelemetry()1449 {1450 ASSERT(RunLoop::isMain());1451 1452 networkProcess().submitTelemetry(m_sessionID, [] { });1453 }1454 1455 1448 void WebsiteDataStore::scheduleClearInMemoryAndPersistent(WallTime modifiedSince, ShouldGrandfatherStatistics shouldGrandfather, CompletionHandler<void()>&& completionHandler) 1456 1449 { -
trunk/Source/WebKit/UIProcess/WebsiteData/WebsiteDataStore.h
r268214 r268458 190 190 void getResourceLoadStatisticsDataSummary(CompletionHandler<void(Vector<WebResourceLoadStatisticsStore::ThirdPartyData>&&)>&&); 191 191 void scheduleStatisticsAndDataRecordsProcessing(CompletionHandler<void()>&&); 192 void submitTelemetry();193 192 void setGrandfathered(const URL&, bool, CompletionHandler<void()>&&); 194 193 void isGrandfathered(const URL&, CompletionHandler<void(bool)>&&); -
trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj
r268457 r268458 4229 4229 7AFBD36421E51BAB005DBACB /* ResourceLoadStatisticsMemoryStore.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ResourceLoadStatisticsMemoryStore.cpp; path = Classifier/ResourceLoadStatisticsMemoryStore.cpp; sourceTree = "<group>"; }; 4230 4230 7AFBD36521E51BAB005DBACB /* ResourceLoadStatisticsMemoryStore.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = ResourceLoadStatisticsMemoryStore.h; path = Classifier/ResourceLoadStatisticsMemoryStore.h; sourceTree = "<group>"; }; 4231 7AFBD36B21E54544005DBACB /* WebResourceLoadStatisticsTelemetry.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = WebResourceLoadStatisticsTelemetry.h; path = Classifier/WebResourceLoadStatisticsTelemetry.h; sourceTree = "<group>"; };4232 7AFBD36C21E54544005DBACB /* WebResourceLoadStatisticsTelemetry.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; name = WebResourceLoadStatisticsTelemetry.cpp; path = Classifier/WebResourceLoadStatisticsTelemetry.cpp; sourceTree = "<group>"; };4233 4231 7AFBD36D21E546E3005DBACB /* PersistencyUtils.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = PersistencyUtils.h; sourceTree = "<group>"; }; 4234 4232 7AFBD36E21E546E3005DBACB /* PersistencyUtils.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = PersistencyUtils.cpp; sourceTree = "<group>"; }; … … 8439 8437 7A843A1A21E41FB200DEF663 /* WebResourceLoadStatisticsStore.cpp */, 8440 8438 7AFBD36221E50F39005DBACB /* WebResourceLoadStatisticsStore.h */, 8441 7AFBD36C21E54544005DBACB /* WebResourceLoadStatisticsTelemetry.cpp */,8442 7AFBD36B21E54544005DBACB /* WebResourceLoadStatisticsTelemetry.h */,8443 8439 ); 8444 8440 name = Classifier; -
trunk/Tools/ChangeLog
r268457 r268458 1 2020-10-14 Kate Cheney <katherine_cheney@apple.com> 2 3 Remove ResourceLoadStatistics telemetry 4 https://bugs.webkit.org/show_bug.cgi?id=217665 5 <rdar://problem/69748988> 6 7 Reviewed by Sam Weinig. 8 9 Telemetry calculation in ITP is unused, very complicated, a potential 10 privacy risk, and it causes flaky tests. We should remove it. 11 12 * WebKitTestRunner/InjectedBundle/Bindings/TestRunner.idl: 13 * WebKitTestRunner/InjectedBundle/InjectedBundle.cpp: 14 (WTR::InjectedBundle::didReceiveMessageToPage): 15 * WebKitTestRunner/InjectedBundle/TestRunner.cpp: 16 (WTR::TestRunner::installStatisticsDidRunTelemetryCallback): Deleted. 17 (WTR::TestRunner::statisticsDidRunTelemetryCallback): Deleted. 18 (WTR::TestRunner::statisticsSubmitTelemetry): Deleted. 19 * WebKitTestRunner/InjectedBundle/TestRunner.h: 20 * WebKitTestRunner/TestController.cpp: 21 (WTR::TestController::statisticsSubmitTelemetry): Deleted. 22 * WebKitTestRunner/TestController.h: 23 * WebKitTestRunner/TestInvocation.cpp: 24 (WTR::TestInvocation::didReceiveSynchronousMessageFromInjectedBundle): 25 1 26 2020-10-14 Kate Cheney <katherine_cheney@apple.com> 2 27 -
trunk/Tools/WebKitTestRunner/InjectedBundle/Bindings/TestRunner.idl
r267761 r268458 300 300 undefined installStatisticsDidModifyDataRecordsCallback(object callback); 301 301 undefined installStatisticsDidScanDataRecordsCallback(object callback); 302 undefined installStatisticsDidRunTelemetryCallback(object callback);303 302 undefined setStatisticsDebugMode(boolean value, object completionHandler); 304 303 undefined setStatisticsPrevalentResourceForDebugMode(DOMString hostName, object completionHandler); … … 329 328 undefined statisticsProcessStatisticsAndDataRecords(); 330 329 undefined statisticsUpdateCookieBlocking(object completionHandler); 331 undefined statisticsSubmitTelemetry();332 330 undefined setStatisticsIsRunningTest(boolean value); 333 331 undefined setStatisticsShouldClassifyResourcesBeforeDataRecordsRemoval(boolean value); -
trunk/Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.cpp
r268321 r268458 456 456 return; 457 457 } 458 459 if (WKStringIsEqualToUTF8CString(messageName, "ResourceLoadStatisticsTelemetryFinished")) {460 auto messageBodyDictionary = dictionaryValue(messageBody);461 auto numberOfPrevalentResources = uint64Value(messageBodyDictionary, "NumberOfPrevalentResources");462 auto numberOfPrevalentResourcesWithUserInteraction = uint64Value(messageBodyDictionary, "NumberOfPrevalentResourcesWithUserInteraction");463 auto numberOfPrevalentResourcesWithoutUserInteraction = uint64Value(messageBodyDictionary, "NumberOfPrevalentResourcesWithoutUserInteraction");464 auto topPrevalentResourceWithUserInteractionDaysSinceUserInteraction = uint64Value(messageBodyDictionary, "TopPrevalentResourceWithUserInteractionDaysSinceUserInteraction");465 auto medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction = uint64Value(messageBodyDictionary, "MedianDaysSinceUserInteractionPrevalentResourceWithUserInteraction");466 auto top3NumberOfPrevalentResourcesWithUI = uint64Value(messageBodyDictionary, "Top3NumberOfPrevalentResourcesWithUI");467 auto top3MedianSubFrameWithoutUI = uint64Value(messageBodyDictionary, "Top3MedianSubFrameWithoutUI");468 auto top3MedianSubResourceWithoutUI = uint64Value(messageBodyDictionary, "Top3MedianSubResourceWithoutUI");469 auto top3MedianUniqueRedirectsWithoutUI = uint64Value(messageBodyDictionary, "Top3MedianUniqueRedirectsWithoutUI");470 auto top3MedianDataRecordsRemovedWithoutUI = uint64Value(messageBodyDictionary, "Top3MedianDataRecordsRemovedWithoutUI");471 472 m_testRunner->statisticsDidRunTelemetryCallback(numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI);473 return;474 }475 458 476 459 if (WKStringIsEqualToUTF8CString(messageName, "DidGetApplicationManifest")) { -
trunk/Tools/WebKitTestRunner/InjectedBundle/TestRunner.cpp
r268391 r268458 733 733 StatisticsDidModifyDataRecordsCallbackID, 734 734 StatisticsDidScanDataRecordsCallbackID, 735 StatisticsDidRunTelemetryCallbackID,736 735 StatisticsDidClearThroughWebsiteDataRemovalCallbackID, 737 736 StatisticsDidClearInMemoryAndPersistentStoreCallbackID, … … 1553 1552 } 1554 1553 1555 void TestRunner::installStatisticsDidRunTelemetryCallback(JSValueRef callback)1556 {1557 cacheTestRunnerCallback(StatisticsDidRunTelemetryCallbackID, callback);1558 }1559 1560 void TestRunner::statisticsDidRunTelemetryCallback(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)1561 {1562 String string = makeString("{ \"numberOfPrevalentResources\" : ", numberOfPrevalentResources, ", \"numberOfPrevalentResourcesWithUserInteraction\" : ", numberOfPrevalentResourcesWithUserInteraction, ", \"numberOfPrevalentResourcesWithoutUserInteraction\" : ", numberOfPrevalentResourcesWithoutUserInteraction, ", \"topPrevalentResourceWithUserInteractionDaysSinceUserInteraction\" : ", topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, ", \"medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction\" : ", medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, ", \"top3NumberOfPrevalentResourcesWithUI\" : ", top3NumberOfPrevalentResourcesWithUI, ", \"top3MedianSubFrameWithoutUI\" : ", top3MedianSubFrameWithoutUI, ", \"top3MedianSubResourceWithoutUI\" : ", top3MedianSubResourceWithoutUI, ", \"top3MedianUniqueRedirectsWithoutUI\" : ", top3MedianUniqueRedirectsWithoutUI, ", \"top3MedianDataRecordsRemovedWithoutUI\" : ", top3MedianDataRecordsRemovedWithoutUI, " }");1563 1564 JSValueRef result = JSValueMakeFromJSONString(mainFrameJSContext(), createJSString(string.utf8().data()).get());1565 1566 callTestRunnerCallback(StatisticsDidRunTelemetryCallbackID, 1, &result);1567 }1568 1569 1554 bool TestRunner::statisticsNotifyObserver() 1570 1555 { … … 1587 1572 { 1588 1573 callTestRunnerCallback(StatisticsDidSetBlockCookiesForHostCallbackID); 1589 }1590 1591 void TestRunner::statisticsSubmitTelemetry()1592 {1593 postSynchronousMessage("StatisticsSubmitTelemetry");1594 1574 } 1595 1575 -
trunk/Tools/WebKitTestRunner/InjectedBundle/TestRunner.h
r267761 r268458 389 389 void installStatisticsDidModifyDataRecordsCallback(JSValueRef callback); 390 390 void installStatisticsDidScanDataRecordsCallback(JSValueRef callback); 391 void installStatisticsDidRunTelemetryCallback(JSValueRef callback);392 391 void statisticsDidModifyDataRecordsCallback(); 393 392 void statisticsDidScanDataRecordsCallback(); 394 void statisticsDidRunTelemetryCallback(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI);395 393 bool statisticsNotifyObserver(); 396 394 void statisticsProcessStatisticsAndDataRecords(); 397 395 void statisticsUpdateCookieBlocking(JSValueRef completionHandler); 398 396 void statisticsCallDidSetBlockCookiesForHostCallback(); 399 void statisticsSubmitTelemetry();400 397 void setStatisticsDebugMode(bool value, JSValueRef completionHandler); 401 398 void statisticsCallDidSetDebugModeCallback(); -
trunk/Tools/WebKitTestRunner/TestController.cpp
r268400 r268458 3225 3225 } 3226 3226 3227 void TestController::statisticsSubmitTelemetry()3228 {3229 WKWebsiteDataStoreStatisticsSubmitTelemetry(websiteDataStore());3230 }3231 3232 3227 void TestController::setStatisticsNotifyPagesWhenDataRecordsWereScanned(bool value) 3233 3228 { -
trunk/Tools/WebKitTestRunner/TestController.h
r268370 r268458 239 239 void statisticsProcessStatisticsAndDataRecords(); 240 240 void statisticsUpdateCookieBlocking(); 241 void statisticsSubmitTelemetry();242 241 void setStatisticsNotifyPagesWhenDataRecordsWereScanned(bool); 243 242 void setStatisticsIsRunningTest(bool); -
trunk/Tools/WebKitTestRunner/TestInvocation.cpp
r268391 r268458 1144 1144 } 1145 1145 1146 if (WKStringIsEqualToUTF8CString(messageName, "StatisticsSubmitTelemetry")) {1147 TestController::singleton().statisticsSubmitTelemetry();1148 return nullptr;1149 }1150 1151 1146 if (WKStringIsEqualToUTF8CString(messageName, "StatisticsNotifyPagesWhenDataRecordsWereScanned")) { 1152 1147 TestController::singleton().setStatisticsNotifyPagesWhenDataRecordsWereScanned(booleanValue(messageBody));
Note: See TracChangeset
for help on using the changeset viewer.