Changeset 182856 in webkit


Ignore:
Timestamp:
Apr 15, 2015 12:54:21 PM (9 years ago)
Author:
Antti Koivisto
Message:

Network Cache: Inline small body data to record file
​https://bugs.webkit.org/show_bug.cgi?id=143783

Reviewed by Chris Dumez.

We currently save all body data as separate files. We can improve space efficiency and do less reads and writes
by inlining smaller resource bodies with the header.

  • NetworkProcess/cache/NetworkCacheIOChannel.h:
  • NetworkProcess/cache/NetworkCacheIOChannelCocoa.mm:

(WebKit::NetworkCache::IOChannel::read):
(WebKit::NetworkCache::IOChannel::readSync):
(WebKit::NetworkCache::IOChannel::write):

Add WorkQueue argument to allow specifying which queue the result is submitted to.

  • NetworkProcess/cache/NetworkCacheStorage.cpp:

(WebKit::NetworkCache::decodeRecordMetaData):

Add a boolean indicating whether the body is inlined.

(WebKit::NetworkCache::decodeRecordHeader):
(WebKit::NetworkCache::Storage::decodeRecord):
(WebKit::NetworkCache::encodeRecordMetaData):
(WebKit::NetworkCache::Storage::storeBodyAsBlob):
(WebKit::NetworkCache::Storage::encodeRecord):
(WebKit::NetworkCache::Storage::dispatchReadOperation):

Read the record first, then read the blob if needed.
Submit the read operation directly from the main queue. Only thing we do is opening an IO channel
and that uses O_NONBLOCK.
Process the read results in the IO work queue where we now do the blob retrieval.

(WebKit::NetworkCache::shouldStoreBodyAsBlob):

The current threshold for saving a separate blob is 16KB.

(WebKit::NetworkCache::Storage::dispatchWriteOperation):
(WebKit::NetworkCache::Storage::traverse):
(WebKit::NetworkCache::createRecord): Deleted.
(WebKit::NetworkCache::encodeRecordHeader): Deleted.

  • NetworkProcess/cache/NetworkCacheStorage.h:
Location:
trunk/Source/WebKit2
Files:
5 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebKit2/ChangeLog

    r182855 r182856  
     12015-04-15  Antti Koivisto  <antti@apple.com>
     2
     3        Network Cache: Inline small body data to record file
     4        https://bugs.webkit.org/show_bug.cgi?id=143783
     5
     6        Reviewed by Chris Dumez.
     7
     8        We currently save all body data as separate files. We can improve space efficiency and do less reads and writes
     9        by inlining smaller resource bodies with the header.
     10
     11        * NetworkProcess/cache/NetworkCacheIOChannel.h:
     12        * NetworkProcess/cache/NetworkCacheIOChannelCocoa.mm:
     13        (WebKit::NetworkCache::IOChannel::read):
     14        (WebKit::NetworkCache::IOChannel::readSync):
     15        (WebKit::NetworkCache::IOChannel::write):
     16
     17            Add WorkQueue argument to allow specifying which queue the result is submitted to.
     18
     19        * NetworkProcess/cache/NetworkCacheStorage.cpp:
     20        (WebKit::NetworkCache::decodeRecordMetaData):
     21
     22            Add a boolean indicating whether the body is inlined.
     23
     24        (WebKit::NetworkCache::decodeRecordHeader):
     25        (WebKit::NetworkCache::Storage::decodeRecord):
     26        (WebKit::NetworkCache::encodeRecordMetaData):
     27        (WebKit::NetworkCache::Storage::storeBodyAsBlob):
     28        (WebKit::NetworkCache::Storage::encodeRecord):
     29        (WebKit::NetworkCache::Storage::dispatchReadOperation):
     30
     31            Read the record first, then read the blob if needed.
     32            Submit the read operation directly from the main queue. Only thing we do is opening an IO channel
     33            and that uses O_NONBLOCK.
     34            Process the read results in the IO work queue where we now do the blob retrieval.
     35
     36        (WebKit::NetworkCache::shouldStoreBodyAsBlob):
     37
     38            The current threshold for saving a separate blob is 16KB.
     39
     40        (WebKit::NetworkCache::Storage::dispatchWriteOperation):
     41        (WebKit::NetworkCache::Storage::traverse):
     42        (WebKit::NetworkCache::createRecord): Deleted.
     43        (WebKit::NetworkCache::encodeRecordHeader): Deleted.
     44        * NetworkProcess/cache/NetworkCacheStorage.h:
     45
    1462015-04-15  Tim Horton  <timothy_horton@apple.com>
    247

  • trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheIOChannel.h

    r181700 r182856  
    3232#include <functional>
    3333#include <wtf/ThreadSafeRefCounted.h>
     34#include <wtf/WorkQueue.h>
    3435#include <wtf/text/WTFString.h>
    3536
    … …  
    4243    static Ref<IOChannel> open(const String& file, Type);
    4344
    44     void read(size_t offset, size_t, std::function<void (Data&, int error)>);
    45     void readSync(size_t offset, size_t, std::function<void (Data&, int error)>);
    46     void write(size_t offset, const Data&, std::function<void (int error)>);
     45    // Using nullptr as queue submits the result to the main queue.
     46    // FIXME: We should add WorkQueue::main() instead.
     47    void read(size_t offset, size_t, WorkQueue*, std::function<void (Data&, int error)>);
     48    void readSync(size_t offset, size_t, WorkQueue*, std::function<void (Data&, int error)>);
     49    void write(size_t offset, const Data&, WorkQueue*, std::function<void (int error)>);
    4750
    4851    const String& path() const { return m_path; }

  • trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheIOChannelCocoa.mm

    r182602 r182856  
    8080}
    8181
    82 void IOChannel::read(size_t offset, size_t size, std::function<void (Data&, int error)> completionHandler)
     82void IOChannel::read(size_t offset, size_t size, WorkQueue* queue, std::function<void (Data&, int error)> completionHandler)
    8383{
    8484    RefPtr<IOChannel> channel(this);
    8585    bool didCallCompletionHandler = false;
    86     dispatch_io_read(m_dispatchIO.get(), offset, size, dispatch_get_main_queue(), [channel, completionHandler, didCallCompletionHandler](bool done, dispatch_data_t fileData, int error) mutable {
     86    auto dispatchQueue = queue ? queue->dispatchQueue() : dispatch_get_main_queue();
     87    dispatch_io_read(m_dispatchIO.get(), offset, size, dispatchQueue, [channel, completionHandler, didCallCompletionHandler](bool done, dispatch_data_t fileData, int error) mutable {
    8788        ASSERT_UNUSED(done, done || !didCallCompletionHandler);
    8889        if (didCallCompletionHandler)
    … …  
    9697
    9798// FIXME: It would be better to do without this.
    98 void IOChannel::readSync(size_t offset, size_t size, std::function<void (Data&, int error)> completionHandler)
     99void IOChannel::readSync(size_t offset, size_t size, WorkQueue* queue, std::function<void (Data&, int error)> completionHandler)
    99100{
    100101    auto semaphore = adoptDispatch(dispatch_semaphore_create(0));
    101     read(offset, size, [semaphore, &completionHandler](Data& data, int error) {
     102    read(offset, size, queue, [semaphore, &completionHandler](Data& data, int error) {
    102103        completionHandler(data, error);
    103104        dispatch_semaphore_signal(semaphore.get());
    … …  
    106107}
    107108
    108 void IOChannel::write(size_t offset, const Data& data, std::function<void (int error)> completionHandler)
     109void IOChannel::write(size_t offset, const Data& data, WorkQueue* queue, std::function<void (int error)> completionHandler)
    109110{
    110111    RefPtr<IOChannel> channel(this);
    111112    auto dispatchData = data.dispatchData();
    112     dispatch_io_write(m_dispatchIO.get(), offset, dispatchData, dispatch_get_main_queue(), [channel, completionHandler](bool done, dispatch_data_t fileData, int error) {
     113    auto dispatchQueue = queue ? queue->dispatchQueue() : dispatch_get_main_queue();
     114    dispatch_io_write(m_dispatchIO.get(), offset, dispatchData, dispatchQueue, [channel, completionHandler](bool done, dispatch_data_t fileData, int error) {
    113115        ASSERT_UNUSED(done, done);
    114116        completionHandler(error);

  • trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.cpp

    r182841 r182856  
    226226    SHA1::Digest bodyHash;
    227227    uint64_t bodySize;
     228    bool isBodyInline;
    228229};
    229230
    … …  
    247248        if (!decoder.decode(metaData.bodySize))
    248249            return false;
     250        if (!decoder.decode(metaData.isBodyInline))
     251            return false;
    249252        if (!decoder.verifyChecksum())
    250253            return false;
    … …  
    256259}
    257260
    258 static bool decodeRecordHeader(const Data& fileData, RecordMetaData& metaData, Data& data)
     261static bool decodeRecordHeader(const Data& fileData, RecordMetaData& metaData, Data& headerData)
    259262{
    260263    if (!decodeRecordMetaData(metaData, fileData)) {
    … …  
    268271    }
    269272
    270     auto headerData = fileData.subrange(metaData.headerOffset, metaData.headerSize);
     273    headerData = fileData.subrange(metaData.headerOffset, metaData.headerSize);
    271274    if (metaData.headerChecksum != hashData(headerData)) {
    272275        LOG(NetworkCacheStorage, "(NetworkProcess) header checksum mismatch");
    273276        return false;
    274277    }
    275     data = { headerData };
    276278    return true;
    277279}
    278280
    279 static std::unique_ptr<Storage::Record> createRecord(const Data& recordData, const BlobStorage::Blob& bodyBlob, const Key& key)
    280 {
     281std::unique_ptr<Storage::Record> Storage::decodeRecord(const Data& recordData, const Key& key)
     282{
     283    ASSERT(!RunLoop::isMain());
     284
    281285    RecordMetaData metaData;
    282286    Data headerData;
    … …  
    291295    if (timeStamp > std::chrono::system_clock::now())
    292296        return nullptr;
    293     if (metaData.bodySize != bodyBlob.data.size())
    294         return nullptr;
    295     if (metaData.bodyHash != bodyBlob.hash)
    296         return nullptr;
     297
     298    Data bodyData;
     299    if (metaData.isBodyInline) {
     300        size_t bodyOffset = metaData.headerOffset + headerData.size();
     301        if (bodyOffset + metaData.bodySize != recordData.size())
     302            return nullptr;
     303        bodyData = recordData.subrange(bodyOffset, metaData.bodySize);
     304        if (metaData.bodyHash != computeSHA1(bodyData))
     305            return nullptr;
     306    } else {
     307        auto bodyPath = bodyPathForKey(key, recordsPath());
     308        auto bodyBlob = m_blobStorage.get(bodyPath);
     309        if (metaData.bodySize != bodyBlob.data.size())
     310            return nullptr;
     311        if (metaData.bodyHash != bodyBlob.hash)
     312            return nullptr;
     313        bodyData = bodyBlob.data;
     314    }
    297315
    298316    return std::make_unique<Storage::Record>(Storage::Record {
    … …  
    300318        timeStamp,
    301319        headerData,
    302         bodyBlob.data
     320        bodyData
    303321    });
    304322}
    … …  
    315333    encoder << metaData.bodyHash;
    316334    encoder << metaData.bodySize;
     335    encoder << metaData.isBodyInline;
    317336
    318337    encoder.encodeChecksum();
    … …  
    321340}
    322341
    323 static Data encodeRecordHeader(const Storage::Record& record, SHA1::Digest bodyHash)
    324 {
     342Optional<BlobStorage::Blob> Storage::storeBodyAsBlob(const Record& record, const MappedBodyHandler& mappedBodyHandler)
     343{
     344    auto bodyPath = bodyPathForKey(record.key, recordsPath());
     345
     346    // Store the body.
     347    auto blob = m_blobStorage.add(bodyPath, record.body);
     348    if (blob.data.isNull())
     349        return { };
     350
     351    // Tell the client we now have a disk-backed map for this data.
     352    if (mappedBodyHandler) {
     353        RunLoop::main().dispatch([blob, mappedBodyHandler] {
     354            mappedBodyHandler(blob.data);
     355        });
     356    }
     357    return blob;
     358}
     359
     360Data Storage::encodeRecord(const Record& record, Optional<BlobStorage::Blob> blob)
     361{
     362    ASSERT(!blob || bytesEqual(blob.value().data, record.body));
     363
    325364    RecordMetaData metaData(record.key);
    326365    metaData.epochRelativeTimeStamp = std::chrono::duration_cast<std::chrono::milliseconds>(record.timeStamp.time_since_epoch());
    327366    metaData.headerChecksum = hashData(record.header);
    328367    metaData.headerSize = record.header.size();
    329     metaData.bodyHash = bodyHash;
     368    metaData.bodyHash = blob ? blob.value().hash : computeSHA1(record.body);
    330369    metaData.bodySize = record.body.size();
     370    metaData.isBodyInline = !blob;
    331371
    332372    auto encodedMetaData = encodeRecordMetaData(metaData);
    333373    auto headerData = concatenate(encodedMetaData, record.header);
     374
     375    if (metaData.isBodyInline)
     376        return concatenate(headerData, record.body);
     377
    334378    return { headerData };
    335379}
    … …  
    363407    ASSERT(m_activeReadOperations.contains(&read));
    364408
    365     ioQueue().dispatch([this, &read] {
    366         auto recordsPath = this->recordsPath();
    367         auto recordPath = recordPathForKey(read.key, recordsPath);
    368         auto bodyPath = bodyPathForKey(read.key, recordsPath);
    369         // FIXME: Body and header retrieves can be done in parallel.
    370         auto bodyBlob = m_blobStorage.get(bodyPath);
    371 
    372         RefPtr<IOChannel> channel = IOChannel::open(recordPath, IOChannel::Type::Read);
    373         channel->read(0, std::numeric_limits<size_t>::max(), [this, &read, bodyBlob](Data& fileData, int error) {
    374             auto record = error ? nullptr : createRecord(fileData, bodyBlob, read.key);
     409    auto recordsPath = this->recordsPath();
     410    auto recordPath = recordPathForKey(read.key, recordsPath);
     411
     412    RefPtr<IOChannel> channel = IOChannel::open(recordPath, IOChannel::Type::Read);
     413    channel->read(0, std::numeric_limits<size_t>::max(), &ioQueue(), [this, &read](const Data& fileData, int error) {
     414        auto record = error ? nullptr : decodeRecord(fileData, read.key);
     415
     416        auto* recordPtr = record.release();
     417        RunLoop::main().dispatch([this, &read, recordPtr] {
     418            auto record = std::unique_ptr<Record>(recordPtr);
    375419            finishReadOperation(read, WTF::move(record));
    376420        });
    … …  
    449493}
    450494
     495static bool shouldStoreBodyAsBlob(const Data& bodyData)
     496{
     497    const size_t maximumInlineBodySize { 16 * 1024 };
     498    return bodyData.size() > maximumInlineBodySize;
     499}
     500
    451501void Storage::dispatchWriteOperation(const WriteOperation& write)
    452502{
    … …  
    461511        auto partitionPath = partitionPathForKey(write.record.key, recordsPath);
    462512        auto recordPath = recordPathForKey(write.record.key, recordsPath);
    463         auto bodyPath = bodyPathForKey(write.record.key, recordsPath);
    464513
    465514        WebCore::makeAllDirectories(partitionPath);
    466515
    467         // Store the body.
    468         auto blob = m_blobStorage.add(bodyPath, write.record.body);
    469         if (blob.data.isNull()) {
    470             RunLoop::main().dispatch([this, &write] {
    471                 finishWriteOperation(write);
    472             });
    473             return;
    474         }
    475 
    476         // Tell the client we now have a disk-backed map for this data.
    477         size_t minimumMapSize = pageSize();
    478         if (blob.data.size() >= minimumMapSize && blob.data.isMap() && write.mappedBodyHandler) {
    479             auto& mappedBodyHandler = write.mappedBodyHandler;
    480             RunLoop::main().dispatch([blob, mappedBodyHandler] {
    481                 mappedBodyHandler(blob.data);
    482             });
    483         }
    484 
    485         // Store the header and meta data.
    486         auto encodedHeader = encodeRecordHeader(write.record, blob.hash);
     516        bool shouldStoreAsBlob = shouldStoreBodyAsBlob(write.record.body);
     517        auto bodyBlob = shouldStoreAsBlob ? storeBodyAsBlob(write.record, write.mappedBodyHandler) : Nullopt;
     518
     519        auto recordData = encodeRecord(write.record, bodyBlob);
     520
    487521        auto channel = IOChannel::open(recordPath, IOChannel::Type::Create);
    488         int fd = channel->fileDescriptor();
    489         size_t headerSize = encodedHeader.size();
    490         channel->write(0, encodedHeader, [this, &write, headerSize, fd](int error) {
     522        size_t recordSize = recordData.size();
     523        channel->write(0, recordData, nullptr, [this, &write, recordSize](int error) {
    491524            // On error the entry still stays in the contents filter until next synchronization.
    492             m_approximateSize += headerSize;
     525            m_approximateSize += recordSize;
    493526            finishWriteOperation(write);
    494527
    … …  
    562595            auto channel = IOChannel::open(recordPath, IOChannel::Type::Read);
    563596            // FIXME: Traversal is slower than it should be due to lack of parallelism.
    564             channel->readSync(0, std::numeric_limits<size_t>::max(), [this, &traverseHandler, &info](Data& fileData, int) {
     597            channel->readSync(0, std::numeric_limits<size_t>::max(), nullptr, [this, &traverseHandler, &info](Data& fileData, int) {
    565598                RecordMetaData metaData;
    566599                Data headerData;

  • trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.h

    r182841 r182856  
    112112    void finishWriteOperation(const WriteOperation&);
    113113
     114    Optional<BlobStorage::Blob> storeBodyAsBlob(const Record&, const MappedBodyHandler&);
     115    Data encodeRecord(const Record&, Optional<BlobStorage::Blob>);
     116    std::unique_ptr<Record> decodeRecord(const Data&, const Key&);
     117
    114118    void updateFileModificationTime(const String& path);
    115119
Note: See TracChangeset for help on using the changeset viewer.