Changes between Version 5 and Version 6 of QtWebKitMirrorGuide

Nov 26, 2009 4:44:12 AM (10 years ago)

Mention the Math.random() problem with the test content.


  • QtWebKitMirrorGuide

    v5 v6  
    1212Example of building and running:
    14 $ cd mirror
     14$ cd host-tools/mirror
    1515$ qmake
    1616$ make
    3838 5. ''crawl_db.db'' now contains a copy of It can be served with the '''http_server'''
     40=== Generate more stable loading times ===
     41Some webpages use Math.random() to randomize which content to be displayed. In the case of Wikipedia this can be one of the various announcement, in case of the website this is the image to be displayed on the front page and the advertisement query. The problem with this randomisation is that the loading time may vary from page view to page view and that the resulting loading time is not stable. On way to deal with it is to remove all calls to Math.random() with a constant.
     43The mirror utility contains a script called ?`` which will store all files from the db to disk. This allows to search for the Math.random() and replace it with a constant. Using the `` utility one can put a new version into the DB. You will have to remove the URL from the top of the file and use it as argument to ``. Once you are done you will need to update the headers of the table, this can be done by invoking the `` script.
     45Another similiar source of trouble comes from using the current date to fetch resources or fetch resources depending on the useragent. Currently there are no hints on how to deal with that. The problem might be that on different dates, or different platforms a 404 will be returned instead of the real content making a comparison hard.
    4047=== Step through to mirror and everything on a screen cast ===
    4148 * A video on mirroring can be seen [ here].