• Lukas 'Eipi' Eipert's avatar
    Add incremental compiler to webpack dev server · 76f9be20
    Lukas 'Eipi' Eipert authored
    In order to have quicker compilations while developing we are adding an
    incremental compiler to webpack to render routes on demand.
    
    When the developer is working on the dev server and enables the
    incremental compiler with DEV_SERVER_INCREMENTAL=true, the new
    functionality is enabled.
    
    The biggest problem to solve here: How can we _know_ which entry point
    to render and which not? Our current webpack integration with rails
    requires the webpack manifest to have a list of all existing entry
    points. So our incremental compiler takes the following approach:
    
    Every compilation of webpack will run `generateEntries` which generates
    a list of all our entry points. In that function we are able to replace
    all page specific entrypoints and point them to an empty file,
    unless we explicitly want them to compile.
    
    In the webpack-dev-server itself we register a middleware which keeps
    track of all the page specific bundles requested. Whenever a page
    specific bundle is requested that hasn't been requested before, we add
    it to the list of bundles we want compiled.
    
    This approach allows us to dynamically change the entry points without a
    need to restart webpack alltogether _and_ it works with hot module
    reloading.
    
    Rather than pointing to a blank javascript we are pointing to one which
    renders an overlay to let the user know that webpack compiles the page
    for the first time.
    
    Additionally we keep a history of requested routes in `tmp/cache` in
    order to keep the list of compiled entry points between sessions. In a
    next iteration we can add a bit of logic and e.g. remove entry points
    the developer hasn't been visiting in a week.
    
    First results are really promising (on my machine):
     - Memory consumption when idling: 1600MB => 340MB
     - Max memory: ~2200MB => ~1000MB
     - Initial Compilation time: 58s => 15s
     - Recompile afer a file change: 13s => 3s
     - Visiting a new page that hasn't been visited before, it takes about
       four seconds to reload the page, seven seconds to completely load the
       page.
    
    Currently the technique still watches all of the source files, so
    changing an unrelated file will trigger a recompilation. This is however
    a minor caveat and the same behavior that we currently have, maybe we
    can optimize in the future.
    76f9be20
incremental_webpack_compiler.js 3.6 KB