rustbuild rebuilds too much on modifications to rustdoc

[ollie27]

Modifying just rustdoc and rebuilding with the Makefiles by running make doc results in the following:

cfg: version 1.15.0-dev (368e092c2 2016-12-11)
cfg: build triple x86_64-pc-windows-msvc
cfg: host triples x86_64-pc-windows-msvc
cfg: target triples x86_64-pc-windows-msvc
cfg: host for x86_64-pc-windows-msvc is x86_64
cfg: os for x86_64-pc-windows-msvc is pc-windows-msvc
cfg: no good valgrind for x86_64-pc-windows-msvc
cfg: using CC=gcc (CFG_CC)
cfg: disabling valgrind run-pass tests
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/lib/librustdoc
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/bin/rustbook.exe
cp: x86_64-pc-windows-msvc/stage2/bin/librustdoc
cp: x86_64-pc-windows-msvc/stage2/bin/rustbook.exe
rustbook: doc/book/index.html
rustbook: doc/nomicon/index.html
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/bin/error_index_generator.exe
cp: x86_64-pc-windows-msvc/stage2/bin/error_index_generator.exe
error_index_generator: doc/error-index.html
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/bin/rustdoc.exe
cp: x86_64-pc-windows-msvc/stage2/bin/rustdoc.exe
rustdoc: doc/not_found.html
rustdoc: doc/index.html
rustdoc: doc/complement-lang-faq.html
rustdoc: doc/complement-design-faq.html
rustdoc: doc/complement-project-faq.html
rustdoc: doc/rustdoc.html
rustdoc: doc/reference.html
rustdoc: doc/grammar.html
rustdoc: doc/guide-crates.html
rustdoc: doc/guide-error-handling.html
rustdoc: doc/guide-ffi.html
rustdoc: doc/guide-macros.html
rustdoc: doc/guide.html
rustdoc: doc/guide-ownership.html
rustdoc: doc/guide-plugins.html
rustdoc: doc/guide-pointers.html
rustdoc: doc/guide-strings.html
rustdoc: doc/guide-tasks.html
rustdoc: doc/guide-testing.html
rustdoc: doc/tutorial.html
rustdoc: doc/intro.html
rustdoc: doc/std/index.html
rustdoc: doc/alloc/index.html
rustdoc: doc/collections/index.html
rustdoc: doc/core/index.html
rustdoc: doc/libc/index.html
rustdoc: doc/rustc_unicode/index.html

Which doesn't take long. However with rustbuild python x.py doc --stage 1 results in:

    Finished debug [unoptimized] target(s) in 0.0 secs
Synchronizing submodule url for 'src/compiler-rt'
Synchronizing submodule url for 'src/jemalloc'
Synchronizing submodule url for 'src/liblibc'
Synchronizing submodule url for 'src/llvm'
Synchronizing submodule url for 'src/rt/hoedown'
Synchronizing submodule url for 'src/rust-installer'
HEAD is now at a8fc4c16 Merge pull request #28 from xen0n/preprocessor-firefighting
HEAD is now at 0ac39c5cc Auto merge of #462 - semarie:openbsd-i686-correct, r=alexcrichton
HEAD is now at d7342a9a957 Merge pull request #59 from pftbest/msp430_backport
HEAD is now at a3736a0 Merge pull request #6 from intelfx/patch-1
HEAD is now at 4f99485 Merge pull request #54 from brson/docdir
Building stage0 std artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Building stage0 test artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Building stage0 compiler artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling rustdoc v0.0.0 (file:///E:/Source/rust/src/librustdoc)
   Compiling rustc-main v0.0.0 (file:///E:/Source/rust/src/rustc)
    Finished release [optimized] target(s) in 95.30 secs
Building stage1 std artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling core v0.0.0 (file:///E:/Source/rust/src/libcore)
   Compiling unwind v0.0.0 (file:///E:/Source/rust/src/libunwind)
   Compiling libc v0.0.0 (file:///E:/Source/rust/src/rustc/libc_shim)
   Compiling compiler_builtins v0.0.0 (file:///E:/Source/rust/src/libcompiler_builtins)
   Compiling std v0.0.0 (file:///E:/Source/rust/src/libstd)
   Compiling alloc v0.0.0 (file:///E:/Source/rust/src/liballoc)
   Compiling rand v0.0.0 (file:///E:/Source/rust/src/librand)
   Compiling rustc_unicode v0.0.0 (file:///E:/Source/rust/src/librustc_unicode)
   Compiling alloc_system v0.0.0 (file:///E:/Source/rust/src/liballoc_system)
   Compiling panic_abort v0.0.0 (file:///E:/Source/rust/src/libpanic_abort)
   Compiling panic_unwind v0.0.0 (file:///E:/Source/rust/src/libpanic_unwind)
   Compiling collections v0.0.0 (file:///E:/Source/rust/src/libcollections)
   Compiling std_shim v0.1.0 (file:///E:/Source/rust/src/rustc/std_shim)
    Finished release [optimized] target(s) in 37.87 secs
Building stage1 test artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling getopts v0.0.0 (file:///E:/Source/rust/src/libgetopts)
   Compiling term v0.0.0 (file:///E:/Source/rust/src/libterm)
   Compiling test v0.0.0 (file:///E:/Source/rust/src/libtest)
   Compiling test_shim v0.1.0 (file:///E:/Source/rust/src/rustc/test_shim)
    Finished release [optimized] target(s) in 13.52 secs
Building stage1 compiler artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling rustc_platform_intrinsics v0.0.0 (file:///E:/Source/rust/src/librustc_platform_intrinsics)
   Compiling rustc_bitflags v0.0.0 (file:///E:/Source/rust/src/librustc_bitflags)
   Compiling log v0.0.0 (file:///E:/Source/rust/src/liblog)
   Compiling arena v0.0.0 (file:///E:/Source/rust/src/libarena)
   Compiling fmt_macros v0.0.0 (file:///E:/Source/rust/src/libfmt_macros)
   Compiling graphviz v0.0.0 (file:///E:/Source/rust/src/libgraphviz)
   Compiling flate v0.0.0 (file:///E:/Source/rust/src/libflate)
   Compiling rustc_llvm v0.0.0 (file:///E:/Source/rust/src/librustc_llvm)
   Compiling rustdoc v0.0.0 (file:///E:/Source/rust/src/librustdoc)
   Compiling serialize v0.0.0 (file:///E:/Source/rust/src/libserialize)
   Compiling rustc_data_structures v0.0.0 (file:///E:/Source/rust/src/librustc_data_structures)
   Compiling syntax_pos v0.0.0 (file:///E:/Source/rust/src/libsyntax_pos)
   Compiling rustc_errors v0.0.0 (file:///E:/Source/rust/src/librustc_errors)
   Compiling syntax v0.0.0 (file:///E:/Source/rust/src/libsyntax)
   Compiling rustc_back v0.0.0 (file:///E:/Source/rust/src/librustc_back)
   Compiling proc_macro v0.0.0 (file:///E:/Source/rust/src/libproc_macro)
   Compiling proc_macro_tokens v0.0.0 (file:///E:/Source/rust/src/libproc_macro_tokens)
   Compiling rustc_const_math v0.0.0 (file:///E:/Source/rust/src/librustc_const_math)
   Compiling syntax_ext v0.0.0 (file:///E:/Source/rust/src/libsyntax_ext)
   Compiling rustc v0.0.0 (file:///E:/Source/rust/src/librustc)
   Compiling rustc_const_eval v0.0.0 (file:///E:/Source/rust/src/librustc_const_eval)
   Compiling rustc_metadata v0.0.0 (file:///E:/Source/rust/src/librustc_metadata)
   Compiling rustc_privacy v0.0.0 (file:///E:/Source/rust/src/librustc_privacy)
   Compiling rustc_incremental v0.0.0 (file:///E:/Source/rust/src/librustc_incremental)
   Compiling rustc_resolve v0.0.0 (file:///E:/Source/rust/src/librustc_resolve)
   Compiling rustc_save_analysis v0.0.0 (file:///E:/Source/rust/src/librustc_save_analysis)
   Compiling rustc_mir v0.0.0 (file:///E:/Source/rust/src/librustc_mir)
   Compiling rustc_passes v0.0.0 (file:///E:/Source/rust/src/librustc_passes)
   Compiling rustc_lint v0.0.0 (file:///E:/Source/rust/src/librustc_lint)
   Compiling rustc_typeck v0.0.0 (file:///E:/Source/rust/src/librustc_typeck)
   Compiling rustc_trans v0.0.0 (file:///E:/Source/rust/src/librustc_trans)
   Compiling rustc_borrowck v0.0.0 (file:///E:/Source/rust/src/librustc_borrowck)
   Compiling rustc_plugin v0.0.0 (file:///E:/Source/rust/src/librustc_plugin)
   Compiling proc_macro_plugin v0.0.0 (file:///E:/Source/rust/src/libproc_macro_plugin)
   Compiling rustc_driver v0.0.0 (file:///E:/Source/rust/src/librustc_driver)
   Compiling rustc-main v0.0.0 (file:///E:/Source/rust/src/rustc)
    Finished release [optimized] target(s) in 676.85 secs
Building stage1 tool rustbook (x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Documenting stage1 standalone (x86_64-pc-windows-msvc)
Documenting stage1 std (x86_64-pc-windows-msvc)
 Documenting core v0.0.0 (file:///E:/Source/rust/src/libcore)
 Documenting libc v0.0.0 (file:///E:/Source/rust/src/rustc/libc_shim)
 Documenting rustc_unicode v0.0.0 (file:///E:/Source/rust/src/librustc_unicode)
 Documenting compiler_builtins v0.0.0 (file:///E:/Source/rust/src/libcompiler_builtins)
 Documenting alloc v0.0.0 (file:///E:/Source/rust/src/liballoc)
 Documenting rand v0.0.0 (file:///E:/Source/rust/src/librand)
 Documenting alloc_system v0.0.0 (file:///E:/Source/rust/src/liballoc_system)
 Documenting panic_abort v0.0.0 (file:///E:/Source/rust/src/libpanic_abort)
 Documenting unwind v0.0.0 (file:///E:/Source/rust/src/libunwind)
 Documenting panic_unwind v0.0.0 (file:///E:/Source/rust/src/libpanic_unwind)
 Documenting collections v0.0.0 (file:///E:/Source/rust/src/libcollections)
 Documenting std v0.0.0 (file:///E:/Source/rust/src/libstd)
 Documenting std_shim v0.1.0 (file:///E:/Source/rust/src/rustc/std_shim)
Building stage1 tool error_index_generator (x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Documenting stage1 error index (x86_64-pc-windows-msvc)
Documenting stage1 test (x86_64-pc-windows-msvc)
 Documenting getopts v0.0.0 (file:///E:/Source/rust/src/libgetopts)
 Documenting term v0.0.0 (file:///E:/Source/rust/src/libterm)
 Documenting test v0.0.0 (file:///E:/Source/rust/src/libtest)
 Documenting test_shim v0.1.0 (file:///E:/Source/rust/src/rustc/test_shim)
Build completed in 0:14:33

Is there a faster way to rebuild rustdoc and the docs when using rustbuild?

[alexcrichton]

Yes rustbuild builds rustdoc in all three stages as opposed to just the end (which the makefiles did).

We may be able to get away with only detecting/cleaning between stages if rustc itself changes, and ignoring changes to rustdoc except for when we think we need to clean docs.

An alternative would be to work with ./x.py doc --stage 0 ...

[Mark-Simulacrum]

Would #38331 help with this? In theory you could then --keep-stage 1 and then rustbuild would only rebuild post-stage 1, I believe. This might be wrong in practice, though.

[QuietMisdreavus]

It seems that ./x.py doc doesn't interact with either the --stage or the --keep-stage arguments well at all. Like the default (--stage 2), --stage 1 unilaterally compiles all of stage 1, even if the only change was within librustdoc. --stage 0 seems to only rebuild rustdoc, but then doesn't use it to rebuild the std docs. It just automatically treats it as complete.

--stage 1 --keep-stage 0, when run immediately after --stage 0, proceeds to rebuild the entirety of stage 1. Otherwise, it rebuilds only what's necessary in stage1 and skips actually regenerating the docs, just like --stage 0.

Confusingly, before I decided to systematically dive into this, these combinations would also use old builds of rustdoc for its documentation rendering, or would generate dynamic-link errors by trying to reference .so files from earlier stages but using hashes from later on. I would dive further into this, but my development environment is a tiny server that takes 15 minutes to rebuild a stage, so I'll leave the report at this for now. For the moment it seems like I need to build all of std and rustc, at least once, just to test a rustdoc change.