Message ID | 20230427172516.19003-1-farosas@suse.de |
---|---|
State | New |
Headers | show |
Series | [v2] meson: Pass -j option to sphinx | expand |
On Thu, Apr 27, 2023 at 02:25:16PM -0300, Fabiano Rosas wrote: > Save a bit of build time by passing the number of jobs option to > sphinx. > > We cannot use the -j option from make because meson does not support > setting build time parameters for custom targets. Use nproc instead or > the equivalent sphinx option "-j auto", if that is available. > > Also make sure our plugins support parallelism and report it properly > to sphinx. Particularly, implement the merge_domaindata method in > DBusDomain that is used to merge in data from other subprocesses. > > before: > $ time make man html > ... > [1/2] Generating docs/QEMU manual with a custom command > [2/2] Generating docs/QEMU man pages with a custom command > > real 0m43.157s > user 0m42.642s > sys 0m0.576s > > after: > $ time make man html > ... > [1/2] Generating docs/QEMU manual with a custom command > [2/2] Generating docs/QEMU man pages with a custom command > > real 0m25.014s > user 0m51.288s > sys 0m2.085s On my 12 CPU laptop I see a similar magnitude benefit - about 20 seconds is cut from the docs build time - 50 down to 30 secs. Watching the CPU usage I see sphinx is not very good at keeping all CPUs busy. For perhaps 2 seconds I'll see 8 sphinx processes burning CPUs, but the majority of the time it'll only be 1 or 2 sphinx processes. IOW, we do get a benefit, but it is not nearly as good as one might hope for given the number of CPUs potentially available. > Signed-off-by: Fabiano Rosas <farosas@suse.de> > --- > docs/meson.build | 12 ++++++++++++ > docs/sphinx/dbusdomain.py | 4 ++++ > docs/sphinx/fakedbusdoc.py | 5 +++++ > docs/sphinx/qmp_lexer.py | 5 +++++ > 4 files changed, 26 insertions(+) Tested-by: Daniel P. Berrangé <berrange@redhat.com> > > diff --git a/docs/meson.build b/docs/meson.build > index f220800e3e..9e4bed6fa0 100644 > --- a/docs/meson.build > +++ b/docs/meson.build > @@ -10,6 +10,18 @@ if sphinx_build.found() > SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ] > endif > > + sphinx_version = run_command(SPHINX_ARGS + ['--version'], > + check: false).stdout().split()[1] > + if sphinx_version.version_compare('>=5.1.2') > + SPHINX_ARGS += ['-j', 'auto'] > + else > + nproc = find_program('nproc') > + if nproc.found() > + jobs = run_command(nproc, check:false).stdout() > + SPHINX_ARGS += ['-j', jobs] > + endif > + endif ANy reason for check: false in these 2 run_command calls ? They'll both return 0 on success, so I would have though 'check: true' was more robust at error reporting ? With regards, Daniel
Daniel P. Berrangé <berrange@redhat.com> writes: > On Thu, Apr 27, 2023 at 02:25:16PM -0300, Fabiano Rosas wrote: >> Save a bit of build time by passing the number of jobs option to >> sphinx. >> >> We cannot use the -j option from make because meson does not support >> setting build time parameters for custom targets. Use nproc instead or >> the equivalent sphinx option "-j auto", if that is available. >> >> Also make sure our plugins support parallelism and report it properly >> to sphinx. Particularly, implement the merge_domaindata method in >> DBusDomain that is used to merge in data from other subprocesses. >> >> before: >> $ time make man html >> ... >> [1/2] Generating docs/QEMU manual with a custom command >> [2/2] Generating docs/QEMU man pages with a custom command >> >> real 0m43.157s >> user 0m42.642s >> sys 0m0.576s >> >> after: >> $ time make man html >> ... >> [1/2] Generating docs/QEMU manual with a custom command >> [2/2] Generating docs/QEMU man pages with a custom command >> >> real 0m25.014s >> user 0m51.288s >> sys 0m2.085s > > On my 12 CPU laptop I see a similar magnitude benefit - about > 20 seconds is cut from the docs build time - 50 down to 30 secs. > > Watching the CPU usage I see sphinx is not very good at keeping > all CPUs busy. For perhaps 2 seconds I'll see 8 sphinx processes > burning CPUs, but the majority of the time it'll only be 1 or 2 > sphinx processes. > > IOW, we do get a benefit, but it is not nearly as good as one > might hope for given the number of CPUs potentially available. > >> Signed-off-by: Fabiano Rosas <farosas@suse.de> >> --- >> docs/meson.build | 12 ++++++++++++ >> docs/sphinx/dbusdomain.py | 4 ++++ >> docs/sphinx/fakedbusdoc.py | 5 +++++ >> docs/sphinx/qmp_lexer.py | 5 +++++ >> 4 files changed, 26 insertions(+) > > Tested-by: Daniel P. Berrangé <berrange@redhat.com> > >> >> diff --git a/docs/meson.build b/docs/meson.build >> index f220800e3e..9e4bed6fa0 100644 >> --- a/docs/meson.build >> +++ b/docs/meson.build >> @@ -10,6 +10,18 @@ if sphinx_build.found() >> SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ] >> endif >> >> + sphinx_version = run_command(SPHINX_ARGS + ['--version'], >> + check: false).stdout().split()[1] >> + if sphinx_version.version_compare('>=5.1.2') >> + SPHINX_ARGS += ['-j', 'auto'] >> + else >> + nproc = find_program('nproc') >> + if nproc.found() >> + jobs = run_command(nproc, check:false).stdout() >> + SPHINX_ARGS += ['-j', jobs] >> + endif >> + endif > > ANy reason for check: false in these 2 run_command calls ? > No, I haven't thought about it. I'll change them to true.
diff --git a/docs/meson.build b/docs/meson.build index f220800e3e..9e4bed6fa0 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -10,6 +10,18 @@ if sphinx_build.found() SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ] endif + sphinx_version = run_command(SPHINX_ARGS + ['--version'], + check: false).stdout().split()[1] + if sphinx_version.version_compare('>=5.1.2') + SPHINX_ARGS += ['-j', 'auto'] + else + nproc = find_program('nproc') + if nproc.found() + jobs = run_command(nproc, check:false).stdout() + SPHINX_ARGS += ['-j', jobs] + endif + endif + # This is a bit awkward but works: create a trivial document and # try to run it with our configuration file (which enforces a # version requirement). This will fail if sphinx-build is too old. diff --git a/docs/sphinx/dbusdomain.py b/docs/sphinx/dbusdomain.py index 2ea95af623..9872fd5bf6 100644 --- a/docs/sphinx/dbusdomain.py +++ b/docs/sphinx/dbusdomain.py @@ -400,6 +400,10 @@ def get_objects(self) -> Iterator[Tuple[str, str, str, str, str, int]]: for refname, obj in self.objects.items(): yield (refname, refname, obj.objtype, obj.docname, obj.node_id, 1) + def merge_domaindata(self, docnames, otherdata): + for name, obj in otherdata['objects'].items(): + if obj.docname in docnames: + self.data['objects'][name] = obj def setup(app): app.add_domain(DBusDomain) diff --git a/docs/sphinx/fakedbusdoc.py b/docs/sphinx/fakedbusdoc.py index d2c5079046..2d2e6ef640 100644 --- a/docs/sphinx/fakedbusdoc.py +++ b/docs/sphinx/fakedbusdoc.py @@ -23,3 +23,8 @@ def run(self): def setup(app: Sphinx) -> Dict[str, Any]: """Register a fake dbus-doc directive with Sphinx""" app.add_directive("dbus-doc", FakeDBusDocDirective) + + return dict( + parallel_read_safe = True, + parallel_write_safe = True + ) diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py index f7e4c0e198..a59de8a079 100644 --- a/docs/sphinx/qmp_lexer.py +++ b/docs/sphinx/qmp_lexer.py @@ -41,3 +41,8 @@ def setup(sphinx): sphinx.add_lexer('QMP', QMPExampleLexer) except errors.VersionRequirementError: sphinx.add_lexer('QMP', QMPExampleLexer()) + + return dict( + parallel_read_safe = True, + parallel_write_safe = True + )
Save a bit of build time by passing the number of jobs option to sphinx. We cannot use the -j option from make because meson does not support setting build time parameters for custom targets. Use nproc instead or the equivalent sphinx option "-j auto", if that is available. Also make sure our plugins support parallelism and report it properly to sphinx. Particularly, implement the merge_domaindata method in DBusDomain that is used to merge in data from other subprocesses. before: $ time make man html ... [1/2] Generating docs/QEMU manual with a custom command [2/2] Generating docs/QEMU man pages with a custom command real 0m43.157s user 0m42.642s sys 0m0.576s after: $ time make man html ... [1/2] Generating docs/QEMU manual with a custom command [2/2] Generating docs/QEMU man pages with a custom command real 0m25.014s user 0m51.288s sys 0m2.085s Signed-off-by: Fabiano Rosas <farosas@suse.de> --- docs/meson.build | 12 ++++++++++++ docs/sphinx/dbusdomain.py | 4 ++++ docs/sphinx/fakedbusdoc.py | 5 +++++ docs/sphinx/qmp_lexer.py | 5 +++++ 4 files changed, 26 insertions(+)