Speeding up org-publish

I publish my [[Digital garden]] using [[org-publish]]. ([[Publishing org-roam via GitLab CI]]).

It is shockingly slow. Around when I first introduced the pipeline, it took about 6 minutes - pretty slow. Currently, as of Feb 2022 with over 3000 org files, it's consistently taking around 30 minutes!!! This is horrendous.

Speed over time

I can pull out the builds time from my gitlab pipeline runs. I can pull out the number of files over time from git (https://blog.benoitblanchon.fr/git-file-count-vs-time/)

⥅ [[Speed_over_time/2022-02-05_12-55-25_screenshot.png]]

It judders around a bit, likely down to the fact that gitlab runners can just take different amount of time, but it's a reasonably linear trend. So it kind of looks like the more files, the more build time. So the amount of time publishing a file in the problem.

Profiling

Bit of a tip here on profiling the publish operation: Speeding Up Org Mode Publishing

I did that, and a big chunk of the process (22%?) seems to be coming from web-mode hooks.

Some things to try

Turning off hooks

I've tried a couple of ways of turning off off web-mode hooks.

(rassq-delete-all 'web-mode auto-mode-alist)
(fset 'web-mode (symbol-function 'fundamental-mode))

(see https://emacs.stackexchange.com/questions/21381/how-to-never-have-a-major-mode-enabled-automatically)

(defun org-publish-ignore-mode-hooks (orig-func &rest args)
  (let ((lexical-binding nil))
    (cl-letf (((symbol-function #'run-mode-hooks) #'ignore))
      (apply orig-func args))))

(advice-add 'org-publish :around #'org-publish-ignore-mode-hooks)

It might be doing something, but I haven't a proper before and after profile of that yet.

Suppressing output messages

Every single file that is published outputs a message like:

Publishing X with org-html-publish-to-html

It's quite nice to see progress through the files, but not really neccesary and output is usually a performance problem.

(advice-add 'org-publish-all :around 'silence)

(defun silence (orig-func &rest args)
  (let ((inhibit-message t))
    (apply orig-func args)))

This reduced the time from about 30 to 24 minutes.

It would be useful to see error messages though - I'll see if those still get output when I encounter one.

Incremental builds

One problem is potentially that the rebuild is from scratch each time. If I could make it incremental, I could to some extent bypass the problem of org-publish slowness, as I'd only be building a few files each time. But goes a bit against the principle of CI builds.

I make be able to make use of pipeline caching for this. What would I cache?

Resources

• How to speed up org export/publish? : emacs