Hugo Ulimit macOS No Space Left

4 minute read Published

How to increase the open files limit on macOS to workaround the too many files open socket exception with Hugo.

Trying to build the Hugo site textfiles.habd.as on a 2019 MBP and kept encountering the too many open file socket exception. Hugo seemed to have completed the build but was unable to subsequently serve the files:

Built in 110398 ms
Watching for changes in /Volumes/Inetpub/textfiles.bit/{archetypes,assets,content,data,la
youts,static,themes}
Watching for config changes in /Volumes/Inetpub/textfiles.bit/config/_default
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop
ERROR 2020/11/09 14:11:00 Error: listen tcp 127.0.0.1:1313: socket: too many open files

Running ulimit -a I see the following output:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       2048
-n: file descriptors                524288

To rule out the likelihood of a possible regression in the installed Hugo version I did a quick brew upgrade hugo to download the latest hugo version, bumping the hugo binary from 0.74.3 up to 0.78.1 and rebuilt only to see:

Built in 199877 ms
Watching for changes in /Volumes/Inetpub/textfiles.bit/{archetypes,assets,content,data,layouts,static,themes}
Watching for config changes in /Volumes/Inetpub/textfiles.bit/config/_default
fatal error: pipe failed

And the following stack trace:

Expand to show stack trace
goroutine 1 [running]:
runtime.throw(0x5d079aa, 0xb)
        /usr/local/Cellar/go/1.15.3/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc26a503858 sp=0xc26a503828 pc=0x403c212
runtime.sigNoteSetup(0x6e41c00)
        /usr/local/Cellar/go/1.15.3/libexec/src/runtime/os_darwin.go:98 +0xc5 fp=0xc26a503880 sp=0xc26a503858 pc=0x4038e65
os/signal.signal_enable(0x7ba4796400000002)
        /usr/local/Cellar/go/1.15.3/libexec/src/runtime/sigqueue.go:198 +0xa5 fp=0xc26a5038a0 sp=0xc26a503880 pc=0x4070b85
os/signal.enableSignal(...)
        /usr/local/Cellar/go/1.15.3/libexec/src/os/signal/signal_unix.go:49
os/signal.Notify.func1(0x2)
        /usr/local/Cellar/go/1.15.3/libexec/src/os/signal/signal.go:144 +0x88 fp=0xc26a5038c0 sp=0xc26a5038a0 pc=0x5615888
os/signal.Notify(0xc1bdd2cd20, 0xc26a503ad8, 0x2, 0x2)
        /usr/local/Cellar/go/1.15.3/libexec/src/os/signal/signal.go:164 +0x162 fp=0xc26a503938 sp=0xc26a5038c0 pc=0x56152a2
github.com/gohugoio/hugo/commands.(*commandeer).serve(0xc000adae00, 0xc000c00fc0, 0x2a9, 0x400)
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/commands/server.go:498 +0x625 fp=0xc26a503b58 sp=0xc26a503938 pc=0x564e5a5
github.com/gohugoio/hugo/commands.(*serverCmd).server(0xc000c00fc0, 0xc0002c9600, 0xc000597260, 0x0, 0x2, 0x0, 0x0)
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/commands/server.go:274 +0x2b6 fp=0xc26a503ca0 sp=0xc26a503b58 pc=0x564cd76
github.com/gohugoio/hugo/commands.(*serverCmd).server-fm(0xc0002c9600, 0xc000597260, 0x0, 0x2, 0x0, 0x0)
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/commands/server.go:131 +0x52 fp=0xc26a503ce8 sp=0xc26a503ca0 pc=0x565d472
github.com/spf13/cobra.(*Command).execute(0xc0002c9600, 0xc000597220, 0x2, 0x2, 0xc0002c9600, 0xc000597220)
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:850 +0x47c fp=0xc26a503dc0 sp=0xc26a503ce8 pc=0x41e0bbc
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003718c0, 0xc000c88c20, 0x8, 0xc0004bfa00)
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958 +0x375 fp=0xc26a503e98 sp=0xc26a503dc0 pc=0x41e1735
github.com/gohugoio/hugo/commands.Execute(0xc00019c010, 0x3, 0x3, 0x400b985, 0xc0000b0058, 0x0, 0x0)
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/commands/hugo.go:90 +0xb9 fp=0xc26a503f28 sp=0xc26a503e98 pc=0x56397f9
main.main()
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/main.go:23 +0x76 fp=0xc26a503f88 sp=0xc26a503f28 pc=0x565ebb6
runtime.main()
        /usr/local/Cellar/go/1.15.3/libexec/src/runtime/proc.go:204 +0x209 fp=0xc26a503fe0 sp=0xc26a503f88 pc=0x403e9e9
runtime.goexit()
        /usr/local/Cellar/go/1.15.3/libexec/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc26a503fe8 sp=0xc26a503fe0 pc=0x4074921

goroutine 40 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0003965a0)
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:32 +0x57

goroutine 56512 [select]:
github.com/gohugoio/hugo/watcher.(*Batcher).run(0xc1d29b1460)
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/watcher/batcher.go:53 +0x15f
created by github.com/gohugoio/hugo/watcher.New
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/watcher/batcher.go:42 +0x125

goroutine 56511 [syscall]:
syscall.syscall6(0x423c7c0, 0xb, 0x0, 0x0, 0xc207b6ee88, 0xa, 0x6e41850, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.15.3/libexec/src/runtime/sys_darwin.go:85 +0x2e
golang.org/x/sys/unix.kevent(0xb, 0x0, 0x0, 0xc207b6ee88, 0xa, 0x6e41850, 0xffffffffffffffff, 0x61b3c40, 0x6d86ac0)
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/zsyscall_darwin_amd64.go:292 +0xa6
golang.org/x/sys/unix.Kevent(0xb, 0x0, 0x0, 0x0, 0xc207b6ee88, 0xa, 0xa, 0x6e41850, 0x0, 0x61b3c00, ...)
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/syscall_bsd.go:413 +0x71
github.com/fsnotify/fsnotify.read(0xb, 0xc207b6ee88, 0xa, 0xa, 0x6e41850, 0x0, 0x0, 0x0, 0x61b3c40, 0x6d86ac0)
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:511 +0x6e
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc1d2a191a0)
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:274 +0x831
created by github.com/fsnotify/fsnotify.NewWatcher
        /Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:62 +0x199

goroutine 56513 [select]:
github.com/gohugoio/hugo/commands.(*commandeer).newWatcher.func1(0xc1d29b1460, 0xc000adae00, 0xc1cd9ac618, 0xc1d4f85050)
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/commands/hugo.go:873 +0xd9
created by github.com/gohugoio/hugo/commands.(*commandeer).newWatcher
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/commands/hugo.go:871 +0x2cc

goroutine 56858 [select]:
github.com/gohugoio/hugo/livereload.(*hub).run(0x6e02660)
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/livereload/hub.go:39 +0x1e9
created by github.com/gohugoio/hugo/livereload.Initialize
        /private/tmp/hugo-20201105-31737-102qobj/hugo-0.78.1/src/github.com/gohugoio/hugo/livereload/livereload.go:98 +0x45

Which isn’t incredibly helpful and harder to debug than the last error. Given I’m running the Big Sur beta I figure it’s possible there may be some kind of corner case there so I decided not to open an issue against Hugo. Leaving me at a loss for how to proceed. So I tried some percussive maintenance by increasing max limits at boot by tinkering with launchctl and using recommended defaults:

sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist && \
sudo launchctl load -w /Library/LaunchDaemons/limit.maxproc.plist

And that finally fixed the issue:

                   |  EN    
-------------------+--------
  Pages            | 57227  
  Paginator pages  |     0  
  Non-page files   | 57289  
  Static files     |    37  
  Processed images |     0  
  Aliases          |     0  
  Sitemaps         |     1  
  Cleaned          |     0  

Built in 147644 ms
Watching for changes in /Volumes/Inetpub/textfiles.bit/{archetypes,assets,content,data,la
youts,static,themes}
Watching for config changes in /Volumes/Inetpub/textfiles.bit/config/_default
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:64992/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Giving me the ability to update 57,000+ pages I hadn’t touched since I created them on Invisible Arch Linux where things JustWorked™️, reminding me once again just how finicky MacBooks can be and why I prefer to develop software on Linux.

Advertisement
Ledger Nano X - The secure hardware wallet