Sign in Subscribe
Paperless-NGX

Paperless-NGX, ERRNO 28 No Space Left on Device

David Buhlman

• 5 min read
Paperless-NGX, ERRNO 28 No Space Left on Device
Photo by Aldebaran S / Unsplash

While upload some documents to Paperless-NGX it gave me the above error message. My initial investigation shows the operating system is not out of space, but the underlying container is showing 99% usage.

  • lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
NAME                      FSTYPE        SIZE MOUNTPOINT        LABEL
loop0                     squashfs     63.2M /snap/core20/1738
loop1                     squashfs     63.3M /snap/core20/1778
loop2                     squashfs      103M /snap/lxd/23541
loop3                     squashfs    111.9M /snap/lxd/24322
loop4                     squashfs     49.6M /snap/snapd/17883
loop5                     squashfs     49.8M /snap/snapd/17950
sda                                      30G
├─sda1                    vfat            1G /boot/efi
├─sda2                    ext4            2G /boot
└─sda3                    LVM2_member  26.9G
  └─ubuntu--vg-ubuntu--lv ext4         13.5G /
sr0                                    1024M
  • df -H
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              405M  1.5M  404M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   15G   14G  140M  99% /
tmpfs                              2.1G     0  2.1G   0% /dev/shm
tmpfs                              5.3M     0  5.3M   0% /run/lock
/dev/sda2                          2.1G  260M  1.7G  14% /boot
/dev/sda1                          1.2G  5.5M  1.2G   1% /boot/efi
//10.10.10.XXX/Data/Documents       68T   28T   41T  41% /mnt/paperless
overlay                             15G   14G  140M  99% /var/lib/docker/overlay2/1870dcf18687be0d4ff644ed02568d40b1dfab90117eab823f0d280e45cc51f0/merged
overlay                             15G   14G  140M  99% /var/lib/docker/overlay2/b992693656b3cb7ce069c3de350f6519497175a800757b5a28df284af33b3364/merged
overlay                             15G   14G  140M  99% /var/lib/docker/overlay2/be9f57f8d9da32e83c95a7b909c90c62c6ac774db6a856cd8e2f1e6781510e87/merged
overlay                             15G   14G  140M  99% /var/lib/docker/overlay2/e8732edcc7aa8b210f88d11c0fb92bacf005acfeeecf6d65d44645151e8b540b/merged
overlay                             15G   14G  140M  99% /var/lib/docker/overlay2/37885a2d8f6bd4085d47b57a42b83da6543d7b74e37fb5c13b57ed3e1a2a9b78/merged
tmpfs                              405M  4.1k  405M   1% /run/user/1000

Looking around online all my searches come back to everyone giving the same answer to prune the docker system. But there is nothing to prune, because this is the only container I've ever run on this machine. So I'm completely out of space. I'm going to bump the virtual disk up to 100GB so I can get Paperless-NGX back up and running. Then I can look into optimizing the amount of space it is using.
Steps performed on PAPERLESS-01

  • sudo shutdown now

Steps performed on HV-03 via Windows Admin Center

  • Navigate to Virtual Machines
  • Select PAPERLESS-01
  • Click on Settings
  • Navigate to Disk
  • Select Edit
  • Select Resize
  • Set to 100GB
  • Save disk settings

Weird, after completing these steps I realized that before I started Windows Admin Center was showing only 50% usage. Huh.

  • I create a checkpoint for PAPERLESS-01
  • I power on PAPERLESS-01

Steps performed on PAPERLESS-01

  • sudo docker-compose down
  • cfdisk
  • I select /dev/sda3 as the partition to work with and select resize from the menu options.
  • I set the New size to 96.9GB and hit enter
  • I select Write from the menu and type yes to confirm
  • I select Quit from the menu
  • sudo reboot
  • df -H
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              405M  979k  404M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   15G   13G  679M  95% /
tmpfs                              2.1G     0  2.1G   0% /dev/shm
tmpfs                              5.3M     0  5.3M   0% /run/lock
/dev/sda2                          2.1G  260M  1.7G  14% /boot
/dev/sda1                          1.2G  5.5M  1.2G   1% /boot/efi
//10.10.10.XXX/Data/Documents       68T   28T   41T  41% /mnt/paperless
tmpfs                              405M  4.1k  405M   1% /run/user/1000
  • sudo resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
    • The filesystem is already 3531776 (4k) blocks long. Nothing to do!

That's weird. What have I messed up? Can I do it the older way?

  • pvcreate /dev/sda3
    • Can't initialize physical volume "/dev/sda3" of volume group "ubuntu-vg" without -ff
  • pvcreate /dev/sda3 -ff
    • Can't open /dev/sda3 exclusively. Mounted filesystem?
  • vgextend ubuntu-vg /dev/sda3
    • Physical volume 'dev/sda3' is already in volume group 'ubuntu-vg'
    • Unable to add physical volume 'dev/sda3' to volume group 'ubuntu-vg'
    • /dev/sda3: physical volume not initialized
  • lvextend -|+ 100%FREE /dev/ubuntu-vg/ubuntu-lv
    • Size of logical volume ubuntu-vg/ubuntu-lv changed from 13.47GB to <26.95GiB.
  • df -H
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              405M  984k  404M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   15G   13G  678M  95% /
tmpfs                              2.1G     0  2.1G   0% /dev/shm
tmpfs                              5.3M     0  5.3M   0% /run/lock
/dev/sda2                          2.1G  260M  1.7G  14% /boot
/dev/sda1                          1.2G  5.5M  1.2G   1% /boot/efi
//10.10.10.XXX/Data/Documents       68T   28T   41T  41% /mnt/paperless
tmpfs                              405M  4.1k  405M   1% /run/user/1000
  • resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
Filesystem at /dev/mapper/ubuntu--vg-ubuntu--lv is mounted on /; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 4
The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now 7063552 (4k) blocks long.
  • df -H
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              405M  984k  404M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   29G   13G   15G  47% /
tmpfs                              2.1G     0  2.1G   0% /dev/shm
tmpfs                              5.3M     0  5.3M   0% /run/lock
/dev/sda2                          2.1G  260M  1.7G  14% /boot
/dev/sda1                          1.2G  5.5M  1.2G   1% /boot/efi
//10.10.10.XXX/Data/Documents       68T   28T   41T  41% /mnt/paperless
tmpfs                              405M  4.1k  405M   1% /run/user/1000

Well that's something, but why did it not increase it all the way to 100GB? What did I mess up?

  • vgs
VG    # PV # LV # SN Attr  VSize  VFree
ubuntu-vg  1  1  0 wz--n- <96.5g 70.00g

So the volume group is fully sized at this point, but why didn't the logical volume increase with my command?

  • lvextend -L+10G /dev/ubuntu-vg/ubuntu-lv
    • Size of logical volume ubuntu-vg/ubuntu-lv changed from 26.95 GiB to 36.95 Gib

So my syntax must have been wrong on my original attempt.

  • lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
    • Size of logical volume ubuntu-vg/ubuntu-lv changed from 36.95 GiB to 96.95 GiB
  • df -H
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              405M  984k  404M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   29G   13G   15G  47% /
tmpfs                              2.1G     0  2.1G   0% /dev/shm
tmpfs                              5.3M     0  5.3M   0% /run/lock
/dev/sda2                          2.1G  260M  1.7G  14% /boot
/dev/sda1                          1.2G  5.5M  1.2G   1% /boot/efi
//10.10.10.XXX/Data/Documents       68T   28T   41T  41% /mnt/paperless
tmpfs                              405M  4.1k  405M   1% /run/user/1000
  • resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
    • Filesystem at /dev/mapper/ubuntu--vg-ubuntu--lv is mounted on /; on-line resizing required
    • old_desc_blocks = 4, new_desc_blocks = 13
    • The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now 25413632 (4k) blocks long.
  • df -H
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              405M  984k  404M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  103G   13G   86G  13% /
tmpfs                              2.1G     0  2.1G   0% /dev/shm
tmpfs                              5.3M     0  5.3M   0% /run/lock
/dev/sda2                          2.1G  260M  1.7G  14% /boot
/dev/sda1                          1.2G  5.5M  1.2G   1% /boot/efi
//10.10.10.XXX/Data/Documents       68T   28T   41T  41% /mnt/paperless
tmpfs                              405M  4.1k  405M   1% /run/user/1000

So the syntax on lvenxtend is weird. If you are using the switch -L to extend it by a certain amount there is no space between the switch and the +, but when using the -l switch to extend it by a percentage there is a space between the switch and the +. Hopefully if I run into trouble again in the future I can find this and save myself some headache.