PostInstall is broken in GoboLinux 017

[Nuc1eoN]

I have noticed that PostInstall is completely broken on GoboLinux 017.

Not sure why it has not yet been discovered. I have tried on a vanilla GoboLinux 017 installation as well as on my updated system and could reproduce on both. So this is a major issue.

I went down quite a rabbit hole.

The issue is as follows:

Assume we take a simple Program like Tree and add a PostInstall script to /Data/Compile/Recipes/Tree/2.1.3/Resources/PostInstall:

#!/bin/bash

echo "Hello World"

make it executable: chmod +x /Data/Compile/Recipes/Tree/2.1.3/Resources/PostInstall.

Upon trying to Compile the recipe, it fails with:

Compile: Running PostInstall for Tree
mount: /.union_mp.tmp.apYC71GFfsq/Programs: special device none does not exist.
       dmesg(1) may have more information after failed mount system call.
UnionSandbox: Unable to mount unionfs
UnionSandbox: Cleaning up.
umount: /.union_mp.tmp.apYC71GFfsq//proc: no mount point specified.
umount: /.union_mp.tmp.apYC71GFfsq//dev: no mount point specified.
umount: /.union_mp.tmp.apYC71GFfsq/Programs/Tree/Settings: no mount point specified.
Error unmounting /.union_mp.tmp.apYC71GFfsq/Programs/Tree/Settings
umount: /.union_mp.tmp.apYC71GFfsq/Programs/Tree/6.6.6: no mount point specified.
Error unmounting /.union_mp.tmp.apYC71GFfsq/Programs/Tree/6.6.6
umount: /.union_mp.tmp.apYC71GFfsq/usr: no mount point specified.
Error unmounting /.union_mp.tmp.apYC71GFfsq/usr
umount: /.union_mp.tmp.apYC71GFfsq/Users: no mount point specified.
Error unmounting /.union_mp.tmp.apYC71GFfsq/Users
umount: /.union_mp.tmp.apYC71GFfsq/Data: no mount point specified.
Error unmounting /.union_mp.tmp.apYC71GFfsq/Data
umount: /.union_mp.tmp.apYC71GFfsq/System: no mount point specified.
Error unmounting /.union_mp.tmp.apYC71GFfsq/System
umount: /.union_mp.tmp.apYC71GFfsq/Programs: not mounted.
Error unmounting /.union_mp.tmp.apYC71GFfsq/Programs
rmdir: failed to remove './dev': No such file or directory
rmdir: failed to remove './proc': No such file or directory
rmdir: failed to remove './run': No such file or directory

However I have "seemingly" discovered a workaround:
In /System/Settings/Scripts/Directories.conf we have to move overlayfs to the bottom:

# UnionFS implementation to be used in the scripts, in order of preference.
# Supported ones are 'unionfs', 'funionfs' and 'unionfs-fuse'.
unionImplementations=(
    "unionfs-fuse"
    "unionfs"
    "funionfs"
    "overlayfs"
)

Now the installation seems to succeed:

SandboxInstall: Postprocessing Sandbox
Compile: Stripping executables...
Compile: Running PostInstall for Tree
Failed to parse RO/RW flag, setting RO.
Failed to parse RO/RW flag, setting RO.
Failed to parse RO/RW flag, setting RO.
Failed to parse RO/RW flag, setting RO.
Hello World!
UnionSandbox: Cleaning up.
UnionSandbox: Moving entries to: /Programs/Tree/6.6.6/.PostInstall_Root
rmdir: failed to remove './run': No such file or directory
Compile: Postprocessing PostInstall changes.
UpdateSettings: Current and default settings match
SymlinkProgram: Symlinking Tree 6.6.6.
SymlinkProgram: Symlinking global settings...
SymlinkProgram: Symlinking tasks...
SymlinkProgram: Storing variable files...
SymlinkProgram: Symlinking libraries...
SymlinkProgram: Updating library database (ldconfig)...
SymlinkProgram: Symlinking headers...
SymlinkProgram: Symlinking info...
SymlinkProgram: Updating info dir...
SymlinkProgram: Symlinking manuals...
SymlinkProgram: Processed 1 file.
SymlinkProgram: Symlinking executables...
SymlinkProgram: Processed 1 file.
SymlinkProgram: Symlinking wrappers...
SymlinkProgram: Symlinking libexec..
SymlinkProgram: Symlinking shared...
SymlinkProgram: Processed 2 files.
SymlinkProgram: Removing unused directories...
SymlinkProgram: Done.
Compile: Compiling Tree version 6.6.6.
Compile: Generating package's build information...
GenBuildInformation: Generating dependency report. Please wait:  |

Compile: You are encouraged to submit your recipe for inclusion into the distribution.
Compile: Please run 'ContributeRecipe Tree 6.6.6' to do this.

We see "Hello World!" being printed here and the installation does not register as fail.

However we also see some weird messages:

Failed to parse RO/RW flag, setting RO.

And actually that happens, because in:

Compile/bin/UnionSandbox

Lines 137 to 145 in c251900

	function mount_union() {
	dir=$1
	Assert_Dir ${sandbox_mp}${dir}
	Assert_Dir ${sandbox_rw}${dir}
	Assert_Dir ${sandbox_tmp}${dir}
	uniondirs=`echo "${sandbox_rw}${dir}=rw:${dir}=ro:${sandbox_tmp}${dir}=tmp" | sed "s,$unionfsPackageDir=ro,$unionfsPackageDir=rw,g"`
	Union_Mount "$unionbackend" "$uniondirs" "${sandbox_mp}${dir}"
	[ "$?" = "0" ] || sandbox_die "Unable to mount unionfs"
	}

in line 142 | sed "s,$unionfsPackageDir=ro,$unionfsPackageDir=rw,g" does not correctly execute, because during PostInstall the variable $unionfsPackageDir is empty (while it is correctly set during the normal install operation)! So the sed command is actually executed as | sed "s,=ro,=rw,g". This is definitely not intended.

Therefore the value of $uniondirs is wrong at this point.

The overlayfs implementation fails for that reason. But for unionfs-fuse the operation is slightly different, as seen in lines 35-37:

Compile/Functions/UnionFS

Lines 24 to 39 in c251900

	function Union_Mount() {
	local union="$1"
	local rw_ro=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\1=rw:\2=ro/g'`
	case "$union" in
	"funionfs") funionfs -o dirs="$rw_ro" -o nonempty none "$3" ;;
	"overlayfs")
	upper=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\1/g'`
	lower=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\2/g'`
	work=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\3/g'`
	mount -t overlay none -o "lowerdir=$lower,upperdir=$upper,workdir=$work" "$3" ;;
	"unionfs") mount -t unionfs -o dirs="$rw_ro" none "$3" ;;
	"unionfs-fuse")
	dirs=`echo "$rw_ro" | sed 's/=[^:]*//g'`
	unionfs -o cow -o nonempty "$rw_ro" "$3";;
	esac
	}

Note that $2 in this code is the wrongly assiged $uniondirs from before.

And for that reason, when using unionfs-fuse the output shows:

Failed to parse RO/RW flag, setting RO.

(Note that the correct behaviour would be to assign rw.)

When using overlayfs instead, it will completely fail with (above mentioned):

mount: /.union_mp.tmp.apYC71GFfsq/Programs: special device none does not exist.
       dmesg(1) may have more information after failed mount system call.

So in my opinion the unionfs-fuse operation is just as flawed as with overlayfs. It seems to "work" for a simple echo "Hello World" but I doubt it would be able to actually manipulate the file system (as a proper PostInstall operation would).

So at this point I am wondering:

Should the PostInstall operation even be run inside the UnionSandbox??

If I understand correctly, it should just directly execute on the actual file system. Also the fact that the variable $unionfsPackageDir is empty during PostInstall indicates, that PostInstall was never intended to run in the UnionSandbox. I could be wrong.

@lucasvr You seem to have written the code for this. Could you provide any input?

I do not know what the proper way to fix this would be at this point. It took me quite a lot of debugging to get to this point.

Thanks in advance! :)

[lucasvr]

The different UnionFS implementations have always been challenging: at times, some of them would not work on the Live environment, whereas some others would show runtime bugs on the installed system. IIRC we even had different versions of Directories.conf for the Live and installed systems.

Could you try to change this line:
mount -t overlay none -o "lowerdir=$lower,upperdir=$upper,workdir=$work" "$3" ;;
to:
mount -t overlay overlay -o "lowerdir=$lower,upperdir=$upper,workdir=$work" "$3" ;;

and move overlay back to the top of Directories.conf? That backend should be the preferred one.

[Nuc1eoN]

TL;DR:

Changing none -> overlay does not fix it. But in my process I was able to figure out why the mount command is erroneous (two failed sed operations in a row).

Mount executes:

mount -v -t overlay overlay -o lowerdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp,upperdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp,workdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp /.union_mp.tmp.9irjB1HRaH7/Programs

but the actual command (if seds would work) should be:

mount -v -t overlay overlay -o lowerdir=/usr,upperdir=/.union_rw.tmp.MZF64mfUQ6x/usr,workdir=/.union_rw.tmp.5hs4MXnYVfG/usr /.union_mp.tmp.HDMuYbmyjtO/usr

But fixing the mount command does not fix the PostInstall in the end.

Which is why I suspect that PostInstall is supposed to run standalone outside of UnionSandbox.

Before (or if :p) you read any further excuse me for the wall of text.
I tried to be as concise as as brief as possible, but not sure I succeeded.

---

Thanks for your input, that is actually the first thing I've tried.

But the only thing it cosmetically changes is the error message:

mount: /.union_mp.tmp.apYC71GFfsq/Programs: special device overlay does not exist.

The logical conclusion would be to understand why the mount command fails. But that is the wrong rabbit hole to get into 🥲

I am still gonna try to explain.

The reason the mount command actually fails is, because it is being passed the wrong arguments:

mount -v -t overlay overlay -o lowerdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp,upperdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp,workdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp /.union_mp.tmp.9irjB1HRaH7/Programs

That is completely unreadable because the same command is getting passed thrice (to upper, lower and workdir):

/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp

Actually it is supposed to be split like this:

mount -v -t overlay overlay -o \
lowerdir=/.union_rw.tmp.GejezIVMqrK/Programs, \
upperdir=/Programs, \
workdir=/.union_rw.tmp.WcOeZ8ZUJJU/Programs \
/.union_mp.tmp.9irjB1HRaH7/Programs

The reason that /.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp is not getting split into 3 parts (as well as getting removed (rw|row|tmp) strings) is because these sed commands fail:

Compile/Functions/UnionFS

Lines 24 to 39 in c251900

	function Union_Mount() {
	local union="$1"
	local rw_ro=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\1=rw:\2=ro/g'`
	case "$union" in
	"funionfs") funionfs -o dirs="$rw_ro" -o nonempty none "$3" ;;
	"overlayfs")
	upper=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\1/g'`
	lower=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\2/g'`
	work=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\3/g'`
	mount -t overlay none -o "lowerdir=$lower,upperdir=$upper,workdir=$work" "$3" ;;
	"unionfs") mount -t unionfs -o dirs="$rw_ro" none "$3" ;;
	"unionfs-fuse")
	dirs=`echo "$rw_ro" | sed 's/=[^:]*//g'`
	unionfs -o cow -o nonempty "$rw_ro" "$3";;
	esac
	}

The seds in lines 30, 31, 32 are supposed to strip /.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp
into essentially

upper=/.union_rw.tmp.GejezIVMqrK/Programs
lower= /Programs
work= /.union_rw.tmp.WcOeZ8ZUJJU/Programs

But that does not happen, because the sed commands are looking for a ro..rw..tmp pattern. But the passed string has a rw..rw..tmp pattern.

At first I "fixed" the seds to look for both patterns. But that is the wrong thing to do.
Because actually /.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp, which is the same as $2, is being passed from a previous call:

Compile/bin/UnionSandbox

Lines 137 to 145 in c251900

	function mount_union() {
	dir=$1
	Assert_Dir ${sandbox_mp}${dir}
	Assert_Dir ${sandbox_rw}${dir}
	Assert_Dir ${sandbox_tmp}${dir}
	uniondirs=`echo "${sandbox_rw}${dir}=rw:${dir}=ro:${sandbox_tmp}${dir}=tmp" | sed "s,$unionfsPackageDir=ro,$unionfsPackageDir=rw,g"`
	Union_Mount "$unionbackend" "$uniondirs" "${sandbox_mp}${dir}"
	[ "$?" = "0" ] || sandbox_die "Unable to mount unionfs"
	}

In line 142 the variable $uniondirs is assigned. $uniondirs is being seded with | sed "s,$unionfsPackageDir=ro,$unionfsPackageDir=rw,g", but as mentioned in my first post $unionfsPackageDir is an empty variable during the PostInstall execution! (but is correctly assigned during install step)

So sed "s,$unionfsPackageDir=ro,$unionfsPackageDir=rw,g" actually becomes sed s,=ro,rw,g.

So what happens is:

echo "/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=ro:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp" | sed "s,=ro,rw,g"

This basically replaces rw..ro..tmp with rw..rw..tmp (which is wrong). Now at this point the variable is mangled!

Because as mentioned before in

Compile/Functions/UnionFS

Lines 24 to 39 in c251900

	function Union_Mount() {
	local union="$1"
	local rw_ro=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\1=rw:\2=ro/g'`
	case "$union" in
	"funionfs") funionfs -o dirs="$rw_ro" -o nonempty none "$3" ;;
	"overlayfs")
	upper=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\1/g'`
	lower=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\2/g'`
	work=`echo $2 | sed 's/\(.*\)=rw:\(.*\)=ro:\(.*\)=tmp/\3/g'`
	mount -t overlay none -o "lowerdir=$lower,upperdir=$upper,workdir=$work" "$3" ;;
	"unionfs") mount -t unionfs -o dirs="$rw_ro" none "$3" ;;
	"unionfs-fuse")
	dirs=`echo "$rw_ro" | sed 's/=[^:]*//g'`
	unionfs -o cow -o nonempty "$rw_ro" "$3";;
	esac
	}

the three seds in lines 30, 31, 32 (correctly) expect a rw..ro..tmp pattern. Therefore sed doesn't do anything here.

So case $overlayfs completely fails with the bogus command:

mount -v -t overlay overlay -o lowerdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp,upperdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp,workdir=/.union_rw.tmp.GejezIVMqrK/Programs=rw:/Programs=rw:/.union_rw.tmp.WcOeZ8ZUJJU/Programs=tmp /.union_mp.tmp.9irjB1HRaH7/Programs

which results in:

mount: /.union_mp.tmp.apYC71GFfsq/Programs: special device overlay does not exist.

The only reason that case "$unionfs-fuse" (line 34) "succeeds" ( = throws no error) is because it is seemingly more lash about it, it just throws:

Failed to parse RO/RW flag, setting RO.

After that it simply proceeds.

---

I have then 'fixed' all of this by removing the offending sed which is causing all of this (the one with the empty var). This fixes the actual mount commands.

So basically I have managed to "fix":

mount: /.union_mp.tmp.apYC71GFfsq/Programs: special device overlay does not exist.

as well as

Failed to parse RO/RW flag, setting RO.

But that will still result in a failed PostInstall:

Compile: Postprocessing PostInstall changes.
Compile: PostInstall tried to change:
Data
Data/Variable
Data/Variable/run
Data/Variable/run/mount
Compile: Restoring safe copy *
Compile: Tree 2.1.3 - Failure in post-install operation.

???

That is why I suspect that PostInstall is not actually supposed to run in the unionSandbox.

[lucasvr]

The last error you see is actually just stating that some files (var/mount) were modified when the UnionSandbox didn't expect them to be. There's an allow-list in its implementation that must account for changes to the mount table. Try changing the following line in Compile/bin/SandboxInstall:

rm -rf ./$goboVariable/{run,}/mount/utab
to:
rm -rf ./$goboVariable/{run,}/mount

That should fix this particular problem.

[Nuc1eoN]

The last error you see is actually just stating that some files (var/mount) were modified when the UnionSandbox didn't expect them to be. There's an allow-list in its implementation that must account for changes to the mount table. Try changing the following line in Compile/bin/SandboxInstall:

Thanks for making me aware of this!

Sadly it didn't help..

Just to be sure I have changed it to rm --verbose -rf ./$goboVariable/*

Here's the output with some debug info:

SandboxInstall: overlayfs is available. Using UnionSandbox.
SandboxInstall: Installing Tree...
mount: overlay mounted on /.union_mp.tmp.aszJurlF3c6/Programs.
mount: overlay mounted on /.union_mp.tmp.aszJurlF3c6/System.
mount: overlay mounted on /.union_mp.tmp.aszJurlF3c6/Data.
mount: overlay mounted on /.union_mp.tmp.aszJurlF3c6/Users.
mount: overlay mounted on /.union_mp.tmp.aszJurlF3c6/usr.
install -d /usr/local/bin
install -d /usr/local/man/man1
install tree /usr/local/bin/tree; \
install -m 644 doc/tree.1 /usr/local/man/man1/tree.1
UnionSandbox: Cleaning up.
UnionSandbox: Moving entries to: /Programs/Tree/6.6.6/.SandboxInstall_Root
sandbox_mp:  /.union_mp.tmp.aszJurlF3c6
# ls -l $sandbox_mp
total 0
SandboxInstall: Postprocessing Sandbox
goboVariable /Data/Variable
removed directory './/Data/Variable/run/mount'
removed directory './/Data/Variable/run'
removed directory './/Data/Variable/tmp'
Compile: Stripping executables...
Compile: Running PostInstall for Tree
mount: overlay mounted on /.union_mp.tmp.FWu4Lkqq6sZ/Programs.
mount: overlay mounted on /.union_mp.tmp.FWu4Lkqq6sZ/System.
mount: overlay mounted on /.union_mp.tmp.FWu4Lkqq6sZ/Data.
mount: overlay mounted on /.union_mp.tmp.FWu4Lkqq6sZ/Users.
mount: overlay mounted on /.union_mp.tmp.FWu4Lkqq6sZ/usr.
Hello World!
UnionSandbox: Cleaning up.
UnionSandbox: Moving entries to: /Programs/Tree/6.6.6/.PostInstall_Root
sandbox_mp:  /.union_mp.tmp.FWu4Lkqq6sZ
# ls -l $sandbox_mp
total 0
rmdir: failed to remove './run': No such file or directory
Compile: Postprocessing PostInstall changes.
Compile: PostInstall tried to change:
Data
Data/Variable
Data/Variable/run
Data/Variable/run/mount
Compile: Restoring safe copy *
Compile: Tree 6.6.6 - Failure in post-install operation.

The cleanup seems to run for the first sandbox:

removed directory './/Data/Variable/run/mount'
removed directory './/Data/Variable/run'
removed directory './/Data/Variable/tmp'

But there is apparently no cleanup happening for the PostInstall sandbox (I have verified this by placing an echo log between the rm commands) 🤔

However frankly upon checking ls -l $sandbox_mp, the sandbox is empty.
That's also why it says
rmdir: failed to remove './run': No such file or directory
in the end.

But Compile still classifies it as a failure in the end.

[Nuc1eoN]

@lucasvr Are you sure PostInstall should be run inside the union sandbox?

This is what the documentation states:

PostInstall

A bash script which is executed by Compile (or InstallPackage) after installation. This is for one-time actions which should not be associated with any stage of the compilation or installation process, but run after the Program is symlinked. They are kept separate from the Recipe file so that they are retained in binary packages which may be distributed.

https://wiki.gobolinux.org/Recipes/Recipe-Format-Specification/index.html#postinstall