globus-net-manager FTBFS during python 3.10 transition

[Mattias Ellert]

Hi!

The globus-net-manager package failed to build from source on m64k and
sh4 during the python 3.10 transition. I tried giving the package back,
but the second attempt failed in the same way.

I did build the package successfully on the m68k porterbox
mitchy.debian.net, so I can not reproduce the failure there.

The build fails, for both m68k and sh4, because doxygen fails to
produce any output files. The package uses a perl script as a filter
during the doxygen run. The failures are consistent with what would
happen if the filter failed to produce any output.

Doxygen uses popen to run filters. Is there an issue with using popen
to run perl scripts on the buildds for m68k and sh4?

The reason I ask is that I found this report in BTS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=974004
which seems to be a similar issue.
Popen will fork and exec the perl script as:

/bin/sh -c "./perlscript.pl arg"

where ./perlscript.pl is a text file with a #!/usr/bin/perl shebang.

Or maybe the issue is something completely different.

Did something change between 2021-08-18 and 2022-03-28 that would cause
a package that succeeded to build on the first date, to fail a binNMU
(i.e. no changes to the source package) on the second date. The binNMU
was successful for all other architectures.

Mattias

[John Paul Adrian Glaubitz]

Hi!

Post by Mattias Ellert
Doxygen uses popen to run filters. Is there an issue with using popen
to run perl scripts on the buildds for m68k and sh4?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=974004
which seems to be a similar issue.
/bin/sh -c "./perlscript.pl arg"
where ./perlscript.pl is a text file with a #!/usr/bin/perl shebang.

It's most likely related to emulation issues with qmeu-user. Either because of
the related glibc bug [1] or because of the bug you mentioned #974004 [2].

I will rebuild the glibc package with the preliminary fix for [1] that has not
been merged upstream yet, unfortunately.

Adrian

Post by Mattias Ellert
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=974004

[Laurent Vivier]

Hi!

...

Post by Mattias Ellert
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=974004

Normally, since qemu v6.0.0 you can preserve argv[0]:

commit 6e1c0d7b951e19c53b8467e8bc4b71ee73a394ea
Author: Laurent Vivier <***@vivier.eu>
Date: Mon Feb 22 11:50:04 2021 +0100

linux-user: manage binfmt-misc preserve-arg[0] flag

Add --preserve-argv0 in qemu-binfmt-conf.sh to configure the preserve-argv0
flag.

This patch allows to use new flag in AT_FLAGS to detect if
preserve-argv0 is configured for this interpreter:
argv[0] (the full pathname provided by binfmt-misc) is removed and
replaced by argv[1] (the original argv[0] provided by binfmt-misc when
'P'/preserve-arg[0] is set)

For instance with this patch and kernel support for AT_FLAGS:

$ sudo chroot m68k-chroot sh -c 'echo $0'
sh

without this patch:

$ sudo chroot m68k-chroot sh -c 'echo $0'
/usr/bin/sh

The new flag is available in kernel (v5.12) since:
2347961b11d4 ("binfmt_misc: pass binfmt_misc flags to the interpreter")

This can be tested with something like:

# cp ..../qemu-ppc /chroot/powerpc/jessie

# qemu-binfmt-conf.sh --qemu-path / --systemd ppc --credential yes \
--persistent no --preserve-argv0 yes
# systemctl restart systemd-binfmt.service
# cat /proc/sys/fs/binfmt_misc/qemu-ppc
enabled
interpreter //qemu-ppc
flags: POC
offset 0
magic 7f454c4601020100000000000000000000020014
mask ffffffffffffff00fffffffffffffffffffeffff
# chroot /chroot/powerpc/jessie sh -c 'echo $0'
sh

# qemu-binfmt-conf.sh --qemu-path / --systemd ppc --credential yes \
--persistent no --preserve-argv0 no
# systemctl restart systemd-binfmt.service
# cat /proc/sys/fs/binfmt_misc/qemu-ppc
enabled
interpreter //qemu-ppc
flags: OC
offset 0
magic 7f454c4601020100000000000000000000020014
mask ffffffffffffff00fffffffffffffffffffeffff
# chroot /chroot/powerpc/jessie sh -c 'echo $0'
/bin/sh

Signed-off-by: Laurent Vivier <***@vivier.eu>
Message-Id: <20210222105004.1642234-1-***@vivier.eu>

You need also kernel v5.12:

commit 2347961b11d4079deace3c81dceed460c08a8fc1
Author: Laurent Vivier <***@vivier.eu>
Date: Tue Jan 28 14:25:39 2020 +0100

binfmt_misc: pass binfmt_misc flags to the interpreter

It can be useful to the interpreter to know which flags are in use.

For instance, knowing if the preserve-argv[0] is in use would
allow to skip the pathname argument.

This patch uses an unused auxiliary vector, AT_FLAGS, to add a
flag to inform interpreter if the preserve-argv[0] is enabled.

Note by Helge Deller:
The real-world user of this patch is qemu-user, which needs to know
if it has to preserve the argv[0]. See Debian bug #970460.

Signed-off-by: Laurent Vivier <***@vivier.eu>
Reviewed-by: YunQiang Su <***@wavecomp.com>
URL: http://bugs.debian.org/970460
Signed-off-by: Helge Deller <***@gmx.de>

Thanks,
Laurent

[John Paul Adrian Glaubitz]

Hi Laurent!

(...)

OK, I'm installing QEMU 6.2 and kernel 5.16 from backports on all QEMU
buildds now. Let's see if that helps.

Adrian

[John Paul Adrian Glaubitz]

Hi!

Post by John Paul Adrian Glaubitz

(...)

OK, I'm installing QEMU 6.2 and kernel 5.16 from backports on all QEMU
buildds now. Let's see if that helps.

Tested with QEMU 7.0 and glibc patches from [1], still fails.

Need to keep digging.

Adrian

Post by John Paul Adrian Glaubitz
[1] https://github.com/zatrazz/glibc/tree/azanella/redir-refactor

[Laurent Vivier]

Post by John Paul Adrian Glaubitz
Hi!

Post by John Paul Adrian Glaubitz

(...)

OK, I'm installing QEMU 6.2 and kernel 5.16 from backports on all QEMU
buildds now. Let's see if that helps.

Tested with QEMU 7.0 and glibc patches from [1], still fails.
Need to keep digging.

Did you configure binfmt_misc to preserve argv0 [2] ?

$ cat /proc/sys/fs/binfmt_misc/qemu-m68k
enabled
interpreter //qemu-m68k
flags: POC
offset 0
magic 7f454c4601020100000000000000000000020004
mask fffffffffffffe00fffffffffffffffffffeffff


Thanks,
Laurent

Post by John Paul Adrian Glaubitz
Adrian

Post by John Paul Adrian Glaubitz
[1] https://github.com/zatrazz/glibc/tree/azanella/redir-refactor

[2] ``P`` - preserve-argv[0]
Legacy behavior of binfmt_misc is to overwrite
the original argv[0] with the full path to the binary. When this
flag is included, binfmt_misc will add an argument to the argument
vector for this purpose, thus preserving the original ``argv[0]``.
e.g. If your interp is set to ``/bin/foo`` and you run ``blah``

[John Paul Adrian Glaubitz]

Hi Laurent!

Post by Laurent Vivier
Did you configure binfmt_misc to preserve argv0 [2] ?
$ cat /proc/sys/fs/binfmt_misc/qemu-m68k
enabled
interpreter //qemu-m68k
flags: POC
offset 0
magic 7f454c4601020100000000000000000000020004
mask fffffffffffffe00fffffffffffffffffffeffff

Looks like this for me:

***@nofan:~> cat /proc/sys/fs/binfmt_misc/qemu-m68k
enabled
interpreter /usr/libexec/qemu-binfmt/m68k-binfmt-P
flags: POCF
offset 0
magic 7f454c4601020100000000000000000000020004
mask ffffffffffffff00fffffffffffffffffffeffff
***@nofan:~>

Adrian

[Laurent Vivier]

Post by John Paul Adrian Glaubitz
Hi Laurent!
The issue can be reproduced on a freshly installed Debian system.
Is the problem the additional "F" flag?
enabled
interpreter /usr/libexec/qemu-binfmt/m68k-binfmt-P
flags: POCF
offset 0
magic 7f454c4601020100000000000000000000020004
mask ffffffffffffff00fffffffffffffffffffeffff
Adrian

No, the 'F' means 'fix-binary':

The interpreter is loaded in memory once when the binfmt_misc is configured (when the configuration
is written to /proc/sys/fs/binfmt_misc/register) from the host filesystem. So you don't need to put
it in the chroot.

Thanks,
Laurent

[John Paul Adrian Glaubitz]

Post by Laurent Vivier
The interpreter is loaded in memory once when the binfmt_misc is configured (when
the configuration is written to /proc/sys/fs/binfmt_misc/register) from the host
filesystem. So you don't need to put it in the chroot.

So, the flags are correct then? Is there any way to properly test the argument issue?

Adrian

[Laurent Vivier]

Post by John Paul Adrian Glaubitz

Post by Laurent Vivier
The interpreter is loaded in memory once when the binfmt_misc is configured (when
the configuration is written to /proc/sys/fs/binfmt_misc/register) from the host
filesystem. So you don't need to put it in the chroot.

So, the flags are correct then? Is there any way to properly test the argument issue?
Adrian

The flags are correct.

This can be tested with:

$ sudo chroot m68k-chroot sh -c 'echo $0'
sh

if the argument is not properly managed, you would have "/usr/bin/sh".

Thanks,
Laurent

[John Paul Adrian Glaubitz]

Hi!

Post by Laurent Vivier
The flags are correct.
$ sudo chroot m68k-chroot sh -c 'echo $0'
sh
if the argument is not properly managed, you would have "/usr/bin/sh".

OK, this is working properly:

***@z6:/srv/chroot> chroot unstable-m68k-sbuild/ sh -c 'echo $0'
sh
***@z6:/srv/chroot>

I guess it's related to the glibc issue then [1].

Adrian

Post by Laurent Vivier
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960

[Laurent Vivier]

Post by John Paul Adrian Glaubitz
Hi!

Hi :)

Post by John Paul Adrian Glaubitz

Post by Laurent Vivier
The flags are correct.
$ sudo chroot m68k-chroot sh -c 'echo $0'
sh
if the argument is not properly managed, you would have "/usr/bin/sh".

sh
I guess it's related to the glibc issue then [1].

You can try by mounting your chroot on a non ext4 partition. It works well with btrfs.

Ext4 stores a hash in a field that is normally an index, so a 64bit (host kernel long) hash cannot
be passed to the 32bit guest.

btrfs uses it as a real index, so it will overflow only with 2³² entries.

Thanks,
Laurent

Post by John Paul Adrian Glaubitz
Adrian

Post by Laurent Vivier
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960

[John Paul Adrian Glaubitz]

Hi Laurent!

Post by Laurent Vivier

Post by John Paul Adrian Glaubitz
I guess it's related to the glibc issue then [1].

You can try by mounting your chroot on a non ext4 partition. It works well with btrfs.
Ext4 stores a hash in a field that is normally an index, so a 64bit (host kernel long) hash cannot be passed to the 32bit guest.
btrfs uses it as a real index, so it will overflow only with 2³² entries.

Yes, you are absolutely right. Using btrfs for the chroot fixes the problem.

Adrian

[Mattias Ellert]

Hi!

Now the openssl3 transition is underway, and many more of the globus
packages fail in the same way.

My original idea that it was failing due to doxygen using an input
filter, seems to be incorrect. Only 4 of the globus packages uses an
input filter, an many more of them are failing than those ones.

I did not try all the failed builds on mitchy.debian.net (the m68k
porterbox), but I did try 3 of them and the builds succeeded there.

I have noticed that in addition to m68k and sh4, the builds also fail
in the same way on the hppa host known as pasta, but when given back
and run on a different hppa host they succeed.

E.g.
https://buildd.debian.org/status/logs.php?pkg=globus-common&arch=hppa
One failure on pasta - give back succeded on pacific

https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer&arch=hppa
Two failures on pasta - third attempt on physik succeeded

Is this related?

The logs look OK, the configure scripts find /usr/bin/doxygen, and
there in a call to /usr/bin/doxygen in the log. There are no errors
reported in the log from the doxygen call, but the files that the call
is supposed ro produce are not there later during the install phase.

rm -rf doc
/usr/bin/doxygen
touch doc-stamp

Mattias

Post by Laurent Vivier

Post by John Paul Adrian Glaubitz

Post by Laurent Vivier
The interpreter is loaded in memory once when the binfmt_misc is configured (when
the configuration is written to
/proc/sys/fs/binfmt_misc/register) from the host
filesystem. So you don't need to put it in the chroot.

So, the flags are correct then? Is there any way to properly test the argument issue?
Adrian

The flags are correct.
       $ sudo chroot m68k-chroot sh -c 'echo $0'
       sh
if the argument is not properly managed, you would have
"/usr/bin/sh".
Thanks,
Laurent

[John Paul Adrian Glaubitz]

Hi Matthias!

Post by Mattias Ellert
I have noticed that in addition to m68k and sh4, the builds also fail
in the same way on the hppa host known as pasta, but when given back
and run on a different hppa host they succeed.
E.g.
https://buildd.debian.org/status/logs.php?pkg=globus-common&arch=hppa
One failure on pasta - give back succeded on pacific
https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer&arch=hppa
Two failures on pasta - third attempt on physik succeeded
Is this related?

pasta is using a similar setup like m68k and sh4.

The underlying bug is an issue with glibc and ext4 [1] and can be resolved
by switching the buildroot to btrfs. I have verified that this fixes the
issue.

I will do that later this week when I find the time.

Adrian

Post by Mattias Ellert
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960

[Rob Landley]

Post by John Paul Adrian Glaubitz
Hi Matthias!

Post by Mattias Ellert
I have noticed that in addition to m68k and sh4, the builds also fail
in the same way on the hppa host known as pasta, but when given back
and run on a different hppa host they succeed.
E.g.
https://buildd.debian.org/status/logs.php?pkg=globus-common&arch=hppa
One failure on pasta - give back succeded on pacific
https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer&arch=hppa
Two failures on pasta - third attempt on physik succeeded
Is this related?

pasta is using a similar setup like m68k and sh4.
The underlying bug is an issue with glibc and ext4 [1] and can be resolved
by switching the buildroot to btrfs. I have verified that this fixes the
issue.

I've generally found qemu's application emulation to be MUCH more brittle than
its system emulation, so I usually create little system emulation versions to
native compile under instead. (Currently
https://landley.net/toybox/downloads/binaries/mkroot/latest/ ala
https://landley.net/toybox/faq.html#mkroot previously
https://landley.net/aboriginal/downloads/binaries/ ala
https://landley.net/aboriginal/about.html .)

QEMU's sh4 system emulation only has 64 megs ram, but you can add as much swap
space as you like and the compiler's been happy with it so far. (Even when it
gets a little thrashy, it stays in the host's page cache so it's all
memory-to-memory copying. I have a todo item to map some more memory into the
virtual board and kernel, but both sides involve a lot of digging.) The m68k
board I'm using has 256 megs ram, that's generally enough for a -j1 build. (Or
even -j3 with distcc calling out to the cross compiler running on the host's
loopback instead of running in the emulator.) I haven't tried to get an hppa
system working yet because musl doesn't support it.

That said, I've never figured out how to do a fully from-source debootstrap.
(I.E. build a Debian root filesystem from source under a Linux From Scratch
system and use that to populate a binary package repository.) Everything always
assumes "fetch binary packages from an existing repository to populate a
chroot", which means the architecture and libc combination has to already exist.

I'd really like to try doing a musl version of debian, but every documented
process I can find starts with "have this script wget a bunch of glibc binaries
off a server". Is there documentation on actually doing it entirely from source
somewhere?

Rob

[Geert Uytterhoeven]

Hi Rob,

Post by Rob Landley
QEMU's sh4 system emulation only has 64 megs ram, but you can add as much swap
space as you like and the compiler's been happy with it so far. (Even when it
gets a little thrashy, it stays in the host's page cache so it's all
memory-to-memory copying. I have a todo item to map some more memory into the
virtual board and kernel, but both sides involve a lot of digging.) The m68k
board I'm using has 256 megs ram, that's generally enough for a -j1 build. (Or
even -j3 with distcc calling out to the cross compiler running on the host's
loopback instead of running in the emulator.) I haven't tried to get an hppa
system working yet because musl doesn't support it.

With Linux v5.19, you will be able to use the new virtual m68k machine
with 3.2 GiB of RAM.

https://lore.kernel.org/all/20220406201523.243733-1-***@vivier.eu

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ***@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

[Mattias Ellert]

Post by John Paul Adrian Glaubitz
Hi Matthias!

Post by Mattias Ellert
I have noticed that in addition to m68k and sh4, the builds also fail
in the same way on the hppa host known as pasta, but when given back
and run on a different hppa host they succeed.
E.g.
https://buildd.debian.org/status/logs.php?pkg=globus-common&arch=hppa
One failure on pasta - give back succeded on pacific
https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer&arch=hppa
Two failures on pasta - third attempt on physik succeeded
Is this related?

pasta is using a similar setup like m68k and sh4.
The underlying bug is an issue with glibc and ext4 [1] and can be resolved
by switching the buildroot to btrfs. I have verified that this fixes the
issue.
I will do that later this week when I find the time.
Adrian

Post by Mattias Ellert
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960

Was this implemented?
Mainly I'm asking if there is any point in clicking the give-back link
on the builds that failed with the old set-up.

Mattias

[John Paul Adrian Glaubitz]

Hi Matthias!

Post by Mattias Ellert

Post by John Paul Adrian Glaubitz
Hi Matthias!

Post by Mattias Ellert
I have noticed that in addition to m68k and sh4, the builds also fail
in the same way on the hppa host known as pasta, but when given back
and run on a different hppa host they succeed.
E.g.
https://buildd.debian.org/status/logs.php?pkg=globus-common&arch=hppa
One failure on pasta - give back succeded on pacific
https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer&arch=hppa
Two failures on pasta - third attempt on physik succeeded
Is this related?

pasta is using a similar setup like m68k and sh4.
The underlying bug is an issue with glibc and ext4 [1] and can be resolved
by switching the buildroot to btrfs. I have verified that this fixes the
issue.
I will do that later this week when I find the time.
Adrian

Post by Mattias Ellert
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960

Was this implemented?
Mainly I'm asking if there is any point in clicking the give-back link
on the builds that failed with the old set-up.

Sorry, not yet, I had a busy week.

I’ll try to get it done this week and will let you know.

Adrian

[Mattias Ellert]

Post by John Paul Adrian Glaubitz
Hi Matthias!

On May 23, 2022, at 9:43 AM, Mattias Ellert
mån 2022-05-16 klockan 10:28 +0200 skrev John Paul Adrian

Post by John Paul Adrian Glaubitz
Hi Matthias!

Post by Mattias Ellert
I have noticed that in addition to m68k and sh4, the builds
also
fail
in the same way on the hppa host known as pasta, but when given back
and run on a different hppa host they succeed.
E.g.
https://buildd.debian.org/status/logs.php?pkg=globus-common&arch=hppa
One failure on pasta - give back succeded on pacific
https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer&arch=hppa
Two failures on pasta - third attempt on physik succeeded
Is this related?

pasta is using a similar setup like m68k and sh4.
The underlying bug is an issue with glibc and ext4 [1] and can be resolved
by switching the buildroot to btrfs. I have verified that this
fixes
the
issue.
I will do that later this week when I find the time.
Adrian

Post by Mattias Ellert
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960

Was this implemented?
Mainly I'm asking if there is any point in clicking the give-back link
on the builds that failed with the old set-up.

Sorry, not yet, I had a busy week.
I’ll try to get it done this week and will let you know.
Adrian

Hi.

Any news?

Mattias

[John Paul Adrian Glaubitz]

Post by Mattias Ellert

Post by John Paul Adrian Glaubitz

Post by Mattias Ellert
Was this implemented?
Mainly I'm asking if there is any point in clicking the give-back link
on the builds that failed with the old set-up.

Sorry, not yet, I had a busy week.
I’ll try to get it done this week and will let you know.

Any news?

Not yet, sorry. It takes quite some time to get the filesystems converted
and I haven't found the time for that yet. It's frustrating.

Adrian

[John Paul Adrian Glaubitz]

Hi Laurent!

The issue can be reproduced on a freshly installed Debian system.

Is the problem the additional "F" flag?

***@z6:/tmp/test> cat /proc/sys/fs/binfmt_misc/qemu-m68k
enabled
interpreter /usr/libexec/qemu-binfmt/m68k-binfmt-P
flags: POCF
offset 0
magic 7f454c4601020100000000000000000000020004
mask ffffffffffffff00fffffffffffffffffffeffff
***@z6:/tmp/test>

Adrian