Question on the m68k Calling Convention (RTD)

[John Paul Adrian Glaubitz]

Hello!

I'm currently trying to fix a bug [1] in the M68k backend in LLVM.

The problem is that the backend tries to emit RTD when the baseline
is raised to at least 68020 using the -m68020 command line flag and
then crashes.

Looking at the code [2], this happens because the backend runs into

llvm_unreachable("RTD is not implemented");

since RTD is indeed not implemented yet.

Now, the interesting part is the question why the compiler is even trying
to take this codepath since RTD should normally be turned off in the M68k
backend by default as it's the case for GCC.

Use a different function-calling convention, in which functions that take a
fixed number of arguments return with the rtd instruction, which pops their
arguments while returning. This saves one instruction in the caller since
there is no need to pop the arguments there.
This calling convention is incompatible with the one normally used on Unix,
so you cannot use it if you need to call libraries compiled with the Unix compiler.
Also, you must provide function prototypes for all functions that take variable
numbers of arguments (including printf); otherwise incorrect code is generated
for calls to those functions.
In addition, seriously incorrect code results if you call a function with too many
arguments. (Normally, extra arguments are harmlessly ignored.)
The rtd instruction is supported by the 68010, 68020, 68030, 68040, 68060 and CPU32
processors, but not by the 68000 or 5200.
The default is -mno-rtd.

LLVM/Clang actually suppors the -mno-rtd switch as well.

-mrtd, -mno-rtd
Make StdCall calling convention the default

Looking at the code in [2] again, it the RTD codepath is taken when the stack
adjustment is non-zero which raises the question what the proper stack adjustment
on m68k is when using GCC with -mno-rtd (the default).

Does the m68k calling convention actually require the callee to clean up the stack
similar to the stdcall calling convention on x86 or should the stack adjustment
be zero here unless -mrtd is set?

Thanks,
Adrian

[1] https://github.com/llvm/llvm-project/issues/60554
[2] https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/M68k/M68kExpandPseudo.cpp#L245
[3] https://gcc.gnu.org/onlinedocs/gcc/M680x0-Options.html
[4] https://clang.llvm.org/docs/ClangCommandLineReference.html

[Richard Z]

Post by John Paul Adrian Glaubitz
Now, the interesting part is the question why the compiler is even trying
to take this codepath since RTD should normally be turned off in the M68k
backend by default as it's the case for GCC.

As you say.. looks like that codepath should never be taken unless -mrtd is explicitly asked for.. so the problem seems to be somewhere in the argument/options parsing logic of the compiler.

Richard