One problem of grab_file() is that it cannot distinguish the following
two cases:

 - It cannot read the file (the file does not exist, or read permission
   is not set)

 - It can read the file, but the file size is zero

This is because grab_file() calls mmap(), which requires the mapped
length is greater than 0. Hence, grab_file() fails for both cases.

If an empty header file were included for checksum calculation, the
following warning would be printed:

  WARNING: modpost: could not open ...: Invalid argument

An empty file is a valid source file, so it should not fail.

Use read_text_file() instead. It can read a zero-length file.
Then, parse_file() will succeed with doing nothing.

Going forward, the first case (it cannot read the file) is a fatal
error. If the source file from which an object was compiled is missing,
something went wrong.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

I do not know how reliably this function works, but it looks dangerous
to me.

    strchr(sources, '\n');

... continues searching until it finds '\n' or it reaches the '\0'
terminator. In other words, 'sources' should be a null-terminated
string.

However, grab_file() just mmaps a file, so 'sources' is not terminated
with null byte. If the file does not contain '\n' at all, strchr() will
go beyond the mmap'ed memory.

Use read_text_file(), which loads the file content into a malloc'ed
buffer, appending null byte.

Here we are interested only in the first line of *.mod files. Use
get_line() helper to get the first line.

This also makes missing *.mod file a fatal error.

Commit 4be40e22 ("kbuild: do not emit src version warning for
non-modules") ignored missing *.mod files.

I do not fully understand what that commit addressed, but commit
91341d4b ("kbuild: introduce new option to enhance section mismatch
analysis") introduced partial section checks by using modpost. built-in.o
was parsed by modpost. Even modules had a problem because *.mod files
were created after the modpost check.

Commit b7dca6dd ("kbuild: create *.mod with full directory path and
remove MODVERDIR") stopped doing that. Now that modpost is only invoked
after the directory descend, *.mod files should always exist at the
modpost stage.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

modpost uses grab_file() to open a file, but it is not suitable for
a text file because the mmap'ed file is not terminated by null byte.
Actually, I see some issues for the use of grab_file().

The new helper, read_text_file() loads the whole file content into a
malloc'ed buffer, and appends a null byte. Then, get_line() reads
each line.

To handle text files, I intend to replace as follows:

  grab_file()    -> read_text_file()
  get_new_line() -> get_line()
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The three calls of get_modinfo() ("license", "import_ns", "version")
always return NULL for vmlinux(.o) because the built-in module info is
prefixed with __MODULE_INFO_PREFIX.

It is harmless to call get_modinfo(), but there is no point to search
for what apparently does not exist.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

As far as I understood, this code gets rid of '$Revision$' or '$Revision:'
of CVS, RCS or whatever in MODULE_VERSION() tags.

Remove the primeval code.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

If modpost fails to load a symbol dump file, it cannot check unresolved
symbols, hence module dependency will not be added. Nor CRCs can be added.

Currently, external module builds check only $(objtree)/Module.symvers,
but it should check files specified by KBUILD_EXTRA_SYMBOLS as well.

Move the warning message from the top Makefile to scripts/Makefile.modpost
and print the warning if any dump file is missing.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

check_exports() does not print warnings about unresolved symbols if
vmlinux is missing because there would be too many.

This situation happens when you do 'make modules' from the clean
tree, or compile external modules against a kernel tree that has
not been completely built.

It is dangerous to not check unresolved symbols because you might be
building useless modules. At least it should be warned.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Currently, the second pass of modpost is always invoked when you run
'make' or 'make modules' even if none of modules is changed.

Use if_changed to invoke it only when it is necessary.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The full build runs modpost twice, first for vmlinux.o and second for
modules.

The first pass dumps all the vmlinux symbols into Module.symvers, but
the second pass parses vmlinux again instead of reusing the dump file,
presumably because it needs to avoid accumulating stale symbols.

Loading symbol info from a dump file is faster than parsing an ELF object.
Besides, modpost deals with various issues to parse vmlinux in the second
pass.

A solution is to make the first pass dumps symbols into a separate file,
vmlinux.symvers. The second pass reads it, and parses module .o files.
The merged symbol information is dumped into Module.symvers in the same
way as before.

This makes further modpost cleanups possible.

Also, it fixes the problem of 'make vmlinux', which previously overwrote
Module.symvers, throwing away module symbols.

I slightly touched scripts/link-vmlinux.sh so that vmlinux is re-linked
when you cross this commit. Otherwise, vmlinux.symvers would not be
generated.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Prepare to use -i for in-tree modpost as well.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The symbol dump *.symvers is the output of modpost. Print it in
the short log.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Previously, the -i option had two functions; load a symbol dump file,
and set the external_module flag.

I want to assign a dedicate option for each of them.

Going forward, the -i is used to load a symbol dump file, and the -e
to set the external_module flag.

With this, we will be able to use -i for loading in-kernel symbols.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The -i option is used to include Modules.symver as well as files from
$(KBUILD_EXTRA_SYMBOLS).

Make the struct and variable names more generic.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Now that there is no difference between -i and -e, they can be unified.

Make modpost accept the -i option multiple times, then remove -e.

I will reuse -e for a different purpose.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The meaning of sym->kernel is obscure; it is set for in-kernel symbols
loaded from Modules.symvers. This happens only when we are building
external modules, and it is used to determine whether to dump symbols
to $(KBUILD_EXTMOD)/Modules.symvers

It is clearer to remember whether the symbol or module came from a dump
file or ELF object.

This changes the KBUILD_EXTRA_SYMBOLS behavior. Previously, symbols
loaded from KBUILD_EXTRA_SYMBOLS are accumulated into the current
$(KBUILD_EXTMOD)/Modules.symvers

Going forward, they will be only used to check symbol references, but
not dumped into the current $(KBUILD_EXTMOD)/Modules.symvers. I believe
this makes more sense.

sym->vmlinux will have no user. Remove it too.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Currently, modpost reads extra symbol dump files in the reverse order.
If '-e foo -e bar' is given, modpost reads bar, foo, in this order.

This is probably not a big deal, but there is no good reason to reverse
the order. Read files in the given order.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The built-in only code is not required to have MODULE_IMPORT_NS() to
use symbols. So, the namespace is not checked for vmlinux(.o).

Do not pass the meaningless -N option to the first pass of modpost.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The '-T -' option reads the file list from stdin.

It is clearer to put it close to the piped command.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

$(filter -i,$(MAKEFLAGS)) works only in limited use-cases.

The representation of $(MAKEFLAGS) depends on various factors:
  - GNU Make version (version 3.8x or version 4.x)
  - The presence of other flags like -j

In my experiments, $(MAKEFLAGS) is expanded as follows:

  * GNU Make 3.8x:

    * without -j option:
      --no-print-directory -Rri

    * with -j option:
      --no-print-directory -Rr --jobserver-fds=3,4 -j -i

  * GNU Make 4.x:

    * without -j option:
      irR --no-print-directory

    * with -j option:
      irR -j --jobserver-fds=3,4 --no-print-directory

For GNU Make 4.x, the flags are grouped as 'irR', which does not work.

For the single thread build with GNU Make 3.8x, the flags are grouped
as '-Rri', which does not work either.

To make it work for all cases, do likewise as commit 6f0fa58e
("kbuild: simplify silent build (-s) detection").

BTW, since commit ff9b45c5 ("kbuild: modpost: read modules.order
instead of $(MODVERDIR)/*.mod"), you also need to pass -k option to
build final *.ko files. 'make -i -k' ignores compile errors in modules,
and build as many remaining *.ko as possible.

Please note this feature is kind of dangerous if other modules depend
on the broken module because the generated modules will lack the correct
module dependency or CRC. Honestly, I am not a big fan of it, but I am
keeping this feature.

Fixes: eed380f3 ("modpost: Optionally ignore secondary errors seen if a single module build fails")
Cc: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Make modules.order depend on $(obj-m), and use if_changed to build it.
This will avoid unneeded update of modules.order, which will be useful
to optimize the modpost stage.

Currently, the second pass of modpost is always invoked. By checking the
timestamp of modules.order, we can avoid the unneeded modpost.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Do not overwrite core-y or drivers-y. Remove libs-y1 and libs-y2.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

This will slightly change the link order; drivers-y from arch Makefile
will be linked after virt/built-in.a, but I guess this is not a big
deal.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

No arch Makefile specifies init-y.

Merge init-y into core-y. This does not change the link order.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Remove lib-target, builtin-target, modorder-target, and modtargets.

Instead, add targets-for-builtin and targets-for-modules.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Remove the unneeded variables, __subdir-y and __subdir-m.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

This hunk has two 'ifdef CONFIG_TRIM_UNUSED_KSYMS ... endif' blocks
with no other code interleaved. Merge them.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

This code does not work as stated in the comment.

$(CONFIG_MODVERSIONS) is always empty because it is expanded before
include/config/auto.conf is included. Hence, 'make modules' with
CONFIG_MODVERSION=y cannot record the version CRCs.

This has been broken since 2003, commit ("kbuild: Enable modules to be
build using the "make dir/" syntax"). [1]

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/commit/?id=15c6240cdc44bbeef3c4797ec860f9765ef4f1a7
Cc: linux-stable <stable@vger.kernel.org> # v2.5.71+
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

As debug information gets larger and larger, it helps significantly save
the size of vmlinux images to compress the information in the debug
information sections. Note: this debug info is typically split off from
the final compressed kernel image, which is why vmlinux is what's used
in conjunction with GDB. Minimizing the debug info size should have no
impact on boot times, or final compressed kernel image size.

All of the debug sections will have a `C` flag set.
$ readelf -S <object file>

$ bloaty vmlinux.gcc75.compressed.dwarf4 -- \
    vmlinux.gcc75.uncompressed.dwarf4

    FILE SIZE        VM SIZE
 --------------  --------------
  +0.0%     +18  [ = ]       0    [Unmapped]
 -73.3%  -114Ki  [ = ]       0    .debug_aranges
 -76.2% -2.01Mi  [ = ]       0    .debug_frame
 -73.6% -2.89Mi  [ = ]       0    .debug_str
 -80.7% -4.66Mi  [ = ]       0    .debug_abbrev
 -82.9% -4.88Mi  [ = ]       0    .debug_ranges
 -70.5% -9.04Mi  [ = ]       0    .debug_line
 -79.3% -10.9Mi  [ = ]       0    .debug_loc
 -39.5% -88.6Mi  [ = ]       0    .debug_info
 -18.2%  -123Mi  [ = ]       0    TOTAL

$ bloaty vmlinux.clang11.compressed.dwarf4 -- \
    vmlinux.clang11.uncompressed.dwarf4

    FILE SIZE        VM SIZE
 --------------  --------------
  +0.0%     +23  [ = ]       0    [Unmapped]
 -65.6%    -871  [ = ]       0    .debug_aranges
 -77.4% -1.84Mi  [ = ]       0    .debug_frame
 -82.9% -2.33Mi  [ = ]       0    .debug_abbrev
 -73.1% -2.43Mi  [ = ]       0    .debug_str
 -84.8% -3.07Mi  [ = ]       0    .debug_ranges
 -65.9% -8.62Mi  [ = ]       0    .debug_line
 -86.2% -40.0Mi  [ = ]       0    .debug_loc
 -42.0% -64.1Mi  [ = ]       0    .debug_info
 -22.1%  -122Mi  [ = ]       0    TOTAL

For x86_64 defconfig + LLVM=1 (before):
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:22.03
Maximum resident set size (kbytes): 43856

For x86_64 defconfig + LLVM=1 (after):
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:32.52
Maximum resident set size (kbytes): 1566776

Thanks to:
Nick Clifton helped us to provide the minimal binutils version.
Sedat Dilek found an increase in size of debug .deb package.

Cc: Nick Clifton <nickc@redhat.com>
Suggested-by: David Blaikie <blaikie@google.com>
Reviewed-by: Fangrui Song <maskray@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Use sym_get_data_by_offset() helper to get access to the .shstrtab
section data. No functional change is intended because
elf->sechdrs[elf->secindex_strings].sh_addr is 0 for both ET_REL
and ET_EXEC object types.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

This may not be a practical problem, but the second pass of ARCH=i386
modpost causes segmentation fault if the -s option is not passed.

    MODPOST 12 modules
  Segmentation fault (core dumped)
  make[2]: *** [scripts/Makefile.modpost:94: __modpost] Error 139
  make[1]: *** [Makefile:1339: modules] Error 2
  make[1]: *** Waiting for unfinished jobs....

The segmentation fault occurs when section_rel() is called for vmlinux,
which is untested in regular builds. The cause of the problem is
reloc_location() returning a wrong pointer for ET_EXEC object type.
In this case, you need to subtract sechdr->sh_addr, otherwise it would
get access beyond the mmap'ed memory.

Add sym_get_data_by_offset() helper to avoid code duplication.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

$(firstword ...) in scripts/Makefile.modpost was added by commit
3f3fd3c0 ("[PATCH] kbuild: allow multi-word $M in Makefile.modpost")
to build multiple external module directories.

It was a solution to resolve symbol dependencies when an external
module depends on another external module.

Commit 0d96fb20 ("kbuild: Add new Kbuild variable
KBUILD_EXTRA_SYMBOLS") introduced another solution by passing symbol
info via KBUILD_EXTRA_SYMBOLS, then broke the multi-word M= support.

  include $(if $(wildcard $(KBUILD_EXTMOD)/Kbuild), \
               $(KBUILD_EXTMOD)/Kbuild, $(KBUILD_EXTMOD)/Makefile)

... does not work if KBUILD_EXTMOD contains multiple words.

This feature has been broken for more than a decade. Remove the
bitrotten code, and stop parsing if M or KBUILD_EXTMOD contains
multiple words.

As Documentation/kbuild/modules.rst explains, if your module depends
on another one, there are two solutions:
  - add a common top-level Kbuild file
  - use KBUILD_EXTRA_SYMBOLS
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

built-in.a contains the built-in object paths from the current and sub
directories.

module.order collects the module paths from the current and sub
directories.

Make their build rules look more symmetrical.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

I think subdir-builtin is clearer.

While I was here, I made its build rule explicit.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Save $(addprefix ...) for subdir-obj-y.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Do not try to build any module-related artifacts when CONFIG_MODULES
is disabled.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

I do not see a good reason to add ifdef here.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

This is a left-over of commit 39808e45 ("kbuild: do not read
$(KBUILD_EXTMOD)/Module.symvers").

Kbuild no longer supports this way.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

I think all the warnings have been fixed by now. Make it a fatal error.

Check it before modpost because we need to stop building *.ko files.
Also, pass modules.order via a script parameter.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Some targets are internal-use only.

It is tedious to care about "what if __build_one_by_one is contained
in $(MAKECMDGOALS)?" etc.

Prefix internal targets with double underscores. Stop parsing Makefile
if they are directly run.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Make it clearer, and self-documenting.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>