I strongly second Dmitry's proposal that something like .NOTINTERMEDIATE: (preferably with a no-dependencies-means-all interpretation) should be added.  The IMO weird behavior of .SECONDARY where it effectively creates strange weak dependencies just bit me for the second time. This time I just said a short prayer and removed .SECONDARY:, but what I really want is to disable the eccentric handling of intermediate files entirely.

This might add more implementation complexity and one more feature to consider, but it gives a road to simpler behavior.

I may be wrong about headers not being intermediate after $(file); I thought I had tested this but I just realized I hadn't saved the last edit to my makefile buffer.  I have to run now I'll look into this again later.

Let me provide a verbose description of .NOTINTERMEDIATE here.


This piece of make code allows to get rid from include directive with generated dep files.

Motivation for this piece of make code is described as 1,2,3 and 4 in update 6.


.SECONDEXPANSION: %.o

%.o: %.c %.d $$(file <%.d)
    gcc $(CPPFLAGS) $(CFLAGS) -MD -MF $*.td -o $@ -c $<
    read obj src headers <$*.td; echo "$$headers" >$*.d
    touch -c $@

%.d: ;
%.h: ;

The only missing piece is that make considers .d and .h files intermediate.

In order for this piece of code to work we need to tell make that  files which match %.d and %.h are not intermediate.

.SECONDARY allows us to prevent make from deleting these files.

But, preventing removal is not enough.

.SECONDARY prevents deletion, but the file is still intermediate and thus, still gives make a green light to not rebuild a target when one of the intermediate prerequisites is missing.


When a .d file or .h is missing (not deleted by make, but for some other reason) we need to have the related rule run to generate a new .d file.


So, a mechanism is needed to accompany implicit rules to let the user mark chosen patterns as not intermediate (not secondary, but  full opposite of intermediate).

> But since the targets you are referring to are already intermediate this isn't an issue.

It is an issue. Because as long as .d and .h files are intermedaite make won't rebuild, if some .d or .h file is missing.

> As far as I can tell, that's the purpose of the .NOTINTERMEDIATE target you introduced: to prevent files from being removed so that $(file ...) can read them.

This is one of 2 purposes of .NOTINTERMEDIATE. The other purpose is to force a rebuild when .d or .h file is missing. That's why there are these 2 rules.
%.d: ;
%.h: ;


i am not describing why a rebuild is needed when .d or .h file is missing. You already described that well in your Auto-Dependency Generation article.

Hope this makes it clear.

> Yes, I'm saying we already have a way to mark things "not intermediate" (.SECONDARY)  The only difference between that and a brand new .NOTINTERMEDIATE you have proposed is that .SECONDARY doesn't handle patterns and .NOTINTERMEDIATE does.

Paul, this must be some sort of blindness on my side. I cannot see how .SECONDARY can be used to mark files as not intermediate.
As far as i can see, .SECONDARY does the opposite, it marks files as intermediate. With the only difference from .INTERMEDIATE is that the files are not getting removed.

That may indeed be useful to have .SECONDARY accept patterns.

Do you mean, that when .SECONDARY depends on a pattern the behavior is the same as that of .NOTINTERMEDIATE (as proposed here)?
That would be quite surprising for users, would not it?

My look at this is the following
1. make provides explicit rules.
2. make provides .INTERMEDIATE to accompany explicit rules to let the user mark a target of choice as intermediate, when it otherwise would be not intermediate.
3. make provides pattern rules with implicit search.
4. make provides .NOTINTERMEDIATE to accompany implicit rules to let the user mark a pattern of choice as not intermediate, when it otherwise would be intermediate.

i read that article several times and indeed found it interesting.
In fact, i was using the technique described in your article, until my use cases forced me to come up with the technique described here.

The modern technique of tracking dependencies uses include directive.  While
this technique is infinitely superior to the manual maintenance of deps there
is still room for improvement.


1. include is not a part of the dag.


src:=$(wildcard *.c)
dfiles:=$(src:.c=.d)
%.o: %.c %.d
    gcc -I. -MMD -o $@ -c $<
%.d: ;
include $(dfiles)

When the user runs

$ make hello.o

make includes all dep files in the current directory, even though only hello.d
is needed. This is not optimal.



2. One makefile for multiple programs.

include is especially problematic when a project has one test driver per
source file. E.g. a lib can have implementation files api.c, util.c, and
engine.c and test programs api.t.c, util.t.c and engine.t.c. Each test program
contains function main and is supposed to be compiled and linked with the
module that it tests.


%.t.tsk: %.o %.t.o
    $(CC) -o $@ $^

When the user runs

$ make api.t.tsk

Only api.d and api.t.d are required to be included. However, our makefile shown
above will include all dep files.



3. One makefile for multiple projects.

Another scenario is a makefile containing a set of implicit rules and shared
between multiple projects.

include takes filenames as parameters. This hinders reuse of the makefile. We
would rather not hardcode filenames of included files in a makefile, but let
implicit rule search figure it out.



4. Phony targets.

Another issue with unconditional include is phony targets, such as clean, gzip,
etc.

The usual workaround is to to figure out the specified target and have a set of
ifeq statements to avoid including dep files. This piece of ifeq code
is difficult, especially when there are a lot of phony targets and the user
can specify multiple targets on the command line. E.g.

$ make api.t.tsk install



Solution.

It is possible to achieve the same automatic tracking of dependencies with
include (or rather its substitue) being a part of the dag.


.SECONDEXPANSION: %.o

%.o: %.c %.d $$(file <%.d)
    gcc $(CPPFLAGS) $(CFLAGS) -MD -MF $*.td -o $@ -c $<
    read obj src headers <$*.td; echo "$$headers" >$*.d
    touch -c $@

%.d: ;
%.h: ;

1. -MD generates a regular dep file.
Note, gcc option -MP is not used. The contents of dep files is of the form

api.o: api.c api.h <other header files>...\
    <more header files>...


2. Postprocessing.

read obj src headers <$*.td; echo "$$headers" >$*.d

is used to extract header files from the generated .td file and store this list of
headers files to a .d file.

The contents of the generated .d file is a space separated list of headers
files, all on one line.


3. $(file) appends the contents of %.d to the list of prerequisites.


4. Second expansion ensures $$(file) is expanded only when this rule is used to build
the current target. This is the magic maker here.



.NOTINTERMEIDATE.

There is still one missing piece in this makefile. Make considers generated dep
files and all headers files to be intermediate.  We need a mechanism to tell
make that all files which match %.d and %.h are not to be treated as
intermediate.


So, we introduce special target .NOTINTERMEDIATE and our makefile becomes


.NOTINTERMEDIATE: %.d %.h
.SECONDEXPANSION: %.o

%.o: %.c %.d $$(file <%.d)
    gcc $(CPPFLAGS) $(CFLAGS) -MD -MF $*.td -o $@ -c $<
    read obj src headers <$*.td; echo "$$headers" >$*.d
    touch -c $@

%.d: ;
%.h: ;


This makefile solves all the above described issues of unconditional include.


An additional bonus is that $(file) is faster than include. When the file
exists, include parses the file and evals its contents, and when the file is
missing include searches a list of directories. $(file) does none of that.


Another use case for .NOTINTERMEDIATE.

gcc has option -MP to generate an explicit target for each header file.

There are compilers which do not have such an option.

.NOTINTERMEDIATE: %.h

can be used to mark header files not intermediate with those compilers.


$$*.d vs %.d.

With the fix from https://savannah.gnu.org/bugs/?60188 it is possible to make
dep files and header files explicit in certain cases with


%.o: %.c $$*.d $$(file <$$*.d)

However, this workaround loses when the object file is not located in the
current directory. With %.d the implicit search algo prepends directory name to
the stem.


Another advantage of .NOTINTERMEDIATE over $$* is ability to be used with
built-in rules, if needed.



Notes.

Postprocessing in this example uses bash code and handles the gcc format of dep
files. To handle the one-rule-per-line format, that other compilers use, read
can be run in a loop.

Typo.

sv60297_notintermediate.diff contains implementation of special target .NOTINTERMEDIATE.

sv60297_notintermediate_test.diff is a test.

The modern technique of tracking dependencies uses include directive. While this technique is infinitely superior to the manual maintenance of deps there is still room for improvement.

1. include is not a part of the dag.

src:=$(wildcard *.c)
dfiles:=$(src:.c=.d)
%.o: %.c %.d
gcc -I. -MMD -o $@ -c $<
%.d: ;
include $(dfiles)

When the user runs

$ make hello.o

make includes all dep files in the current directory, even though only hello.d is needed. This is not optimal.

2. One makefile for multiple programs.

include is especially problematic when a project has one test driver per source file. E.g. a lib can have implementation files api.c, util.c, and engine.c and test programs api.t.c, util.t.c and engine.t.c. Each test program contains function main and is supposed to be compiled and linked with the module that it tests.

%.t.tsk: %.o %.t.o
$(CC) -o $@ $^

When the user runs

$ make api.t.tsk

Only api.d and api.t.d are required to be included. However, our makefile shown above will include all dep files.

3. One makefile for multiple projects.

Another scenario is a makefile containing a set of implicit rules and shared between multiple projects.

include takes filenames as parameters. This hinders reuse of the makefile. We would rather not hardcode filenames of included files in a makefile, but let implicit rule search figure it out.

4. Phony targets.

Another issue with unconditional include is phony targets, such as clean, gzip, etc.

The usual workaround is to to figure out the specified target and have a set of ifeq statements to avoid including dep files. This piece of ifeq code is difficult, especially when there are a lot of phony targets and the user can specify multiple targets on the command line. E.g.

$ make api.t.tsk install

Solution.

It is possible to achieve the same automatic tracking of dependencies with include (or rather its substitue) being a part of the dag.

.SECONDEXPANSION: %.o

%.o: %.c %.d $$(file <%.d)
gcc $(CPPFLAGS) $(CFLAGS) -MD -MF $*.td -o $@ -c $<
read obj src headers <$*.td; echo "$$headers" >$*.d
touch -c $@

%.d: ;
%.h: ;

    -MD generates a regular dep file. Note, gcc option -MP is not used. The contents of dep files is of the form

api.o: api.c api.h <other header files>...\
    <more header files>...

    Postprocessing.

read obj src headers <$*.td; echo "$$headers" >$*.d

is used to extract header files from the generated .td file and store this list of headers files to a .d file.

The contents of the generated .d file is a space separated list of headers files, all on one line.

    $(file) appends the contents of %.d to the list of prerequisites.

    Second expansion ensures $$(file) is expanded only when this rule is used to build the current target. This is the magic maker here.

.NOTINTERMEIDATE.

There is still one missing piece in this makefile. Make considers generated dep files and all headers files to be intermediate. We need a mechanism to tell make that all files which match %.d and %.h are not to be treated as intermediate.

So, we introduce special target .NOTINTERMEDIATE and our makefile becomes

.NOTINTERMEDIATE: %.d %.h
.SECONDEXPANSION: %.o

%.o: %.c %.d $$(file <%.d)
gcc $(CPPFLAGS) $(CFLAGS) -MD -MF $*.td -o $@ -c $<
read obj src headers <$*.td; echo "$$headers" >$*.d
touch -c $@

%.d: ;
%.h: ;

This makefile solves all the above described issues of unconditional include.

An additional bonus is that $(file) is faster than include. When the file exists, include parses the file and evals its contents, and when the file is missing include searches a list of directories. $(file) does none of that.
Another use case for .NOTINTERMEDIATE.

gcc has option -MP to generate an explicit target for each header file.

There are compilers which do not have such an option.

.NOTINTERMEDIATE: %.h

can be used to mark header files not intermediate with those compilers.
$$*.d vs %.d.

With the fix from https://savannah.gnu.org/bugs/?60188 it is possible to make dep files and header files explicit in certain cases with

%.o: %.c $$*.d $$(file <$$*.d)

However, this workaround loses when the object file is not located in the current directory. With %.d the implicit search algo prepends directory name to the stem.

Another advantage of .NOTINTERMEDIATE over $$* is ability to be used with built-in rules, if needed.
Notes.

Postprocessing in this example uses bash code and handles the gcc format of dep files. To handle the one-rule-per-line format, that other compilers use, read can be run in a loop.