Add compilation db for use by clang tools and cquery (fix issue #3355) #3662

bdbaddog · 2020-05-19T20:23:50Z

Closes #3355

Adds tool 'compilation_db', which adds Builder CompilationDatabase

env['COMPILATIONDB_USE_ABSPATH'] decides whether target and source file paths are relative or absolute.

Contributor Checklist:

I have created a new test or updated the unit tests to cover the new/changed functionality.
I have updated CHANGES.txt (and read the README.rst)
I have updated the appropriate documentation

…c and cxx tools respectively

…ord so couldn't be imported into compilation_db tool. Add initial bits of test and docs

SCons/Tool/compilation_db.xml

test/CompilationDatabase/basic.py

…for CompilationDatabase tests

…ress sider complaint

….py. Handle when not all of cc,cxx and as tools are configured

bdbaddog · 2020-05-22T20:55:26Z

@ivankravets - just pushed removal of debug output.

…ation database

…r to clear source and set default target as 'compile_commands.json'.

…relative paths. If user doesn't specify target CompilationDatabase('my_output.json') SCons will swap what is by default the source to be the target and clear the source list. If no target specified use compile_commands.json. If more than one item specified as source, source is set to empty list

bdbaddog · 2020-06-06T19:30:34Z

Ok. This is more or less ready to merge.

Notes on usage and a question.

If you call as env.CompilationDatabase('my_file.json') that argument will get assigned to the source list. I've set the emitter to use the source list (only if it has one element) as the target.
If you call as env.CompilationDatabase() the emitter will fill in a default target as 'compile_commands.json'
If you call as env.CompilationDatabase(['source1','source2']), the emitter will drop your list of sources.

I think all the above are reasonable, but my query is should any of the above magic transformations yield a warning from SCons?

@acmorrow @mwichmann thoughts?

bdbaddog · 2020-06-06T20:52:46Z

Almost forgot.. Need to add some more to the test.

…pilationDatabase

mwichmann · 2020-06-07T16:52:51Z

SCons/Tool/compilation_db.xml

+    <tool name="compilation_db">
+        <summary>
+            <para>
+                Sets up &b-CompilationDatabase; builder which generates a clang tooling compatible compilation database.


I'd reorder this:

Sets up the &b-CompilationDatabase; builder which generates a compilation database for clang tooling.

Since this will be separated from the Builder spec in the output documentation, use a link: &b-link-CompilationDatabase;

mwichmann · 2020-06-07T16:56:49Z

SCons/Tool/compilation_db.xml

+        <summary>
+            <para>
+                The &b-CompilationDatabase; builder writes a
+                <ulink url="https://clang.llvm.org/docs/JSONCompilationDatabase.html">clang formatted compilation


"clang formatted" doesn't seem to mean anything. If it wants to refer to the data format, say "JSON formatted". If it want to just make reference to clang, say something like "Compilation database according to the LLVM specification"

mwichmann · 2020-06-07T17:04:42Z

SCons/Tool/compilation_db.xml

+                the C, C++, assembly source/target pairs.
+            </para>
+            <para>
+                NOTE: You must load the compilation_db tool prior to specifying any part of your build or some source/target


Use markup: &t-compilation_db;

mwichmann · 2020-06-07T17:06:20Z

SCons/Tool/compilation_db.xml

+            <para>
+                This is a boolean flag to instruct &b-link-CompilationDatabase; to
+                write the <literal>file</literal> and <literal>target</literal> members
+                in the target file with absolute or relative path.


This is confusing: target and targets refer to different things. Suggest make the second one "compilation database".

mwichmann · 2020-06-07T17:13:22Z

SCons/Tool/compilation_db.xml

+                in the target file with absolute or relative path.
+            </para>
+            <para>
+                The default value is False.


Reword? boolean... with absolute or relative path. ... The default value is False. Text should indicate what False means (yes, you can deduce from name of var, but...). I'd propose (considering the previous comment also): "

This is a boolean flag to instruct the &b-link-CompilationDatabase; builder to record the <literal>file</literal> and <literal>target</literal> members in the compilation database using absolute paths. The default value is <constant>False</constant>, indicating relative paths will be used.

mwichmann · 2020-06-07T17:21:13Z

SCons/Tool/compilation_db.xml

+                The &b-CompilationDatabase; builder writes a
+                <ulink url="https://clang.llvm.org/docs/JSONCompilationDatabase.html">clang formatted compilation
+                    database
+                </ulink> which is consumed by a number of clang tools, editors, and other tools.


Might be useful to say for what... "which is consumed by" - to what end?

mwichmann · 2020-06-07T17:26:30Z

SCons/Tool/compilation_db.xml

+                </ulink> which is consumed by a number of clang tools, editors, and other tools.
+            </para>
+            <para>
+                If you don't specify any files, the builder will default to <filename>compile_commands.json</filename>.


This must be where you had the question about supplying args to a builder spec, which our tooling doesn't provide for. Still, this sounds awkward: "if you don't speciffy any files"? The Builder Objects section in the manpage talks about calling builders, but doesn't seem to correlate with this usage?

See the emitter..

mwichmann · 2020-06-07T17:30:42Z

doc/user/misc.xml

+
+            Currently SCons supports two variations. One is to output source and target files with paths relative to the
+            top
+            of the SCons build, or to output those files with their absolute path.


Usage mismatch: two variations. One is... or to.... I'd express this as:

&SCons; can output source and target files either with paths relative to the top of the build, or with absolute paths

The next para can be combined into this one as it's the same concept being described...

Or

The compilation database can be populated with source and target files either using paths relative to the top of the build, or using absolute paths

mwichmann · 2020-06-07T19:11:47Z

SCons/Tool/compilation_db.py

+        "directory": env.Dir("#").abspath,
+        "command": command,
+        "file": env["__COMPILATIONDB_USOURCE"][0],
+        "target": env['__COMPILATIONDB_UTARGET'][0]


'target' is not called out in the LLVM Spec, will that cause problems?

The contracts for each field in the command object are:

directory: The working directory of the compilation. All paths specified in the command or file fields must be either absolute or relative to this directory.

file: The main translation unit source processed by this compilation step. This is used by tools as the key into the compilation database. There can be multiple command objects for the same file, for example if the same source file is compiled with different configurations.

command: The compile command executed. After JSON unescaping, this must be a valid command to rerun the exact compilation step for the translation unit in the environment the build system uses. Parameters use shell quoting and shell escaping of quotes, with ‘"’ and ‘\’ being the only special characters. Shell expansion is not supported.

arguments: The compile command executed as list of strings. Either arguments or command is required.

output: The name of the output created by this compilation step. This field is optional. It can be used to distinguish different processing modes of the same input file.

Hmm.. No idea. I know the current logic is in use and has been pulled in from MongoDB's source to several other projects without changing the field name?
@acmorrow @ivankravets thoughts?

bdbaddog · 2020-06-08T19:24:46Z

Merging.

dmoody256 · 2020-06-09T17:16:31Z

test/fixture/mylink.py

-                if not line.startswith(b'#link'):
+opts, args = getopt.getopt(sys.argv[1:], 'o:s:')
+for opt, arg in opts:
+    if opt == '-o':


I think this broke some windows tests using mylink.py which get fed MSVC like args. Originally the code handled MSVC like args with things like a[:5].lower() == '/out:': out = a[5:]

bdbaddog added 3 commits May 12, 2020 13:57

Initial import from mongodb tree. https://github.com/mongodb/mongo

686672a

refactor function names per PEP8. Pull C & C++ file extensions from c…

bf15138

…c and cxx tools respectively

Rename as.py to asm.py and add to TOOL_ALIASES. 'as' is reserved keyw…

02dbb38

…ord so couldn't be imported into compilation_db tool. Add initial bits of test and docs

bdbaddog requested a review from mwichmann May 19, 2020 20:23

bdbaddog self-assigned this May 19, 2020

mwichmann reviewed May 19, 2020

View reviewed changes

SCons/Tool/compilation_db.xml Outdated Show resolved Hide resolved

mwichmann reviewed May 19, 2020

View reviewed changes

test/CompilationDatabase/basic.py Outdated Show resolved Hide resolved

bdbaddog changed the title ~~Add compilation db for use by clang tools and cquery (fix issue #3355)~~ [WIP] Add compilation db for use by clang tools and cquery (fix issue #3355) May 19, 2020

bdbaddog added 7 commits May 19, 2020 14:53

Updates to doc. Still WIP

a1a47b7

extract fake gcc and link python scripts to fixture. We'll use these …

97bc344

…for CompilationDatabase tests

more test updates

9d1cae7

address sider issues

97650c6

fix typo

876f98d

Don't use from..import, rather import and copy generate,exists to add…

7d8ea41

…ress sider complaint

More doc updates. Hide internal env variables. Fix broken test/import…

6878db7

….py. Handle when not all of cc,cxx and as tools are configured

bdbaddog mentioned this pull request May 22, 2020

Can we export compile_commands.json? #3355

Closed

bdbaddog added 3 commits May 22, 2020 11:19

resolve sider complaints in test fixtures

45df0d7

[skip appveyor] [skip travis] resolve sider complaints in test fixtures

10b7f6b

Remove debug output

4b4eadb

LunarWatcher mentioned this pull request May 26, 2020

Add _real_ compilation DB support LunarWatcher/SConsStandard#4

Closed

bdbaddog added 7 commits May 26, 2020 18:54

add comment to indicate which sections are appendix

a56a563

add commented out XSLTFLAGS info which may enable creating PDF bookmarks

8c28b83

[ci skip] Add initial section for Users Guide misc section for compil…

897e738

…ation database

[ci skip] a little formatting change

ef7265e

change CompilationDatabase from psuedo builder to builder. Add emitte…

d53f612

…r to clear source and set default target as 'compile_commands.json'.

[ci skip] Updates to docs

f65a1d8

bdbaddog changed the title ~~[WIP] Add compilation db for use by clang tools and cquery (fix issue #3355)~~ Add compilation db for use by clang tools and cquery (fix issue #3355) Jun 6, 2020

bdbaddog added the clang clang support label Jun 6, 2020

bdbaddog added 2 commits June 6, 2020 16:25

Added tests which should cover all the variations of ways to call Com…

cddeec3

…pilationDatabase

Fix test SConstruct formatting

12739bb

mwichmann reviewed Jun 7, 2020

View reviewed changes

[ci skip] updates based on mwichmann's review of the PR

c282967

mwichmann reviewed Jun 7, 2020

View reviewed changes

bdbaddog merged commit ce32ce7 into SCons:master Jun 8, 2020

bdbaddog deleted the add_compilation_db branch June 8, 2020 19:24

dmoody256 reviewed Jun 9, 2020

View reviewed changes

Uh oh!

Add compilation db for use by clang tools and cquery (fix issue #3355) #3662

Add compilation db for use by clang tools and cquery (fix issue #3355) #3662

Uh oh!

Conversation

bdbaddog commented May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor Checklist:

Uh oh!

Uh oh!

Uh oh!

bdbaddog commented May 22, 2020

Uh oh!

bdbaddog commented Jun 6, 2020

Uh oh!

bdbaddog commented Jun 6, 2020

Uh oh!

mwichmann Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mwichmann Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mwichmann Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mwichmann Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdbaddog commented Jun 8, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bdbaddog commented May 19, 2020 •

edited

Loading

mwichmann Jun 7, 2020 •

edited

Loading

mwichmann Jun 7, 2020 •

edited

Loading

mwichmann Jun 7, 2020 •

edited

Loading

mwichmann Jun 7, 2020 •

edited

Loading