diff --git a/doc/attributes.md b/doc/attributes.md new file mode 100644 index 0000000..df0fe59 --- /dev/null +++ b/doc/attributes.md @@ -0,0 +1,154 @@ +## 1. `cppast.net 0.12` Support for `attributes` +The original support of `cppast.net` for various types of `attributes`, including the `meta attribute` of `c++17`, is restricted due to the limitation of `libclang` itself `Api`. We need to rely on token-level parsing to implement related functions. In the implementation of `cppast.net 0.12` and previous versions, we used parsing `token` to implement related functions. Even some `attributes` that `libclang` supports well, such as `dllexport`, `dllimport`, etc., `cppast.net` most of the time also uses token parsing. Although this approach is flexible and we can always try to parse the related `attributes` from the `token` level, it also brings some problems and restrictions, including: +1. `ParseAttributes()` is extremely time-consuming, which led to the addition of the `ParseAttributes` parameter in later versions to control whether to parse `attributes`. However, in some cases, we need to rely on `attributes` to complete the related functions, which is obviously inconvenient. +2. There are defects in the parsing of `meta attribute` - `[[]]`. For `meta attribute` defined above `Function` and `Field`, it is obviously legal at the semantic level, but `cppast.net` does not support this type of `meta attribute` defined above the object very well (there are some exceptions here, like `namespace`, `class`, `enum` these `attribute` declarations, the attribute definition itself cannot be at the top, the compiler will report an error directly for the related usage, it can only be after the related keywords, such as `class [[deprecated]] Abc{};`). +3. Individual parameters of `meta attribute` use macros. Because our original implementation is based on `token` parsing, macros during compilation obviously cannot be correctly handled in this case. + +--- +## 2. A brief introduction to cases where `attribute` is needed + +--- +### 2.1 System-level `attribute` +Taking the code segment in the `cppast.net` test case as an example: +```cpp +#ifdef WIN32 +#define EXPORT_API __declspec(dllexport) +#else +#define EXPORT_API __attribute__((visibility(""default""))) +#endif +``` +For `attributes` like `dllexport` and `visibility` that control interface visibility, we definitely use them more often, not only when `ParseAttributes` is turned on to make it work. We need to provide a high-performance solution for these basic system attributes, and the implementation should not be affected by the switch. + +--- +### 2.2 Injection of Additional Information by Export Tools and Other Tools +  Let's take the following class definition as an example: +```cpp +#if !defined(__cppast) +#define __cppast(...) +#endif + +struct __cppast(msgid = 1) TestPBMessage { + public: + __cppast(id = 1) + float x; + __cppast(id = 2) + double y; + __cppast(id = 3) + uint64_t z; +}; +``` +To better support serialization and deserialization of `TestPBMessage`, and to have a certain degree of fault tolerance, we have added some additional information based on the original struct definition: +1. The msgid of `TestPBMessage`, here it is directly specified as integer `1`. +2. The `id` of x, y, and z, here directly using `1`, `2`, and `3` respectively. +This way, if we use `cppast.net` to create our offline processing tools, we definitely need to conveniently read out the various 'meta attributes' injected by `__cppast()` which do not directly impact the original code compilation in the tool, and use them appropriately. However, the performance of this part in `cppast.net 0.12` and previous versions is rather poor and has limitations. For example, it can't support cases like the one above where the `attribute` is directly defined on the `Field`. + +--- +## 3. New Implementation and Adjustment +  The new implementation is mainly based on the limitations of the current implementation mentioned earlier, and the various application scenarios mentioned in the previous chapter. We have re-categorized the `attribute` into three types: +1. `AttributeKind.CxxSystemAttribute` - It corresponds to various system `attributes` that `libclang` itself can parse very well, such as `visibility` mentioned above, as well as `[[deprecated]]`, `[[noreturn]]`, etc. With the help of `ClangSharp`, we can efficiently parse and handle them, so there is no need to worry about switch issues. +2. `AttributeKind.TokenAttribute` - As the name suggests, this corresponds to the `attribute` in the original version of `cppast.net`. It has been marked as `deprecated`, but the parsing of `token` is always a fallback implementation mechanism. We will keep the relevant `Tokenizer` code and use them cautiously to implement some complex features when `ClangSharp` is unable to implement related functions. +3. `AttributeKind.AnnotateAttribute` - This is used to replace the original `meta attribute` implemented based on `token` parsing, aiming to inject methods for classes and members with high performance and low restrictions as introduced earlier. + +Next, we will briefly introduce the implementation ideas and usage of various types of `attributes`. + +--- +### 3.1 `AttributeKind.CxxSystemAttribute` +  We added a function to handle various `attributes` that `ClangSharp` itself supports: +```cs + private List ParseSystemAndAnnotateAttributeInCursor(CXCursor cursor) + { + List collectAttributes = new List(); + cursor.VisitChildren((argCursor, parentCursor, clientData) => + { + var sourceSpan = new CppSourceSpan(GetSourceLocation(argCursor.SourceRange.Start), GetSourceLocation(argCursor.SourceRange.End)); + var meta = argCursor.Spelling.CString; + switch (argCursor.Kind) + { + case CXCursorKind.CXCursor_VisibilityAttr: + //... + break; + case CXCursorKind.CXCursor_AnnotateAttr: + //... + break; + case CXCursorKind.CXCursor_AlignedAttr: + //... + break; + //... + default: + break; + } + + return CXChildVisitResult.CXChildVisit_Continue; + + }, new CXClientData((IntPtr)0)); + return collectAttributes; + } +``` +With the existing features of `ClangSharp`, such as `visibility attribute`, can be efficiently handled here. Note that here the use of `AnnotateAttr` and `meta attribute` will be introduced. It is also the key to our high-performance `meta attribute` usage. We can directly access the relevant `cursor` on `libclang`'s `AST`, thus avoiding handling related data at the high performance-cost `token` level. + +--- +### 3.2 `AttributeKind.TokenAttribute` +  For the original `attribute` implemented based on `token` parsing, for compatibility with older versions, it has temporarily been moved from the original `Attributes` property to the `TokenAttributes` property. The new `CxxSystemAttribute` and `AnnotateAttribute` are stored in the original `Attributes` property. You can refer to the relevant test cases to understand their specific usage. + +--- +### 3.3 `AttributeKind.AnnotateAttribute` +  We need a mechanism to implement `meta attribute` that bypasses `token` parsing. Here we cleverly use the `annotate` + + attribute to achieve this. From the several new built-in macros, we can see how it works: +```cs + //Add a default macro here for CppAst.Net + Defines = new List() { + "__cppast_run__", //Help us for identify the CppAst.Net handler + @"__cppast_impl(...)=__attribute__((annotate(#__VA_ARGS__)))", //Help us for use annotate attribute convenience + @"__cppast(...)=__cppast_impl(__VA_ARGS__)", //Add a macro wrapper here, so the argument with macro can be handle right for compiler. + }; +``` +> [!note] +> These three system macros will not be parsed into `CppMacro` and added to the final parsing result to avoid polluting the output. + +In the end, we simply convert the variable argument `__VA_ARGS__` to a string and use `__attribute__((annotate(???)))` to inject information. Thus, if we, like the test code, add the following at the right place: +```cpp +#if !defined(__cppast) +#define __cppast(...) +#endif +``` +When the code is parsed by `cppast.net`, the relevant input will be correctly identified and read as an `annotate attribute`. In non-`cppast.net` scenarios, the data injected in `__cppast()` will be correctly ignored to avoid interfering with the actual compilation and execution of the code. In this way, we indirectly achieve the purpose of injecting and reading `meta attribute`. + +For the macro case: +```cpp +#if !defined(__cppast) +#define __cppast(...) +#endif + +#define UUID() 12345 + +__cppast(id=UUID(), desc=""a function with macro"") +void TestFunc() +{ +} +``` +Relevant test code: +```cs +//annotate attribute support on namespace +var func = compilation.Functions[0]; + Assert.AreEqual(1, func.Attributes.Count); + Assert.AreEqual(func.Attributes[0].Kind, AttributeKind.AnnotateAttribute); + Assert.AreEqual(func.Attributes[0].Arguments, "id=12345, desc=\"a function with macro\""); +``` +Because we did a `wrapper` packaging when defining `__cppast`, we find that macros also work well in `meta attribute` state. + +As for the case of `outline attribute`, like `Function`, `Field`, it can support well, and even you can define multiple `attributes` on an object, which is also legal: +```cpp +__cppast(id = 1) +__cppast(name = "x") +__cppast(desc = "???") +float x; +``` + +--- +## 4. Conclusion +  This article mainly introduces the new `attributes` supported by `cppast.net`, which are mainly divided into three categories: +1. CxxSystemAttribute +2. TokenAttribute +3. AnnotateAttribute +We recommend using `CxxSystemAttribute` and `AnnotateAttribute`, which do not require switch control. The existence of `TokenAttribute` is mainly for compatibility with old implementations. The related attributes have been moved into a separate `TokenAttributes` to distinguish from the first two. And `CppParserOptions` corresponding switch is adjusted to `ParseTokenAttributes`. Due to performance and usage limitations, it is not recommended to continue to use it. \ No newline at end of file diff --git a/src/CppAst.Tests/AttributesTest/TestAnnotateAttributes.cs b/src/CppAst.Tests/AttributesTest/TestAnnotateAttributes.cs new file mode 100644 index 0000000..c8f9a14 --- /dev/null +++ b/src/CppAst.Tests/AttributesTest/TestAnnotateAttributes.cs @@ -0,0 +1,142 @@ +// Copyright (c) Alexandre Mutel. All rights reserved. +// Licensed under the BSD-Clause 2 license. +// See license.txt file in the project root for full license information. + +using System; +using NUnit.Framework; + +namespace CppAst.Tests +{ + public class TestAnnotateAttributes : InlineTestBase + { + [Test] + public void TestAnnotateAttribute() + { + var text = @" + +#if !defined(__cppast) +#define __cppast(...) +#endif + +__cppast(script, is_browsable=true, desc=""a function"") +void TestFunc() +{ +} + +enum class __cppast(script, is_browsable=true, desc=""a enum"") TestEnum +{ +}; + +class __cppast(script, is_browsable=true, desc=""a class"") TestClass +{ + public: + __cppast(desc=""a member function"") + void TestMemberFunc(); + + __cppast(desc=""a member field"") + int X; +}; +"; + + ParseAssert(text, + compilation => + { + Assert.False(compilation.HasErrors); + + //annotate attribute support on global function + var cppFunc = compilation.Functions[0]; + Assert.AreEqual(1, cppFunc.Attributes.Count); + Assert.AreEqual(cppFunc.Attributes[0].Kind, AttributeKind.AnnotateAttribute); + Assert.AreEqual(cppFunc.Attributes[0].Arguments, "script, is_browsable=true, desc=\"a function\""); + + //annotate attribute support on enum + var cppEnum = compilation.Enums[0]; + Assert.AreEqual(1, cppEnum.Attributes.Count); + Assert.AreEqual(cppEnum.Attributes[0].Kind, AttributeKind.AnnotateAttribute); + Assert.AreEqual(cppEnum.Attributes[0].Arguments, "script, is_browsable=true, desc=\"a enum\""); + + //annotate attribute support on class + var cppClass = compilation.Classes[0]; + Assert.AreEqual(1, cppClass.Attributes.Count); + Assert.AreEqual(cppClass.Attributes[0].Kind, AttributeKind.AnnotateAttribute); + Assert.AreEqual(cppClass.Attributes[0].Arguments, "script, is_browsable=true, desc=\"a class\""); + + Assert.AreEqual(1, cppClass.Functions.Count); + var memFunc = cppClass.Functions[0]; + Assert.AreEqual(1, memFunc.Attributes.Count); + Assert.AreEqual(memFunc.Attributes[0].Arguments, "desc=\"a member function\""); + + + Assert.AreEqual(1, cppClass.Fields.Count); + var memField = cppClass.Fields[0]; + Assert.AreEqual(1, memField.Attributes.Count); + Assert.AreEqual(memField.Attributes[0].Arguments, "desc=\"a member field\""); + } + ); + } + + + [Test] + public void TestAnnotateAttributeInNamespace() + { + var text = @" + +#if !defined(__cppast) +#define __cppast(...) +#endif + +namespace __cppast(script, is_browsable=true, desc=""a namespace test"") TestNs{ + +} + +"; + + ParseAssert(text, + compilation => + { + Assert.False(compilation.HasErrors); + + //annotate attribute support on namespace + var ns = compilation.Namespaces[0]; + Assert.AreEqual(1, ns.Attributes.Count); + Assert.AreEqual(ns.Attributes[0].Kind, AttributeKind.AnnotateAttribute); + Assert.AreEqual(ns.Attributes[0].Arguments, "script, is_browsable=true, desc=\"a namespace test\""); + + } + ); + } + + [Test] + public void TestAnnotateAttributeWithMacro() + { + var text = @" + +#if !defined(__cppast) +#define __cppast(...) +#endif + +#define UUID() 12345 + +__cppast(id=UUID(), desc=""a function with macro"") +void TestFunc() +{ +} + +"; + + ParseAssert(text, + compilation => + { + Assert.False(compilation.HasErrors); + + //annotate attribute support on namespace + var func = compilation.Functions[0]; + Assert.AreEqual(1, func.Attributes.Count); + Assert.AreEqual(func.Attributes[0].Kind, AttributeKind.AnnotateAttribute); + Assert.AreEqual(func.Attributes[0].Arguments, "id=12345, desc=\"a function with macro\""); + + } + ); + } + } +} diff --git a/src/CppAst.Tests/AttributesTest/TestSystemAttributes.cs b/src/CppAst.Tests/AttributesTest/TestSystemAttributes.cs new file mode 100644 index 0000000..20e645c --- /dev/null +++ b/src/CppAst.Tests/AttributesTest/TestSystemAttributes.cs @@ -0,0 +1,366 @@ +// Copyright (c) Alexandre Mutel. All rights reserved. +// Licensed under the BSD-Clause 2 license. +// See license.txt file in the project root for full license information. + +using System; +using NUnit.Framework; + +namespace CppAst.Tests +{ + public class TestSystemAttributes : InlineTestBase + { + [Test] + public void TestSimple() + { + ParseAssert(@" +__declspec(dllimport) int i; +__declspec(dllexport) void func0(); +extern ""C"" void __stdcall func1(int a, int b, int c); +void *fun2(int align) __attribute__((alloc_align(1))); +", + compilation => + { + + // Print diagnostic messages + foreach (var message in compilation.Diagnostics.Messages) + Console.WriteLine(message); + + // Print All enums + foreach (var cppEnum in compilation.Enums) + Console.WriteLine(cppEnum); + + // Print All functions + foreach (var cppFunction in compilation.Functions) + Console.WriteLine(cppFunction); + + // Print All classes, structs + foreach (var cppClass in compilation.Classes) + Console.WriteLine(cppClass); + + // Print All typedefs + foreach (var cppTypedef in compilation.Typedefs) + Console.WriteLine(cppTypedef); + + + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Fields.Count); + Assert.NotNull(compilation.Fields[0].Attributes); + Assert.AreEqual("dllimport", compilation.Fields[0].Attributes[0].Name); + + Assert.AreEqual(3, compilation.Functions.Count); + Assert.NotNull(compilation.Functions[0].Attributes); + Assert.AreEqual(1, compilation.Functions[0].Attributes.Count); + Assert.AreEqual("dllexport", compilation.Functions[0].Attributes[0].Name); + + Assert.AreEqual(CppCallingConvention.X86StdCall, compilation.Functions[1].CallingConvention); + + Assert.NotNull(compilation.Functions[2].Attributes); + Assert.AreEqual(1, compilation.Functions[2].Attributes.Count); + Assert.AreEqual("allocalign", compilation.Functions[2].Attributes[0].Name); + + }, + new CppParserOptions() { }.ConfigureForWindowsMsvc() // Force using X86 to get __stdcall calling convention + ); + } + + [Test] + public void TestStructAttributes() + { + ParseAssert(@" +struct __declspec(uuid(""1841e5c8-16b0-489b-bcc8-44cfb0d5deae"")) __declspec(novtable) Test{ + int a; + int b; +};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Classes.Count); + + Assert.NotNull(compilation.Classes[0].Attributes); + + Assert.AreEqual(2, compilation.Classes[0].Attributes.Count); + + { + var attr = compilation.Classes[0].Attributes[0]; + Assert.AreEqual("uuid", attr.Name); + } + + { + var attr = compilation.Classes[0].Attributes[1]; + Assert.AreEqual("msnovtable", attr.Name); + } + }, + new CppParserOptions() { }.ConfigureForWindowsMsvc()); + } + + [Test] + public void TestCpp11VarAlignas() + { + ParseAssert(@" +alignas(128) char cacheline[128];", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Fields.Count); + Assert.AreEqual(1, compilation.Fields[0].Attributes.Count); + { + var attr = compilation.Fields[0].Attributes[0]; + Assert.AreEqual("alignas", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + [Test] + public void TestCpp11StructAlignas() + { + ParseAssert(@" +struct alignas(8) S {};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Classes.Count); + Assert.AreEqual(1, compilation.Classes[0].Attributes.Count); + { + var attr = compilation.Classes[0].Attributes[0]; + Assert.AreEqual("alignas", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + [Test] + public void TestCpp11StructAlignasWithAttribute() + { + ParseAssert(@" +struct [[deprecated(""abc"")]] alignas(8) S {};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Classes.Count); + Assert.AreEqual(2, compilation.Classes[0].Attributes.Count); + { + var attr = compilation.Classes[0].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + + { + var attr = compilation.Classes[0].Attributes[1]; + Assert.AreEqual("alignas", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }} + ); + } + + [Test] + public void TestCpp11StructAttributes() + { + ParseAssert(@" +struct [[deprecated]] Test{ + int a; + int b; +}; + +struct [[deprecated(""old"")]] TestMessage{ + int a; + int b; +};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(2, compilation.Classes.Count); + Assert.AreEqual(1, compilation.Classes[0].Attributes.Count); + { + var attr = compilation.Classes[0].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + + Assert.AreEqual(1, compilation.Classes[1].Attributes.Count); + { + var attr = compilation.Classes[1].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + [Test] + public void TestCpp11VariablesAttributes() + { + ParseAssert(@" +struct Test{ + [[deprecated]] int a; + int b; +}; + +[[deprecated]] int x;", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Classes.Count); + Assert.AreEqual(2, compilation.Classes[0].Fields.Count); + Assert.AreEqual(1, compilation.Classes[0].Fields[0].Attributes.Count); + { + var attr = compilation.Classes[0].Fields[0].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + + Assert.AreEqual(1, compilation.Fields.Count); + Assert.AreEqual(1, compilation.Fields[0].Attributes.Count); + { + var attr = compilation.Fields[0].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + [Test] + public void TestCpp11FunctionsAttributes() + { + ParseAssert(@" +[[noreturn]] void x() {};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Functions.Count); + Assert.AreEqual(1, compilation.Functions[0].Attributes.Count); + { + var attr = compilation.Functions[0].Attributes[0]; + Assert.AreEqual("cxx11noreturn", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + [Test] + public void TestCpp11NamespaceAttributes() + { + ParseAssert(@" +namespace [[deprecated]] cppast {};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Namespaces.Count); + Assert.AreEqual(1, compilation.Namespaces[0].Attributes.Count); + { + var attr = compilation.Namespaces[0].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + [Test] + public void TestCpp11EnumAttributes() + { + ParseAssert(@" +enum [[deprecated]] E { };", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Enums.Count); + Assert.AreEqual(1, compilation.Enums[0].Attributes.Count); + { + var attr = compilation.Enums[0].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + [Test] + public void TestCpp11TemplateStructAttributes() + { + ParseAssert(@" +template struct X {}; +template<> struct [[deprecated]] X {};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(2, compilation.Classes.Count); + Assert.AreEqual(0, compilation.Classes[0].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[1].Attributes.Count); + { + var attr = compilation.Classes[1].Attributes[0]; + Assert.AreEqual("deprecated", attr.Name); + } + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" } } + ); + } + + + [Test] + public void TestCppNoParseOptionsAttributes() + { + ParseAssert(@" +[[noreturn]] void x() {};", compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(1, compilation.Functions.Count); + Assert.AreEqual(1, compilation.Functions[0].Attributes.Count); + }, + // we are using a C++14 attribute because it can be used everywhere + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }} + ); + } + + [Test] + public void TestClassPublicExportAttribute() + { + var text = @" +#ifdef WIN32 +#define EXPORT_API __declspec(dllexport) +#else +#define EXPORT_API __attribute__((visibility(""default""))) +#endif +class EXPORT_API TestClass +{ +}; +"; + ParseAssert(text, + compilation => + { + Assert.False(compilation.HasErrors); + + var cppClass = compilation.Classes[0]; + Assert.AreEqual(1, cppClass.Attributes.Count); + Assert.True(cppClass.IsPublicExport()); + + }, + new CppParserOptions() { } + ); + ParseAssert(text, + compilation => + { + Assert.False(compilation.HasErrors); + + var cppClass = compilation.Classes[0]; + Assert.AreEqual(1, cppClass.Attributes.Count); + Assert.True(cppClass.IsPublicExport()); + }, new CppParserOptions() { }.ConfigureForWindowsMsvc() + ); + } + + } +} diff --git a/src/CppAst.Tests/TestAttributes.cs b/src/CppAst.Tests/AttributesTest/TestTokenAttributes.cs similarity index 66% rename from src/CppAst.Tests/TestAttributes.cs rename to src/CppAst.Tests/AttributesTest/TestTokenAttributes.cs index 384da68..11ef50d 100644 --- a/src/CppAst.Tests/TestAttributes.cs +++ b/src/CppAst.Tests/AttributesTest/TestTokenAttributes.cs @@ -7,7 +7,7 @@ namespace CppAst.Tests { - public class TestAttributes : InlineTestBase + public class TestTokenAttributes : InlineTestBase { [Test] public void TestSimple() @@ -45,22 +45,22 @@ public void TestSimple() Assert.False(compilation.HasErrors); Assert.AreEqual(1, compilation.Fields.Count); - Assert.NotNull(compilation.Fields[0].Attributes); - Assert.AreEqual("dllimport", compilation.Fields[0].Attributes[0].ToString()); + Assert.NotNull(compilation.Fields[0].TokenAttributes); + Assert.AreEqual("dllimport", compilation.Fields[0].TokenAttributes[0].Name); Assert.AreEqual(3, compilation.Functions.Count); - Assert.NotNull(compilation.Functions[0].Attributes); - Assert.AreEqual(1, compilation.Functions[0].Attributes.Count); - Assert.AreEqual("dllexport", compilation.Functions[0].Attributes[0].ToString()); + Assert.NotNull(compilation.Functions[0].TokenAttributes); + Assert.AreEqual(1, compilation.Functions[0].TokenAttributes.Count); + Assert.AreEqual("dllexport", compilation.Functions[0].TokenAttributes[0].Name); Assert.AreEqual(CppCallingConvention.X86StdCall, compilation.Functions[1].CallingConvention); - Assert.NotNull(compilation.Functions[2].Attributes); - Assert.AreEqual(1, compilation.Functions[2].Attributes.Count); - Assert.AreEqual("alloc_align(1)", compilation.Functions[2].Attributes[0].ToString()); + Assert.NotNull(compilation.Functions[2].TokenAttributes); + Assert.AreEqual(1, compilation.Functions[2].TokenAttributes.Count); + Assert.AreEqual("alloc_align(1)", compilation.Functions[2].TokenAttributes[0].ToString()); }, - new CppParserOptions() { ParseAttributes = true }.ConfigureForWindowsMsvc() // Force using X86 to get __stdcall calling convention + new CppParserOptions() { ParseTokenAttributes = true }.ConfigureForWindowsMsvc() // Force using X86 to get __stdcall calling convention ); } @@ -77,23 +77,23 @@ struct __declspec(uuid(""1841e5c8-16b0-489b-bcc8-44cfb0d5deae"")) __declspec(nov Assert.AreEqual(1, compilation.Classes.Count); - Assert.NotNull(compilation.Classes[0].Attributes); + Assert.NotNull(compilation.Classes[0].TokenAttributes); - Assert.AreEqual(2, compilation.Classes[0].Attributes.Count); + Assert.AreEqual(2, compilation.Classes[0].TokenAttributes.Count); { - var attr = compilation.Classes[0].Attributes[0]; + var attr = compilation.Classes[0].TokenAttributes[0]; Assert.AreEqual("uuid", attr.Name); Assert.AreEqual("\"1841e5c8-16b0-489b-bcc8-44cfb0d5deae\"", attr.Arguments); } { - var attr = compilation.Classes[0].Attributes[1]; + var attr = compilation.Classes[0].TokenAttributes[1]; Assert.AreEqual("novtable", attr.Name); Assert.Null(attr.Arguments); } }, - new CppParserOptions() { ParseAttributes = true }.ConfigureForWindowsMsvc()); + new CppParserOptions() { ParseTokenAttributes = true }.ConfigureForWindowsMsvc()); } [Test] @@ -105,59 +105,14 @@ public void TestCpp11VarAlignas() Assert.False(compilation.HasErrors); Assert.AreEqual(1, compilation.Fields.Count); - Assert.AreEqual(1, compilation.Fields[0].Attributes.Count); + Assert.AreEqual(1, compilation.Fields[0].TokenAttributes.Count); { - var attr = compilation.Fields[0].Attributes[0]; + var attr = compilation.Fields[0].TokenAttributes[0]; Assert.AreEqual("alignas", attr.Name); } }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } - ); - } - - [Test] - public void TestCpp11StructAlignas() - { - ParseAssert(@" -struct alignas(8) S {};", compilation => - { - Assert.False(compilation.HasErrors); - - Assert.AreEqual(1, compilation.Classes.Count); - Assert.AreEqual(1, compilation.Classes[0].Attributes.Count); - { - var attr = compilation.Classes[0].Attributes[0]; - Assert.AreEqual("alignas", attr.Name); - } - }, - // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } - ); - } - - [Test] - public void TestCpp11StructAlignasWithAttribute() - { - ParseAssert(@" -struct [[deprecated]] alignas(8) S {};", compilation => - { - Assert.False(compilation.HasErrors); - - Assert.AreEqual(1, compilation.Classes.Count); - Assert.AreEqual(2, compilation.Classes[0].Attributes.Count); - { - var attr = compilation.Classes[0].Attributes[0]; - Assert.AreEqual("deprecated", attr.Name); - } - - { - var attr = compilation.Classes[0].Attributes[1]; - Assert.AreEqual("alignas", attr.Name); - } - }, - // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = true } ); } @@ -178,21 +133,21 @@ struct [[deprecated(""old"")]] TestMessage{ Assert.False(compilation.HasErrors); Assert.AreEqual(2, compilation.Classes.Count); - Assert.AreEqual(1, compilation.Classes[0].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[0].TokenAttributes.Count); { - var attr = compilation.Classes[0].Attributes[0]; + var attr = compilation.Classes[0].TokenAttributes[0]; Assert.AreEqual("deprecated", attr.Name); } - Assert.AreEqual(1, compilation.Classes[1].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[1].TokenAttributes.Count); { - var attr = compilation.Classes[1].Attributes[0]; + var attr = compilation.Classes[1].TokenAttributes[0]; Assert.AreEqual("deprecated", attr.Name); Assert.AreEqual("\"old\"", attr.Arguments); } }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = true } ); } @@ -211,21 +166,21 @@ struct Test{ Assert.AreEqual(1, compilation.Classes.Count); Assert.AreEqual(2, compilation.Classes[0].Fields.Count); - Assert.AreEqual(1, compilation.Classes[0].Fields[0].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[0].Fields[0].TokenAttributes.Count); { - var attr = compilation.Classes[0].Fields[0].Attributes[0]; + var attr = compilation.Classes[0].Fields[0].TokenAttributes[0]; Assert.AreEqual("deprecated", attr.Name); } Assert.AreEqual(1, compilation.Fields.Count); - Assert.AreEqual(1, compilation.Fields[0].Attributes.Count); + Assert.AreEqual(1, compilation.Fields[0].TokenAttributes.Count); { - var attr = compilation.Fields[0].Attributes[0]; + var attr = compilation.Fields[0].TokenAttributes[0]; Assert.AreEqual("deprecated", attr.Name); } }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = true } ); } @@ -238,14 +193,14 @@ public void TestCpp11FunctionsAttributes() Assert.False(compilation.HasErrors); Assert.AreEqual(1, compilation.Functions.Count); - Assert.AreEqual(1, compilation.Functions[0].Attributes.Count); + Assert.AreEqual(1, compilation.Functions[0].TokenAttributes.Count); { - var attr = compilation.Functions[0].Attributes[0]; + var attr = compilation.Functions[0].TokenAttributes[0]; Assert.AreEqual("noreturn", attr.Name); } }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = true } ); } @@ -258,14 +213,14 @@ namespace [[deprecated]] cppast {};", compilation => Assert.False(compilation.HasErrors); Assert.AreEqual(1, compilation.Namespaces.Count); - Assert.AreEqual(1, compilation.Namespaces[0].Attributes.Count); + Assert.AreEqual(1, compilation.Namespaces[0].TokenAttributes.Count); { - var attr = compilation.Namespaces[0].Attributes[0]; + var attr = compilation.Namespaces[0].TokenAttributes[0]; Assert.AreEqual("deprecated", attr.Name); } }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = true } ); } @@ -278,14 +233,14 @@ enum [[deprecated]] E { };", compilation => Assert.False(compilation.HasErrors); Assert.AreEqual(1, compilation.Enums.Count); - Assert.AreEqual(1, compilation.Enums[0].Attributes.Count); + Assert.AreEqual(1, compilation.Enums[0].TokenAttributes.Count); { - var attr = compilation.Enums[0].Attributes[0]; + var attr = compilation.Enums[0].TokenAttributes[0]; Assert.AreEqual("deprecated", attr.Name); } }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = true } ); } @@ -299,15 +254,15 @@ public void TestCpp11TemplateStructAttributes() Assert.False(compilation.HasErrors); Assert.AreEqual(2, compilation.Classes.Count); - Assert.AreEqual(0, compilation.Classes[0].Attributes.Count); - Assert.AreEqual(1, compilation.Classes[1].Attributes.Count); + Assert.AreEqual(0, compilation.Classes[0].TokenAttributes.Count); + Assert.AreEqual(1, compilation.Classes[1].TokenAttributes.Count); { - var attr = compilation.Classes[1].Attributes[0]; + var attr = compilation.Classes[1].TokenAttributes[0]; Assert.AreEqual("deprecated", attr.Name); } }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = true } ); } @@ -328,15 +283,15 @@ struct [[cppast(""old"")]] TestMessage{ Assert.False(compilation.HasErrors); Assert.AreEqual(2, compilation.Classes.Count); - Assert.AreEqual(1, compilation.Classes[0].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[0].TokenAttributes.Count); { - var attr = compilation.Classes[0].Attributes[0]; + var attr = compilation.Classes[0].TokenAttributes[0]; Assert.AreEqual("cppast", attr.Name); } - Assert.AreEqual(1, compilation.Classes[1].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[1].TokenAttributes.Count); { - var attr = compilation.Classes[1].Attributes[0]; + var attr = compilation.Classes[1].TokenAttributes[0]; Assert.AreEqual("cppast", attr.Name); Assert.AreEqual("\"old\"", attr.Arguments); } @@ -344,7 +299,7 @@ struct [[cppast(""old"")]] TestMessage{ // C++17 says if the compile encounters a attribute it doesn't understand // it will ignore that attribute and not throw an error, we still want to // parse this. - new CppParserOptions() { AdditionalArguments = { "-std=c++17" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++17" }, ParseTokenAttributes = true } ); } @@ -367,9 +322,9 @@ public void TestCommentParen() resultText = resultText?.Replace("\r\n", "\n"); Assert.AreEqual(expectedText, resultText); - Assert.AreEqual(0, compilation.Functions[0].Attributes.Count); + Assert.AreEqual(0, compilation.Functions[0].TokenAttributes.Count); }, - new CppParserOptions() { ParseAttributes = true }); + new CppParserOptions() { ParseTokenAttributes = true }); } [Test] @@ -391,9 +346,9 @@ public void TestCommentParenWithAttribute() resultText = resultText?.Replace("\r\n", "\n"); Assert.AreEqual(expectedText, resultText); - Assert.AreEqual(1, compilation.Functions[0].Attributes.Count); + Assert.AreEqual(1, compilation.Functions[0].TokenAttributes.Count); }, - new CppParserOptions() { ParseAttributes = true }); + new CppParserOptions() { ParseTokenAttributes = true }); } [Test] @@ -418,9 +373,9 @@ [[infinite loop]] resultText = resultText?.Replace("\r\n", "\n"); Assert.AreEqual(expectedText, resultText); - Assert.AreEqual(0, compilation.Functions[0].Attributes.Count); + Assert.AreEqual(0, compilation.Functions[0].TokenAttributes.Count); }, - new CppParserOptions() { ParseAttributes = true }); + new CppParserOptions() { ParseTokenAttributes = true }); } [Test] @@ -431,9 +386,9 @@ public void TestAttributeInvalidBracketEnd() int function1(int a, int b);", compilation => { Assert.False(compilation.HasErrors); - Assert.AreEqual(0, compilation.Functions[0].Attributes.Count); + Assert.AreEqual(0, compilation.Functions[0].TokenAttributes.Count); }, - new CppParserOptions() { ParseAttributes = true }); + new CppParserOptions() { ParseTokenAttributes = true }); } [Test] @@ -444,9 +399,9 @@ public void TestAttributeInvalidParenEnd() int function1(int a, int b);", compilation => { Assert.False(compilation.HasErrors); - Assert.AreEqual(0, compilation.Functions[0].Attributes.Count); + Assert.AreEqual(0, compilation.Functions[0].TokenAttributes.Count); }, - new CppParserOptions() { ParseAttributes = true }); + new CppParserOptions() { ParseTokenAttributes = true }); } [Test] @@ -465,16 +420,16 @@ struct Test{ Assert.AreEqual(3, compilation.Classes.Count); Assert.AreEqual(1, compilation.Classes[1].Fields.Count); - Assert.AreEqual(1, compilation.Classes[1].Fields[0].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[1].Fields[0].TokenAttributes.Count); { - var attr = compilation.Classes[1].Fields[0].Attributes[0]; + var attr = compilation.Classes[1].Fields[0].TokenAttributes[0]; Assert.AreEqual("cppast", attr.Name); } }, // C++17 says if the compile encounters a attribute it doesn't understand // it will ignore that attribute and not throw an error, we still want to // parse this. - new CppParserOptions() { AdditionalArguments = { "-std=c++17" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++17" }, ParseTokenAttributes = true } ); } @@ -490,16 +445,16 @@ struct Test{ Assert.AreEqual(1, compilation.Classes.Count); Assert.AreEqual(1, compilation.Classes[0].Functions.Count); - Assert.AreEqual(1, compilation.Classes[0].Functions[0].Attributes.Count); + Assert.AreEqual(1, compilation.Classes[0].Functions[0].TokenAttributes.Count); { - var attr = compilation.Classes[0].Functions[0].Attributes[0]; + var attr = compilation.Classes[0].Functions[0].TokenAttributes[0]; Assert.AreEqual("cppast", attr.Name); } }, // C++17 says if the compile encounters a attribute it doesn't understand // it will ignore that attribute and not throw an error, we still want to // parse this. - new CppParserOptions() { AdditionalArguments = { "-std=c++17" }, ParseAttributes = true } + new CppParserOptions() { AdditionalArguments = { "-std=c++17" }, ParseTokenAttributes = true } ); } @@ -512,48 +467,13 @@ public void TestCppNoParseOptionsAttributes() Assert.False(compilation.HasErrors); Assert.AreEqual(1, compilation.Functions.Count); - Assert.AreEqual(0, compilation.Functions[0].Attributes.Count); + Assert.AreEqual(0, compilation.Functions[0].TokenAttributes.Count); }, // we are using a C++14 attribute because it can be used everywhere - new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseAttributes = false } + new CppParserOptions() { AdditionalArguments = { "-std=c++14" }, ParseTokenAttributes = false } ); } - - [Test] - public void TestClassPublicExportAttribute() - { - var text = @" -#ifdef WIN32 -#define EXPORT_API __declspec(dllexport) -#else -#define EXPORT_API __attribute__((visibility(""default""))) -#endif -class EXPORT_API TestClass -{ -}; -"; - ParseAssert(text, - compilation => - { - Assert.False(compilation.HasErrors); - var cppClass = compilation.Classes[0]; - Assert.AreEqual(1, cppClass.Attributes.Count); - Assert.True(cppClass.IsPublicExport()); - - }, - new CppParserOptions() { ParseAttributes = true } - ); - ParseAssert(text, - compilation => - { - Assert.False(compilation.HasErrors); - var cppClass = compilation.Classes[0]; - Assert.AreEqual(1, cppClass.Attributes.Count); - Assert.True(cppClass.IsPublicExport()); - }, new CppParserOptions() { ParseAttributes = true }.ConfigureForWindowsMsvc() - ); - } } } diff --git a/src/CppAst.Tests/TestComments.cs b/src/CppAst.Tests/TestComments.cs index bee6e16..9cf2a26 100644 --- a/src/CppAst.Tests/TestComments.cs +++ b/src/CppAst.Tests/TestComments.cs @@ -136,7 +136,7 @@ public void TestParen() resultText = resultText?.Replace("\r\n", "\n"); Assert.AreEqual(expectedText, resultText); }, - new CppParserOptions() { ParseAttributes = true }); + new CppParserOptions() { ParseTokenAttributes = true }); } } } \ No newline at end of file diff --git a/src/CppAst.Tests/TestFunctions.cs b/src/CppAst.Tests/TestFunctions.cs index c4b01af..eed91b0 100644 --- a/src/CppAst.Tests/TestFunctions.cs +++ b/src/CppAst.Tests/TestFunctions.cs @@ -191,7 +191,7 @@ public void TestFunctionExport() Assert.True(cppFunction.IsPublicExport()); } }, - new CppParserOptions() { ParseAttributes = true } + new CppParserOptions() { } ); ParseAssert(text, @@ -211,7 +211,7 @@ public void TestFunctionExport() Assert.AreEqual(0, cppFunction.Attributes.Count); Assert.True(cppFunction.IsPublicExport()); } - }, new CppParserOptions() { ParseAttributes = true }.ConfigureForWindowsMsvc() + }, new CppParserOptions() { }.ConfigureForWindowsMsvc() ); } diff --git a/src/CppAst.Tests/TestTypes.cs b/src/CppAst.Tests/TestTypes.cs index 8d8c36a..69b2fe7 100644 --- a/src/CppAst.Tests/TestTypes.cs +++ b/src/CppAst.Tests/TestTypes.cs @@ -147,6 +147,60 @@ class Derived : public ::BaseTemplate<::Derived> ); } + + [Test] + public void TestTemplatePartialSpecialization() + { + ParseAssert(@" +template +struct foo {}; + +template +struct foo {}; + +foo foobar; +", + compilation => + { + Assert.False(compilation.HasErrors); + + Assert.AreEqual(3, compilation.Classes.Count); + Assert.AreEqual(1, compilation.Fields.Count); + + var baseTemplate = compilation.Classes[0]; + var fullSpecializedClass = compilation.Classes[1]; + var partialSpecializedTemplate = compilation.Classes[2]; + + var field = compilation.Fields[0]; + Assert.AreEqual(field.Name, "foobar"); + + Assert.AreEqual(baseTemplate.TemplateKind, CppAst.CppTemplateKind.TemplateClass); + Assert.AreEqual(fullSpecializedClass.TemplateKind, CppAst.CppTemplateKind.TemplateSpecializedClass); + Assert.AreEqual(partialSpecializedTemplate.TemplateKind, CppAst.CppTemplateKind.PartialTemplateClass); + + //Need be a specialized for partial template here + Assert.AreEqual(fullSpecializedClass.SpecializedTemplate, partialSpecializedTemplate); + + //Need be a full specialized class for this field + Assert.AreEqual(field.Type, fullSpecializedClass); + + Assert.AreEqual(partialSpecializedTemplate.TemplateSpecializedArguments.Count, 2); + //The first argument is integer now + Assert.AreEqual(partialSpecializedTemplate.TemplateSpecializedArguments[0].ArgString, "int"); + //The second argument is not a specialized argument, we do not specialized a `B` template parameter here(partial specialized template) + Assert.AreEqual(partialSpecializedTemplate.TemplateSpecializedArguments[1].IsSpecializedArgument, false); + + //The field use type is a full specialized type here~, so we can have two `int` template parmerater here + //It's a not template or partial template class, so we can instantiate it, see `foo foobar;` before. + Assert.AreEqual(fullSpecializedClass.TemplateSpecializedArguments.Count, 2); + //The first argument is integer now + Assert.AreEqual(fullSpecializedClass.TemplateSpecializedArguments[0].ArgString, "int"); + //The second argument is not a specialized argument + Assert.AreEqual(fullSpecializedClass.TemplateSpecializedArguments[1].ArgString, "int"); + } + ); + } + [Test] public void TestClassPrototype() { diff --git a/src/CppAst/CppAttribute.cs b/src/CppAst/CppAttribute.cs index 29a2092..f3a29ae 100644 --- a/src/CppAst/CppAttribute.cs +++ b/src/CppAst/CppAttribute.cs @@ -12,9 +12,10 @@ namespace CppAst /// public class CppAttribute : CppElement { - public CppAttribute(string name) + public CppAttribute(string name, AttributeKind kind) { Name = name ?? throw new ArgumentNullException(nameof(name)); + Kind = kind; } /// @@ -37,15 +38,14 @@ public CppAttribute(string name) /// public bool IsVariadic { get; set; } + public AttributeKind Kind { get; } + /// public override string ToString() { var builder = new StringBuilder(); - if (Scope != null) - { - builder.Append(Scope).Append("::"); - } + ////builder.Append("[["); builder.Append(Name); if (Arguments != null) @@ -58,6 +58,15 @@ public override string ToString() builder.Append("..."); } + ////builder.Append("]]"); + + ////if (Scope != null) + ////{ + //// builder.Append(" { scope:"); + //// builder.Append(Scope).Append("::"); + //// builder.Append("}"); + ////} + return builder.ToString(); } } diff --git a/src/CppAst/CppAttributeKind.cs b/src/CppAst/CppAttributeKind.cs new file mode 100644 index 0000000..a1459a4 --- /dev/null +++ b/src/CppAst/CppAttributeKind.cs @@ -0,0 +1,21 @@ +// Copyright (c) Alexandre Mutel. All rights reserved. +// Licensed under the BSD-Clause 2 license. +// See license.txt file in the project root for full license information. + +using System; +using System.Text; + +namespace CppAst +{ + /// + /// Attribute kind enum here + /// + public enum AttributeKind + { + CxxSystemAttribute, + ////CxxCustomAttribute, + AnnotateAttribute, + CommentAttribute, + TokenAttribute, //the attribute is parse from token, and the parser is slow. + } +} \ No newline at end of file diff --git a/src/CppAst/CppClass.cs b/src/CppAst/CppClass.cs index df3e569..cff4f41 100644 --- a/src/CppAst/CppClass.cs +++ b/src/CppAst/CppClass.cs @@ -28,7 +28,8 @@ public CppClass(string name) : base(CppTypeKind.StructOrClass) Classes = new CppContainerList(this); Typedefs = new CppContainerList(this); TemplateParameters = new List(); - Attributes = new CppContainerList(this); + Attributes = new List(); + TokenAttributes = new List(); } /// @@ -98,7 +99,10 @@ public override string FullName public CppVisibility Visibility { get; set; } /// - public CppContainerList Attributes { get; } + public List Attributes { get; } + + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + public List TokenAttributes { get; } /// /// Gets or sets a boolean indicating if this type is a definition. If false the type was only declared but is not defined. diff --git a/src/CppAst/CppEnum.cs b/src/CppAst/CppEnum.cs index 43f6273..c5715de 100644 --- a/src/CppAst/CppEnum.cs +++ b/src/CppAst/CppEnum.cs @@ -11,7 +11,7 @@ namespace CppAst /// /// A C++ standard or scoped enum. /// - public sealed class CppEnum : CppTypeDeclaration, ICppMemberWithVisibility + public sealed class CppEnum : CppTypeDeclaration, ICppMemberWithVisibility, ICppAttributeContainer { /// /// Creates a new instance of this enum. @@ -21,7 +21,8 @@ public CppEnum(string name) : base(CppTypeKind.Enum) { Name = name; Items = new CppContainerList(this); - Attributes = new CppContainerList(this); + Attributes = new List(); + TokenAttributes = new List(); } /// @@ -67,7 +68,9 @@ public override string FullName /// /// Gets the list of attached attributes. /// - public CppContainerList Attributes { get; } + public List Attributes { get; } + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + public List TokenAttributes { get; } private bool Equals(CppEnum other) { diff --git a/src/CppAst/CppEnumItem.cs b/src/CppAst/CppEnumItem.cs index 0d820b0..8095ea5 100644 --- a/src/CppAst/CppEnumItem.cs +++ b/src/CppAst/CppEnumItem.cs @@ -3,13 +3,14 @@ // See license.txt file in the project root for full license information. using System; +using System.Collections.Generic; namespace CppAst { /// /// An enum item of . /// - public sealed class CppEnumItem : CppDeclaration, ICppMember + public sealed class CppEnumItem : CppDeclaration, ICppMember, ICppAttributeContainer { /// /// Creates a new instance of this enum item. @@ -35,6 +36,11 @@ public CppEnumItem(string name, long value) /// public CppExpression ValueExpression { get; set; } + /// + public List Attributes { get; } = new List(); + + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + public List TokenAttributes { get; } = new List(); /// public override string ToString() diff --git a/src/CppAst/CppField.cs b/src/CppAst/CppField.cs index 47670bf..830c541 100644 --- a/src/CppAst/CppField.cs +++ b/src/CppAst/CppField.cs @@ -11,12 +11,14 @@ namespace CppAst /// /// A C++ field (of a struct/class) or global variable. /// - public sealed class CppField : CppDeclaration, ICppMemberWithVisibility + public sealed class CppField : CppDeclaration, ICppMemberWithVisibility , ICppAttributeContainer { public CppField(CppType type, string name) { Type = type ?? throw new ArgumentNullException(nameof(type)); Name = name; + Attributes = new List(); + TokenAttributes = new List(); } /// @@ -30,7 +32,10 @@ public CppField(CppType type, string name) /// /// Gets attached attributes. Might be null. /// - public List Attributes { get; set; } + public List Attributes { get; } + + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + public List TokenAttributes { get; } /// /// Gets the type of this field/variable. diff --git a/src/CppAst/CppFunction.cs b/src/CppAst/CppFunction.cs index a1ca7a3..ca94a83 100644 --- a/src/CppAst/CppFunction.cs +++ b/src/CppAst/CppFunction.cs @@ -11,7 +11,7 @@ namespace CppAst /// /// A C++ function/method declaration. /// - public sealed class CppFunction : CppDeclaration, ICppMemberWithVisibility, ICppTemplateOwner, ICppContainer + public sealed class CppFunction : CppDeclaration, ICppMemberWithVisibility, ICppTemplateOwner, ICppContainer, ICppAttributeContainer { /// /// Creates a new instance of a function/method with the specified name. @@ -22,7 +22,8 @@ public CppFunction(string name) Name = name; Parameters = new CppContainerList(this); TemplateParameters = new List(); - Attributes = new CppContainerList(this); + Attributes = new List(); + TokenAttributes = new List(); } /// @@ -36,7 +37,10 @@ public CppFunction(string name) /// /// Gets the attached attributes. /// - public CppContainerList Attributes { get; } + public List Attributes { get; } + + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + public List TokenAttributes { get; } /// /// Gets or sets the storage qualifier. diff --git a/src/CppAst/CppGlobalDeclarationContainer.cs b/src/CppAst/CppGlobalDeclarationContainer.cs index 29fdd10..6f5060d 100644 --- a/src/CppAst/CppGlobalDeclarationContainer.cs +++ b/src/CppAst/CppGlobalDeclarationContainer.cs @@ -30,7 +30,8 @@ public CppGlobalDeclarationContainer() Classes = new CppContainerList(this); Typedefs = new CppContainerList(this); Namespaces = new CppContainerList(this); - Attributes = new CppContainerList(this); + Attributes = new List(); + TokenAttributes = new List(); } /// @@ -60,7 +61,10 @@ public CppGlobalDeclarationContainer() public CppContainerList Namespaces { get; } /// - public CppContainerList Attributes { get; } + public List Attributes { get; } + + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + public List TokenAttributes { get; } /// public virtual IEnumerable Children() diff --git a/src/CppAst/CppModelBuilder.cs b/src/CppAst/CppModelBuilder.cs index f5aa55e..c562bbf 100644 --- a/src/CppAst/CppModelBuilder.cs +++ b/src/CppAst/CppModelBuilder.cs @@ -33,7 +33,9 @@ public CppModelBuilder() public bool ParseSystemIncludes { get; set; } - public bool ParseAttributeEnabled { get; set; } + public bool ParseTokenAttributeEnabled { get; set; } + + public bool ParseCommentAttributeEnabled { get; set; } public CppCompilation RootCompilation { get; } @@ -207,7 +209,7 @@ private CppContainerContext GetOrCreateDeclarationContainer(CXCursor cursor, voi { var argh = arg.AsType; var argType = GetCppType(argh.Declaration, argh, cursor, data); - cppClass.TemplateSpecializedArguments.Add(new CppTemplateArgument(tempParams[(int)i], argType)); + cppClass.TemplateSpecializedArguments.Add(new CppTemplateArgument(tempParams[(int)i], argType, argh.TypeClass != CX_TypeClass.CX_TypeClass_TemplateTypeParm)); } break; case CXTemplateArgumentKind.CXTemplateArgumentKind_Integral: @@ -265,7 +267,7 @@ private CppNamespace VisitNamespace(CXCursor cursor, void* data) { // Create the container if not already created var ns = GetOrCreateDeclarationContainer(cursor, data, out var context); - ns.Attributes.AddRange(ParseAttributes(cursor)); + ParseAttributes(cursor, ns, false); cursor.VisitChildren(VisitMember, new CXClientData((IntPtr)data)); return ns; } @@ -275,7 +277,7 @@ private CppClass VisitClassDecl(CXCursor cursor, void* data) var cppStruct = GetOrCreateDeclarationContainer(cursor, data, out var context); if (cursor.IsDefinition && !cppStruct.IsDefinition) { - cppStruct.Attributes.AddRange(ParseAttributes(cursor)); + ParseAttributes(cursor, cppStruct, false); cppStruct.IsDefinition = true; cppStruct.SizeOf = (int)cursor.Type.SizeOf; cppStruct.AlignOf = (int)cursor.Type.AlignOf; @@ -303,6 +305,7 @@ private CXChildVisitResult VisitMember(CXCursor cursor, CXCursor parent, void* d var containerContext = GetOrCreateDeclarationContainer(parent, data); var cppEnum = (CppEnum)containerContext.Container; var enumItem = new CppEnumItem(GetCursorSpelling(cursor), cursor.EnumConstantDeclValue); + ParseAttributes(cursor, enumItem, true); VisitInitValue(cursor, data, out var enumItemExpression, out var enumValue); enumItem.ValueExpression = enumItemExpression; @@ -347,8 +350,8 @@ private CXChildVisitResult VisitMember(CXCursor cursor, CXCursor parent, void* d StorageQualifier = GetStorageQualifier(cursor), IsAnonymous = true, Offset = offset, - Attributes = ParseAttributes(cursor) }; + ParseAttributes(cursor, cppField, true); containerContext.DeclarationContainer.Fields.Add(cppField); element = cppField; } @@ -412,50 +415,13 @@ private CXChildVisitResult VisitMember(CXCursor cursor, CXCursor parent, void* d break; case CXCursorKind.CXCursor_MacroExpansion: break; - - case CXCursorKind.CXCursor_VisibilityAttr: - { - var containerContext = GetOrCreateDeclarationContainer(parent, data).Container; - var cppClass = containerContext as CppClass; - if (cppClass != null) - { - CppAttribute attribute = new CppAttribute("visibility"); - AssignSourceSpan(cursor, attribute); - attribute.Arguments = string.Format("\"{0}\"", cursor.DisplayName.ToString()); - cppClass.Attributes.Add(attribute); - } - } - break; - case CXCursorKind.CXCursor_DLLImport: - { - var containerContext = GetOrCreateDeclarationContainer(parent, data).Container; - var cppClass = containerContext as CppClass; - if (cppClass != null) - { - CppAttribute attribute = new CppAttribute("dllimport"); - AssignSourceSpan(cursor, attribute); - cppClass.Attributes.Add(attribute); - } - } - break; - case CXCursorKind.CXCursor_DLLExport: - { - var containerContext = GetOrCreateDeclarationContainer(parent, data).Container; - var cppClass = containerContext as CppClass; - if (cppClass != null) - { - CppAttribute attribute = new CppAttribute("dllexport"); - AssignSourceSpan(cursor, attribute); - cppClass.Attributes.Add(attribute); - } - } - break; case CXCursorKind.CXCursor_InclusionDirective: // Don't emit warning for this directive break; case CXCursorKind.CXCursor_Destructor: case CXCursorKind.CXCursor_TemplateTypeParameter: + case CXCursorKind.CXCursor_AnnotateAttr: // Don't emit warning break; default: @@ -471,6 +437,13 @@ private CXChildVisitResult VisitMember(CXCursor cursor, CXCursor parent, void* d if (element is ICppDeclaration cppDeclaration) { cppDeclaration.Comment = GetComment(cursor); + + var attrContainer = cppDeclaration as ICppAttributeContainer; + //Only handle commnet attribute when we need + if(attrContainer != null && ParseCommentAttributeEnabled) + { + TryToParseAttributesFromComment(cppDeclaration.Comment, attrContainer); + } } return CXChildVisitResult.CXChildVisit_Continue; @@ -733,6 +706,12 @@ private CppMacro ParseMacro(CXCursor cursor) var tokens = tu.Tokenize(range); var name = GetCursorSpelling(cursor); + if(name.StartsWith("__cppast")) + { + //cppast system macros, just ignore here + return null; + } + var cppMacro = new CppMacro(name); uint previousLine = 0; @@ -870,7 +849,7 @@ private CppField VisitFieldOrVariable(CppContainerContext containerContext, CXCu Offset = cursor.OffsetOfField / 8, }; containerContext.DeclarationContainer.Fields.Add(cppField); - cppField.Attributes = ParseAttributes(cursor); + ParseAttributes(cursor, cppField, true); if (cursor.Kind == CXCursorKind.CXCursor_VarDecl) { @@ -894,7 +873,7 @@ private void AddAnonymousTypeWithField(CppContainerContext containerContext, CXC Offset = cursor.OffsetOfField / 8, }; containerContext.DeclarationContainer.Fields.Add(cppField); - cppField.Attributes = ParseAttributes(cursor); + ParseAttributes(cursor, cppField, true); } private void VisitInitValue(CXCursor cursor, void* data, out CppExpression expression, out CppValue value) @@ -994,7 +973,7 @@ private CppExpression VisitExpression(CXCursor cursor, void* data) visitChildren = true; break; case CXCursorKind.CXCursor_UnaryOperator: - var tokens = new Tokenizer(cursor); + var tokens = new CppTokenUtil.Tokenizer(cursor); expr = new CppUnaryExpression(CppExpressionKind.UnaryOperator) { Operator = tokens.Count > 0 ? tokens.GetString(0) : string.Empty @@ -1184,7 +1163,7 @@ private void AppendTokensToExpression(CXCursor cursor, CppExpression expression) { if (expression is CppRawExpression tokensExpr) { - var tokenizer = new Tokenizer(cursor); + var tokenizer = new CppTokenUtil.Tokenizer(cursor); for (int i = 0; i < tokenizer.Count; i++) { tokensExpr.Tokens.Add(tokenizer[i]); @@ -1201,7 +1180,7 @@ private CppEnum VisitEnumDecl(CXCursor cursor, void* data) var integralType = cursor.EnumDecl_IntegerType; cppEnum.IntegerType = GetCppType(integralType.Declaration, integralType, cursor, data); cppEnum.IsScoped = cursor.EnumDecl_IsScoped; - cppEnum.Attributes.AddRange(ParseAttributes(cursor)); + ParseAttributes(cursor, cppEnum, false); context.IsChildrenVisited = true; cursor.VisitChildren(VisitMember, new CXClientData((IntPtr)data)); } @@ -1322,7 +1301,7 @@ private CppFunction VisitFunctionDecl(CXCursor cursor, CXCursor parent, void* da var returnType = GetCppType(cursor.ResultType.Declaration, cursor.ResultType, cursor, data); cppFunction.ReturnType = returnType; - cppFunction.Attributes.AddRange(ParseFunctionAttributes(cursor, cppFunction.Name)); + ParseAttributes(cursor, cppFunction, true); cppFunction.CallingConvention = GetCallingConvention(cursor.Type); int i = 0; @@ -1429,294 +1408,157 @@ private static CppCallingConvention GetCallingConvention(CXType type) } } - private void SkipTemplates(TokenIterator iter) + private List ParseSystemAndAnnotateAttributeInCursor(CXCursor cursor) { - if (iter.CanPeek) + List collectAttributes = new List(); + cursor.VisitChildren((argCursor, parentCursor, clientData) => { - if (iter.Skip("template")) + var sourceSpan = new CppSourceSpan(GetSourceLocation(argCursor.SourceRange.Start), GetSourceLocation(argCursor.SourceRange.End)); + var meta = argCursor.Spelling.CString; + switch (argCursor.Kind) { - iter.Next(); // skip the first > - int parentCount = 1; - while (parentCount > 0 && iter.CanPeek) - { - var text = iter.PeekText(); - if (text == ">") + case CXCursorKind.CXCursor_VisibilityAttr: { - parentCount--; + CppAttribute attribute = new CppAttribute("visibility", AttributeKind.CxxSystemAttribute); + AssignSourceSpan(argCursor, attribute); + attribute.Arguments = string.Format("\"{0}\"", argCursor.DisplayName.ToString()); + collectAttributes.Add(attribute); } - iter.Next(); - } - } - } - } - - private List ParseAttributes(CXCursor cursor) - { - if (!ParseAttributeEnabled) return null; - - var tokenizer = new AttributeTokenizer(cursor); - var tokenIt = new TokenIterator(tokenizer); - - // if this is a template then we need to skip that ? - if (tokenIt.CanPeek && tokenIt.PeekText() == "template") - SkipTemplates(tokenIt); - - List attributes = null; - while (tokenIt.CanPeek) - { - if (ParseAttributes(tokenIt, ref attributes)) - { - continue; - } - - // If we have a keyword, try to skip it and process following elements - // for example attribute put right after a struct __declspec(uuid("...")) Test {...} - if (tokenIt.Peek().Kind == CppTokenKind.Keyword) - { - tokenIt.Next(); - continue; - } - break; - } - return attributes; - } - - private List ParseFunctionAttributes(CXCursor cursor, string functionName) - { - if (!ParseAttributeEnabled) return null; - - // TODO: This function is not 100% correct when parsing tokens up to the function name - // we assume to find the function name immediately followed by a `(` - // but some return type parameter could actually interfere with that - // Ideally we would need to parse more properly return type and skip parenthesis for example - var tokenizer = new AttributeTokenizer(cursor); - var tokenIt = new TokenIterator(tokenizer); - - // if this is a template then we need to skip that ? - if (tokenIt.CanPeek && tokenIt.PeekText() == "template") - SkipTemplates(tokenIt); + break; + case CXCursorKind.CXCursor_AnnotateAttr: + { + var attribute = new CppAttribute("annotate", AttributeKind.AnnotateAttribute) + { + Span = sourceSpan, + Scope = "", + Arguments = meta, + IsVariadic = false, + }; - // Parse leading attributes - List attributes = null; - while (tokenIt.CanPeek) - { - if (ParseAttributes(tokenIt, ref attributes)) - { - continue; - } - break; - } + collectAttributes.Add(attribute); + } + break; + case CXCursorKind.CXCursor_AlignedAttr: + { + var attrKindSpelling = argCursor.AttrKindSpelling.ToLower(); + var attribute = new CppAttribute("alignas", AttributeKind.CxxSystemAttribute) + { + Span = sourceSpan, + Scope = "", + Arguments = "", + IsVariadic = false, + }; - if (!tokenIt.CanPeek) - { - return attributes; - } + collectAttributes.Add(attribute); + } + break; + case CXCursorKind.CXCursor_FirstAttr: + { + var attrKind = argCursor.AttrKind; + var attrKindSpelling = argCursor.AttrKindSpelling.ToLower(); + + var attribute = new CppAttribute(attrKindSpelling, AttributeKind.CxxSystemAttribute) + { + Span = sourceSpan, + Scope = "", + Arguments = "", + IsVariadic = false, + }; - // Find function name (We only support simple function name declaration) - if (!tokenIt.Find(functionName, "(")) - { - return attributes; - } + collectAttributes.Add(attribute); + } + break; + case CXCursorKind.CXCursor_DLLImport: + case CXCursorKind.CXCursor_DLLExport: + { + var attrKind = argCursor.AttrKind; + var attrKindSpelling = argCursor.AttrKindSpelling.ToLower(); - Debug.Assert(tokenIt.PeekText() == functionName); - tokenIt.Next(); - Debug.Assert(tokenIt.PeekText() == "("); - tokenIt.Next(); + var attribute = new CppAttribute(attrKindSpelling, AttributeKind.CxxSystemAttribute) + { + Span = sourceSpan, + Scope = "", + Arguments = "", + IsVariadic = false, + }; - int parentCount = 1; - while (parentCount > 0 && tokenIt.CanPeek) - { - var text = tokenIt.PeekText(); - if (text == "(") - { - parentCount++; - } - else if (text == ")") - { - parentCount--; + collectAttributes.Add(attribute); + } + break; + + // Don't generate a warning for unsupported cursor + default: + break; } - tokenIt.Next(); - } - if (parentCount != 0) - { - return attributes; - } - - while (tokenIt.CanPeek) - { - if (ParseAttributes(tokenIt, ref attributes)) - { - continue; - } - // Skip the token if we can parse it. - tokenIt.Next(); - } + return CXChildVisitResult.CXChildVisit_Continue; - return attributes; + }, new CXClientData((IntPtr)0)); + return collectAttributes; } - private bool ParseAttributes(TokenIterator tokenIt, ref List attributes) + private void TryToParseAttributesFromComment(CppComment comment, ICppAttributeContainer attrContainer) { - // Parse C++ attributes - // [[]] - if (tokenIt.Skip("[", "[")) - { - while (ParseAttribute(tokenIt, out var attribute)) - { - if (attributes == null) - { - attributes = new List(); - } - attributes.Add(attribute); - - tokenIt.Skip(","); - } - - return tokenIt.Skip("]", "]"); - } - - // Parse GCC or clang attributes - // __attribute__(()) - if (tokenIt.Skip("__attribute__", "(", "(")) - { - while (ParseAttribute(tokenIt, out var attribute)) - { - if (attributes == null) - { - attributes = new List(); - } - attributes.Add(attribute); - - tokenIt.Skip(","); - } - - return tokenIt.Skip(")", ")"); - } + if (comment == null) return; - // Parse MSVC attributes - // __declspec() - if (tokenIt.Skip("__declspec", "(")) + if(comment is CppCommentText ctxt) { - while (ParseAttribute(tokenIt, out var attribute)) + var txt = ctxt.Text.Trim(); + if(txt.StartsWith("[[") && txt.EndsWith("]]")) { - if (attributes == null) + attrContainer.Attributes.Add(new CppAttribute("comment", AttributeKind.CommentAttribute) { - attributes = new List(); - } - attributes.Add(attribute); - - tokenIt.Skip(","); + Arguments = txt, + Scope = "", + IsVariadic = false, + }) ; } - return tokenIt.Skip(")"); - } - - // Parse C++11 alignas attribute - // alignas(expression) - if (tokenIt.PeekText() == "alignas") - { - while (ParseAttribute(tokenIt, out var attribute)) - { - if (attributes == null) - { - attributes = new List(); - } - attributes.Add(attribute); - - break; - } - - return tokenIt.Skip(")"); ; } - return false; - } - - private bool ParseDirectAttribute(CXCursor cursor, ref List attributes) - { - var tokenizer = new AttributeTokenizer(cursor); - var tokenIt = new TokenIterator(tokenizer); - if (ParseAttribute(tokenIt, out var attribute)) + if (comment.Children != null) { - if (attributes == null) + foreach (var child in comment.Children) { - attributes = new List(); + TryToParseAttributesFromComment(child, attrContainer); } - attributes.Add(attribute); - return true; } - - return false; } - private bool ParseAttribute(TokenIterator tokenIt, out CppAttribute attribute) + private void ParseAttributes(CXCursor cursor, ICppAttributeContainer attrContainer, bool needOnlineSeek = false) { - // (identifier ::)? identifier ('(' tokens ')' )? (...)? - attribute = null; - var token = tokenIt.Peek(); - if (token == null || !token.Kind.IsIdentifierOrKeyword()) - { - return false; - } - tokenIt.Next(out token); + //Try to handle annotate in cursor first + //Low spend handle here, just open always + attrContainer.Attributes.AddRange(ParseSystemAndAnnotateAttributeInCursor(cursor)); - var firstToken = token; + //Low performance tokens handle here + if (!ParseTokenAttributeEnabled) return; - // try (identifier ::)? - string scope = null; - if (tokenIt.Skip("::")) + var tokenAttributes = new List(); + //Parse attributes online + if (needOnlineSeek) { - scope = token.Text; - - token = tokenIt.Peek(); - if (token == null || !token.Kind.IsIdentifierOrKeyword()) + bool hasOnlineAttribute = CppTokenUtil.TryToSeekOnlineAttributes(cursor, out var onLineRange); + if (hasOnlineAttribute) { - return false; + CppTokenUtil.ParseAttributesInRange(cursor.TranslationUnit, onLineRange, ref tokenAttributes); } - tokenIt.Next(out token); } - // identifier - string tokenIdentifier = token.Text; - - string arguments = null; - - // ('(' tokens ')' )? - if (tokenIt.Skip("(")) + //Parse attributes contains in cursor + if(attrContainer is CppFunction) { - var builder = new StringBuilder(); - var previousTokenKind = CppTokenKind.Punctuation; - while (tokenIt.PeekText() != ")" && tokenIt.Next(out token)) - { - if (token.Kind.IsIdentifierOrKeyword() && previousTokenKind.IsIdentifierOrKeyword()) - { - builder.Append(" "); - } - previousTokenKind = token.Kind; - builder.Append(token.Text); - } - - if (!tokenIt.Skip(")")) - { - return false; - } - arguments = builder.ToString(); - } - - var isVariadic = tokenIt.Skip("..."); - - var previousToken = tokenIt.PreviousToken(); - - attribute = new CppAttribute(tokenIdentifier) + var func = attrContainer as CppFunction; + CppTokenUtil.ParseFunctionAttributes(cursor, func.Name, ref tokenAttributes); + } + else { - Span = new CppSourceSpan(firstToken.Span.Start, previousToken.Span.End), - Scope = scope, - Arguments = arguments, - IsVariadic = isVariadic, - }; - return true; + CppTokenUtil.ParseCursorAttributs(cursor, ref tokenAttributes); + } + + attrContainer.TokenAttributes.AddRange(tokenAttributes); } + private CppType VisitTypeAliasDecl(CXCursor cursor, void* data) { var fulltypeDefName = clang.getCursorUSR(cursor).CString; @@ -1809,11 +1651,11 @@ private CppType VisitElaboratedDecl(CXCursor cursor, CXType type, CXCursor paren return GetCppType(type.CanonicalType.Declaration, type.CanonicalType, parent, data); } - private static string GetCursorAsText(CXCursor cursor) => new Tokenizer(cursor).TokensToString(); + private static string GetCursorAsText(CXCursor cursor) => new CppTokenUtil.Tokenizer(cursor).TokensToString(); private string GetCursorAsTextBetweenOffset(CXCursor cursor, int startOffset, int endOffset) { - var tokenizer = new Tokenizer(cursor); + var tokenizer = new CppTokenUtil.Tokenizer(cursor); var builder = new StringBuilder(); var previousTokenKind = CppTokenKind.Punctuation; for (int i = 0; i < tokenizer.Count; i++) @@ -2080,575 +1922,6 @@ private List ParseTemplateSpecializedArguments(CXCursor cursor, CXType return templateCppTypes; } - /// - /// Internal class to iterate on tokens - /// - private class TokenIterator - { - private readonly Tokenizer _tokens; - private int _index; - - public TokenIterator(Tokenizer tokens) - { - _tokens = tokens; - } - - public bool Skip(string expectedText) - { - if (_index < _tokens.Count) - { - if (_tokens.GetString(_index) == expectedText) - { - _index++; - return true; - } - } - - return false; - } - - public CppToken PreviousToken() - { - if (_index > 0) - { - return _tokens[_index - 1]; - } - - return null; - } - - public bool Skip(params string[] expectedTokens) - { - var startIndex = _index; - foreach (var expectedToken in expectedTokens) - { - if (startIndex < _tokens.Count) - { - if (_tokens.GetString(startIndex) == expectedToken) - { - startIndex++; - continue; - } - } - return false; - } - _index = startIndex; - return true; - } - - public bool Find(params string[] expectedTokens) - { - var startIndex = _index; - restart: - while (startIndex < _tokens.Count) - { - var firstIndex = startIndex; - foreach (var expectedToken in expectedTokens) - { - if (startIndex < _tokens.Count) - { - if (_tokens.GetString(startIndex) == expectedToken) - { - startIndex++; - continue; - } - } - startIndex = firstIndex + 1; - goto restart; - } - _index = firstIndex; - return true; - } - return false; - } - - public bool Next(out CppToken token) - { - token = null; - if (_index < _tokens.Count) - { - token = _tokens[_index]; - _index++; - return true; - } - return false; - } - - public bool CanPeek => _index < _tokens.Count; - - public bool Next() - { - if (_index < _tokens.Count) - { - _index++; - return true; - } - return false; - } - - public CppToken Peek() - { - if (_index < _tokens.Count) - { - return _tokens[_index]; - } - return null; - } - - public string PeekText() - { - if (_index < _tokens.Count) - { - return _tokens.GetString(_index); - } - return null; - } - } - - /// - /// Internal class to tokenize - /// - [DebuggerTypeProxy(typeof(TokenizerDebuggerType))] - private class Tokenizer - { - private readonly CXToken[] _tokens; - private CppToken[] _cppTokens; - protected readonly CXTranslationUnit _tu; - - public Tokenizer(CXCursor cursor) - { - _tu = cursor.TranslationUnit; - var range = GetRange(cursor); - _tokens = _tu.Tokenize(range).ToArray(); - } - - public Tokenizer(CXTranslationUnit tu, CXSourceRange range) - { - _tu = tu; - _tokens = _tu.Tokenize(range).ToArray(); - } - - public virtual CXSourceRange GetRange(CXCursor cursor) - { - return cursor.Extent; - } - - public int Count => _tokens?.Length ?? 0; - - public CppToken this[int i] - { - get - { - // Only create a tokenizer if necessary - if (_cppTokens == null) - { - _cppTokens = new CppToken[_tokens.Length]; - } - - ref var cppToken = ref _cppTokens[i]; - if (cppToken != null) - { - return cppToken; - } - - var token = _tokens[i]; - CppTokenKind cppTokenKind = 0; - switch (token.Kind) - { - case CXTokenKind.CXToken_Punctuation: - cppTokenKind = CppTokenKind.Punctuation; - break; - case CXTokenKind.CXToken_Keyword: - cppTokenKind = CppTokenKind.Keyword; - break; - case CXTokenKind.CXToken_Identifier: - cppTokenKind = CppTokenKind.Identifier; - break; - case CXTokenKind.CXToken_Literal: - cppTokenKind = CppTokenKind.Literal; - break; - case CXTokenKind.CXToken_Comment: - cppTokenKind = CppTokenKind.Comment; - break; - default: - break; - } - - var tokenStr = token.GetSpelling(_tu).CString; - var tokenLocation = token.GetLocation(_tu); - - var tokenRange = token.GetExtent(_tu); - cppToken = new CppToken(cppTokenKind, tokenStr) - { - Span = new CppSourceSpan(GetSourceLocation(tokenRange.Start), GetSourceLocation(tokenRange.End)) - }; - return cppToken; - } - } - - public string GetString(int i) - { - var token = _tokens[i]; - return token.GetSpelling(_tu).CString; - } - - public string TokensToString() - { - if (_tokens == null) - { - return null; - } - - var tokens = new List(_tokens.Length); - - for (int i = 0; i < _tokens.Length; i++) - { - tokens.Add(this[i]); - } - - return CppToken.TokensToString(tokens); - } - - public string GetStringForLength(int length) - { - StringBuilder result = new StringBuilder(length); - for (var cur = 0; cur < Count; ++cur) - { - result.Append(GetString(cur)); - if (result.Length >= length) - return result.ToString(); - } - return result.ToString(); - } - } - - private class AttributeTokenizer : Tokenizer - { - public AttributeTokenizer(CXCursor cursor) : base(cursor) - { - } - - public AttributeTokenizer(CXTranslationUnit tu, CXSourceRange range) : base(tu, range) - { - - } - - private uint IncOffset(int inc, uint offset) - { - if (inc >= 0) - offset += (uint)inc; - else - offset -= (uint)-inc; - return offset; - } - - private Tuple GetExtent(CXTranslationUnit tu, CXCursor cur) - { - var cursorExtend = cur.Extent; - var begin = cursorExtend.Start; - var end = cursorExtend.End; - - bool CursorIsFunction(CXCursorKind inKind) - { - return inKind == CXCursorKind.CXCursor_FunctionDecl || inKind == CXCursorKind.CXCursor_CXXMethod - || inKind == CXCursorKind.CXCursor_Constructor || inKind == CXCursorKind.CXCursor_Destructor - || inKind == CXCursorKind.CXCursor_ConversionFunction; - } - - bool CursorIsVar(CXCursorKind inKind) - { - return inKind == CXCursorKind.CXCursor_VarDecl || inKind == CXCursorKind.CXCursor_FieldDecl; - } - - bool IsInRange(CXSourceLocation loc, CXSourceRange range) - { - var xbegin = range.Start; - var xend = range.End; - - loc.GetSpellingLocation(out var fileLocation, out var lineLocation, out var u1, out var u2); - xbegin.GetSpellingLocation(out var fileBegin, out var lineBegin, out u1, out u2); - xend.GetSpellingLocation(out var fileEnd, out var lineEnd, out u1, out u2); - - return lineLocation >= lineBegin && lineLocation < lineEnd && (fileLocation.Equals(fileBegin)); - } - - bool HasInlineTypeDefinition(CXCursor varDecl) - { - var typeDecl = varDecl.Type.Declaration; - if (typeDecl.IsNull) - return false; - - var typeLocation = typeDecl.Location; - var varRange = typeDecl.Extent; - return IsInRange(typeLocation, varRange); - } - - CXSourceLocation GetNextLocation(CXSourceLocation loc, int inc = 1) - { - CXSourceLocation value; - loc.GetSpellingLocation(out var f, out var u, out var z, out var originalOffset); - var offset = IncOffset(inc, z); - var shouldUseLine = (z != 0 && (offset != 0 || offset != uint.MaxValue)); - if (shouldUseLine) - { - value = tu.GetLocation(f, u, offset); - } - else - { - offset = IncOffset(inc, originalOffset); - value = tu.GetLocationForOffset(f, offset); - } - - return value; - } - - CXSourceLocation GetPrevLocation(CXSourceLocation loc, int tokenLength) - { - var inc = 1; - while (true) - { - var locBefore = GetNextLocation(loc, -inc); - CXToken* tokens; - uint size; - clang.tokenize(tu, clang.getRange(locBefore, loc), &tokens, &size); - if (size == 0) - return CXSourceLocation.Null; - - var tokenLocation = tokens[0].GetLocation(tu); - if (locBefore.Equals(tokenLocation)) - { - return GetNextLocation(loc, -1 * (inc + tokenLength - 1)); - } - else - ++inc; - } - } - - bool TokenIsBefore(CXSourceLocation loc, string tokenString) - { - var length = tokenString.Length; - var locBefore = GetPrevLocation(loc, length); - - var tokenizer = new Tokenizer(tu, clang.getRange(locBefore, loc)); - if (tokenizer.Count == 0) return false; - - return tokenizer.GetStringForLength(length) == tokenString; - } - - bool TokenAtIs(CXSourceLocation loc, string tokenString) - { - var length = tokenString.Length; - - var locAfter = GetNextLocation(loc, length); - var tokenizer = new Tokenizer(tu, clang.getRange(locAfter, loc)); - - return tokenizer.GetStringForLength(length) == tokenString; - } - - bool ConsumeIfTokenAtIs(ref CXSourceLocation loc, string tokenString) - { - var length = tokenString.Length; - - var locAfter = GetNextLocation(loc, length); - var tokenizer = new Tokenizer(tu, clang.getRange(locAfter, loc)); - if (tokenizer.Count == 0) - return false; - - if (tokenizer.GetStringForLength(length) == tokenString) - { - loc = locAfter; - return true; - } - else - return false; - } - - bool ConsumeIfTokenBeforeIs(ref CXSourceLocation loc, string tokenString) - { - var length = tokenString.Length; - - var locBefore = GetPrevLocation(loc, length); - - var tokenizer = new Tokenizer(tu, clang.getRange(locBefore, loc)); - if (tokenizer.GetStringForLength(length) == tokenString) - { - loc = locBefore; - return true; - } - else - return false; - } - - bool CheckIfValidOrReset(ref CXSourceLocation checkedLocation, CXSourceLocation resetLocation) - { - bool isValid = true; - if (checkedLocation.Equals(CXSourceLocation.Null)) - { - checkedLocation = resetLocation; - isValid = false; - } - - return isValid; - } - - var kind = cur.Kind; - if (CursorIsFunction(kind) || CursorIsFunction(cur.TemplateCursorKind) - || kind == CXCursorKind.CXCursor_VarDecl || kind == CXCursorKind.CXCursor_FieldDecl || kind == CXCursorKind.CXCursor_ParmDecl - || kind == CXCursorKind.CXCursor_NonTypeTemplateParameter) - { - while (TokenIsBefore(begin, "]]") || TokenIsBefore(begin, ")")) - { - var saveBegin = begin; - if (ConsumeIfTokenBeforeIs(ref begin, "]]")) - { - bool isValid = true; - while (!ConsumeIfTokenBeforeIs(ref begin, "[[") && isValid) - { - begin = GetPrevLocation(begin, 1); - isValid = CheckIfValidOrReset(ref begin, saveBegin); - } - - if (!isValid) - { - break; - } - } - else if (ConsumeIfTokenBeforeIs(ref begin, ")")) - { - var parenCount = 1; - for (var lastBegin = begin; parenCount != 0; lastBegin = begin) - { - if (TokenIsBefore(begin, "(")) - --parenCount; - else if (TokenIsBefore(begin, ")")) - ++parenCount; - - begin = GetPrevLocation(begin, 1); - - // We have reached the end of the source of trying to deal - // with the potential of alignas, so we just break, which - // will cause ConsumeIfTokenBeforeIs(ref begin, "alignas") to be false - // and thus fall back to saveBegin which is the correct behavior - if (!CheckIfValidOrReset(ref begin, saveBegin)) - break; - } - - if (!ConsumeIfTokenBeforeIs(ref begin, "alignas")) - { - begin = saveBegin; - break; - } - } - } - - if (CursorIsVar(kind) || CursorIsVar(cur.TemplateCursorKind)) - { - if (HasInlineTypeDefinition(cur)) - { - var typeCursor = clang.getTypeDeclaration(clang.getCursorType(cur)); - var typeExtent = clang.getCursorExtent(typeCursor); - - var typeBegin = clang.getRangeStart(typeExtent); - var typeEnd = clang.getRangeEnd(typeExtent); - - return new Tuple(clang.getRange(begin, typeBegin), clang.getRange(typeEnd, end)); - } - } - else if (kind == CXCursorKind.CXCursor_TemplateTypeParameter && TokenAtIs(end, "(")) - { - var next = GetNextLocation(end, 1); - var prev = end; - for (var parenCount = 1; parenCount != 0; next = GetNextLocation(next, 1)) - { - if (TokenAtIs(next, "(")) - ++parenCount; - else if (TokenAtIs(next, ")")) - --parenCount; - prev = next; - } - end = next; - } - else if (kind == CXCursorKind.CXCursor_TemplateTemplateParameter && TokenAtIs(end, "<")) - { - var next = GetNextLocation(end, 1); - for (var angleCount = 1; angleCount != 0; next = GetNextLocation(next, 1)) - { - if (TokenAtIs(next, ">")) - --angleCount; - else if (TokenAtIs(next, ">>")) - angleCount -= 2; - else if (TokenAtIs(next, "<")) - ++angleCount; - } - - while (!TokenAtIs(next, ">") && !TokenAtIs(next, ",")) - next = GetNextLocation(next, 1); - - end = GetPrevLocation(next, 1); - } - else if ((kind == CXCursorKind.CXCursor_TemplateTypeParameter || kind == CXCursorKind.CXCursor_NonTypeTemplateParameter - || kind == CXCursorKind.CXCursor_TemplateTemplateParameter)) - { - ConsumeIfTokenAtIs(ref end, "..."); - } - else if (kind == CXCursorKind.CXCursor_EnumConstantDecl && !TokenAtIs(end, ",")) - { - var parent = clang.getCursorLexicalParent(cur); - end = clang.getRangeEnd(clang.getCursorExtent(parent)); - } - } - - return new Tuple(clang.getRange(begin, end), clang.getNullRange()); - } - - public override CXSourceRange GetRange(CXCursor cursor) - { - /* This process is complicated when parsing attributes that use - C++11 syntax, essentially even if libClang understands them - it doesn't always return them back as parse of the token range. - - This is kind of frustrating when you want to be able to do something - with custom or even compiler attributes in your parsing. Thus we have - to do things a little manually in order to make this work. - - This code supports stepping back when its valid to parse attributes, it - doesn't currently support all cases but it supports most valid cases. - */ - var range = GetExtent(_tu, cursor); - - var beg = range.Item1.Start; - var end = range.Item1.End; - if (!range.Item2.Equals(CXSourceRange.Null)) - end = range.Item2.End; - - return clang.getRange(beg, end); - } - } - - private class TokenizerDebuggerType - { - private readonly Tokenizer _tokenizer; - - public TokenizerDebuggerType(Tokenizer tokenizer) - { - _tokenizer = tokenizer; - } - - [DebuggerBrowsable(DebuggerBrowsableState.RootHidden)] - public object[] Items - { - get - { - var array = new object[_tokenizer.Count]; - for (int i = 0; i < _tokenizer.Count; i++) - { - array[i] = _tokenizer[i]; - } - return array; - } - } - } - private class CppContainerContext { public CppContainerContext(ICppContainer container) diff --git a/src/CppAst/CppNamespace.cs b/src/CppAst/CppNamespace.cs index 4a42b17..88349e8 100644 --- a/src/CppAst/CppNamespace.cs +++ b/src/CppAst/CppNamespace.cs @@ -25,7 +25,8 @@ public CppNamespace(string name) Classes = new CppContainerList(this); Typedefs = new CppContainerList(this); Namespaces = new CppContainerList(this); - Attributes = new CppContainerList(this); + Attributes = new List(); + TokenAttributes = new List(); } /// @@ -57,7 +58,10 @@ public CppNamespace(string name) public CppContainerList Namespaces { get; } /// - public CppContainerList Attributes { get; } + public List Attributes { get; } + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + public List TokenAttributes { get; } + protected bool Equals(CppNamespace other) { diff --git a/src/CppAst/CppParser.cs b/src/CppAst/CppParser.cs index 6f631d3..68fb40d 100644 --- a/src/CppAst/CppParser.cs +++ b/src/CppAst/CppParser.cs @@ -128,7 +128,8 @@ private static unsafe CppCompilation ParseInternal(List cppFile { AutoSquashTypedef = options.AutoSquashTypedef, ParseSystemIncludes = options.ParseSystemIncludes, - ParseAttributeEnabled = options.ParseAttributes, + ParseTokenAttributeEnabled = options.ParseTokenAttributes, + ParseCommentAttributeEnabled = options.ParseCommentAttribute, }; var compilation = builder.RootCompilation; diff --git a/src/CppAst/CppParserOptions.cs b/src/CppAst/CppParserOptions.cs index e9a6abc..9a46517 100644 --- a/src/CppAst/CppParserOptions.cs +++ b/src/CppAst/CppParserOptions.cs @@ -20,7 +20,13 @@ public CppParserOptions() ParseAsCpp = true; SystemIncludeFolders = new List(); IncludeFolders = new List(); - Defines = new List(); + + //Add a default macro here for CppAst.Net + Defines = new List() { + "__cppast_run__", //Help us for identify the CppAst.Net handler + @"__cppast_impl(...)=__attribute__((annotate(#__VA_ARGS__)))", //Help us for use annotate attribute convenience + @"__cppast(...)=__cppast_impl(__VA_ARGS__)", //Add a macro wrapper here, so the argument with macro can be handle right for compiler. + }; AdditionalArguments = new List() { "-Wno-pragma-once-outside-header" @@ -29,7 +35,8 @@ public CppParserOptions() ParseMacros = false; ParseComments = true; ParseSystemIncludes = true; - ParseAttributes = false; + ParseTokenAttributes = false; + ParseCommentAttribute = false; // Default triple targets TargetCpu = IntPtr.Size == 8 ? CppTargetCpu.X86_64 : CppTargetCpu.X86; @@ -85,9 +92,14 @@ public CppParserOptions() public bool ParseSystemIncludes { get; set; } /// - /// Gets or sets a boolean indicating whether to parse Attributes. Default is false + /// Gets or sets a boolean indicating whether to parse meta attributes. Default is false + /// + public bool ParseTokenAttributes { get; set; } + + /// + /// Gets or sets a boolean indicating whether to parse comment attributes. Default is false /// - public bool ParseAttributes { get; set; } + public bool ParseCommentAttribute { get; set; } /// /// Sets to true and return this instance. diff --git a/src/CppAst/CppTemplateArgument.cs b/src/CppAst/CppTemplateArgument.cs index 2f97827..4efa3aa 100644 --- a/src/CppAst/CppTemplateArgument.cs +++ b/src/CppAst/CppTemplateArgument.cs @@ -12,11 +12,12 @@ namespace CppAst /// public class CppTemplateArgument : CppType { - public CppTemplateArgument(CppType sourceParam, CppType typeArg) : base(CppTypeKind.TemplateArgumentType) + public CppTemplateArgument(CppType sourceParam, CppType typeArg, bool isSpecializedArgument) : base(CppTypeKind.TemplateArgumentType) { SourceParam = sourceParam ?? throw new ArgumentNullException(nameof(sourceParam)); ArgAsType = typeArg ?? throw new ArgumentNullException(nameof(typeArg)); ArgKind = CppTemplateArgumentKind.AsType; + IsSpecializedArgument = isSpecializedArgument; } public CppTemplateArgument(CppType sourceParam, long intArg) : base(CppTypeKind.TemplateArgumentType) @@ -24,6 +25,7 @@ public CppTemplateArgument(CppType sourceParam, long intArg) : base(CppTypeKind. SourceParam = sourceParam ?? throw new ArgumentNullException(nameof(sourceParam)); ArgAsInteger = intArg; ArgKind = CppTemplateArgumentKind.AsInteger; + IsSpecializedArgument = true; } public CppTemplateArgument(CppType sourceParam, string unknownStr) : base(CppTypeKind.TemplateArgumentType) @@ -31,6 +33,7 @@ public CppTemplateArgument(CppType sourceParam, string unknownStr) : base(CppTyp SourceParam = sourceParam ?? throw new ArgumentNullException(nameof(sourceParam)); ArgAsUnknown = unknownStr; ArgKind = CppTemplateArgumentKind.Unknown; + IsSpecializedArgument = true; } public CppTemplateArgumentKind ArgKind { get; } @@ -65,6 +68,7 @@ public string ArgString /// public CppType SourceParam { get; } + public bool IsSpecializedArgument { get; } /// public override int SizeOf diff --git a/src/CppAst/CppTokenUtil.cs b/src/CppAst/CppTokenUtil.cs new file mode 100644 index 0000000..c8b2384 --- /dev/null +++ b/src/CppAst/CppTokenUtil.cs @@ -0,0 +1,1065 @@ +// Copyright (c) Alexandre Mutel. All rights reserved. +// Licensed under the BSD-Clause 2 license. +// See license.txt file in the project root for full license information. + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq.Expressions; +using System.Text; +using ClangSharp; +using ClangSharp.Interop; + +namespace CppAst +{ + static internal unsafe class CppTokenUtil + { + public static void ParseCursorAttributs(CXCursor cursor, ref List attributes) + { + var tokenizer = new AttributeTokenizer(cursor); + var tokenIt = new TokenIterator(tokenizer); + + // if this is a template then we need to skip that ? + if (tokenIt.CanPeek && tokenIt.PeekText() == "template") + SkipTemplates(tokenIt); + + while (tokenIt.CanPeek) + { + if (ParseAttributes(tokenIt, ref attributes)) + { + continue; + } + + // If we have a keyword, try to skip it and process following elements + // for example attribute put right after a struct __declspec(uuid("...")) Test {...} + if (tokenIt.Peek().Kind == CppTokenKind.Keyword) + { + tokenIt.Next(); + continue; + } + break; + } + } + + + public static void ParseFunctionAttributes(CXCursor cursor, string functionName, ref List attributes) + { + // TODO: This function is not 100% correct when parsing tokens up to the function name + // we assume to find the function name immediately followed by a `(` + // but some return type parameter could actually interfere with that + // Ideally we would need to parse more properly return type and skip parenthesis for example + var tokenizer = new AttributeTokenizer(cursor); + var tokenIt = new TokenIterator(tokenizer); + + // if this is a template then we need to skip that ? + if (tokenIt.CanPeek && tokenIt.PeekText() == "template") + SkipTemplates(tokenIt); + + // Parse leading attributes + while (tokenIt.CanPeek) + { + if (ParseAttributes(tokenIt, ref attributes)) + { + continue; + } + break; + } + + if (!tokenIt.CanPeek) + { + return; + } + + // Find function name (We only support simple function name declaration) + if (!tokenIt.Find(functionName, "(")) + { + return; + } + + Debug.Assert(tokenIt.PeekText() == functionName); + tokenIt.Next(); + Debug.Assert(tokenIt.PeekText() == "("); + tokenIt.Next(); + + int parentCount = 1; + while (parentCount > 0 && tokenIt.CanPeek) + { + var text = tokenIt.PeekText(); + if (text == "(") + { + parentCount++; + } + else if (text == ")") + { + parentCount--; + } + tokenIt.Next(); + } + + if (parentCount != 0) + { + return; + } + + while (tokenIt.CanPeek) + { + if (ParseAttributes(tokenIt, ref attributes)) + { + continue; + } + // Skip the token if we can parse it. + tokenIt.Next(); + } + + return; + } + + + public static void ParseAttributesInRange(CXTranslationUnit tu, CXSourceRange range, ref List collectAttributes) + { + var tokenizer = new AttributeTokenizer(tu, range); + var tokenIt = new TokenIterator(tokenizer); + + var tokenIt2 = new TokenIterator(tokenizer); + StringBuilder sb = new StringBuilder(); + while (tokenIt.CanPeek) + { + sb.Append(tokenIt.PeekText()); + tokenIt.Next(); + } + + // if this is a template then we need to skip that ? + if (tokenIt.CanPeek && tokenIt.PeekText() == "template") + SkipTemplates(tokenIt); + + while (tokenIt.CanPeek) + { + if (ParseAttributes(tokenIt, ref collectAttributes)) + { + continue; + } + + // If we have a keyword, try to skip it and process following elements + // for example attribute put right after a struct __declspec(uuid("...")) Test {...} + if (tokenIt.Peek().Kind == CppTokenKind.Keyword) + { + tokenIt.Next(); + continue; + } + break; + } + } + + public static bool TryToSeekOnlineAttributes(CXCursor cursor, out CXSourceRange range) + { + + + int SkipWhiteSpace(ReadOnlySpan cnt, int cntOffset) + { + while (cntOffset > 0) + { + char ch = (char)cnt[cntOffset]; + if (ch == ' ' || ch == '\r' || ch == '\n' || ch == '\t') + { + cntOffset--; + } + else + { + break; + } + } + + return cntOffset; + }; + + int ToLineStart(ReadOnlySpan cnt, int cntOffset) + { + for (int i = cntOffset; i >= 0; i--) + { + char ch = (char)cnt[i]; + if (ch == '\n') + { + return i + 1; + } + } + return 0; + }; + + bool IsAttributeEnd(ReadOnlySpan cnt, int cntOffset) + { + if (cntOffset < 1) return false; + + char ch0 = (char)cnt[cntOffset]; + char ch1 = (char)cnt[cntOffset - 1]; + + return ch0 == ch1 && ch0 == ']'; + }; + + bool IsAttributeStart(ReadOnlySpan cnt, int cntOffset) + { + if (cntOffset < 1) return false; + + char ch0 = (char)cnt[cntOffset]; + char ch1 = (char)cnt[cntOffset - 1]; + + return ch0 == ch1 && ch0 == '['; + }; + + bool SeekAttributeStartSingleChar(ReadOnlySpan cnt, int cntOffset, out int outSeekOffset) + { + outSeekOffset = cntOffset; + while (cntOffset > 0) + { + char ch = (char)cnt[cntOffset]; + if (ch == '[') + { + outSeekOffset = cntOffset; + return true; + } + cntOffset--; + } + return false; + }; + + int SkipAttributeStartOrEnd(ReadOnlySpan cnt, int cntOffset) + { + cntOffset -= 2; + return cntOffset; + }; + + string QueryLineContent(ReadOnlySpan cnt, int startOffset, int endOffset) + { + StringBuilder sb = new StringBuilder(); + for (int i = startOffset; i <= endOffset; i++) + { + sb.Append((char)cnt[i]); + } + return sb.ToString(); + }; + + CXSourceLocation location = cursor.Extent.Start; + location.GetFileLocation(out var file, out var line, out var column, out var offset); + var contents = cursor.TranslationUnit.GetFileContents(file, out var fileSize); + + AttributeLexerParseStatus status = AttributeLexerParseStatus.SeekAttributeEnd; + int offsetStart = (int)offset - 1; //Try to ignore start char here + int lastSeekOffset = offsetStart; + int curOffset = offsetStart; + while (curOffset > 0) + { + curOffset = SkipWhiteSpace(contents, curOffset); + + switch (status) + { + case AttributeLexerParseStatus.SeekAttributeEnd: + { + if (!IsAttributeEnd(contents, curOffset)) + { + status = AttributeLexerParseStatus.Error; + } + else + { + curOffset = SkipAttributeStartOrEnd(contents, curOffset); + status = AttributeLexerParseStatus.SeekAttributeStart; + } + } + break; + case AttributeLexerParseStatus.SeekAttributeStart: + { + if (!SeekAttributeStartSingleChar(contents, curOffset, out var queryOffset)) + { + status = AttributeLexerParseStatus.Error; + } + else + { + if (IsAttributeStart(contents, queryOffset)) + { + curOffset = SkipAttributeStartOrEnd(contents, queryOffset); + lastSeekOffset = curOffset + 1; + status = AttributeLexerParseStatus.SeekAttributeEnd; + } + else + { + status = AttributeLexerParseStatus.Error; + } + } + } + break; + } + + if (status == AttributeLexerParseStatus.Error) + { + break; + } + } + if (lastSeekOffset == offsetStart) + { + range = new CXSourceRange(); + return false; + } + else + { + var startLoc = cursor.TranslationUnit.GetLocationForOffset(file, (uint)lastSeekOffset); + var endLoc = cursor.TranslationUnit.GetLocationForOffset(file, (uint)offsetStart); + range = clang.getRange(startLoc, endLoc); + return true; + } + } + + #region "Nested Types" + + /// + /// Internal class to iterate on tokens + /// + private class TokenIterator + { + private readonly Tokenizer _tokens; + private int _index; + + public TokenIterator(Tokenizer tokens) + { + _tokens = tokens; + } + + public bool Skip(string expectedText) + { + if (_index < _tokens.Count) + { + if (_tokens.GetString(_index) == expectedText) + { + _index++; + return true; + } + } + + return false; + } + + public CppToken PreviousToken() + { + if (_index > 0) + { + return _tokens[_index - 1]; + } + + return null; + } + + public bool Skip(params string[] expectedTokens) + { + var startIndex = _index; + foreach (var expectedToken in expectedTokens) + { + if (startIndex < _tokens.Count) + { + if (_tokens.GetString(startIndex) == expectedToken) + { + startIndex++; + continue; + } + } + return false; + } + _index = startIndex; + return true; + } + + public bool Find(params string[] expectedTokens) + { + var startIndex = _index; + restart: + while (startIndex < _tokens.Count) + { + var firstIndex = startIndex; + foreach (var expectedToken in expectedTokens) + { + if (startIndex < _tokens.Count) + { + if (_tokens.GetString(startIndex) == expectedToken) + { + startIndex++; + continue; + } + } + startIndex = firstIndex + 1; + goto restart; + } + _index = firstIndex; + return true; + } + return false; + } + + public bool Next(out CppToken token) + { + token = null; + if (_index < _tokens.Count) + { + token = _tokens[_index]; + _index++; + return true; + } + return false; + } + + public bool CanPeek => _index < _tokens.Count; + + public bool Next() + { + if (_index < _tokens.Count) + { + _index++; + return true; + } + return false; + } + + public CppToken Peek() + { + if (_index < _tokens.Count) + { + return _tokens[_index]; + } + return null; + } + + public string PeekText() + { + if (_index < _tokens.Count) + { + return _tokens.GetString(_index); + } + return null; + } + } + + /// + /// Internal class to tokenize + /// + [DebuggerTypeProxy(typeof(TokenizerDebuggerType))] + internal class Tokenizer + { + private readonly CXToken[] _tokens; + private CppToken[] _cppTokens; + protected readonly CXTranslationUnit _tu; + + public Tokenizer(CXCursor cursor) + { + _tu = cursor.TranslationUnit; + var range = GetRange(cursor); + _tokens = _tu.Tokenize(range).ToArray(); + } + + public Tokenizer(CXTranslationUnit tu, CXSourceRange range) + { + _tu = tu; + _tokens = _tu.Tokenize(range).ToArray(); + } + + public virtual CXSourceRange GetRange(CXCursor cursor) + { + return cursor.Extent; + } + + public int Count => _tokens?.Length ?? 0; + + public CppToken this[int i] + { + get + { + // Only create a tokenizer if necessary + if (_cppTokens == null) + { + _cppTokens = new CppToken[_tokens.Length]; + } + + ref var cppToken = ref _cppTokens[i]; + if (cppToken != null) + { + return cppToken; + } + + var token = _tokens[i]; + CppTokenKind cppTokenKind = 0; + switch (token.Kind) + { + case CXTokenKind.CXToken_Punctuation: + cppTokenKind = CppTokenKind.Punctuation; + break; + case CXTokenKind.CXToken_Keyword: + cppTokenKind = CppTokenKind.Keyword; + break; + case CXTokenKind.CXToken_Identifier: + cppTokenKind = CppTokenKind.Identifier; + break; + case CXTokenKind.CXToken_Literal: + cppTokenKind = CppTokenKind.Literal; + break; + case CXTokenKind.CXToken_Comment: + cppTokenKind = CppTokenKind.Comment; + break; + default: + break; + } + + var tokenStr = token.GetSpelling(_tu).CString; + var tokenLocation = token.GetLocation(_tu); + + var tokenRange = token.GetExtent(_tu); + cppToken = new CppToken(cppTokenKind, tokenStr) + { + Span = new CppSourceSpan(CppModelBuilder.GetSourceLocation(tokenRange.Start), CppModelBuilder.GetSourceLocation(tokenRange.End)) + }; + return cppToken; + } + } + + public string GetString(int i) + { + var token = _tokens[i]; + return token.GetSpelling(_tu).CString; + } + + public string TokensToString() + { + if (_tokens == null) + { + return null; + } + + var tokens = new List(_tokens.Length); + + for (int i = 0; i < _tokens.Length; i++) + { + tokens.Add(this[i]); + } + + return CppToken.TokensToString(tokens); + } + + public string GetStringForLength(int length) + { + StringBuilder result = new StringBuilder(length); + for (var cur = 0; cur < Count; ++cur) + { + result.Append(GetString(cur)); + if (result.Length >= length) + return result.ToString(); + } + return result.ToString(); + } + } + + + private class AttributeTokenizer : Tokenizer + { + public AttributeTokenizer(CXCursor cursor) : base(cursor) + { + } + + public AttributeTokenizer(CXTranslationUnit tu, CXSourceRange range) : base(tu, range) + { + + } + + private uint IncOffset(int inc, uint offset) + { + if (inc >= 0) + offset += (uint)inc; + else + offset -= (uint)-inc; + return offset; + } + + private Tuple GetExtent(CXTranslationUnit tu, CXCursor cur) + { + var cursorExtend = cur.Extent; + var begin = cursorExtend.Start; + var end = cursorExtend.End; + + bool CursorIsFunction(CXCursorKind inKind) + { + return inKind == CXCursorKind.CXCursor_FunctionDecl || inKind == CXCursorKind.CXCursor_CXXMethod + || inKind == CXCursorKind.CXCursor_Constructor || inKind == CXCursorKind.CXCursor_Destructor + || inKind == CXCursorKind.CXCursor_ConversionFunction; + } + + bool CursorIsVar(CXCursorKind inKind) + { + return inKind == CXCursorKind.CXCursor_VarDecl || inKind == CXCursorKind.CXCursor_FieldDecl; + } + + bool IsInRange(CXSourceLocation loc, CXSourceRange range) + { + var xbegin = range.Start; + var xend = range.End; + + loc.GetSpellingLocation(out var fileLocation, out var lineLocation, out var u1, out var u2); + xbegin.GetSpellingLocation(out var fileBegin, out var lineBegin, out u1, out u2); + xend.GetSpellingLocation(out var fileEnd, out var lineEnd, out u1, out u2); + + return lineLocation >= lineBegin && lineLocation < lineEnd && (fileLocation.Equals(fileBegin)); + } + + bool HasInlineTypeDefinition(CXCursor varDecl) + { + var typeDecl = varDecl.Type.Declaration; + if (typeDecl.IsNull) + return false; + + var typeLocation = typeDecl.Location; + var varRange = typeDecl.Extent; + return IsInRange(typeLocation, varRange); + } + + CXSourceLocation GetNextLocation(CXSourceLocation loc, int inc = 1) + { + CXSourceLocation value; + loc.GetSpellingLocation(out var f, out var u, out var z, out var originalOffset); + var offset = IncOffset(inc, z); + var shouldUseLine = (z != 0 && (offset != 0 || offset != uint.MaxValue)); + if (shouldUseLine) + { + value = tu.GetLocation(f, u, offset); + } + else + { + offset = IncOffset(inc, originalOffset); + value = tu.GetLocationForOffset(f, offset); + } + + return value; + } + + CXSourceLocation GetPrevLocation(CXSourceLocation loc, int tokenLength) + { + var inc = 1; + while (true) + { + var locBefore = GetNextLocation(loc, -inc); + CXToken* tokens; + uint size; + clang.tokenize(tu, clang.getRange(locBefore, loc), &tokens, &size); + if (size == 0) + return CXSourceLocation.Null; + + var tokenLocation = tokens[0].GetLocation(tu); + if (locBefore.Equals(tokenLocation)) + { + return GetNextLocation(loc, -1 * (inc + tokenLength - 1)); + } + else + ++inc; + } + } + + bool TokenIsBefore(CXSourceLocation loc, string tokenString) + { + var length = tokenString.Length; + var locBefore = GetPrevLocation(loc, length); + + var tokenizer = new Tokenizer(tu, clang.getRange(locBefore, loc)); + if (tokenizer.Count == 0) return false; + + return tokenizer.GetStringForLength(length) == tokenString; + } + + bool TokenAtIs(CXSourceLocation loc, string tokenString) + { + var length = tokenString.Length; + + var locAfter = GetNextLocation(loc, length); + var tokenizer = new Tokenizer(tu, clang.getRange(locAfter, loc)); + + return tokenizer.GetStringForLength(length) == tokenString; + } + + bool ConsumeIfTokenAtIs(ref CXSourceLocation loc, string tokenString) + { + var length = tokenString.Length; + + var locAfter = GetNextLocation(loc, length); + var tokenizer = new Tokenizer(tu, clang.getRange(locAfter, loc)); + if (tokenizer.Count == 0) + return false; + + if (tokenizer.GetStringForLength(length) == tokenString) + { + loc = locAfter; + return true; + } + else + return false; + } + + bool ConsumeIfTokenBeforeIs(ref CXSourceLocation loc, string tokenString) + { + var length = tokenString.Length; + + var locBefore = GetPrevLocation(loc, length); + + var tokenizer = new Tokenizer(tu, clang.getRange(locBefore, loc)); + if (tokenizer.GetStringForLength(length) == tokenString) + { + loc = locBefore; + return true; + } + else + return false; + } + + bool CheckIfValidOrReset(ref CXSourceLocation checkedLocation, CXSourceLocation resetLocation) + { + bool isValid = true; + if (checkedLocation.Equals(CXSourceLocation.Null)) + { + checkedLocation = resetLocation; + isValid = false; + } + + return isValid; + } + + var kind = cur.Kind; + if (CursorIsFunction(kind) || CursorIsFunction(cur.TemplateCursorKind) + || kind == CXCursorKind.CXCursor_VarDecl || kind == CXCursorKind.CXCursor_FieldDecl || kind == CXCursorKind.CXCursor_ParmDecl + || kind == CXCursorKind.CXCursor_NonTypeTemplateParameter) + { + while (TokenIsBefore(begin, "]]") || TokenIsBefore(begin, ")")) + { + var saveBegin = begin; + if (ConsumeIfTokenBeforeIs(ref begin, "]]")) + { + bool isValid = true; + while (!ConsumeIfTokenBeforeIs(ref begin, "[[") && isValid) + { + begin = GetPrevLocation(begin, 1); + isValid = CheckIfValidOrReset(ref begin, saveBegin); + } + + if (!isValid) + { + break; + } + } + else if (ConsumeIfTokenBeforeIs(ref begin, ")")) + { + var parenCount = 1; + for (var lastBegin = begin; parenCount != 0; lastBegin = begin) + { + if (TokenIsBefore(begin, "(")) + --parenCount; + else if (TokenIsBefore(begin, ")")) + ++parenCount; + + begin = GetPrevLocation(begin, 1); + + // We have reached the end of the source of trying to deal + // with the potential of alignas, so we just break, which + // will cause ConsumeIfTokenBeforeIs(ref begin, "alignas") to be false + // and thus fall back to saveBegin which is the correct behavior + if (!CheckIfValidOrReset(ref begin, saveBegin)) + break; + } + + if (!ConsumeIfTokenBeforeIs(ref begin, "alignas")) + { + begin = saveBegin; + break; + } + } + } + + if (CursorIsVar(kind) || CursorIsVar(cur.TemplateCursorKind)) + { + if (HasInlineTypeDefinition(cur)) + { + var typeCursor = clang.getTypeDeclaration(clang.getCursorType(cur)); + var typeExtent = clang.getCursorExtent(typeCursor); + + var typeBegin = clang.getRangeStart(typeExtent); + var typeEnd = clang.getRangeEnd(typeExtent); + + return new Tuple(clang.getRange(begin, typeBegin), clang.getRange(typeEnd, end)); + } + } + else if (kind == CXCursorKind.CXCursor_TemplateTypeParameter && TokenAtIs(end, "(")) + { + var next = GetNextLocation(end, 1); + var prev = end; + for (var parenCount = 1; parenCount != 0; next = GetNextLocation(next, 1)) + { + if (TokenAtIs(next, "(")) + ++parenCount; + else if (TokenAtIs(next, ")")) + --parenCount; + prev = next; + } + end = next; + } + else if (kind == CXCursorKind.CXCursor_TemplateTemplateParameter && TokenAtIs(end, "<")) + { + var next = GetNextLocation(end, 1); + for (var angleCount = 1; angleCount != 0; next = GetNextLocation(next, 1)) + { + if (TokenAtIs(next, ">")) + --angleCount; + else if (TokenAtIs(next, ">>")) + angleCount -= 2; + else if (TokenAtIs(next, "<")) + ++angleCount; + } + + while (!TokenAtIs(next, ">") && !TokenAtIs(next, ",")) + next = GetNextLocation(next, 1); + + end = GetPrevLocation(next, 1); + } + else if ((kind == CXCursorKind.CXCursor_TemplateTypeParameter || kind == CXCursorKind.CXCursor_NonTypeTemplateParameter + || kind == CXCursorKind.CXCursor_TemplateTemplateParameter)) + { + ConsumeIfTokenAtIs(ref end, "..."); + } + else if (kind == CXCursorKind.CXCursor_EnumConstantDecl && !TokenAtIs(end, ",")) + { + var parent = clang.getCursorLexicalParent(cur); + end = clang.getRangeEnd(clang.getCursorExtent(parent)); + } + } + + return new Tuple(clang.getRange(begin, end), clang.getNullRange()); + } + + public override CXSourceRange GetRange(CXCursor cursor) + { + /* This process is complicated when parsing attributes that use + C++11 syntax, essentially even if libClang understands them + it doesn't always return them back as parse of the token range. + + This is kind of frustrating when you want to be able to do something + with custom or even compiler attributes in your parsing. Thus we have + to do things a little manually in order to make this work. + + This code supports stepping back when its valid to parse attributes, it + doesn't currently support all cases but it supports most valid cases. + */ + var range = GetExtent(_tu, cursor); + + var beg = range.Item1.Start; + var end = range.Item1.End; + if (!range.Item2.Equals(CXSourceRange.Null)) + end = range.Item2.End; + + return clang.getRange(beg, end); + } + } + + private class TokenizerDebuggerType + { + private readonly Tokenizer _tokenizer; + + public TokenizerDebuggerType(Tokenizer tokenizer) + { + _tokenizer = tokenizer; + } + + [DebuggerBrowsable(DebuggerBrowsableState.RootHidden)] + public object[] Items + { + get + { + var array = new object[_tokenizer.Count]; + for (int i = 0; i < _tokenizer.Count; i++) + { + array[i] = _tokenizer[i]; + } + return array; + } + } + } + + #endregion + + + #region "Private Functions" + + private static void SkipTemplates(TokenIterator iter) + { + if (iter.CanPeek) + { + if (iter.Skip("template")) + { + iter.Next(); // skip the first > + int parentCount = 1; + while (parentCount > 0 && iter.CanPeek) + { + var text = iter.PeekText(); + if (text == ">") + { + parentCount--; + } + iter.Next(); + } + } + } + } + + private enum AttributeLexerParseStatus + { + SeekAttributeEnd, + SeekAttributeStart, + Error, + } + + + private static bool ParseAttributes(TokenIterator tokenIt, ref List attributes) + { + // Parse C++ attributes + // [[]] + if (tokenIt.Skip("[", "[")) + { + while (ParseAttribute(tokenIt, out var attribute)) + { + if (attributes == null) + { + attributes = new List(); + } + attributes.Add(attribute); + + tokenIt.Skip(","); + } + + return tokenIt.Skip("]", "]"); + } + + // Parse GCC or clang attributes + // __attribute__(()) + if (tokenIt.Skip("__attribute__", "(", "(")) + { + while (ParseAttribute(tokenIt, out var attribute)) + { + if (attributes == null) + { + attributes = new List(); + } + attributes.Add(attribute); + + tokenIt.Skip(","); + } + + return tokenIt.Skip(")", ")"); + } + + // Parse MSVC attributes + // __declspec() + if (tokenIt.Skip("__declspec", "(")) + { + while (ParseAttribute(tokenIt, out var attribute)) + { + if (attributes == null) + { + attributes = new List(); + } + attributes.Add(attribute); + + tokenIt.Skip(","); + } + return tokenIt.Skip(")"); + } + + // Parse C++11 alignas attribute + // alignas(expression) + if (tokenIt.PeekText() == "alignas") + { + while (ParseAttribute(tokenIt, out var attribute)) + { + if (attributes == null) + { + attributes = new List(); + } + attributes.Add(attribute); + + break; + } + + return tokenIt.Skip(")"); ; + } + + return false; + } + + private static bool ParseAttribute(TokenIterator tokenIt, out CppAttribute attribute) + { + // (identifier ::)? identifier ('(' tokens ')' )? (...)? + attribute = null; + var token = tokenIt.Peek(); + if (token == null || !token.Kind.IsIdentifierOrKeyword()) + { + return false; + } + tokenIt.Next(out token); + + var firstToken = token; + + // try (identifier ::)? + string scope = null; + if (tokenIt.Skip("::")) + { + scope = token.Text; + + token = tokenIt.Peek(); + if (token == null || !token.Kind.IsIdentifierOrKeyword()) + { + return false; + } + tokenIt.Next(out token); + } + + // identifier + string tokenIdentifier = token.Text; + + string arguments = null; + + // ('(' tokens ')' )? + if (tokenIt.Skip("(")) + { + var builder = new StringBuilder(); + var previousTokenKind = CppTokenKind.Punctuation; + while (tokenIt.PeekText() != ")" && tokenIt.Next(out token)) + { + if (token.Kind.IsIdentifierOrKeyword() && previousTokenKind.IsIdentifierOrKeyword()) + { + builder.Append(" "); + } + previousTokenKind = token.Kind; + builder.Append(token.Text); + } + + if (!tokenIt.Skip(")")) + { + return false; + } + arguments = builder.ToString(); + } + + var isVariadic = tokenIt.Skip("..."); + + var previousToken = tokenIt.PreviousToken(); + + attribute = new CppAttribute(tokenIdentifier, AttributeKind.TokenAttribute) + { + Span = new CppSourceSpan(firstToken.Span.Start, previousToken.Span.End), + Scope = scope, + Arguments = arguments, + IsVariadic = isVariadic, + }; + return true; + } + + + #endregion + } + +} + + diff --git a/src/CppAst/ICppAttributeContainer.cs b/src/CppAst/ICppAttributeContainer.cs new file mode 100644 index 0000000..cc5fa7a --- /dev/null +++ b/src/CppAst/ICppAttributeContainer.cs @@ -0,0 +1,22 @@ +// Copyright (c) Alexandre Mutel. All rights reserved. +// Licensed under the BSD-Clause 2 license. +// See license.txt file in the project root for full license information. +using System; +using System.Collections.Generic; + +namespace CppAst +{ + /// + /// Base interface for all with attribute element. + /// + public interface ICppAttributeContainer + { + /// + /// Gets the attributes from element. + /// + List Attributes { get; } + + [Obsolete("TokenAttributes is deprecated. please use system attributes and annotate attributes")] + List TokenAttributes { get; } + } +} \ No newline at end of file diff --git a/src/CppAst/ICppDeclarationContainer.cs b/src/CppAst/ICppDeclarationContainer.cs index b8d1a5a..25ca93d 100644 --- a/src/CppAst/ICppDeclarationContainer.cs +++ b/src/CppAst/ICppDeclarationContainer.cs @@ -8,7 +8,7 @@ namespace CppAst /// Base interface of a containing fields, functions, enums, classes, typedefs. /// /// - public interface ICppDeclarationContainer : ICppContainer + public interface ICppDeclarationContainer : ICppContainer, ICppAttributeContainer { /// /// Gets the fields/variables. @@ -35,9 +35,7 @@ public interface ICppDeclarationContainer : ICppContainer /// CppContainerList Typedefs { get; } - /// - /// Gets the attributes. - /// - CppContainerList Attributes { get; } + //Just use ICppAttributeContainer here(enum can support attribute, so we just use ICppAttributeContainer here)~~ + //CppContainerList Attributes { get; } } } \ No newline at end of file