[lldb][RPC] Upstream LLDB to RPC converstion Python script #138028

chelcassanova · 2025-04-30T21:28:57Z

As part of upstreaming LLDB RPC, this commit adds a python script that
is used by LLDB RPC to modify the public lldb header files for use with
RPC.

https://discourse.llvm.org/t/rfc-upstreaming-lldb-rpc/85804

llvmbot · 2025-04-30T21:29:32Z

@llvm/pr-subscribers-lldb

Author: Chelsea Cassanova (chelcassanova)

Changes

As part of upstreaming LLDB RPC, this commit adds python scripts that are used by LLDB RPC to modify the public lldb header files for use with RPC.

https://discourse.llvm.org/t/rfc-upstreaming-lldb-rpc/85804

Full diff: https://github.com/llvm/llvm-project/pull/138028.diff

3 Files Affected:

(added) lldb/scripts/convert-lldb-header-to-rpc-header.py (+65)
(added) lldb/scripts/framework-header-include-fix.py (+44)
(added) lldb/scripts/framework-header-version-fix.py (+65)

diff --git a/lldb/scripts/convert-lldb-header-to-rpc-header.py b/lldb/scripts/convert-lldb-header-to-rpc-header.py
new file mode 100755
index 0000000000000..4550837d8e08d
--- /dev/null
+++ b/lldb/scripts/convert-lldb-header-to-rpc-header.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+# Usage: convert-lldb-header-to-rpc-header.py <path/to/input-header.h> <path/to/output-header.h>
+# This scripts takes common LLDB headers (such as lldb-defines.h) and replaces references to LLDB
+# with those for RPC. This happens for:
+# - namespace definitions
+# - namespace usage
+# - version string macros
+# - ifdef/ifndef lines
+
+import argparse
+import os
+import re
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input")
+    parser.add_argument("output")
+    args = parser.parse_args()
+    input_path = str(args.input)
+    output_path = str(args.output)
+    with open(input_path, "r") as input_file:
+        lines = input_file.readlines()
+
+    with open(output_path, "w") as output_file:
+        for line in lines:
+            # NOTE: We do not use lldb-forward.h or lldb-versioning.h in RPC, so remove
+            # all includes that are found for these files.
+            if re.match(
+                r'#include "lldb/lldb-forward|#include "lldb/lldb-versioning', line
+            ):
+                continue
+            # For lldb-rpc-defines.h, replace the ifndef LLDB_LLDB_ portion with LLDB_RPC_ as we're not
+            # using LLDB private definitions in RPC.
+            elif re.match(r".+LLDB_LLDB_", line):
+                output_file.write(re.sub(r"LLDB_LLDB_", r"LLDB_RPC_", line))
+            # Similarly to lldb-rpc-defines.h, replace the ifndef for LLDB_API in SBDefines.h to LLDB_RPC_API_ for the same reason.
+            elif re.match(r".+LLDB_API_", line):
+                output_file.write(re.sub(r"LLDB_API_", r"LLDB_RPC_API_", line))
+            # Replace the references for the macros that define the versioning strings in
+            # lldb-rpc-defines.h.
+            elif re.match(r".+LLDB_VERSION", line):
+                output_file.write(re.sub(r"LLDB_VERSION", r"LLDB_RPC_VERSION", line))
+            elif re.match(r".+LLDB_REVISION", line):
+                output_file.write(re.sub(r"LLDB_REVISION", r"LLDB_RPC_REVISION", line))
+            elif re.match(r".+LLDB_VERSION_STRING", line):
+                output_file.write(
+                    re.sub(r"LLDB_VERSION_STRING", r"LLDB_RPC_VERSION_STRING", line)
+                )
+            # For local #includes
+            elif re.match(r'#include "lldb/lldb-', line):
+                output_file.write(re.sub(r"lldb/lldb-", r"lldb-rpc-", line))
+            # Rename the lldb namespace definition to lldb-rpc.
+            elif re.match(r"namespace lldb", line):
+                output_file.write(re.sub(r"lldb", r"lldb_rpc", line))
+            # Rename namespace references
+            elif re.match(r".+lldb::", line):
+                output_file.write(re.sub(r"lldb::", r"lldb_rpc::", line))
+            else:
+                # Write any line that doesn't need to be converted
+                output_file.write(line)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/lldb/scripts/framework-header-include-fix.py b/lldb/scripts/framework-header-include-fix.py
new file mode 100755
index 0000000000000..e214ce005923f
--- /dev/null
+++ b/lldb/scripts/framework-header-include-fix.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+# Usage: framework-header-include-fix.py <path/to/input-header.h> <path/to/output-header.h>
+# This script modifies all #include lines in all lldb-rpc headers
+# from either filesystem or local includes to liblldbrpc includes.
+
+import argparse
+import os
+import re
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input")
+    parser.add_argument("output")
+    args = parser.parse_args()
+    input_path = str(args.input)
+    output_path = str(args.output)
+    with open(input_path, "r+") as input_file:
+        lines = input_file.readlines()
+
+    with open(output_path, "w+") as output_file:
+        for line in lines:
+            # Replace includes from RPCCommon to liblldbrpc includes.
+            # e.g. #include <lldb-rpc/common/RPCArgument.h> -> #include <LLDBRPC/RPCArgument.h>
+            if re.match(r".+<lldb-rpc/common", line):
+                output_file.write(re.sub(r"<lldb-rpc/common", r"<LLDBRPC", line))
+                # Replace all local file includes to liblldbrpc includes.
+                # e.g. #include "SBFoo.h" -> #include <LLDBRPC/SBFoo.h>
+            elif re.match(r'#include "(.*)"', line):
+                include_filename = re.search(r'#include "(.*)"', line).groups()[0]
+                output_file.write(
+                    re.sub(
+                        r'#include "(.*)"',
+                        r"#include <LLDBRPC/" + include_filename + ">",
+                        line,
+                    )
+                )
+            else:
+                # Write any line that doesn't need to be converted
+                output_file.write(line)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/lldb/scripts/framework-header-version-fix.py b/lldb/scripts/framework-header-version-fix.py
new file mode 100755
index 0000000000000..72185f8e820ce
--- /dev/null
+++ b/lldb/scripts/framework-header-version-fix.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+# Usage: framework-header-version-fix.py <path/to/input-header.h> <path/to/output-header.h> MAJOR MINOR PATCH
+# This script modifies lldb-rpc-defines.h to uncomment the macro defines used for the LLDB
+# major, minor and patch values as well as populating their definitions.
+
+import argparse
+import os
+import re
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input")
+    parser.add_argument("output")
+    parser.add_argument("lldb_version_major")
+    parser.add_argument("lldb_version_minor")
+    parser.add_argument("lldb_version_patch")
+    args = parser.parse_args()
+    input_path = str(args.input)
+    output_path = str(args.output)
+    lldb_version_major = args.lldb_version_major
+    lldb_version_minor = args.lldb_version_minor
+    lldb_version_patch = args.lldb_version_patch
+
+    with open(input_path, "r") as input_file:
+        lines = input_file.readlines()
+
+    with open(output_path, "w") as output_file:
+        for line in lines:
+            # Uncomment the line that defines the LLDB major version and populate its value.
+            if re.match(r"//#define LLDB_RPC_VERSION$", line):
+                output_file.write(
+                    re.sub(
+                        r"//#define LLDB_RPC_VERSION",
+                        r"#define LLDB_RPC_VERSION " + lldb_version_major,
+                        line,
+                    )
+                )
+            # Uncomment the line that defines the LLDB minor version and populate its value.
+            elif re.match(r"//#define LLDB_RPC_REVISION$", line):
+                output_file.write(
+                    re.sub(
+                        r"//#define LLDB_RPC_REVISION",
+                        r"#define LLDB_RPC_REVISION " + lldb_version_minor,
+                        line,
+                    )
+                )
+            # Uncomment the line that defines the complete LLDB version string and populate its value.
+            elif re.match(r"//#define LLDB_RPC_VERSION_STRING$", line):
+                output_file.write(
+                    re.sub(
+                        r"//#define LLDB_RPC_VERSION_STRING",
+                        r'#define LLDB_RPC_VERSION_STRING "{0}.{1}.{2}"'.format(
+                            lldb_version_major, lldb_version_minor, lldb_version_patch
+                        ),
+                        line,
+                    )
+                )
+            else:
+                # Write any line that doesn't need to be converted
+                output_file.write(line)
+
+
+if __name__ == "__main__":
+    main()

JDevlieghere · 2025-05-01T00:53:34Z

We should add shell tests for these scripts.

chelcassanova · 2025-05-01T22:00:33Z

I can add shell tests 👍🏾

bulbazord

Thanks for adding tests. Please let others take a look before merging.

lldb/test/Shell/RPC/Scripts/TestFrameworkIncludeFixScript/CheckSBClass.test

lldb/scripts/convert-lldb-header-to-rpc-header.py

lldb/scripts/framework-header-include-fix.py

lldb/scripts/framework-header-version-fix.py

lldb/test/Shell/RPC/Scripts/TestVersionFixScript/CheckLLDBDefines.test

chelcassanova · 2025-05-06T18:21:34Z

For what it's worth, while these scripts do work for intended purpose for now, an approach I'd love is to have all this header modification be a subtool of the lldb-rpc-gen tool itself. Subtooling a ClangTool is something I'm not too sure on how to do so maybe a better Clang expert can help guide the way on this? :)

chelcassanova · 2025-05-13T22:48:12Z

@DavidSpickett I pushed here to address most of the outstanding changes I needed to make from your comments, and this change should also fix the CI issue on BuildKite. Could you give this patch another pass over?

JDevlieghere · 2025-05-14T17:49:41Z

lldb/test/Shell/RPC/Scripts/TestConvertScript/CheckLLDBDefines.test

+// Copy lldb-defines.h from source.
+# RUN: mkdir -p %t/input
+# RUN: mkdir -p %t/output
+# RUN: cp %p/../../../../../include/lldb/lldb-defines.h %t/input


You're mixing two comment styles here, and lit doesn't actually treat these any differently (it just look for RUN lines). So let's settle on one and drop them everywhere else.

Suggested change

// Copy lldb-defines.h from source.

# RUN: mkdir -p %t/input

# RUN: mkdir -p %t/output

# RUN: cp %p/../../../../../include/lldb/lldb-defines.h %t/input

// Copy lldb-defines.h from source.

RUN: mkdir -p %t/input

RUN: mkdir -p %t/output

RUN: cp %p/../../../../../include/lldb/lldb-defines.h %t/input

or

Suggested change

// Copy lldb-defines.h from source.

# RUN: mkdir -p %t/input

# RUN: mkdir -p %t/output

# RUN: cp %p/../../../../../include/lldb/lldb-defines.h %t/input

# Copy lldb-defines.h from source.

RUN: mkdir -p %t/input

RUN: mkdir -p %t/output

RUN: cp %p/../../../../../include/lldb/lldb-defines.h %t/input

I think the latter is slightly more common.

Nice, I didn't know that about llvm-lit. I'll change everything to use Python-style comments and remove the hashes from the lit commands

JDevlieghere · 2025-05-14T17:51:29Z

lldb/scripts/convert-lldb-header-to-rpc-header.py

+# Usage: convert-lldb-header-to-rpc-header.py <path/to/input-header.h> <path/to/output-header.h>
+# This scripts takes common LLDB headers (such as lldb-defines.h) and replaces references to LLDB
+# with those for RPC. This happens for:
+# - namespace definitions
+# - namespace usage
+# - version string macros
+# - ifdef/ifndef lines


Python has a PEP that describes how to document files/modules. We should use that here too.

Suggested change

# Usage: convert-lldb-header-to-rpc-header.py <path/to/input-header.h> <path/to/output-header.h>

# This scripts takes common LLDB headers (such as lldb-defines.h) and replaces references to LLDB

# with those for RPC. This happens for:

# - namespace definitions

# - namespace usage

# - version string macros

# - ifdef/ifndef lines

"""

Usage: convert-lldb-header-to-rpc-header.py <path/to/input-header.h> <path/to/output-header.h>

This scripts takes common LLDB headers (such as lldb-defines.h) and replaces references to LLDB

with those for RPC. This happens for:

- namespace definitions

- namespace usage

- version string macros

- ifdef/ifndef lines

"""

JDevlieghere · 2025-05-14T17:53:39Z

lldb/test/Shell/RPC/Scripts/TestFrameworkIncludeFixScript/CheckLLDBDefines.test

+// Copy lldb-rpc-defines.h from source.
+# RUN: mkdir -p %t/input
+# RUN: mkdir -p %t/output
+# RUN: cp %p/../../../../../include/lldb/lldb-defines.h %t/input


Is the goal to run this on the sources to catch regressions, or are we just using the source file as a meaningful input? If it's the latter, I would recommend copying the file into the Inputs directory and reducing it to the bare minimum needed for the test.

If the script failed to work at all, that would be caught by the build step, right?

In which case yes the input for this should be small. It'll save us cutting down the real file to find the place that's failing.

I changed all the tests so that the files are no longer copied from source. Each test should now be using truncated versions of whatever files will be operated on and these files are in the tests' Inputs directory.

DavidSpickett

With Jonas' comments addressed I'm probably fine with this. Need to read the design doc again to be sure.

chelcassanova · 2025-05-28T17:38:22Z

I think this could be ready to land now @DavidSpickett and @JDevlieghere, could you give this another once over?

DavidSpickett

LGTM with the license comment addressed.

lldb/scripts/convert-lldb-header-to-rpc-header.py

DavidSpickett · 2025-05-29T08:07:59Z

CI failure but looks unrelated.

lldb/scripts/convert-lldb-header-to-rpc-header.py

JDevlieghere

LGTM. Might be worth making this match the other scripts and doing this once instead of line-by-line.

github-actions · 2025-06-04T16:27:03Z

✅ With the latest revision this PR passed the Python code formatter.

As part of upstreaming LLDB RPC, this commit adds a python script that is used by LLDB RPC to modify the public lldb header files for use with RPC. https://discourse.llvm.org/t/rfc-upstreaming-lldb-rpc/85804

chelcassanova · 2025-06-04T16:31:24Z

Might be worth making this match the other scripts and doing this once instead of line-by-line.

Updated the patch to do this 👍🏾

As part of upstreaming LLDB RPC, this commit adds a python script that is used by LLDB RPC to modify the public lldb header files for use with RPC. https://discourse.llvm.org/t/rfc-upstreaming-lldb-rpc/85804

As part of upstreaming LLDB RPC, this commit adds a python script that is used by LLDB RPC to modify the public lldb header files for use with RPC. https://discourse.llvm.org/t/rfc-upstreaming-lldb-rpc/85804 (cherry picked from commit a2d2941)

As part of upstreaming LLDB RPC, this commit adds a python script that is used by LLDB RPC to modify the public lldb header files for use with RPC. https://discourse.llvm.org/t/rfc-upstreaming-lldb-rpc/85804

chelcassanova requested review from DavidSpickett, bulbazord and medismailben April 30, 2025 21:28

chelcassanova requested a review from JDevlieghere as a code owner April 30, 2025 21:28

llvmbot added the lldb label Apr 30, 2025

chelcassanova force-pushed the upstream-lldb-rpc/add-python-scripts-for-headers branch from c915f7c to c51a312 Compare May 2, 2025 18:46

bulbazord approved these changes May 2, 2025

View reviewed changes

chelcassanova mentioned this pull request May 5, 2025

[lldb] Upstream lldb-rpc-gen and LLDB RPC server-side emitters #136748

Closed

DavidSpickett reviewed May 6, 2025

View reviewed changes

lldb/test/Shell/RPC/Scripts/TestFrameworkIncludeFixScript/CheckSBClass.test Outdated Show resolved Hide resolved

DavidSpickett reviewed May 6, 2025

View reviewed changes

chelcassanova mentioned this pull request May 13, 2025

[lldb][RPC] Upstream lldb-rpc-gen tool #138031

Open

chelcassanova force-pushed the upstream-lldb-rpc/add-python-scripts-for-headers branch from c51a312 to a00c76d Compare May 13, 2025 22:39

chelcassanova closed this May 13, 2025

chelcassanova reopened this May 13, 2025

JDevlieghere reviewed May 14, 2025

View reviewed changes

DavidSpickett reviewed May 21, 2025

View reviewed changes

chelcassanova force-pushed the upstream-lldb-rpc/add-python-scripts-for-headers branch 4 times, most recently from 01e35b2 to 2b6f460 Compare May 27, 2025 19:34

chelcassanova force-pushed the upstream-lldb-rpc/add-python-scripts-for-headers branch from 2b6f460 to 4520d08 Compare May 28, 2025 21:13

chelcassanova changed the title ~~[lldb][RPC] Upstream Python scripts~~ [lldb][RPC] Upstream LLDB to RPC converstion Python script May 28, 2025

DavidSpickett approved these changes May 29, 2025

View reviewed changes

lldb/scripts/convert-lldb-header-to-rpc-header.py Show resolved Hide resolved

JDevlieghere reviewed May 29, 2025

View reviewed changes

lldb/scripts/convert-lldb-header-to-rpc-header.py Outdated Show resolved Hide resolved

chelcassanova force-pushed the upstream-lldb-rpc/add-python-scripts-for-headers branch from 4520d08 to 208288f Compare May 30, 2025 22:46

JDevlieghere approved these changes Jun 3, 2025

View reviewed changes

chelcassanova force-pushed the upstream-lldb-rpc/add-python-scripts-for-headers branch from 208288f to 4f818cc Compare June 4, 2025 16:24

[lldb][RPC] Upstream LLDB to RPC converstion Python script

ba96f13

As part of upstreaming LLDB RPC, this commit adds a python script that is used by LLDB RPC to modify the public lldb header files for use with RPC. https://discourse.llvm.org/t/rfc-upstreaming-lldb-rpc/85804

chelcassanova force-pushed the upstream-lldb-rpc/add-python-scripts-for-headers branch from 4f818cc to ba96f13 Compare June 4, 2025 16:30

chelcassanova requested a review from JDevlieghere June 10, 2025 17:15

JDevlieghere approved these changes Jun 11, 2025

View reviewed changes

chelcassanova merged commit a2d2941 into llvm:main Jun 11, 2025
10 checks passed

arjunUpatel mentioned this pull request Jun 24, 2025

[lldb] Revive TestSimulatorPlatform.py (#142244) arjunUpatel/llvm-project#1

Closed

[lldb][RPC] Upstream LLDB to RPC converstion Python script #138028

[lldb][RPC] Upstream LLDB to RPC converstion Python script #138028

Uh oh!

Conversation

chelcassanova commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Apr 30, 2025

Uh oh!

JDevlieghere commented May 1, 2025

Uh oh!

chelcassanova commented May 1, 2025

Uh oh!

bulbazord left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chelcassanova commented May 6, 2025

Uh oh!

chelcassanova commented May 13, 2025

Uh oh!

JDevlieghere May 14, 2025

Choose a reason for hiding this comment

Uh oh!

chelcassanova May 22, 2025

Choose a reason for hiding this comment

Uh oh!

JDevlieghere May 14, 2025

Choose a reason for hiding this comment

Uh oh!

JDevlieghere May 14, 2025

Choose a reason for hiding this comment

Uh oh!

DavidSpickett May 21, 2025

Choose a reason for hiding this comment

Uh oh!

chelcassanova May 28, 2025

Choose a reason for hiding this comment

Uh oh!

DavidSpickett left a comment

Choose a reason for hiding this comment

Uh oh!

chelcassanova commented May 28, 2025

Uh oh!

DavidSpickett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DavidSpickett commented May 29, 2025

Uh oh!

Uh oh!

JDevlieghere left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chelcassanova commented Jun 4, 2025

Uh oh!

Uh oh!

Uh oh!

chelcassanova commented Apr 30, 2025 •

edited

Loading

github-actions bot commented Jun 4, 2025 •

edited

Loading