Add the support of ANALYZE command to inspect the performance of RocksDB #1943

chrisxu333 · 2023-12-18T15:01:02Z

This pr tries to close #783 .

To expose essential perf_context and iostats_context from rocksdb, a new command ANALYZE is introduced.

A typical usage would be as follows:

src/commands/cmd_server.cc

mapleFU

This is a bit confusing because it does far less than PROFILE in SQL. Also it might break the previous config about perflevel, so I wonder whether we have better way for this...

src/commands/cmd_server.cc

mapleFU · 2023-12-19T04:26:00Z

Maybe it's better to first having a RAII guard like TiKV: https://github.com/tikv/tikv/blob/master/src/coprocessor/tracker.rs ( We can be easier than them) ?

chrisxu333 · 2023-12-19T06:09:23Z

Maybe it's better to first having a RAII guard like TiKV: https://github.com/tikv/tikv/blob/master/src/coprocessor/tracker.rs ( We can be easier than them) ?

Can you elaborate more on this? I didn't quite get the point.

src/commands/cmd_server.cc

src/server/redis_reply.cc

mapleFU · 2023-12-20T03:28:59Z

Maybe it's better to first having a RAII guard like TiKV: https://github.com/tikv/tikv/blob/master/src/coprocessor/tracker.rs ( We can be easier than them) ?

Can you elaborate more on this? I didn't quite get the point.

Hmm we can just leave it here in this patch. I believe something like a :

class StoragePerfContext {
  
  void close();
  ~StoragePerfContext() {
      if (!reset_) {
         rocksdb::Perfcontext // resume
      }
   }
private:
  Storage* // or rocksdb?
  bool reseted_ = false;
};

Might be better? If other command want to debugging, they can also uses this tool? Also we can considering RAII here.

chrisxu333 · 2023-12-21T07:05:41Z

Maybe it's better to first having a RAII guard like TiKV: https://github.com/tikv/tikv/blob/master/src/coprocessor/tracker.rs ( We can be easier than them) ?

Can you elaborate more on this? I didn't quite get the point.

Hmm we can just leave it here in this patch. I believe something like a :
class StoragePerfContext {
  
  void close();
  ~StoragePerfContext() {
      if (!reset_) {
         rocksdb::Perfcontext // resume
      }
   }
private:
  Storage* // or rocksdb?
  bool reseted_ = false;
};
Might be better? If other command want to debugging, they can also uses this tool? Also we can considering RAII here.

I see what you meant. So just to be clear, do we still need to do this in this patch? Or we wait until when it's actually needed to do so?

mapleFU · 2023-12-21T07:17:50Z

do we still need to do this in this patch

I agree it's not neccessary in this patch. Personally I think metrics can be widely used, but lets first get it in used...

cc @PragmaTwice @git-hulk This command looks like SQL "PROFILE" with rocksdb io/perf stats enabled and output. It can be used in some cases, but for some blocking command I don't know whether it's ok. Would you mind take a look?

chrisxu333 · 2023-12-25T03:34:27Z

do we still need to do this in this patch

I agree it's not neccessary in this patch. Personally I think metrics can be widely used, but lets first get it in used...

cc @PragmaTwice @git-hulk This command looks like SQL "PROFILE" with rocksdb io/perf stats enabled and output. It can be used in some cases, but for some blocking command I don't know whether it's ok. Would you mind take a look?

@git-hulk @PragmaTwice Hello guys, if you have time could you help review this pr? Thank you :)

git-hulk · 2023-12-25T03:58:50Z

Sure, sorry for missing this message. I will take a look recently.

src/commands/cmd_server.cc

git-hulk · 2023-12-25T07:52:12Z

@chrisxu333 Can rename true_args_ to command_args_, rest are good to me. Sorry for missing this point.

mapleFU · 2023-12-25T10:14:21Z

Would you mind fix the lint first?

torwig · 2023-12-25T10:36:56Z

@chrisxu333 ./x.py format should do the trick.

chrisxu333 · 2023-12-25T11:06:00Z

Would you mind fix the lint first?

Sounds good I just did.

mapleFU · 2023-12-25T11:09:09Z

src/commands/cmd_server.cc

+
+    auto s = cmd->Parse(command_args_);


Should we check the arity before Parse? Since wrong command_args_ might make our program crash

We need to reuse the code in ExecuteCommand rather than write them again here.

I just pushed a newer version which added the arity checking logic. As for reusing / refactoring the code in ExecuteCommand, maybe it's better to do it in a separate pr? As the ExecuteCommand code needs to be dissected into several chunks for reusing purpose. @mapleFU @PragmaTwice

mapleFU · 2023-12-26T07:22:55Z

src/commands/cmd_server.cc

+    for (int i = 1; i < args.size(); ++i) {
+      command_args_.push_back(args[i]);
+    }


nit: we can directly insert rather than push back

chrisxu333 · 2023-12-27T03:15:26Z

@chrisxu333 ./x.py format should do the trick.

Hi @torwig, could you help review this pr whenever you're able to? Thx!

chrisxu333 · 2024-01-02T01:28:11Z

Hi @git-hulk @PragmaTwice,

Could you help review and approve this pr if possible? Thx!

git-hulk · 2024-01-02T01:32:35Z

@chrisxu333 Thanks for your follow-up. This PR looks good to me, to see if @PragmaTwice and @torwig has any comments, if not, we can move forward to merge.

sonarqubecloud · 2024-01-02T06:32:10Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

1 New issue
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

torwig reviewed Dec 18, 2023

View reviewed changes

src/commands/cmd_server.cc Outdated Show resolved Hide resolved

src/commands/cmd_server.cc Outdated Show resolved Hide resolved

mapleFU reviewed Dec 18, 2023

View reviewed changes

src/commands/cmd_server.cc Outdated Show resolved Hide resolved

src/commands/cmd_server.cc Show resolved Hide resolved

src/commands/cmd_server.cc Outdated Show resolved Hide resolved

mapleFU reviewed Dec 20, 2023

View reviewed changes

src/commands/cmd_server.cc Outdated Show resolved Hide resolved

src/server/redis_reply.cc Outdated Show resolved Hide resolved

chrisxu333 force-pushed the expose_perf_io_context branch from 84c4794 to b71c53d Compare December 21, 2023 07:03

git-hulk reviewed Dec 25, 2023

View reviewed changes

src/commands/cmd_server.cc Outdated Show resolved Hide resolved

chrisxu333 force-pushed the expose_perf_io_context branch from b71c53d to cb470c0 Compare December 25, 2023 06:51

git-hulk changed the title ~~expose rocksdb perf/iostat context with ANALYZE command~~ Add the support of ANALYZE command to inspect the performance of RocksDB Dec 25, 2023

chrisxu333 force-pushed the expose_perf_io_context branch 2 times, most recently from ce3220b to 614672d Compare December 25, 2023 09:45

git-hulk previously approved these changes Dec 25, 2023

View reviewed changes

git-hulk requested review from mapleFU and torwig December 25, 2023 10:00

chrisxu333 dismissed git-hulk’s stale review via 25392d3 December 25, 2023 11:05

chrisxu333 force-pushed the expose_perf_io_context branch from 614672d to 25392d3 Compare December 25, 2023 11:05

mapleFU reviewed Dec 25, 2023

View reviewed changes

chrisxu333 force-pushed the expose_perf_io_context branch from 25392d3 to 682777c Compare December 26, 2023 06:48

mapleFU approved these changes Dec 26, 2023

View reviewed changes

expose rocksdb perf/iostat context with ANALYZE command

46957db

chrisxu333 force-pushed the expose_perf_io_context branch from 682777c to 46957db Compare December 27, 2023 03:14

git-hulk approved these changes Jan 2, 2024

View reviewed changes

Merge branch 'unstable' into expose_perf_io_context

25778af

PragmaTwice approved these changes Jan 2, 2024

View reviewed changes

git-hulk merged commit 6b9dcd1 into apache:unstable Jan 2, 2024

Add the support of ANALYZE command to inspect the performance of RocksDB #1943

Add the support of ANALYZE command to inspect the performance of RocksDB #1943

Uh oh!

Conversation

chrisxu333 commented Dec 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mapleFU commented Dec 19, 2023

Uh oh!

chrisxu333 commented Dec 19, 2023

Uh oh!

Uh oh!

Uh oh!

mapleFU commented Dec 20, 2023

Uh oh!

chrisxu333 commented Dec 21, 2023

Uh oh!

mapleFU commented Dec 21, 2023

Uh oh!

chrisxu333 commented Dec 25, 2023

Uh oh!

git-hulk commented Dec 25, 2023

Uh oh!

Uh oh!

git-hulk commented Dec 25, 2023

Uh oh!

mapleFU commented Dec 25, 2023

Uh oh!

torwig commented Dec 25, 2023

Uh oh!

chrisxu333 commented Dec 25, 2023

Uh oh!

mapleFU Dec 25, 2023

Choose a reason for hiding this comment

Uh oh!

PragmaTwice Dec 25, 2023

Choose a reason for hiding this comment

Uh oh!

chrisxu333 Dec 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU Dec 26, 2023

Choose a reason for hiding this comment

Uh oh!

chrisxu333 commented Dec 27, 2023

Uh oh!

chrisxu333 commented Jan 2, 2024

Uh oh!

git-hulk commented Jan 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Jan 2, 2024

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chrisxu333 commented Dec 18, 2023 •

edited

Loading

chrisxu333 Dec 26, 2023 •

edited

Loading

git-hulk commented Jan 2, 2024 •

edited

Loading