Add new option in debug tool to get a histogram of key and value sizes. by martinmr · Pull Request #2844 · dgraph-io/dgraph

martinmr · 2018-12-25T02:41:15Z

The new option can be accessed via the --histogram flag. It will print a
histogram of the key and value sizes, as well as the min, max, mean and
standard deviation of their distributions.

This change is

CLAassistant · 2018-12-25T02:41:21Z

All committers have signed the CLA.

martinmr · 2018-12-25T02:42:43Z

Example output below (fyi, it looks justified in the terminal but not when I pasted it here):

Opening DB: ./p
badger2018/12/25 02:41:37 INFO: Replaying file id: 2 at offset: 60334520
badger2018/12/25 02:41:37 INFO: Replay took: 21.41µs
prefix =
Found 1036362 keys

Histogram of key sizes (in bytes)
Min value: 6
Max value: 33
Mean: 23.51
Standard deviation: 5.69
Range Count
[ 0, 32) 910747
[ 32, 64) 125615

Histogram of value sizes (in bytes)
Min value: 0
Max value: 2432
Mean: 37.71
Standard deviation: 45.32
Range Count
[ 0, 32) 635819
[ 32, 64) 261183
[ 64, 128) 131757
[ 128, 256) 5804
[ 256, 512) 850
[ 512, 1024) 690
[ 1024, 2048) 250
[ 2048, 4096) 9

manishrjain

Reviewable status: 0 of 1 files reviewed, 8 unresolved discussions (waiting on @martinmr and @manishrjain)

dgraph/cmd/debug/run.go, line 82 at r1 (raw file):

	flag.BoolVarP(&opt.keyHistory, "history", "y", false, "Show all versions of a key.")
	flag.StringVarP(&opt.pdir, "postings", "p", "", "Directory where posting lists are stored.")
	flag.BoolVar(&opt.sizeHistogram, "histogram", false, "Show a histogram of the key and value sizes.")

100 chars.

dgraph/cmd/debug/run.go, line 530 at r1 (raw file):

// Return a new instance of HistogramData with properly initialized fields.
func NewHistogramData(bounds []float64) *HistogramData {
	histogram := new(HistogramData)

Can also be written as:

return &HistogramData{
Bounds: bounds,
Min: math.MaxInt64,
}

Slightly easier to read.

dgraph/cmd/debug/run.go, line 539 at r1 (raw file):

// Update the Min and Max fields if value is less than or greater than the
// current values.
func (histogram *HistogramData) UpdateMinMax(value int64) {

Also, find the right bucket, and +1 it.

dgraph/cmd/debug/run.go, line 555 at r1 (raw file):

// Simple exporter function that copies the view data to the exporter.
func (exporter HistogramExporter) ExportView(viewData *view.Data) {

Remove all this.

dgraph/cmd/debug/run.go, line 611 at r1 (raw file):

func sizeHistogram(db *badger.DB) {
	txn := db.NewTransactionAt(math.MaxUint64, false)

Use opt.readTs instead of maxuint64.

dgraph/cmd/debug/run.go, line 628 at r1 (raw file):

	keySizeMeasure := stats.Int64("debug/key_sizes",
		"The size of the keys in bytes", "1")
	keySizeView := &view.View{

Get rid of the opencensus stuff.

dgraph/cmd/debug/run.go, line 673 at r1 (raw file):

		keySize := int64(len(item.Key()))
		valueSize := item.ValueSize()
		keySizeHistogram.UpdateMinMax(keySize)

.Update

dgraph/cmd/debug/run.go, line 684 at r1 (raw file):

	// printing.
	sleepDuration, _ := time.ParseDuration("500ms")
	time.Sleep(sleepDuration)

time.Sleep(500*time.Millisecond), or better still time.Sleep(time.Second).

golangcibot · 2018-12-27T00:59:06Z

+var (
+	Debug             x.SubCommand
+	opt               flagOptions
+	keySizeViewName   = "debug/key_size_histogram"


keySizeViewName is unused (from varcheck)

golangcibot · 2018-12-27T00:59:07Z

+	Debug             x.SubCommand
+	opt               flagOptions
+	keySizeViewName   = "debug/key_size_histogram"
+	valueSizeViewName = "debug/value_size_histogram"


valueSizeViewName is unused (from varcheck)

martinmr

Reviewable status: 0 of 1 files reviewed, 10 unresolved discussions (waiting on @manishrjain and @golangcibot)

dgraph/cmd/debug/run.go, line 82 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

100 chars.

Done.

dgraph/cmd/debug/run.go, line 530 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Can also be written as:

return &HistogramData{
Bounds: bounds,
Min: math.MaxInt64,
}

Slightly easier to read.

Done.

dgraph/cmd/debug/run.go, line 539 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Also, find the right bucket, and +1 it.

Done.

dgraph/cmd/debug/run.go, line 555 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Remove all this.

Done.

dgraph/cmd/debug/run.go, line 611 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Use opt.readTs instead of maxuint64.

Done.

dgraph/cmd/debug/run.go, line 628 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Get rid of the opencensus stuff.

Done.

dgraph/cmd/debug/run.go, line 673 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

.Update

Done.

dgraph/cmd/debug/run.go, line 684 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

time.Sleep(500*time.Millisecond), or better still time.Sleep(time.Second).

Done . Got rid of all the opencensus stuff so this is no longer needed.

dgraph/cmd/debug/run.go, line 43 at r2 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

keySizeViewName is unused (from varcheck)

Done.

dgraph/cmd/debug/run.go, line 44 at r2 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

valueSizeViewName is unused (from varcheck)

Done.

golangcibot · 2018-12-27T01:07:08Z

-
-var opt flagOptions
+var (
+	Debug             x.SubCommand


File is not goimports-ed (from goimports)

martinmr · 2018-12-27T01:17:51Z

I compared the output with the previous version using opencensus to verify that they are the same.

martinmr

Reviewable status: 0 of 1 files reviewed, 11 unresolved discussions (waiting on @manishrjain and @golangcibot)

dgraph/cmd/debug/run.go, line 41 at r3 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

File is not goimports-ed (from goimports)

This was due to a formatting issue. I formatted the file and the warning went away.

martinmr

Reviewable status: 0 of 1 files reviewed, 11 unresolved discussions (waiting on @manishrjain and @golangcibot)

dgraph/cmd/debug/run.go, line 41 at r3 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…

This was due to a formatting issue. I formatted the file and the warning went away.

Done.

The new option can be accessed via the --histogram flag. It will print a histogram of the key and value sizes, as well as the min, max, mean and standard deviation of their distributions.

manishrjain

Looks good. Got a few comments. Address them and feel free to merge.

Reviewed 1 of 1 files at r5.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @manishrjain, @golangcibot, and @martinmr)

dgraph/cmd/debug/run.go, line 562 at r5 (raw file):

	fmt.Printf("Min value: %d\n", histogram.Min)
	fmt.Printf("Max value: %d\n", histogram.Max)
	fmt.Printf("Mean: %.2f\n", histogram.Mean)

calculate the mean here.

dgraph/cmd/debug/run.go, line 617 at r5 (raw file):

		item := itr.Item()

		keySize := int64(len(item.Key()))

keySize := ...
keyHistogram.Update(keySize)

valSize := ...
valHistogram.Update(valSize)

Define the variable as close as possible to usage.

dgraph/cmd/debug/run.go, line 620 at r5 (raw file):

		valueSize := item.ValueSize()
		keySizeHistogram.Update(keySize)
		valueSizeHistogram.Update(valueSize)

valueSizeHistogra

martinmr

Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @manishrjain and @golangcibot)

dgraph/cmd/debug/run.go, line 562 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

calculate the mean here.

Done.

dgraph/cmd/debug/run.go, line 617 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

keySize := ...
keyHistogram.Update(keySize)

valSize := ...
valHistogram.Update(valSize)

Define the variable as close as possible to usage.

Done.

dgraph/cmd/debug/run.go, line 620 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

valueSizeHistogra

Done.

…s. (dgraph-io#2844) The new option can be accessed via the --histogram flag. It will print a histogram of the key and value sizes, as well as the min, max, mean and standard deviation of their distributions.

martinmr requested a review from manishrjain December 25, 2018 02:41

manishrjain suggested changes Dec 26, 2018

View reviewed changes

golangcibot reviewed Dec 27, 2018

View reviewed changes

martinmr commented Dec 27, 2018

View reviewed changes

golangcibot reviewed Dec 27, 2018

View reviewed changes

martinmr commented Dec 27, 2018

View reviewed changes

Add new option in debug tool to get a histogram of key and value sizes.

dc9d08e

The new option can be accessed via the --histogram flag. It will print a histogram of the key and value sizes, as well as the min, max, mean and standard deviation of their distributions.

martinmr force-pushed the martinmr/histogram branch from 248b9c0 to 7550e06 Compare December 27, 2018 19:05

Address review comments.

f9c57b4

martinmr force-pushed the martinmr/histogram branch from 7550e06 to f9c57b4 Compare December 27, 2018 22:56

manishrjain approved these changes Dec 27, 2018

View reviewed changes

Address more review comments

13c2709

martinmr commented Dec 27, 2018

View reviewed changes

martinmr merged commit 320fb1c into master Dec 27, 2018

martinmr deleted the martinmr/histogram branch December 27, 2018 23:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new option in debug tool to get a histogram of key and value sizes.#2844

Add new option in debug tool to get a histogram of key and value sizes.#2844
martinmr merged 3 commits intomasterfrom
martinmr/histogram

martinmr commented Dec 25, 2018 •

edited by manishrjain

Loading

Uh oh!

CLAassistant commented Dec 25, 2018 •

edited

Loading

Uh oh!

martinmr commented Dec 25, 2018

Uh oh!

manishrjain left a comment

Uh oh!

golangcibot Dec 27, 2018

Uh oh!

golangcibot Dec 27, 2018

Uh oh!

martinmr left a comment

Uh oh!

golangcibot Dec 27, 2018

Uh oh!

martinmr commented Dec 27, 2018

Uh oh!

martinmr left a comment

Uh oh!

martinmr left a comment

Uh oh!

manishrjain left a comment

Uh oh!

martinmr left a comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Conversation

martinmr commented Dec 25, 2018 • edited by manishrjain Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Dec 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martinmr commented Dec 25, 2018

Uh oh!

manishrjain left a comment

Choose a reason for hiding this comment

Uh oh!

golangcibot Dec 27, 2018

Choose a reason for hiding this comment

Uh oh!

golangcibot Dec 27, 2018

Choose a reason for hiding this comment

Uh oh!

martinmr left a comment

Choose a reason for hiding this comment

Uh oh!

golangcibot Dec 27, 2018

Choose a reason for hiding this comment

Uh oh!

martinmr commented Dec 27, 2018

Uh oh!

martinmr left a comment

Choose a reason for hiding this comment

Uh oh!

martinmr left a comment

Choose a reason for hiding this comment

Uh oh!

manishrjain left a comment

Choose a reason for hiding this comment

Uh oh!

martinmr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

martinmr commented Dec 25, 2018 •

edited by manishrjain

Loading

CLAassistant commented Dec 25, 2018 •

edited

Loading