Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .eslintrc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

"ignorePatterns": [
"dist/",
"benchmark/",
],

"rules": {
Expand Down
65 changes: 65 additions & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: Performance Benchmark

on:
push:
branches: [main]
pull_request:
branches: [main]

jobs:
matrix:
runs-on: ubuntu-latest
outputs:
latest: ${{ steps.set-matrix.outputs.requireds }}
nonlatest: ${{ steps.set-matrix.outputs.optionals }}
steps:
- uses: ljharb/actions/node/matrix@main
id: set-matrix
with:
versionsAsRoot: true
type: majors
preset: '>= 0.8'

benchmark:
needs: [matrix]
name: Run Performance Benchmarks
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
node-version: ${{ fromJson(needs.matrix.outputs.latest) }}

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Install dependencies
uses: ljharb/actions/node/install@main
with:
node-version: ${{ matrix.node-version }}

- name: Install benchmark dependencies
run: cd benchmark && npm install

# For PRs: Run benchmarks and compare against committed baseline
- name: Run benchmarks and compare against baseline
if: ${{ github.event_name == 'pull_request' }}
run: |
echo "==== Running benchmarks and comparing against baseline ===="
# Run benchmarks using the compare script that uses the committed baseline
npm run benchmark:compare

# Print the Node.js version for debugging
echo "Benchmark completed on Node.js version: $(node --version)"

# For pushes to main: Run benchmarks against the committed baseline
- name: Run benchmarks on main branch
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: |
# Run benchmarks against the committed baseline
npm run benchmark:compare

# Print the Node.js version for debugging
echo "Benchmark completed on Node.js version: $(node --version)"
206 changes: 206 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
# QS Library Benchmarking Suite

This directory contains a comprehensive benchmarking infrastructure for the `qs` library, designed to measure performance across different Node.js versions and detect performance regressions.

## Features

- 🚀 **Comprehensive Test Coverage**: Parse and stringify operations across various scenarios
- 📊 **Cross-Node Version Testing**: Automated testing from Node.js 0.8 through current versions
- 📈 **Regression Detection**: Compare performance against baselines
- 🔄 **CI Integration**: Automated benchmarks on every PR and push
- 📋 **Detailed Reporting**: Visual tables with performance metrics
- 💾 **Historical Tracking**: Track performance trends over time

## Quick Start

### Install Dependencies
```bash
cd benchmark
npm install
```

### Run All Benchmarks
```bash
# Run all benchmarks
node index.js

# Run only parse benchmarks
node index.js --parse

# Run only stringify benchmarks
node index.js --stringify
```

### Compare with Baseline
```bash
# Save current results as baseline
node index.js --save-baseline

# Compare against baseline
node index.js --baseline baseline.json
```

## Test Scenarios

### Parse Benchmarks
- **Simple queries**: `a=1&b=2&c=3`
- **Nested objects**: `user[profile][name]=John`
- **Arrays**: `colors[]=red&colors[]=green`
- **Real-world cases**: E-commerce, forms, APIs
- **Edge cases**: Empty values, special characters
- **Different options**: `allowDots`, `comma`, `depth`

### Stringify Benchmarks
- **Simple objects**: `{a: '1', b: '2'}`
- **Nested objects**: `{user: {name: 'John'}}`
- **Arrays**: `{colors: ['red', 'green']}`
- **Array formats**: indices, brackets, repeat, comma
- **Options**: `allowDots`, encoding, delimiters
- **Edge cases**: null values, empty arrays

## Performance Metrics

Each benchmark reports:
- **Operations per second (Hz)**: Primary performance metric
- **Relative margin of error (RME)**: Statistical reliability
- **Sample size**: Number of test runs
- **Fastest/Slowest indicators**: Comparative performance

## Example Output

```
📊 Benchmark Results

┌──────────────────────────────┬──────────────┬──────────┬─────────┬────────────┐
│ Test Name │ Ops/sec │ RME │ Samples │ Status │
├──────────────────────────────┼──────────────┼──────────┼─────────┼────────────┤
│ Parse simple tiny query │ 2,451,234 │ ±1.23% │ 89 │ 🚀 Fastest │
│ Parse simple small query │ 1,876,543 │ ±0.87% │ 92 │ │
│ Parse nested object │ 923,456 │ ±1.45% │ 85 │ │
│ Parse complex array │ 654,321 │ ±2.12% │ 78 │ 🐌 Slowest │
└──────────────────────────────┴──────────────┴──────────┴─────────┴────────────┘
```

## Baseline Comparison

When comparing against a baseline:

```
📈 Performance Comparison

┌──────────────────────────────┬─────────────┬─────────────┬─────────┬─────────────┐
│ Test Name │ Current │ Baseline │ Change │ Status │
├──────────────────────────────┼─────────────┼─────────────┼─────────┼─────────────┤
│ Parse simple query │ 2,451,234 │ 2,234,567 │ +9.70% │ ✅ Improved │
│ Parse nested object │ 923,456 │ 934,123 │ -1.14% │ ➖ No change│
│ Parse complex array │ 654,321 │ 724,891 │ -9.73% │ ⚠️ Regression│
└──────────────────────────────┴─────────────┴─────────────┴─────────┴─────────────┘
```

## Command Line Options

```bash
node benchmark/index.js [options]

Options:
--parse Run only parse benchmarks
--stringify Run only stringify benchmarks
--baseline FILE Compare results against baseline file
--save-baseline Save current results as baseline
--no-summary Skip summary report
--help Show help

Environment Variables:
STRESS_TEST=1 Include stress test scenarios
```

## Adding New Benchmarks

### 1. Add Test Data
Edit `fixtures.js` to add new test scenarios:

```javascript
const newScenarios = {
myNewTest: 'complex=query&string=here'
};
```

### 2. Add Benchmark
In `parse.js` or `stringify.js`:

```javascript
runner.add('My new test case', () => {
qs.parse(fixtures.newScenarios.myNewTest);
});
```

### 3. Test Locally
```bash
node index.js --parse
```

## Performance Guidelines

### What to Benchmark
- ✅ Core functionality (parse/stringify)
- ✅ Common real-world scenarios
- ✅ Edge cases that might be slow
- ✅ Different option combinations
- ✅ Large inputs (stress tests)

### What NOT to Benchmark
- ❌ Trivial operations
- ❌ Error conditions (use unit tests)
- ❌ Platform-specific features
- ❌ Operations that vary by environment

### Interpreting Results

- **5%+ improvement**: Significant performance gain ✅
- **±5% change**: No meaningful change ➖
- **5%+ regression**: Potential performance issue ⚠️

## Troubleshooting

### Inconsistent Results
- Ensure system is not under load
- Run multiple times and average
- Check for background processes

### Memory Issues
- Use `--max-old-space-size=4096` for large tests
- Monitor memory usage during stress tests

### CI Failures
- Check Node version compatibility
- Verify all dependencies are installed
- Review benchmark timeout settings

## File Structure

```
benchmark/
├── index.js # Main benchmark runner
├── runner.js # Benchmark infrastructure
├── fixtures.js # Test data generators
├── parse.js # Parse function benchmarks
├── stringify.js # Stringify function benchmarks
├── package.json # Benchmark dependencies
├── README.md # This documentation
└── results/ # Generated benchmark results
├── benchmark-*.json
└── baseline.json
```

## Contributing

When adding performance optimizations:

1. **Run baseline first**: `node index.js --save-baseline`
2. **Make your changes**: Implement optimization
3. **Run comparison**: `node index.js --baseline baseline.json`
4. **Verify improvements**: Check for positive performance changes
5. **Include stress tests**: `STRESS_TEST=1 node index.js`

This ensures your optimization actually improves performance and doesn't introduce regressions.

Loading