Automate test harness and integrate into build process

[binarylogic]

The Vector test harness is a blackbox test suite that allows for full end-to-end integration testing on log forwarders/routers (not just Vector). It allows for 3 types of tests:

1. Ephemeral performance tests
2. Ephemeral correctness tests
3. Long-running reliability tests

1 and 2 should are finite tests, 3 runs indefinitely. Each time Vector is released we should test it on this suite and compare it to previous versions. Keep in mind, test data is persisted on S3, and standard Athena SQL queries can be run to compare results. This is what the compare and cohort commands do.

Prior art

1. Option::{expect,unwrap} and Result::{expect, expect_err, unwrap, unwrap_err} have #[track_caller] rust-lang/rust#67887 (comment)

Design

1. I'm open to the design of this features. We use CircleCI for Vector itself, but I would not be oppsed to experimenting with Github actions if we find it to be easier. Especially if it's easier to comment on pull requests, etc.
2. You have liberty to change anything in the vector-test-harness repo. I originally built that and I am not an operations expert.

Requirements

• Ability to manually test new Vector versions, ideally within a pull request. We would then wait to merge the PR until we have manually verified that there are no regressions.
• Bonus points if we can post the results as a comment on the PR or in a Slack message.

[LucioFranco]

I was chatting with eliza about what buoyant is doing with linkerd's benchmarks. They are still in the early stages of designing what they CI benches will look like. One thing she suggested was using gh action that triggers a bench build on a bare metal machine. This would ensure the least amount of noisiness.

I wonder if we could capture https://help.github.com/en/actions/automating-your-workflow-with-github-actions/events-that-trigger-workflows#issue-comment-event-issue_comment events then use something like /bench to trigger a remote bench job that will run then report back in a new comment the results of this branch vs master. So this would allow us to then trigger this job either via comment or a prep PR for a release etc.

[MOZGIII]

My understanding is with Github actions it's super easy, cause we can:

• run a custom docker image - and we'll just wrap the test harness itself as a docker image
• react to events like comments in the repo - and it's super easy too
• since we'll be executing the code at the context of a comment hook - we can trivially respond to the comment with the whatever output our command generates

So, I'd just go with Github actions if everyone's ok with this.

Based on the requirements, we might not even need this to run automatically on a PR - just to have a manual trigger by a comment with something like /bench like @LucioFranco wrote.

[binarylogic]

I am ok with that.

[MOZGIII]

I created a tracking issue at vector-test-harness repo - vectordotdev/vector-test-harness#14 - check it out! Also, let's move the Github Action specific automation implementation there.