Start a test suite

[ehmatthes]

Focus on end to end tests.

• Get project running locally.
• Install pytest.
• Test three main subcommands: log, status, merge.
  • log
  • status
  • merge
• Show that a change makes a test fail.
• Share for feedback.

[ehmatthes]

Test log command

• Run git-sim log in test environment.
• Compare test file to reference file.
• Suppress auto-open of output.

Note: Test takes 25.3s, most of which is setup.

[ehmatthes]

Test status

• Write test for status, which should fail.
  • Added a .gitignore file to ignore .venv/ in sample_repo, otherwise git-sim seems to be walking entire dir.
• Copy reference file.
• Test should pass.

Note: Tests take 28.4s.

[ehmatthes]

Test merge

• Write test for merge, which should fail.
• Copy reference file.
• Test should pass.

Note: Tests take 38.9s; takes 30.2s without the -s flag.

[ehmatthes]

Next

• Clean
  • Refactoring
  • Comments
• Additional tests (No, probably better for project owner to write the next tests.) (This is why I got a Studio. Tests run faster in a Windows VM on the Studio than they do on my older intel MacBook.)
• Ensure this runs on Windows as well. (Should just be some cross-platform code to clean up.)
• Ensure aligns with main project's development workflow. ie Docker vs venv?

[ehmatthes]

Speed (slow) issue

• Where is the time spent?
  • Environment setup?
  • Individual tests?
• How can environment setup be made more efficient?
  • Vendoring?
  • Simplify environment?
  • Reuse environment?
  • Container vs venv?

Working notes

• 0.2s sleep happens before tmp_dir is generated; can probably be removed.

• Wherever this goes, make sure that a test will actually fail.

  • ie Add a new commit, make sure log test fails. (reset --hard, failure works.)

• Fix: use existing development venv from the main project.

[ehmatthes]

Tests pass on M1 Mac, fail on intel Mac

• This is probably due to slight compression differences in jpg generation?
• File on M1 is 127626 bytes, Intel is 128091 bytes.
• Try using a png as the test image.

Ugh, image comparison.

• Even png does not guarantee identical output across systems.
• Using a fairly simply numpy pixel comparison with a threshold seems to work.
• All file comparison work is in one function, which can easily be updated.
• Tests do pass on both intel and m1 Macs, and they fail when expected to as well.

[ehmatthes]

Much better now, using existing venv from project.

• Consider writing test_cmd = git-sim log, then replacing git-sim with test git sim command, and adding in flags required just for testing. This lets us write the exact commands we want to test.

[ehmatthes]

Submit PR

• Add brief documentation.
• Squash commits?
• Submit & write up PR.

Notes

• Files look different on Windows! May need to compare against a group of reference files, and only fail if the generated image doesn't match any of them closely enough.

[ehmatthes]

First review

• Use git-dummy instead of copying sample repo to tmp dir.
• Remove sample repo.
• Make sure no unnecessary imports are left in modules.
• Update docs. (not needed)
• Merge use_gitdummy into start_e2e_testing and push; should update PR.
• Note in PR: git-dummy calls are not as fast as copying a static repo; probably wouldn't want one per test function!

Working notes

• Upgrade git-dummy.
• Generate git_sim_path in a utils function.
• Use a get_venv_path() utils function for git-dummy path, and git-sim path.

[ehmatthes]

Merged.