[EXPERIMENTAL] Autogenerated unittests - evaluating usability

[JanKrivanek]

Context

Generated with very early preview of unit tests generation tool. Trying to evaluate if there is any hope of that being useful and what would need to be worked on to achieve usefulness.

Technical data

Baseline (main):

Test summary: total: 45, failed: 0, succeeded: 44, skipped: 1, duration: 610.0s 

Target (generated):

Test summary: total: 606, failed: 51, succeeded: 545, skipped: 10, duration: 605.2s 

AssemblyName	LineRateBefore	LineRateAfter	LineRateDiff	BranchRateBefore	BranchRateAfter	BranchRateDiff
crank	50.64 %	63.29 %	12.65 %	0.00 %	56.39 %	56.39 %
crank-agent	41.13 %	45.62 %	4.50 %	0.00 %	38.75 %	38.75 %
crank-azdo	5.69 %	34.00 %	28.31 %	0.00 %	35.07 %	35.07 %
crank-pr	0.00 %	21.00 %	21.00 %	0.00 %	11.57 %	11.57 %
hello	95.65 %	95.65 %	0.00 %	0.00 %	75.93 %	75.93 %
Microsoft.Crank.EventSources	54.60 %	59.97 %	5.37 %	0.00 %	2.11 %	2.11 %
Microsoft.Crank.JobObjectWrapper	0.00 %	0.00 %	0.00 %	0.00 %	0.00 %	0.00 %
Microsoft.Crank.Jobs.Bombardier	66.92 %	62.49 %	-4.43 %	0.00 %	40.19 %	40.19 %
Microsoft.Crank.Jobs.H2Load	0.00 %	0.00 %	0.00 %	0.00 %	0.00 %	0.00 %
Microsoft.Crank.Jobs.HttpClient	0.00 %	29.70 %	29.70 %	0.00 %	19.42 %	19.42 %
Microsoft.Crank.Jobs.K6	0.00 %	9.96 %	9.96 %	0.00 %	9.52 %	9.52 %
Microsoft.Crank.Jobs.PipeliningClient	0.00 %	59.25 %	59.25 %	0.00 %	48.34 %	48.34 %
Microsoft.Crank.Jobs.Wrk	0.00 %	33.86 %	33.86 %	0.00 %	44.32 %	44.32 %
Microsoft.Crank.Jobs.Wrk2	0.00 %	11.04 %	11.04 %	0.00 %	7.69 %	7.69 %
Microsoft.Crank.Models	60.27 %	88.54 %	28.27 %	0.00 %	65.91 %	65.91 %
Microsoft.Crank.RegressionBot	0.00 %	9.83 %	9.83 %	0.00 %	1.66 %	1.66 %
post	100.00 %	100.00 %	0.00 %	0.00 %	100.00 %	100.00 %

More comments

• Still just heuristical version (so might not follow convention and references of the repo)
• Very few and dummy code fixers - so we fallback a lot to commenting out tests with compilation diagnostic
• No runtime feedback yet - so tests that crash/fail during runtime are kept