“99 failing tests in the queue! 99 failing tests! Check one out, grind it out, 98 failing tests in the queue!”

So I'm facing this nearly twenty-hour long regression test [1] and I had this idea—instead of querying the mocked end point if it did its thing or not, have the mocked endpoint check to see if it did its thing or not.

I now send the mocked endpoint the testcase itself (along with a query flag of true or false). The mocked endpoint will save this information, and set a queried flag for this testcase to false. If it is queried, it updates the queried flag for the given request. At the end of the regression test (and I pause for a few extra seconds to let any pending requests to hopefully finish), the mocked endpoint will then go through the list of all the testcases it was told about, and check to see if the query flag and queried flag match—if not, it logs an error.

Sure, now we have two error logs to check, but I think that's better than waiting nearly twenty hours for results.

I got a baseline time for the subset of the regression test [2] without the mock checks—35 seconds. I'm trying to beat a time of 4 hours, 30 minutes with the mock checks.

The new method ran the subset in 40 seconds. The entirety of the regression test, 15,852 tests, took just a few minutes—about the same as before.

I can live with that.

Now all that's left is to write the validation logic—I still don't have it down yet.

[1] /boston/2021/06/09.1

[2] /boston/2021/06/09.2

Gemini Mention this post

Contact the author