A failing test doesn't fail the build

[mgol]

Expected Behavior

If a browser fails to spawn or a test fails in one of the browsers, the process should exit with code different than 0.

Actual Behavior

The process always exits with code 0.

Steps to Reproduce the Problem

1. git clone git@github.com:jquery/jquery-color.git
2. npm install
3. Now, modify one of the test cases in test/unit/color.js so that it fails, e.g. by changing one 255 to 254.
4. Run npm run browserstack; echo $?

You should see a failure & a number different than 0 as the last output. 0 is printed instead.

browserstack.json

Platform details

1. browserstack-runner version: 0.9.1
2. node version: v12.16.3
3. os type and version: macOS 10.15.4 (19E287)

Details

This is quite a critical bug to me as it can mask errors by exiting the CI successfully. If one doesn't notice, this may lead to releasing a broken version of a project.

[Krinkle]

I believe the exit_with_fail option can be set for this. When I use that, I get exit status 1 when there are failures or time outs etc.

[Namrata-m13]

@Krinkle I would recommend you to use the karma-browserstack-launcher for your tests - https://github.com/karma-runner/karma-browserstack-launcher