support skipped tests

[erichs]

Thanks @rylnd for this awesome project! Any thoughts on how to implement skipped tests?

[AdrieanKhisbe]

I think I might have one:

 pending it 'should do something that is not specified yet' <<end
       assert blablabla
       assert etc
  end

Using the end here doc make that we just have to modify the one line to make the test pending. :)
Then the pending function could just print the test message, and ignore the stdin.

[erichs]

@AdrieanKhisbe: Nice! I definitely like the approach of adding the pending() function, but I'd be concerned that the heredoc would end up being brittle, requiring either leading tab characters, or not working correctly for various indentation styles.

What if pending() set a flag, assert() printed the test message but did nothing else when the flag was set, and end() cleared the flag?

[rylnd]

@erichs the flag seems like the simplest solution given the current behavior of shpec.

However, it's important to note that any other normal expressions within the pending test would still be evaluate, so "side effects" would have to be taken into account, which seems odd for a pending test. In other words, there's currently no way not to run a test (we can only choose whether it 'counts', not whether it's run).

[locochris]

What about xit instead of pending it?
(ie. similar to https://www.relishapp.com/rspec/rspec-core/v/2-0/docs/pending/pending-examples#temporarily-pending-by-changing-"it"-to-"xit")

[rylnd]

@locochris yeah, I was thinking the same thing. xit will be more familiar to folks coming from RSpec.

[AdrieanKhisbe]

Even if I touch a lot of rspec this week, didn't know about xit.

What I would say, is that it cost us nothing to do both! :) Besides event if xit is faster to write, pending is faster to read and spot.

[AdrieanKhisbe]

About remark of the heredoc Noting would prevent the user to use something else than end.
(true I forgot about the act that to close the hererdoc, end can't have trailling spaces)

describe 'My wondrfull super test
    pending it 'should do something that is not specified yet' <<dis
        assert blablabla
        assert etc
    end
dis
end

[erichs]

The above really isn't too bad for manually marking tests pending, @AdrieanKhisbe. I kind of like the simplicity. What I'd love though, is the ability to dynamically mark tests pending at run time, and have all statements ignored until the next end is encountered.

That might technically be possible using a DEBUG trap handler, but it's a horrible hack...

[AdrieanKhisbe]

what do you mean dynamically mark them at pending time?
Matching some criteria, tag attached?

An alternative would be that the test are function definition.

 it_"is suppose to be a some test"() {
        some testing logic
 }

no code is run before function is invoked
and then we retrieve the tests with function matching.

But I'm not sure that is really a direction we should go into.

[erichs]

@AdrieanKhisbe, the other day, I had to do a runtime check, and conditionally execute various tests, like:

if connected_to_vpn; then
  describe "when vpn-connected, hosts are reachable"
    it "pings myhost1"
      assert pingable myhost1 192.168.1.1
    end

    ...

  end
else
  echo "not attached to vpn, skipping"
fi

What I'd love to be able to say, is:

  describe "when vpn-connected, hosts are reachable"
    it "pings myhost1"
      connected_to_vpn || pending
      assert pingable myhost1 192.168.1.1
    end

    ...

  end

Having to surround the tests with if/else conditionals broke the feel of the DSL, and started me thinking.

[AdrieanKhisbe]

Okey. Now I perfectly see the use case :)

[AdrieanKhisbe]

I think the way to go is to "impose" the use of run async command (funciton so reconfigurable, behavior depending a global variable), I started to introduce in the prototyping branch of new language feature I started some week ago.

(problem is that would put a constraint on interpolation)

describe "when vpn-connected, hosts are reachable"
    it "pings myhost1"
      run connected_to_vpn || pending
      assert pingable myhost1 192.168.1.1
    end
# ....

[rylnd]

@erichs @AdrieanKhisbe I think that an xit (that sets a flag and then calls it) and a pending function (that simply sets the flag), along with end unsetting the flag should get us most of the way there.

@AdrieanKhisbe I'm not sure what you're getting at with the run stuff. . . can we move that conversation to #53?