Cucumber Testing Framework

Cucumber is BDD (behaviour driven development) framework. In contrast to other BDD frameworks (like RSpec) the specification is written in a natural human readable language.

Rubocop-yast

I wanted to write tests for the new Rubocop-yast plugin in some nice way. I started with RSpec, but the tests looked ugly (writing multiline indented Ruby code in a string literal requires extra escaping which makes it quite hard to read...).

Then I looked at the original Zombie Killer tests. They are written in Markdown so they are better readable, you can write some additional comments to the test etc... And how to run the Markdown tests? There is a custom Markdown renderer which converts the Markdown specification into a RSpec test.

Looks nice, but having a custom renderer makes it difficult, we have to maintain it and the Markdown format in specific, there is nothing else similar to it...

Then Cucumber come to my mind! It exactly fits our need! The specification allows to write extra comments and notes, it's readable almost like our Markdown and is a standard tool. The killer feature is multiline docstring parameter. It allows to write indented Ruby code directly without any extra escaping.

You can check how the tests looks here, check the *.feature files. Here is the code which converts the specification into the testing code.

Pros & Cons

Here is the summary of pros & cons I found when starting with Cucumber:

Advantages

• Specification in natural language, readable tests and user stories
• Allows to use other testing framework for running the real tests (like RSpec)

Disadvantages

• Extra code for converting the textual specification to Ruby code
• One more layer between test description and the code (you need to make sure the code really matches the description)
• Test descriptions should describe the high level features (usually the user interaction), they should not describe the low-level implementation details

Ideally the features should be written by managers, designers or other non-technical people. They can describe the required features without any programming skills. That's probably the most important Cucumber feature.

Suitable for Yast?

In my opinion not. Why? We are usually focused on low-level features and we would probably need too much code for converting the specifications to tests. And the tests are usually too different, we would need to write extra conversion for each test and maintain it. The overall benefit would be small in my opinion and would not be worth of doing it.

Moreover the feature descriptions we usually get are hard to convert to a testing code, they are usually too generic or cannot be tested in unit tests (e.g. the installer features).

Links

http://cukes.info/

http://cukes.info/step-definitions.html

https://blog.engineyard.com/2009/15-expert-tips-for-using-cucumber/

http://antonymarcano.com/blog/2010/08/youre-almost-cuking-it/

http://www.elabs.se/blog/15-you-re-cuking-it-wrong

Using Rubocop

Introduction

Rubocop is a Ruby static code analyzer which looks for common code smells and checks the coding style.

Installation

The installation is quite easy, just run "sudo gem install rubocop" command (assuming you have Ruby alredy installed).

Initial Run and Creating the Config File

If there is no .rubocop.yml file in your project root then Rubocop uses the default configuration.

It is a good idea to let Rubocop generate the project default for you, simply run "rubocop --auto-gen-config". This will create .rubocop_todo.yml file which can be used as a template for your initial config file.

The default generated config file disables all checks which fail. That means if you run Rubocop with this config file (or if you remove the _todo suffix) it will report success.

Fixing the Issues

Now you can go one by one disabled check in the created template, enable each check and see where the problem is and whether it's a valid issue according to your style or preferences.

There are basically these solutions how to fix an error reported by Rubocop:

• Fix the issue according to the suggestions reported by Rubocop
• Let Rubocop to fix it for you (does not work for all issues found, but majority of coding style issues, like indentation or white space usage, can be fixed automatically), just add "-a" or "--auto-correct" option. You should manually check the changes done ("git diff") after auto correction, just to be sure the fix was correct and had no side effects.
• Change the expected style (e.g. the default Rubocop style is single quoted string literals, if you prefer double quoted strings in your project then set the different default in the config), see the possible options in the default configuration.
• Disable the check locally in the code (e.g. the rule is valid, but the specific place in the code is an exception where breaking the rule is correct, for example you prefer ".nil?" over "== nil", but in a test you want to check your operator= definition correctly handles nil comparison)
• Disable the check globally

Using Rubocop in CI

To ensure that the coding style is honored during development it is a good idea to run Rubocop at CI (Continuous Integration) server like Travis or Jenkins.

Rubocop in Yast

I tried to start with Rubocop in the Yast registration module which is written from scratch and should not contain ugly code parts introduced by YCP to Ruby conversion.

Initially it reported almost 3000 (!) offenses, but many of them were false positives caused by different coding style defaults (e.g. single quote vs. double quote string literals). After adapting the config style (and relaxing some metric checks which would require non-trivial refactoring) the number of issues was decreased to just about 900.

Majority issues were harmless and related to white space, but some of the checks found really bad code, like this "private" modifier issue.

Thanks to the nice auto correction feature the majority of the issues (~830 which is about 92%) could be fixed automatically. So the number of the manual changes was rather small.

I found only two issues with auto correction - at one place it removed a comment which was inside a removed block (moved outside the block) and in another case it added a trailing space at the end of the line (at that time I had the trailing check disabled so it was not fixed by another check).

You can see all the changes in this pull request.

Result

It was a nice experience with Rubocop as it not only complains what is wrong but also suggests the solution how to fix the found issues. More over it has auto correct feature which works very well and can fix almost all coding styles issues automatically.

So let see if we can use it in more Yast modules...