Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run behat fixture features against Cucumber parser #229

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Run behat fixture features against Cucumber parser #229

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
43c01dc
Run behat fixture features against Cucumber parser
ciaranmcnulty Mar 9, 2021
789ed5c
Try to fix gherkin path in GA
ciaranmcnulty Mar 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,15 @@ jobs:
php-version: "${{ matrix.php }}"
coverage: none

- name: Update Symfony version
- name: Update required Symfony version
if: matrix.symfony-version != ''
run: composer require --no-update "symfony/symfony:${{ matrix.symfony-version }}"

- name: Install dependencies
- name: Install composer dependencies
run: composer update ${{ matrix.composer-flags }}

- name: Install gherkin binary
run: wget https://github.com/cucumber/cucumber/releases/download/cucumber-gherkin%2Fv17.0.2/cucumber-gherkin-linux-amd64 -O gherkin && chmod +x gherkin

- name: Run tests (phpunit)
run: ./vendor/bin/phpunit
run: PATH=$PATH:. ./vendor/bin/phpunit
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ vendor
composer.phar
composer.lock
.phpunit.result.cache
gherkin
6 changes: 6 additions & 0 deletions bin/update_cucumber
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,12 @@ file_put_contents($composerFile, $newJson);

echo "Updated composer config:\n$newJson";

$githubFile = __DIR__ . '/../.github/workflows/build.yml';
$githubConfig = file_get_contents($githubFile);
$newConfig = str_replace($oldTag, $newTag, $githubConfig);
file_put_contents($githubFile, $newConfig);

echo "Updated github build action:$newConfig\n";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should put a \n before the new config, to be more readable (otherwise, the first line of the config is stuck with the message)


if (getenv('GITHUB_ACTIONS')) {
echo "::set-output name=cucumber_version::$newTag\n";
Expand Down
13 changes: 6 additions & 7 deletions src/Behat/Gherkin/Loader/CucumberNDJsonAstLoader.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ private static function getScenarios(array $json)
array_map(
static function ($child) {

if (isset($child['scenario']['examples'])) {
if (isset($child['scenario']['examples']) && count($child['scenario']['examples'])) {
return new OutlineNode(
isset($child['scenario']['name']) ? $child['scenario']['name'] : null,
self::getTags($child['scenario']),
Expand All @@ -90,7 +90,7 @@ static function ($child) {
}
else {
return new ScenarioNode(
$child['scenario']['name'],
isset($child['scenario']['name']) ? $child['scenario']['name'] : null,
self::getTags($child['scenario']),
self::getSteps(isset($child['scenario']['steps']) ? $child['scenario']['steps'] : []),
$child['scenario']['keyword'],
Expand Down Expand Up @@ -118,7 +118,7 @@ private static function getBackground(array $json)
array_map(
static function ($child) {
return new BackgroundNode(
$child['background']['name'],
isset($child['background']['name']) ? $child['background']['name'] : null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not using ??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just forgot what versions we support

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, you used ?? in the same diff a few lines below 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a good memory :)

self::getSteps(isset($child['background']['steps']) ? $child['background']['steps'] : []),
$child['background']['keyword'],
$child['background']['location']['line']
Expand Down Expand Up @@ -147,8 +147,7 @@ static function(array $json) {
trim($json['keyword']),
$json['text'],
[],
$json['location']['line'],
trim($json['keyword'])
$json['location']['line']
);
},
$json
Expand All @@ -167,15 +166,15 @@ static function($tableJson) {

$table[$tableJson['tableHeader']['location']['line']] = array_map(
static function($cell) {
return $cell['value'];
return $cell['value'] ?? '';
},
$tableJson['tableHeader']['cells']
);

foreach ($tableJson['tableBody'] as $bodyRow) {
$table[$bodyRow['location']['line']] = array_map(
static function($cell) {
return $cell['value'];
return isset($cell['value']) ? $cell['value'] : '';
},
$bodyRow['cells']
);
Expand Down
109 changes: 93 additions & 16 deletions tests/Behat/Gherkin/Cucumber/CompatibilityTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
use Behat\Gherkin\Loader\ArrayLoader;
use Behat\Gherkin\Loader\CucumberNDJsonAstLoader;
use Behat\Gherkin\Loader\LoaderInterface;
use Behat\Gherkin\Loader\YamlFileLoader;
use Behat\Gherkin\Node\FeatureNode;
use Behat\Gherkin\Node\ScenarioInterface;
use Behat\Gherkin\Node\ScenarioNode;
Expand All @@ -17,15 +18,16 @@
use PHPUnit\Framework\TestCase;

/**
* Tests the parser against the upstream cucumber/gherkin test data
* Tests the Behat and Cucumber parsers against each other
*
* @group cucumber-compatibility
*/
class CompatibilityTest extends TestCase
{
const TESTDATA_PATH = __DIR__ . '/../../../../vendor/cucumber/cucumber/gherkin/testdata';
const CUCUMBER_TEST_DATA = __DIR__ . '/../../../../vendor/cucumber/cucumber/gherkin/testdata';
const BEHAT_TEST_DATA = __DIR__ . '/../Fixtures/etalons';

private $notParsingCorrectly = [
private $cucumberFeaturesNotParsingCorrectly = [
'complex_background.feature' => 'Rule keyword not supported',
'rule.feature' => 'Rule keyword not supported',
'descriptions.feature' => 'Examples table descriptions not supported',
Expand All @@ -41,7 +43,35 @@ class CompatibilityTest extends TestCase
'tags.feature' => 'Tags followed by comments not parsed correctly'
];

private $parsedButShouldNotBe = [
private $behatFeaturesNotParsingCorrectly = [
'issue_13.yml' => 'Scenario descriptions are not supported',
'complex_descriptions.yml' => 'Scenario descriptions are not supported',
'multiline_name_with_newlines.yml' => 'Scenario descriptions are not supported',
'multiline_name.yml' => 'Scenario descriptions are not supported',
'background_title.yml' => 'Background descriptions are not supported',

'empty_scenario_without_linefeed.yml' => 'Feature description has wrong whitespace captured',
'addition.yml' => 'Feature description has wrong whitespace captured',
'test_unit.yml' => 'Feature description has wrong whitespace captured',
'ja_addition.yml' => 'Feature description has wrong whitespace captured',
'ru_addition.yml' => 'Feature description has wrong whitespace captured',
'fibonacci.yml' => 'Feature description has wrong whitespace captured',
'ru_commented.yml' => 'Feature description has wrong whitespace captured',
'empty_scenario.yml' => 'Feature description has wrong whitespace captured',
'start_comments.yml' => 'Feature description has wrong whitespace captured',
'empty_scenarios.yml' => 'Feature description has wrong whitespace captured',
'commented_out.yml' => 'Feature description has wrong whitespace captured',
'ru_division.yml' => 'Feature description has wrong whitespace captured',
'hashes_in_quotes.yml' => 'Feature description has wrong whitespace captured',
'outline_with_spaces.yml' => 'Feature description has wrong whitespace captured',
'ru_consecutive_calculations.yml' => 'Feature description has wrong whitespace captured',
];

private $behatFeaturesCucumberCannotParseCorrectly = [
'comments.yml' => 'see https://github.com/cucumber/cucumber/issues/1413'
];

private $cucumberFeaturesParsedButShouldNotBe = [
'invalid_language.feature' => 'Invalid language is silently ignored',
'whitespace_in_tags.feature' => 'Whitespace in tags is tolerated',
];
Expand All @@ -54,30 +84,35 @@ class CompatibilityTest extends TestCase
/**
* @var LoaderInterface
*/
private $loader;
private $cucumberLoader;

/**
* @var LoaderInterface
*/
private $yamlLoader;

protected function setUp(): void
{
$arrKeywords = include __DIR__ . '/../../../../i18n.php';
$lexer = new Lexer(new Keywords\ArrayKeywords($arrKeywords));
$this->parser = new Parser($lexer);
$this->loader = new CucumberNDJsonAstLoader();
$this->cucumberLoader = new CucumberNDJsonAstLoader();
$this->yamlLoader = new YamlFileLoader();
}

/**
* @dataProvider goodCucumberFeatures
*/
public function testFeaturesParseTheSameAsCucumber(\SplFileInfo $file)
public function testCucumberFeaturesParseTheSame(\SplFileInfo $file)
{

if (isset($this->notParsingCorrectly[$file->getFilename()])){
$this->markTestIncomplete($this->notParsingCorrectly[$file->getFilename()]);
if (isset($this->cucumberFeaturesNotParsingCorrectly[$file->getFilename()])){
$this->markTestIncomplete($this->cucumberFeaturesNotParsingCorrectly[$file->getFilename()]);
}

$gherkinFile = $file->getPathname();

$actual = $this->parser->parse(file_get_contents($gherkinFile), $gherkinFile);
$cucumberFeatures = $this->loader->load($gherkinFile . '.ast.ndjson');
$cucumberFeatures = $this->cucumberLoader->load($gherkinFile . '.ast.ndjson');
$expected = $cucumberFeatures ? $cucumberFeatures[0] : null;

$this->assertEquals(
Expand All @@ -86,13 +121,45 @@ public function testFeaturesParseTheSameAsCucumber(\SplFileInfo $file)
);
}

/**
* @dataProvider behatFeatures
*/
public function testBehatFeaturesParseTheSame(\SplFileInfo $ymlFile)
{
if (isset($this->behatFeaturesNotParsingCorrectly[$ymlFile->getFilename()])){
$this->markTestIncomplete($this->behatFeaturesNotParsingCorrectly[$ymlFile->getFilename()]);
}

if (isset($this->behatFeaturesCucumberCannotParseCorrectly[$ymlFile->getFilename()])){
$this->markTestIncomplete($this->behatFeaturesCucumberCannotParseCorrectly[$ymlFile->getFilename()]);
return;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this return is dead code as markTestIncomplete never returns but always throws

}

exec('which gherkin', $_, $result);
if ($result) {
$this->markTestSkipped("No gherkin executable in path");
}

$filename = $ymlFile->getPathname();
$expected = current($this->yamlLoader->load($filename));

$featureFile = preg_replace('/etalons\/(.*).yml$/', 'features/\\1.feature', $filename);

$tempFile = tempnam(sys_get_temp_dir(), 'behat-cucumber');
exec("gherkin -format ndjson -no-source -no-pickles $featureFile > $tempFile");
$actual = current($this->cucumberLoader->load($tempFile));
unlink($tempFile);

$this->assertEquals($this->normaliseFeature($expected), $this->normaliseFeature($actual));
}

/**
* @dataProvider badCucumberFeatures
*/
public function testBadFeaturesDoNotParse(\SplFileInfo $file)
public function testBadCucumberFeaturesDoNotParse(\SplFileInfo $file)
{
if (isset($this->parsedButShouldNotBe[$file->getFilename()])){
$this->markTestIncomplete($this->parsedButShouldNotBe[$file->getFilename()]);
if (isset($this->cucumberFeaturesParsedButShouldNotBe[$file->getFilename()])){
$this->markTestIncomplete($this->cucumberFeaturesParsedButShouldNotBe[$file->getFilename()]);
}

$this->expectException(ParserException::class);
Expand All @@ -112,13 +179,22 @@ public static function badCucumberFeatures()

private static function getCucumberFeatures($folder)
{
foreach (new \FilesystemIterator(self::TESTDATA_PATH . $folder) as $file) {
foreach (new \FilesystemIterator(self::CUCUMBER_TEST_DATA . $folder) as $file) {
if ($file->isFile() && $file->getExtension() == 'feature') {
yield $file->getFilename() => array($file);
}
}
}

public static function behatFeatures(): iterable
{
foreach (new \FilesystemIterator(self::BEHAT_TEST_DATA) as $file) {
if ($file->isFile() && $file->getExtension() == 'yml') {
yield $file->getFilename() => array($file);
}
}
}

/**
* Renove features that aren't present in the cucumber source
*/
Expand All @@ -129,7 +205,7 @@ private function normaliseFeature($featureNode)
return null;
}

$scenarios = array_map(
array_map(
function(ScenarioInterface $scenarioNode) {
$steps = array_map(
function(StepNode $step) {
Expand All @@ -148,6 +224,7 @@ function(StepNode $step) {
$featureNode->getScenarios()
);

$this->setPrivateProperty($featureNode, 'file', 'file.feature');

return $featureNode;
}
Expand Down