Refactor rspec-puppet tests #43

dfarrell07 · 2015-02-11T16:52:22Z

The current rspec-puppet tests are pretty great, but the architecture isn't going to scale as I continue adding param combinations to test. They basically work like this at the moment:

# tests for param0=x
  # tests for param1=x'
    # punt tests to shared fn
  # tests for param1=y'
    # punt tests to shared fn
# tests for param0=y
  # tests for param1=x'
    # punt tests to shared fn
  # tests for param1=y'
    # punt tests to shared fn

If I add a third layer, like I need to do for #41, it would look like this:

# tests for param0=x
  # tests for param1=x'
    # tests for param2=x''
      # punt tests to shared fn
    # tests for param2=y''
      # punt tests to shared fn
  # tests for param1=y'
    # tests for param2=x''
      # punt tests to shared fn
    # tests for param2=y''
      # punt tests to shared fn
# tests for param0=y
  # tests for param1=x'
    # tests for param2=x''
      # punt tests to shared fn
    # tests for param2=y''
      # punt tests to shared fn
  # tests for param1=y'
    # tests for param2=x''
      # punt tests to shared fn
    # tests for param2=y''
      # punt tests to shared fn

Clearly this doesn't scale, although it provides awesome coverage (every param combo).

I think something like this would provide solid practical coverage, without covering every possible param combo:

# tests for param0=x
  # punt tests to fn specialized in testing for param0
# tests for param0=y
  # punt tests to fn specialized in testing for param0
# tests for param1=x'
  # punt tests to fn specialized in testing for param1
# tests for param1=y'
  # punt tests to fn specialized in testing for param1
<continue pattern>

The text was updated successfully, but these errors were encountered:

dfarrell07 · 2015-02-11T22:14:15Z

This was making good progress until I hit one of those impossible-for-humans-to-parse error messages that makes Puppet so fucking evil.

dfarrell07 · 2015-02-12T17:58:22Z

Bumping to high prio because it blocks so many other things.

dfarrell07 · 2015-02-13T14:03:25Z

I made a solid first attempt at this refactoring, but I'm seeing a persistent error that I don't understand. Since the error message itself is amazingly useless, I'm trying to debug it by reducing the difference between my changed version (broken) and master (works). Amazingly, I reduced it all the way down to a trivial test suite and saw the error at every step along the way.

The simplest version of opendaylight_spec.rb I can make:

require 'spec_helper'

describe 'opendaylight' do
  it { should compile }
end

Fails with:

[~/puppet-opendaylight]$ bundle exec rake test                                                                                              8:57:05
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color

opendaylight class
  should compile into a catalogue without dependency cycles (FAILED - 1)

Failures:

  1) opendaylight class should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation: Could not parse for environment production: Syntax error at end of file at line 4 on node localhost.localdomain
     # ./spec/classes/opendaylight_spec.rb:4:in `block (2 levels) in <top (required)>'

Finished in 0.05337 seconds (files took 0.4825 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/classes/opendaylight_spec.rb:4 # opendaylight class should compile into a catalogue without dependency cycles

Total resources:   0
Touched resources: 0
Resource coverage:   NaN%
Untouched resources:


/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color failed

Everything else is the same as master.

dfarrell07 · 2015-02-13T14:05:34Z

This is the part of the error that I don't understand:

Syntax error at end of file at line 4 on node localhost.localdomain

Line 4 of which file? Clearly not opendaylight_spec.rb, as I've changed that file 10s of times (including what's on line 4) and gotten the same error message every time. Clearly not another file, as they are all the same as the versions on master. But clearly both of those statements aren't likely to be true, as there does seem to be an error somewhere. sigh

dfarrell07 · 2015-02-13T14:22:52Z

Started again with exactly the same opendaylight_spec.rb as is on master (so the same code as master in total), then cut it in ~half repeatedly to reduce it to the failing example above.

Made it all the way to this with the tests still passing:

require 'spec_helper'

describe 'opendaylight' do
  let(:facts) {{
    :osfamily => 'RedHat',
    :operatingsystem => 'Fedora',
    :operatingsystemmajrelease => '20',
  }}

  it { should compile }
end

Taking the final step and removing the only difference between that code and the version described above, I end up with:

require 'spec_helper'

describe 'opendaylight' do
  it { should compile }
end

And see this failure:

[~/puppet-opendaylight]$ bundle exec rake test                                                                                              9:18:27
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color

opendaylight
  should compile into a catalogue without dependency cycles (FAILED - 1)

Failures:

  1) opendaylight should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation: Unsupported OS family:  at /home/daniel/puppet-opendaylight/spec/fixtures/modules/opendaylight/manifests/init.pp:34 on node localhost.localdomain
     # ./spec/classes/opendaylight_spec.rb:4:in `block (2 levels) in <top (required)>'

Finished in 0.65601 seconds (files took 0.47619 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/classes/opendaylight_spec.rb:4 # opendaylight should compile into a catalogue without dependency cycles

Total resources:   0
Touched resources: 0
Resource coverage:   NaN%
Untouched resources:


/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color failed

Note that this is exactly the same code, and it's producing a different error message. This error makes sense, as the checks in init.rb really do need to know the OS family.

dfarrell07 · 2015-02-13T14:50:12Z

One lesson learned: let statements to set facts seem to override each other. So, this works:

require 'spec_helper'

describe 'opendaylight' do
  describe 'OS family Red Hat ' do
    # All tests for Fedora
    describe 'Fedora' do
      operatingsystem = 'Fedora'

      # This is the Fedora Yum repo URL
      yum_repo = 'https://copr-be.cloud.fedoraproject.org/results/dfarrell07/OpenDaylight/fedora-$releasever-$basearch/'

      # All tests for supported versions of Fedora
      ['20', '21'].each do |operatingsystemmajrelease|
        let(:facts) {{
          :osfamily => 'RedHat',
          :operatingsystem => operatingsystem,
          :operatingsystemmajrelease => operatingsystemmajrelease,
        }}
        context "#{operatingsystemmajrelease}" do
          # TODO: Call supported-OS-specific tests
          it { should compile }
        end
      end
    end
  end
end

But this throws errors about OS family not being set:

require 'spec_helper'

describe 'opendaylight' do
  describe 'OS family Red Hat ' do
    let(:facts) {{
      :osfamily => 'RedHat',
    }}
    # All tests for Fedora
    describe 'Fedora' do
      operatingsystem = 'Fedora'
      let(:facts) {{
        :operatingsystem => operatingsystem,
      }}

      # This is the Fedora Yum repo URL
      yum_repo = 'https://copr-be.cloud.fedoraproject.org/results/dfarrell07/OpenDaylight/fedora-$releasever-$basearch/'

      # All tests for supported versions of Fedora
      ['20', '21'].each do |operatingsystemmajrelease|
        let(:facts) {{
          :operatingsystemmajrelease => operatingsystemmajrelease,
        }}
        context "#{operatingsystemmajrelease}" do
          # TODO: Call supported-OS-specific tests
          it { should compile }
        end
      end
    end
  end
end

Errors:

[~/puppet-opendaylight]$ bundle exec rake test                                                                                              9:48:39
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
Cloning into 'spec/fixtures/modules/stdlib'...
remote: Counting objects: 6201, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 6201 (delta 3), reused 3 (delta 0)
Receiving objects: 100% (6201/6201), 1.24 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2646/2646), done.
Checking connectivity... done.
Cloning into 'spec/fixtures/modules/archive'...
remote: Counting objects: 198, done.
remote: Compressing objects: 100% (111/111), done.
remote: Total 198 (delta 69), reused 198 (delta 69)
Receiving objects: 100% (198/198), 39.26 KiB | 0 bytes/s, done.
Resolving deltas: 100% (69/69), done.
Checking connectivity... done.
/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color

opendaylight
  OS family Red Hat
    Fedora
      20
        should compile into a catalogue without dependency cycles (FAILED - 1)
      21
        should compile into a catalogue without dependency cycles (FAILED - 2)

Failures:

  1) opendaylight OS family Red Hat  Fedora 20 should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation: Unsupported OS family:  at /home/daniel/puppet-opendaylight/spec/fixtures/modules/opendaylight/manifests/init.pp:34 on node localhost.localdomain
     # ./spec/classes/opendaylight_spec.rb:25:in `block (6 levels) in <top (required)>'

  2) opendaylight OS family Red Hat  Fedora 21 should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation: Unsupported OS family:  at /home/daniel/puppet-opendaylight/spec/fixtures/modules/opendaylight/manifests/init.pp:34 on node localhost.localdomain
     # ./spec/classes/opendaylight_spec.rb:25:in `block (6 levels) in <top (required)>'

Finished in 0.64973 seconds (files took 0.51095 seconds to load)
2 examples, 2 failures

Failed examples:

rspec ./spec/classes/opendaylight_spec.rb:25 # opendaylight OS family Red Hat  Fedora 20 should compile into a catalogue without dependency cycles
rspec ./spec/classes/opendaylight_spec.rb:25 # opendaylight OS family Red Hat  Fedora 21 should compile into a catalogue without dependency cycles

Total resources:   0
Touched resources: 0
Resource coverage:   NaN%
Untouched resources:


/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color failed

dfarrell07 · 2015-02-13T14:56:47Z

Literally coping+pasting the code from the working example in my most recent comment, I'm now seeing that fucking line 4 error.

Code:

require 'spec_helper'

describe 'opendaylight class' do
  # All tests that check OS support/not-support
  describe 'OS support tests' do
    # All tests for OSs in the Red Hat family (CentOS, Fedora)
    describe 'OS family Red Hat ' do
      osfamily = 'RedHat'
      # All tests for Fedora
      describe 'Fedora' do
        operatingsystem = 'Fedora'

        # This is the Fedora Yum repo URL
        yum_repo = 'https://copr-be.cloud.fedoraproject.org/results/dfarrell07/OpenDaylight/fedora-$releasever-$basearch/'

        # All tests for supported versions of Fedora
        ['20', '21'].each do |operatingsystemmajrelease|
          let(:facts) {{
            :osfamily => osfamily,
            :operatingsystem => operatingsystem,
            :operatingsystemmajrelease => operatingsystemmajrelease,
          }}
          context "#{operatingsystemmajrelease}" do
            # TODO: Call supported-OS-specific tests
            it { should compile }
          end
        end
      end
    end
  end
end

Error:

[~/puppet-opendaylight]$ bundle exec rake test                                                                                              9:56:31
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color

opendaylight class
  OS support tests
    OS family Red Hat
      Fedora
        20
          should compile into a catalogue without dependency cycles (FAILED - 1)
        21
          should compile into a catalogue without dependency cycles (FAILED - 2)

Failures:

  1) opendaylight class OS support tests OS family Red Hat  Fedora 20 should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation: Could not parse for environment production: Syntax error at end of file at line 4 on node localhost.localdomain
     # ./spec/classes/opendaylight_spec.rb:25:in `block (7 levels) in <top (required)>'

  2) opendaylight class OS support tests OS family Red Hat  Fedora 21 should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation: Could not parse for environment production: Syntax error at end of file at line 4 on node localhost.localdomain
     # ./spec/classes/opendaylight_spec.rb:25:in `block (7 levels) in <top (required)>'

Finished in 0.06265 seconds (files took 0.52556 seconds to load)
2 examples, 2 failures

Failed examples:

rspec ./spec/classes/opendaylight_spec.rb:25 # opendaylight class OS support tests OS family Red Hat  Fedora 20 should compile into a catalogue without dependency cycles
rspec ./spec/classes/opendaylight_spec.rb:25 # opendaylight class OS support tests OS family Red Hat  Fedora 21 should compile into a catalogue without dependency cycles

Total resources:   0
Touched resources: 0
Resource coverage:   NaN%
Untouched resources:


/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color failed

dfarrell07 · 2015-02-13T15:05:56Z

Strange data point. This works:

require 'spec_helper'

describe 'opendaylight' do
  let(:facts) {{ 
    :osfamily => 'RedHat',
    :operatingsystem => 'Fedora', 
    :operatingsystemmajrelease => '20',
  }}

  it { should compile }
end

This doesn't:

require 'spec_helper'

describe 'opendaylight class' do
  let(:facts) {{ 
    :osfamily => 'RedHat',
    :operatingsystem => 'Fedora', 
    :operatingsystemmajrelease => '20',
  }}

  it { should compile }
end

Error:

[~/puppet-opendaylight]$ bundle exec rake test                                                                                             10:05:00
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color

opendaylight class
  should compile into a catalogue without dependency cycles (FAILED - 1)

Failures:

  1) opendaylight class should compile into a catalogue without dependency cycles
     Failure/Error: it { should compile }
       error during compilation: Could not parse for environment production: Syntax error at end of file at line 4 on node localhost.localdomain
     # ./spec/classes/opendaylight_spec.rb:10:in `block (2 levels) in <top (required)>'

Finished in 0.05924 seconds (files took 0.47654 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/classes/opendaylight_spec.rb:10 # opendaylight class should compile into a catalogue without dependency cycles

Total resources:   0
Touched resources: 0
Resource coverage:   NaN%
Untouched resources:


/usr/bin/ruby -I/home/daniel/.gem/ruby/gems/rspec-core-3.1.7/lib:/home/daniel/.gem/ruby/gems/rspec-support-3.1.2/lib /home/daniel/.gem/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color failed

The difference is the inclusion of the word class in the test description.

dfarrell07 · 2015-02-13T15:12:41Z

Okay, I now have a fairly substantial subset of the refactored version passing. The two important changes are removing class from the top describe statement and setting all facts via let in the inner-most context.

require 'spec_helper'

describe 'opendaylight' do
  # All tests that check OS support/not-support
  describe 'OS support tests' do
    # All tests for OSs in the Red Hat family (CentOS, Fedora)
    describe 'OS family Red Hat ' do
      osfamily = 'RedHat'
      # All tests for Fedora
      describe 'Fedora' do
        operatingsystem = 'Fedora'

        # This is the Fedora Yum repo URL
        yum_repo = 'https://copr-be.cloud.fedoraproject.org/results/dfarrell07/OpenDaylight/fedora-$releasever-$basearch/'

        # All tests for supported versions of Fedora
        ['20', '21'].each do |operatingsystemmajrelease|
          context "#{operatingsystemmajrelease}" do
            let(:facts) {{
              :osfamily => osfamily,
              :operatingsystem => operatingsystem,
              :operatingsystemmajrelease => operatingsystemmajrelease,
            }}
            # TODO: Call supported-OS-specific tests
            it { should compile }
          end
        end

        # All tests for unsupported versions of Fedora
        ['18', '19', '22'].each do |operatingsystemmajrelease|
          context "#{operatingsystemmajrelease}" do
            let(:facts) {{
              :osfamily => osfamily,
              :operatingsystem => operatingsystem,
              :operatingsystemmajrelease => operatingsystemmajrelease,
            }}
            # Run shared tests applicable to all unsupported OSs
            # Note that this function is defined in spec_helper
            unsupported_os_tests("Unsupported OS: #{operatingsystem} #{operatingsystemmajrelease}")
          end
        end
      end
    end
  end
end

- Relevant to #43 - The word 'class' must not appear in `describe` strings - `let` statements aren't inherited by child contexts Signed-off-by: Daniel Farrell <[email protected]>

- Relevant to #43 Signed-off-by: Daniel Farrell <[email protected]>

* Only testing defaults! See #43 for reasoning. * May add this to other OSs Signed-off-by: Daniel Farrell <[email protected]>

* Only testing defaults! See #43 for reasoning. * By 'all', of course I mean 'all supported' (others fail) * Refactored RPM rspec helper fn to support Fedora * Minor doc fix Signed-off-by: Daniel Farrell <[email protected]>

dfarrell07 self-assigned this Feb 11, 2015

dfarrell07 added prio:normal info:progress-0 type:feature labels Feb 11, 2015

This was referenced Feb 11, 2015

Install ODL from tar/zip #41

Closed

rspec-puppet test coverage for tarball-based installs #44

Closed

rspec-puppet test coverage for ODL port #47

Closed

dfarrell07 added prio:high and removed prio:normal labels Feb 12, 2015

dfarrell07 added a commit that referenced this issue Feb 13, 2015

Refactored rspec tests now passing

004933c

- Relevant to #43 - The word 'class' must not appear in `describe` strings - `let` statements aren't inherited by child contexts Signed-off-by: Daniel Farrell <[email protected]>

dfarrell07 added a commit that referenced this issue Feb 13, 2015

Add Karaf feature tests to refactored RSpec tests

288ef07

- Relevant to #43 Signed-off-by: Daniel Farrell <[email protected]>

dfarrell07 closed this as completed in 2ecf46f Feb 13, 2015

dfarrell07 added a commit that referenced this issue Mar 25, 2015

Do Karaf rspec tests with default values for Deb

7e39fc7

* Only testing defaults! See #43 for reasoning. * May add this to other OSs Signed-off-by: Daniel Farrell <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor rspec-puppet tests #43

Refactor rspec-puppet tests #43

dfarrell07 commented Feb 11, 2015

dfarrell07 commented Feb 11, 2015

dfarrell07 commented Feb 12, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

Refactor rspec-puppet tests #43

Refactor rspec-puppet tests #43

Comments

dfarrell07 commented Feb 11, 2015

dfarrell07 commented Feb 11, 2015

dfarrell07 commented Feb 12, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015

dfarrell07 commented Feb 13, 2015