How to get octocatalog to work with puppet enterprise 2016.1.1

[jamesandrews-iopp]

Description of problem

• What did you do?
  installed octocatalog but had a bit of trouble getting it to work
• What happened?
  In the end I triumphed
• What did you expect to happen?
  I would get bored and give up
• How can someone reproduce the problem?
  Using Centos 6.8, rvm with various versions (2.18, 2.1.9, 2.3.1) of ruby, gem versions of octocatalog or git clone and the documentation on github I attempted to get octocatalog to work. Initially it did not.
  I fixed a couple of things like whitelisting the SSL puppetdb connection on the puppetdb server that aren't in the docs but that are clearly described on puppetlabs.com

However with the following two changes to the documentation, octocatalog-diff did eventually work

1. The puppetdb_ssl_client_key and puppet_ssl_client_cert in .octocatalog.cfg.rb should both be left unset. Instead do this:
   settings[:puppetdb_ssl_client_pem] = File.open("/etc/puppetlabs/puppet/ssl/private_keys/#{fqdn}.pem").read + File.open("/etc/puppetlabs/puppet/ssl/certs/#{fqdn}.pem").read

2. I didn't notice this anywhere in the docs but it is necessary to do this install if you are running on a puppet agent node

sudo /opt/puppetlabs/puppet/bin/puppet resource package puppetdb-termini ensure=latest

Command used and debugging output

If you set it up "by the book" (with paths to puppetdb_ssl_client_key and puppetdb_ssl_client_cert) then this sort of thing happens

[jan@succi puppetmaster_np]$ octocatalog-diff -n 'c1.content-service.cld.iop.org-i-f83c0172' -d --no-parallel --catalog-only -f cloudlive -t cloudlive
D, [2016-10-28T16:44:36.387100 #26758] DEBUG -- : Running octocatalog-diff 0.5.1 with ruby 2.1.8
D, [2016-10-28T16:44:36.387248 #26758] DEBUG -- : Command line arguments: ["-n", "c1.content-service.cld.iop.org-i-f83c0172", "-d", "--no-parallel", "--catalog-only", "-f", "cloudlive", "-t", "cloudlive"]
D, [2016-10-28T16:44:36.387293 #26758] DEBUG -- : Running on host succi.ioppublishing.com (x86_64-linux)
D, [2016-10-28T16:44:36.387345 #26758] DEBUG -- : Compiling catalog --catalog-only for c1.content-service.cld.iop.org-i-f83c0172
D, [2016-10-28T16:44:36.387766 #26758] DEBUG -- : Initialized OctocatalogDiff::Catalog::Noop for from-catalog
D, [2016-10-28T16:44:36.387835 #26758] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for to-catalog
D, [2016-10-28T16:44:36.387942 #26758] DEBUG -- : Begin build_catalog for cloudlive
D, [2016-10-28T16:44:36.387982 #26758] DEBUG -- : Setting up Puppet catalog build for cloudlive
D, [2016-10-28T16:44:36.388013 #26758] DEBUG -- : Catalog for cloudlive will be built with OctocatalogDiff::Catalog::Noop
D, [2016-10-28T16:44:36.388136 #26758] DEBUG -- : Catalog for cloudlive successfully built with OctocatalogDiff::Catalog::Noop in 2.944e-06 seconds
D, [2016-10-28T16:44:36.388191 #26758] DEBUG -- : Success build_catalog for cloudlive
D, [2016-10-28T16:44:36.388223 #26758] DEBUG -- : Begin build_catalog for cloudlive
D, [2016-10-28T16:44:36.388251 #26758] DEBUG -- : Setting up Puppet catalog build for cloudlive
D, [2016-10-28T16:44:36.388279 #26758] DEBUG -- : Catalog for cloudlive will be built with OctocatalogDiff::Catalog::Computed
D, [2016-10-28T16:44:36.388310 #26758] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2016-10-28T16:44:36.388360 #26758] DEBUG -- : Start retrieving facts for c1.content-service.cld.iop.org-i-f83c0172 from OctocatalogDiff::Catalog::Computed
D, [2016-10-28T16:44:36.388395 #26758] DEBUG -- : Retrieving facts from PuppetDB
D, [2016-10-28T16:44:36.389692 #26758] DEBUG -- : Failed build_catalog for cloudlive: OpenSSL::PKey::RSAError Neither PUB key nor PRIV key: nested asn1 error
W, [2016-10-28T16:44:36.389795 #26758] WARN -- : Failed build_catalog for cloudlive
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:209:in add_parallel_result': Catalog for 'to' (cloudlive) failed to compile with OpenSSL::PKey::RSAError: Neither PUB key nor PRIV key: nested asn1 error (OctocatalogDiff::CatalogDiff::Cli::Catalogs::CatalogError) /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/httparty.rb:124:ininitialize'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/httparty.rb:124:in new' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/httparty.rb:124:inssl_options'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/httparty.rb:81:in wrap_ssl_options' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/httparty.rb:16:inget'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/puppetdb.rb:117:in block in _get' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/puppetdb.rb:106:ineach'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/puppetdb.rb:106:in _get' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/puppetdb.rb:86:inget'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/facts/puppetdb.rb:31:in block in fact_retriever' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/facts/puppetdb.rb:29:intimes'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/facts/puppetdb.rb:29:in fact_retriever' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/facts.rb:28:ininitialize'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-util/facts.rb:48:in new' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-util/facts.rb:48:infacts_from_puppetdb'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-util/facts.rb:81:in compute_facts' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-util/facts.rb:25:infacts'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog/computed.rb:57:in build' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog.rb:60:inbuild'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:221:in build_catalog' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:26:incall'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:26:in execute' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:147:inblock in run_tasks_serial'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:144:in each' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:144:inrun_tasks_serial'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/util/parallel.rb:72:in run_tasks' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:98:inbuild_catalog_parallelizer'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:32:in catalogs' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:180:incatalog_only'
/home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:88:in cli' /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/bin/octocatalog-diff:74:in<top (required)>'
/home/jan/.rvm/gems/ruby-2.1.8/bin/octocatalog-diff:23:in load' /home/jan/.rvm/gems/ruby-2.1.8/bin/octocatalog-diff:23:in

'
/home/jan/.rvm/gems/ruby-2.1.8/bin/ruby_executable_hooks:15:in eval' /home/jan/.rvm/gems/ruby-2.1.8/bin/ruby_executable_hooks:15:in'
from /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:118:in block in build_catalog_parallelizer' from /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:112:ineach'
from /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:112:in build_catalog_parallelizer' from /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:32:incatalogs'
from /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:180:in catalog_only' from /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/lib/octocatalog-diff/catalog-diff/cli.rb:88:incli'
from /home/jan/.rvm/gems/ruby-2.1.8/gems/octocatalog-diff-0.5.1/bin/octocatalog-diff:74:in <top (required)>' from /home/jan/.rvm/gems/ruby-2.1.8/bin/octocatalog-diff:23:inload'
from /home/jan/.rvm/gems/ruby-2.1.8/bin/octocatalog-diff:23:in <main>' from /home/jan/.rvm/gems/ruby-2.1.8/bin/ruby_executable_hooks:15:ineval'
from /home/jan/.rvm/gems/ruby-2.1.8/bin/ruby_executable_hooks:15:in `'

fixing the keys using the concatenated certs trick then leads to a baffling terminus puppetdb not found for indirection facts. This is solved by installing the puppetdb-terminus stuff on the host that is running octocatalog-diff

Platform and version information

• Your OS: Centos 6.8

• Your Ruby version: 2.19, 2.1.8 and 2.3.1

• Your version of Puppet: 4.4.1 ( puppet enterprise 2016.1.1)

• Your version of octocatalog-diff: 0.5.1

  Do the tests pass from a clean checkout?

  Yes

Anything else to add that you think will be helpful?

thank you for this wondrous looking tool. I am hoping to be able to simply the over complex puppet hiera config we have here at IOPP

[kpaulisse]

Hey @jamesandrews-iopp ,

Thanks for the report. I've identified a couple clear opportunities for improvement in the documentation.

1. In the example file, I believe we are currently missing a File.read() around the SSL client certificate details. The underlying code around puppetdb_ssl_client_(key|cert|pem) is just this. What you did with puppetdb_ssl_client_pem is legitimate, but could have been accomplished with the missing File.read() around the key and certificate settings. I'll fix up the example file to reflect this.
2. Thanks for the hint about puppetdb-termini. We just had that in our Gemfile (probably from before I ever started alog-diff) and so I didn't explicitly put it in the docs. I'll make sure to add a hint in an appropriate place, as well as direct links to puppet documents on the SSL whitelisting you described.

Pull request with these fixes forthcoming.