versioning - Chef: Cookbook can't depend on older version of other cookbook -

background

note: keep in mind whole problem want use version 0.0.3 of environment-cookbook.

note 2: never had problem before. recent , don't know caused it.

1. environment-cookbook , environment-cookbook-machines

to build domains, have 2 cookbooks:

• one provisioning machines: environment-cookbook-machines
• one domain configuration: environment-cookbook

when check chef-server:

$ knife cookbook list -a | grep environment environment-cookbook 0.0.3 0.0.4 0.0.5 0.0.6 0.0.7 

looking @ company-environment-cookbook 0.0.3's metadata.json, see depends on dir-library 0.13.6:

"dependencies": {   "dir-library": "= 0.13.6" } 

whereas environment-cookbook 0.0.4 and higher depend on dir-library 0.13.7:

"dependencies": {   "dir-library": "=0.13.7" } 

2. wrapper-domain , wrapper-domain-machines

as each domain can depend on specific version of environment-cookbook, use wrapper cookbooks each domain, respectively

• wrapper-domain-machines
• wrapper-domain

checking metadata.json both of wrappers shows dependency on environment-cookbook 0.0.3 (this version want):

"dependencies": {   "environment-cookbook": "= 0.0.3"   ... } 

wrapper-domain-machines shows dependency on environment-cookbook-machines 0.0.3.

the berksfile.lock wrapper cookbooks this:

graph  environment-cookbook (0.0.3)     dir-library (= 0.13.6) environment-cookbook-machines (0.0.3)  # here wrapper-cookbook-machines     dir-library (= 0.13.6) 

3. dependency graph

when run berks wiz on wrapper-domain-machines cookbook, following dependency graph:

so seems fine.

problem

when run domain build through ci job on hudson, see following @ beginning of log file:

info: using dir-library (0.13.6) info: using environment-cookbook (0.0.3) info: using environment-cookbook-machines (0.0.3) info: installing environment-cookbook (0.0.3) chef-server-url info: using dir-library (0.13.6) 

a little bit further:

info: run list [recipe[wrapper-domain-machines::up-machines]] info: run list expands [wrapper-domain-machines::up-machines] info: loading cookbooks [wrapper-domain-machines@0.0.2, dir-library@0.13.6, environment-cookbook-machines@0.0.3] 

so far good. it's using version 0.0.3 environment-cookbook , 0.13.6 dir-library.

later on during build:

info: run list [recipe[environment-cookbook::prepare_machine]] info: run list expands [environment-cookbook::prepare_machine] info: starting chef run domain.company.com info: running start handlers info: start handlers complete. resolving cookbooks run list: ["environment-cookbook::prepare_machine"][0m info: loading cookbooks [environment-cookbook@0.0.7, dir-library@0.13.7] 

stop, ?

info: loading cookbooks [environment-cookbook@0.0.7, dir-library@0.13.7]

what tried far

1. delete cached cookbooks

   • delete cookbooks .berkshelf/cookbooks
   • re-run build (downloads automatically chef-server)
   • still picks 0.0.8

2. check dependencies on environment-cookbook in other cookbooks: none.

3. delete , re-install versions of environment-cookbook 0.0.3 0.0.7: no luck.

4. clean chef clients , nodes before re-running: no luck either.

questions

• why changing latest version of environment-cookbook (0.0.7), picking it's dependency dir-library (0.13.7) ?

• how can troubleshoot ?

• how avoid in future ?

this show-stopper us.

ask me further clarifications , i'll update post.

i suspect same problem chef ran into. boils down run-time revision control, opposed compile time revision control.

lesson learned: unconstrained cookbook versions @ run-time dangerous when running chef @ scale.

background

you're using berkshelf manage cookbook dependencies, that's great , ensure correct versions loaded chef server. subtle problem each cookbook has own dependency tree. @ run-time, when add multiple cookbooks node's run-list, chef server must calculate fresh tree of dependencies. problem can appear random because depends on combination of cookbooks have on run list. more cookbooks, more potential conflict.

we tried fix problem explicitly setting dependencies our cookbook metadata files. discovered chef silently fail resolve dependency tree of our cookbooks , default older version calculate dependencies. puzzling.

we got closer problem when began explicitly set version of cookbook on run-list. began error messages chef unable resolve dependencies. weird problem impacted our production chef server most. determined because on production had historical versions of our cookbooks loaded. purging old cookbooks helped, did not solve our problems.

chef had worked fine 2 years before discovered these problems. time , scale exposed fatal flaw in our system. @ run-time need fix versions of cookbooks match configurations have tested.

analysis

coming java background equated problem how we can run multiple applications on tomcat server.

maven build tool manages each apps dependencies , creates package upload tomcat. in chef berkshelf fufils function.

the big difference @ run-time. tomcat creates separate classpath jars belong each application. provides strong isolation between applications @ run-time, safely allowing them run different versions of same cookbook. impossible problem faced chef, @ runtime chef-client runs single set of cookbooks.

solutions

while i'm not fan of policy files present them option favoured chef.

policy files

while users oblivious problem being solved, chef have developed new feature called policy files:

• https://docs.chef.io/config_rb_policyfile.html
• https://www.chef.io/blog/2015/08/18/policyfiles-a-guided-tour/
• https://www.chef.io/blog/2015/10/05/policyfiles-why-what-and-how/

in nutshell they're doing setting nodes run-list in advance, @ compile time.

one big benefit of policy files result in faster chef run. chef server no longer has figure out large dependency tree, can big saving in chef installations large numbers of cookbooks.

environment cookbook pattern

personally i'm not fan of policy files, because had discovered environment cookbook pattern, poorly understood powerful feature of chef existed:

• http://blog.vialstudios.com/the-environment-cookbook-pattern/

now every time deploy cookbooks use chef environment. natural way provide isolation in chef (did point out poorly understood). here's example using berkshelf:

berks upload  berks apply my_app_cookbook_version1 

the handy "apply" command use berkshelf lock file update cookbook versions in environment "my_app_cookbook_version1". you've fixed run-time match tested conditions.

the consequence of course have environment per application cookbook:

• my_app_cookbook_version1
• my_app_cookbook_version2
• etc

this bonus me, because enables me bootstrap infrastructure against i've tested:

knife bootstrap --environment my_app_cookbook_version1 ... 

it creates predictability , means loading new cookbooks not going magically change servers in production.

a bonus environments provide record of cookbook versions in use , convenient place set override attributes associated deployment, "app_owner", "app_version", etc.

apologies long posting.