Thursday, 2014-06-19

greghayneslifeless: that one got hit by the invalid checksum for trusty to00:00
*** yfaykya has joined #tripleo00:00
lifelessgreghaynes: ack00:00
*** yfaykya has quit IRC00:05
*** yfaykya has joined #tripleo00:06
SpamapSlifeless: Oh the revert for heat is landed.00:07
SpamapSlifeless: 2 hours ago00:07
*** yfaykya has quit IRC00:11
*** yfaykya has joined #tripleo00:12
*** yamahata has joined #tripleo00:14
*** hrushig_ has quit IRC00:15
*** yfaykya has quit IRC00:17
xuhaiweiHi, i rebuilded devtest, but it failed in building undercloud, could anyone see this?
uvirtbotLaunchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New]00:18
*** yfaykya has joined #tripleo00:19
*** matsuhashi has joined #tripleo00:20
vinshwhat does the charset part at then end of this mean?00:21
vinsh          ovs_db: mysql://neutron:unset@localhost/ovs_neutron?charset=utf800:21
vinshand.. shouldn't this point to a virtual ip in a multi controller setup?00:21
greghaynesalthough we havent dont that part (mysql behind haproxy) yet00:21
vinshit seems that the controller nodes can't on setup.. as if its not setup yet00:21
greghaynesmaybe I should implement this00:22
* greghaynes hacks00:22
*** yfaykya has quit IRC00:23
*** yfaykya has joined #tripleo00:25
*** nati_ueno has quit IRC00:26
vinshman.. NeutronDSN..00:27
* vinsh 'splodes00:27
lifelessgreghaynes: stil sploding00:28
*** ramishra has joined #tripleo00:28
lifeless2014-06-19 00:07:46.977 | + grep trusty-server-cloudimg-amd64-root.tar.gz SHA256SUMS.ubuntu.trusty.amd6400:28
lifeless2014-06-19 00:07:48.434 | trusty-server-cloudimg-amd64-root.tar.gz: FAILED00:28
lifeless2014-06-19 00:07:48.437 | sha256sum: WARNING: 1 computed checksum did NOT match00:28
* greghaynes does sha check on personal box00:29
lifelessI bet we have a squid cache in that region00:29
*** yfaykya has quit IRC00:29
greghaynesyep, must be a cache issue00:30
greghaynesthe sha checked out for me00:31
*** yfaykya has joined #tripleo00:31
lifelesswhat we need is cache-bust-on-sha-fail00:31
StevenKlifeless: Ng has a patch up for that00:31
lifelessStevenK: Idon't see it in dib00:32
SpamapSlifeless: I thought there was a patch in flight that did something like that00:32
*** ramishra has quit IRC00:32
lifelessah its there but failed CI00:33
StevenKI just found it myself00:33
SpamapSwe all just wasted time in parallel00:33
lifelessit passed our functional tests00:34
*** yfaykya has quit IRC00:35
*** CaptTofu_ has quit IRC00:36
*** yfaykya has joined #tripleo00:37
*** CaptTofu_ has joined #tripleo00:37
*** yfaykya has quit IRC00:41
*** CaptTofu_ has quit IRC00:41
*** chuckC has quit IRC00:42
*** yfaykya has joined #tripleo00:43
*** nosnos has joined #tripleo00:43
vinshvery confused here.. compute nodes try to talk to ovs_db ending in a url of "neutron" where as the controller nodes are attempting at "ovs_neutron"00:43
vinshyet.. control nodes can not talk to "ovs_neutron" sql00:44
vinshwish I knew what each was. :)00:44
SpamapSAFAIK there is no "neutron" database00:45
SpamapSovs_neutron is the only one00:45
lifelessvinsh: pastebin ?00:46
vinshcheckout overcloud-source.yaml00:46
vinshfor ovs_db on compute nodes they reference:00:47
vinsh        NeutronDSN: {"Fn::Join": ['', ['mysql://neutron:unset@', {'Fn::Select': [ip_address, 'Fn::Select': [0, 'Fn::GetAtt': [ControlVirtualIP, fixed_ips]]]}, '/neutron']]}00:47
SpamapSvinsh: that may be erroneous, because IIRC, the agents only speak over RPC00:47
SpamapSso compute nodes aren't accessing neutron's DB00:47
vinshyet controller try for "          ovs_db: mysql://neutron:unset@localhost/ovs_neutron?charset=utf8"00:47
vinshwhich fails00:47
*** yfaykya has quit IRC00:48
vinshI wonder which element initializes the ovs_neutron entries?00:48
lifelesshuh I thought the agents spoke to DB too00:48
vinshI have all the controllers talking to mysql on one node for now..00:48
lifelessbecause we had races with neutrons special approach to schema applications00:49
vinshand everything works.. but this ovs_db business.00:49
vinshcauses a stack trace on "00:49
vinsh+ neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head"00:49
*** yfaykya has joined #tripleo00:49
lifelessthere's something wrong for sure00:49
vinshorly? ;)00:49
vinshah. ./elements/boot-stack/bin/reset-db:os-db-create ovs_neutron neutron $db_pass00:51
greghaynesits possible were just creating the db and not actually using it?00:51
lifelessok -> gym with C, bbiac hours00:52
vinshcompute nodes are using "neutron" control nodes are trying for "ovs_neutron"00:52
*** chuckC has joined #tripleo00:52
*** yfaykya has quit IRC00:53
lifelesswell this might explain the epic vif plugging failures we have atm00:54
vinshwhat I am rambling about? or something else?00:54
*** yfaykya has joined #tripleo00:55
*** yfaykya has quit IRC01:00
*** TravT has quit IRC01:01
*** yfaykya has joined #tripleo01:01
*** dprince has joined #tripleo01:03
dprinceCI update: the undercloud jobs all seem to fail due to
uvirtbotLaunchpad bug 1331874 in tripleo "trusty-server-cloudimg-amd64 checksum mismatch" [High,Triaged]01:04
dprinceIs there anything we can do to work around that? ^^^01:04
*** yfaykya has quit IRC01:06
*** yfaykya has joined #tripleo01:08
*** CaptTofu_ has joined #tripleo01:08
michchapHi folks, is precise supported for running devtest?01:09
vinsh13.04 -14.1001:09
vinshCan't say for precise :)01:09
StevenKmichchap: It *may* work with the Ubuntu Cloud Archive, but as vinsh says01:09
*** spzala has quit IRC01:10
michchapThanks. Just trying to evaluate tripleo rather than deploy anything serious at this point. Is devtest what I should be using for that?01:10
*** dprince has quit IRC01:11
*** yfaykya has quit IRC01:12
*** yfaykya has joined #tripleo01:14
*** yfaykya has quit IRC01:18
*** yfaykya has joined #tripleo01:20
*** yfaykya has quit IRC01:24
*** eguz has joined #tripleo01:25
*** yfaykya has joined #tripleo01:26
*** ramishra has joined #tripleo01:29
*** eghobo has quit IRC01:29
*** mestery has joined #tripleo01:29
*** yfaykya has quit IRC01:30
*** yfaykya has joined #tripleo01:32
*** ramishra has quit IRC01:34
*** mestery has quit IRC01:34
*** yfaykya has quit IRC01:37
*** yfaykya has joined #tripleo01:38
*** eguz has quit IRC01:42
*** yfaykya has quit IRC01:43
*** yfaykya has joined #tripleo01:44
*** yfaykya has quit IRC01:49
*** CaptTofu_ has quit IRC01:50
*** CaptTofu_ has joined #tripleo01:50
*** yfaykya has joined #tripleo01:50
*** mestery has joined #tripleo01:54
*** CaptTofu_ has quit IRC01:55
*** yfaykya has quit IRC01:55
*** yfaykya has joined #tripleo01:57
*** yfaykya has quit IRC02:01
*** yfaykya has joined #tripleo02:03
*** yfaykya has quit IRC02:07
*** matsuhashi has quit IRC02:07
*** matsuhashi has joined #tripleo02:08
*** matsuhashi has quit IRC02:08
*** matsuhashi has joined #tripleo02:08
*** yfaykya has joined #tripleo02:09
*** yfaykya has quit IRC02:13
*** yfaykya has joined #tripleo02:15
*** yfaykya has quit IRC02:19
*** mestery_ has joined #tripleo02:20
*** yfaykya has joined #tripleo02:21
*** mestery_ has quit IRC02:21
*** mestery_ has joined #tripleo02:22
lifelessok 60 seconds wasn't enoug02:22
*** mestery has quit IRC02:23
*** yfaykya has quit IRC02:26
*** yfaykya has joined #tripleo02:27
*** CaptTofu_ has joined #tripleo02:29
*** ramishra has joined #tripleo02:29
*** bnemec has quit IRC02:30
lifelessgreghaynes: so ping02:31
greghaynessuch ping?02:31
lifelessgreghaynes: AIUI mysql isn't accessible through haproxy yet, right ?02:31
lifelessso all these         NeutronDSN: {"Fn::Join": ['', ['mysql://neutron:unset@', {'Fn::Select': [ip_address, 'Fn::Select': [0, 'Fn::GetAtt': [ControlVirtualIP, fixed_ips]]]}, '/neutron']]}02:31
lifelessare entirely bogus / untested02:31
* greghaynes verifies02:32
*** yfaykya has quit IRC02:32
lifelessor is it working because mysql is bound to * ?02:32
lifelessindeed         neutron_dsn: {Ref: NeutronDSN}02:33
lifelessis not used in tie at all02:33
lifelessmuch bong02:33
greghaynesmysql is infact bound to *02:33
*** yfaykya has joined #tripleo02:33
lifelessvinsh: your local elements must have a substantial delta vs tie02:33
lifelessvinsh: since we used:02:33
lifelesselements/neutron/os-apply-config/etc/neutron/plugins/ml2/ml2_conf.ini:connection = {{neutron.ovs_db}}02:34
lifelesselements/neutron/os-apply-config/etc/neutron/plugins/ml2/ml2_conf.ini~:connection = {{neutron.ovs_db}}02:34
lifelesselements/neutron/os-apply-config/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini:sql_connection = {{neutron.ovs_db}}02:34
*** ramishra has quit IRC02:35
lifelesswe go NeutronDSN -> neutron_dsn -> ovs_db02:35
openstackgerritlifeless proposed a change to openstack/tripleo-heat-templates: Use the right DSN for neutron agents.
vinshso ya.. mysql is bound to *02:37
vinshonly thing having issue using it is ovs_db on controller nodes02:37
vinshall other services going that route (to vip mysql) work.02:37
vinshya.. awful chain of dsn links there :)02:38
*** ramishra has joined #tripleo02:38
vinshlol @10108002:39
greghaynescant argue with that02:39
lifelessfingers cossed it will be the issue02:40
vinshwhere do you see this issue at?02:40
lifelessvinsh: all our CI job are failing for the last ~36 hours02:40
lifelessvinsh: image deployed in the OC can't be pinged.02:40
lifelessvinsh: and folk are having trouble reproducing locally02:41
vinshwell. whatever the case is.. that ovs thing is giving me hell.02:41
vinshnode can connect to it on local host just fine.02:41
vinshtry to use node ip/vip02:41
greghaynesvinsh: is your controller VIP?02:44
vinshSo in .. you have compute nodes using the vip to get ovs_db02:45
lifelessis your mysql bindigto 0000?02:45
greghaynesis controller0 up and done bootstrapping?02:45
vinshbut on line 466 control nodes use localhost for that?02:45
vinshgreghaynes, its fails early on looping on that failure02:45
lifelessvinsh: oh I'm sure there are other issues, I just trying something to see if it unbreaks CI02:45
lifelessvinsh: this is why we need CI to have 3 nodes ctl plane :)02:46
greghaynesvinsh: I bet keepalived hasnt brought up the VIP yet02:46
lifelessvinsh: also note that the hypervisors may try to connect before the ctl plane is even deployed02:46
lifelessvinsh: we sequence the configs, but not the controller machine nodes02:46
lifelessvinsh: so errors like that are entirely normal, in that situation02:46
vinshhm. hard to work around in this case.02:47
vinshuntil I cluster mysql.02:47
vinshgot a set of 6 patches that do that from Therese  that she gathered from upstream.02:48
*** mestery_ is now known as mestery02:48
greghaynesvinsh: try restarting keepalived service on controller002:48
vinshat what point?02:48
greghaynesjust by hand to see if that is the issue02:48
vinshor .. ip addr show br-ex ?02:49
vinshsee if vip exits02:49
greghayneslooks like we restart keepalived in post-config02:49
*** noslzzp has joined #tripleo02:49
greghaynesso that should probably get moved up02:49
greghaynesinto config.d/50- or somesuch02:49
lifelesswe do most of our restarts post-config02:50
lifelessperhaps post-config 10 or something02:50
vinshis it right to want to access mysql on vip though?02:50
greghaynesWe need to hit mysql in config.d/51-02:50
lifelessvinsh: does the whole thing halt for you - it should try to reconfigure 30 seconds later02:50
vinshor should it be done on localhost.. with mysql clustered?02:50
greghaynesvinsh: needs to be VIP02:51
vinshlifeless, ya. it tries every 30 seconds and fails.02:51
lifelessgreghaynes: arguably then the reconfigure should pause until the VIP is pingable02:51
lifelessgreghaynes: or some timeout occurs02:51
lifelesse.g. wait_for 30 1 ping -c 1 $vip02:51
lifelessif it doesn't come up on *some* control node, we can't proceed02:51
*** ramishra has quit IRC02:51
greghaynesthe problem is we need to restart keepalived after weve templated the config when metadata is up, and before we try to hit mysql02:51
greghaynesotherwise we have right config on disk but its not applied02:52
greghaynesso VIP isnt up02:52
lifelessgreghaynes: wait why02:52
greghaynesrestarting keepalived is what actually adds the VIP to the interface02:53
lifelessgreghaynes: popping out for a second, but local db creation etc has to be local because of the cluster init races, no ?02:53
lifelessvinsh: is this error on the controller or the hypervisor?02:53
vinshcontroller, compute nodes do not hit this (probably cause they wait till vip is up on controller)02:54
vinshgot a stack re-launching to check on vip status during this failure loop.02:54
*** ramishra_ has joined #tripleo02:56
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
uvirtbotLaunchpad bug 1329238 in tripleo "OVS isn't persisting mac addresses on OVS bridges" [Undecided,In progress]03:00
uvirtbotLaunchpad bug 1331337 in tripleo "install_dependencies fails if python-software-properties is not installed." [Undecided,In progress]03:00
*** untriaged-bot has quit IRC03:00
uvirtbotLaunchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New]03:00
*** ramishra has joined #tripleo03:00
*** ramishra_ has quit IRC03:01
vinshgreghaynes, so here's the story. keepalived can restart fine.03:12
vinshBUT, br-ex isn't up yet on the controller node03:12
vinshso, there really isn't a vip setup yet.03:13
vinshGood find :)03:13
vinshroot@overcloud-controller0-gw4pdihxsskk:~# ip addr show br-ex03:14
vinshDevice "br-ex" does not exist, error in cloud, stop, drop and roll.03:14
greghaynesError in cloud!03:14
*** rwsu has quit IRC03:27
*** CaptTofu_ has quit IRC03:28
*** CaptTofu_ has joined #tripleo03:29
*** CaptTofu_ has quit IRC03:33
*** ramishra has quit IRC03:33
*** blamar has joined #tripleo03:38
*** blamar has quit IRC03:38
*** matsuhashi has quit IRC03:38
*** matsuhashi has joined #tripleo03:39
*** akuznetsov has joined #tripleo03:43
*** matsuhashi has quit IRC03:45
*** matsuhashi has joined #tripleo03:45
*** matsuhashi has quit IRC03:46
*** victor_lowther has quit IRC04:02
*** nosnos has quit IRC04:02
*** victor_lowther has joined #tripleo04:05
*** mestery has quit IRC04:06
*** mestery has joined #tripleo04:07
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-image-elements: Reserve the keystone high port out of ephemeral
morganfainbergStevenK, hey, for ^ does sysctl-set-value override the current value?04:09
vinshgreghaynes, how would I turn this into syntax to get the ip of controller0 only instead of the vip? :
vinshjust can't get the heat syntax right here04:09
morganfainbergStevenK, or does it do magic to append etc, because if it just does a plain override, you'd need to do port1,port2 instead of setting twice04:09
morganfainbergStevenK, haven't had a chance to look at that underlying code though yet.04:10
*** mestery has quit IRC04:11
StevenKmorganfainberg: Hmm04:12
StevenKmorganfainberg: Let me pull up a seed with the change04:12
morganfainbergStevenK, ++04:12
morganfainbergStevenK, gonna clock out and grab some sleep, let me know if you need anything.04:13
StevenKmorganfainberg: Aye, thanks04:14
greghaynesvinsh: Why would you want to do that?04:15
greghaynesvinsh: also, I dont know off the top of my head04:16
vinshto get around this blocker... I can't use localhost.. because that isolates each node until I cluster mysql04:16
vinshI cant use the vip04:16
vinshbut all other nodes are using controller0 for mysql.. (it usually gets the vip)..04:16
vinshso I figure.. "hey lets hardcode that.. and move along for a while"04:16
greghaynesSo I think the only two sane options are using the vip or localhost04:17
greghaynespreferrably, we use the vip04:17
greghaynesI dont know what bring up that interface though...04:17
greghaynesI also dont know what lifeless was hinting at re: startup race issues... I hope we dont bring up a mysql master people can connect to until its actually clustered04:18
vinshroger that04:19
greghayneswait, have you tried using mysql?04:20
greghayneser sorry, have you tried using localhost?04:20
vinshyes.. and that works.. no error connecting.  But then when the demo vm is launched..04:20
vinshneutron never seems right04:20
vinshdevtest_overcloud fails on the neutron port-list part after nova boot04:21
vinshProbably because each node is talking to ovs_db on its own localhost?04:21
vinshso thats no good.04:21
greghayneshuh. I really dont think we should be proceeding beyond config.d/51- unless mysql is started and joined to the cluster04:22
vinshI guess what I am saying.. is localhost doesn't seem any good until that mysql is clustered.04:22
greghaynesyes, but we shouldnt be starting mysql any other way04:22
vinshok sure, but I don't think there is any clustering of mysql in this yet.04:22
vinshthats wip right?04:22
greghaynesAre you not using ?04:23
vinshnot yet, in what I have04:23
vinshtime to pull that in I think.04:23
greghaynesah, use that, and there are some recent merged things in t-i-e you need too04:24
vinshOk, I follow.  Need clustered mysql before anything else here.  I'll pull that in to my mix tomorrow.  Therese has been working on that clustering part.. and we have all the bits needed for it to work.. just need to merge it into this controller scaling work.  Will report back after that. :)04:26
greghaynesYep, localhost should always work IMO but VIP is preferred so we prevent excess rollbacks04:27
vinshOff for now, have a good night/day/etc!04:29
*** vinsh is now known as vinsh_zzz04:29
lifelessgreghaynes: so a node thats joined the cluster can be connected to before the IST completes, AIUI04:31
lifelessgreghaynes: that was the race I was referring to04:32
greghaynesand it will accept connections?04:33
lifelessthats my understanding04:33
tchaypoI've een having a horribly day;04:33
tchaypoI've been having one of those horrible days where I just can't manage to engage my brain or type properly04:34
tchaypoI'd given up on getting anything useful done today04:34
lifelessI've got diaahorea, fever, dehydration04:34
lifeless-> useless04:34
clarkbeveryone has been sick04:34
greghayneswow, srsly04:35
tchaypothen a friend asks about gettins started with tripleo, i look at the getting started docs, notice some ways they can be improved, suddenly I'm all cylenders go04:35
tchaypoexcept the spelling cylinder04:35
tchaypoDo we have any idea what's breaking CI yet?04:36
tchaypomaybe it's just caught the same virus as everyone else?04:37
*** matsuhashi has joined #tripleo04:37
lifelessbnemec managed to reproduce, once.04:38
tchaypos/reproduce/arrive at same symptoms, possibly (but unprovenly) via the same cause/ ?04:39
*** nosnos has joined #tripleo04:43
greghayneslifeless: So I think the race condition depends on your SST method:
greghaynesif we rsync (I think thats what we do?) its not an issue04:44
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-image-elements: Reserve the keystone high port out of ephemeral
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Fix --heat-env option handling for undercloud.
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Add -c support to
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Fix --build-only.
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Cleanup post controller parameter rename.
StevenKmorganfainberg: Two calls leads to the element erroring because they conflict, so CI (if it was working) would have picked it up. I've pushed a fix04:50
morganfainbergStevenK, ++ [i swear i'm going to bed :P]04:51
lifelessStevenK: beep04:51
StevenKlifeless: Yeees?04:52
lifelessStevenK: check your review :)04:52
StevenKmorganfainberg: Sure sure04:52
*** akuznetsov has quit IRC04:53
StevenKlifeless: Using the sysctl, elements can only say "Make it be this", and not "This should be in the values we set"04:54
lifelessI know04:54
lifelessnote that I said 'reasonable place to fix...'04:55
StevenKI see a rabbit hole beneath me04:55
*** matsuhashi has quit IRC04:56
lifelessferret maybe04:56
*** matsuhashi has joined #tripleo04:57
*** akuznetsov has joined #tripleo04:58
StevenKCan't work out why node["pm_password"] is null on CI04:59
lifelessStevenK: age?05:00
lifelessStevenK: we haven't redeployed in a bit05:00
lifelessthe json dict is shown at the top of the run05:01
tchaypo <<- the most useful thing I've done all day05:01
*** dshulyak_ has joined #tripleo05:13
StevenKThe sysctl element has two useless uses of cat05:14
*** CaptTofu_ has joined #tripleo05:17
*** jtomasek has joined #tripleo05:21
*** CaptTofu_ has quit IRC05:21
*** tzumainn has quit IRC05:22
*** nati_ueno has joined #tripleo05:23
rushiagrfollowing os-collect-config readme gave me this error while installing it via pip05:30
rushiagrconstitutes a (doc) bug?05:31
*** rakesh_hs2 has joined #tripleo05:35
*** bmahalakshmi has joined #tripleo05:37
*** akuznetsov has quit IRC05:43
*** pcrews has quit IRC05:45
*** rbrady has quit IRC05:51
openstackgerritlokesh s proposed a change to openstack/tripleo-incubator: Make install-dependencies use sudo -E to enable use of env proxy.
*** morganfainberg has quit IRC05:57
lifelessyay + install-packages build-essential libz-dev libxslt-dev libxml2-dev python-dev05:58
*** morganfainberg has joined #tripleo05:58
openstackgerritlokesh s proposed a change to openstack/tripleo-incubator: Make install-dependencies use sudo -E to enable use of env proxy.
lifelessrushiagr: it does yes, though if you had wheels of the xml bindings it probably wouldn't error05:59
*** lazy_prince has joined #tripleo06:00
lifeless python-dev : Depends: python (= 2.7.5-5ubuntu1) but 2.7.5-5ubuntu3 is to be installed06:00
rushiagrlifeless: okay. thanks06:00
*** ProfFalk1n has joined #tripleo06:00
lifelesssometimes metapackages give me the s****06:01
*** ProfFalk1n has quit IRC06:03
*** vigneshvar has joined #tripleo06:06
vigneshvarHI, I am running an overcloud , i have seen that my stack still in progress and finally fails.06:07
vigneshvarI logged into the VM and checked the os-collect-config06:07
vigneshvarIt says  key db-password does not exist in ['/var/lib/os-collect-config/heat_local.json', '/var/lib/os-collect-config/ec2.json', '/var/lib/os-collect-config/cfn.json']06:08
vigneshvarand iterates all the time. Can anyone help me06:08
*** akuznetsov has joined #tripleo06:09
lifelesscan you pastebin heat resource-list overcloud ?06:09
vigneshvarsure. Just a min06:10
*** mrunge has joined #tripleo06:11
lifelessvigneshvar: so this06:12
lifeless| controller0Deployment    | OS::Heat::StructuredDeployment | CREATE_IN_PROGRESS | 2014-04-18T19:21:32Z |06:12
lifelessI would have expected that to include db-password06:13
vigneshvarlifeless, yes it never completes06:13
lifelessvigneshvar: what does os-collect-config --print output (be sure to edit out passwords)06:13
lifelessvigneshvar: it will toggle to COMPLETE only once the controller pings-back06:14
*** rakesh_hs2 has quit IRC06:15
*** rakesh_hs2 has joined #tripleo06:16
vigneshvarlifeless, Give me a min, i will just pastebin the output06:19
*** pcrews has joined #tripleo06:20
*** jtomasek has quit IRC06:21
*** andreaf has joined #tripleo06:22
openstackgerritxu-haiwei proposed a change to openstack/tripleo-incubator: Do not set NeutronControlPlaneID when building image only
mariosyah so 'recheck on bug' doesn't work, fyi :)06:24
marioslifeless: are you going to tripleo mid-cycle - and which one (july/august?)06:25
lifelessmarios: I hope so :)06:26
mariosah, just checked again but couldn't see you06:26
mariosare you going to heat one?06:26
lifelesswe're just sorting out paper work internally :(06:27
*** dshulyak_ has quit IRC06:29
*** dshulyak_ has joined #tripleo06:29
vigneshvarDEBUG heat.common.serializers [req-ae9c6a16-4e54-4d14-858f-c42c40e9c0de None demo] JSON response : {"software_deployment": {"status": "IN_PROGRESS", "server_id": "aa8ff79d-179a-41a2-8340-e6b9992e7f2d", "config_id": "2af2db82-9f77-4f1e-ac5e-3c97624ca45c", "output_values": null, "input_values": null, "action": "CREATE", "status_reason": "Deploy data available", "id": "7828c0fc-9391-4fdd-97da-5bf82730e07f"}} from (pid=19864) to_json /opt/stack/heat/heat/06:30
StevenKThis whole backup image thing is terrible06:30
StevenK-rw-r--r-- 1 steven steven 2.4G Jun 12 16:58 seed-2014.06.13-14.12.38.qcow206:30
StevenK-rw-r--r-- 1 steven steven 2.4G Jun 13 14:12 seed-2014.06.13-15.00.41.qcow206:30
StevenK-rw-r--r-- 1 steven steven 2.4G Jun 13 15:00 seed-2014.06.13-15.35.36.qcow206:30
lifelessvigneshvar: interesting -06:30
lifeless     "db-password": "unset",  is in there06:30
lifelessvigneshvar: you will need to reset your admin token and perhaps others since you've pasted them on the internet :)06:31
lifelessStevenK: huh when did we start doing that?06:31
vigneshvarlifeless, ya sure will do that06:31
StevenKlifeless: I'm not sure.06:31
*** jprovazn has joined #tripleo06:32
StevenKlifeless: I'm getting rather tempted to check, since 10 seed images just wiped out all the free space in /home.06:32
mariosStevenK: ouch are you also sending those on the network or at least stored on seed building host? (or attached logical volume?). is that hourly?06:32
StevenKmarios: I'm testing something, so rebuilding the seed06:32
vigneshvarlifeless: "db-password" is expected to be unset ?06:33
lifelessvigneshvar: unset is a (poor) default we have06:33
lifelessvigneshvar: its a value06:33
StevenKlifeless: b740c50c7625162a6ba4499a7dfbc7450deac3eb in dib06:33
StevenKMerged in fa9a561819197c136a3740cdd81c8631e78640d4 on the 12th06:34
vigneshvarlifeless: i can see that there is a key db-password, but why is not getting recognized/parsed06:35
lifelessvigneshvar: can you pastebin the end of os-collect-config.log? From the last time it started a poll06:39
lifelessvigneshvar: thats at least half an hour ago06:41
lifelessvigneshvar: it should be retrying every 30 seconds06:41
lifelessvigneshvar: so there is something very wrong :)06:43
vigneshvarlifeless, that server had a time set differently, i manually started os-collect-config and i get the same06:44
lifelessvigneshvar: time skew can break authentication06:44
lifelessvigneshvar: you should fix that before trying anything else06:44
*** e0ne has joined #tripleo06:45
vigneshvarlifeless, undercloud and overcloud are in sync06:46
lifelessI guess we need to check you have a new enough os-collect-config06:48
lifelessrun /opt/stack/venvs/os-collect-config/bin/pip freeze06:48
lifelessStevenK: you seeing: --trash-my-machine 2>&1 | tail -n 50 | pastebinit06:49
lifelessand now I figure it out06:49
StevenKNot with my local mirror, at least06:51
vigneshvarp freeze06:51
lifelessvigneshvar: uhm pastebint!06:52
StevenKvigneshvar: Pastebin!06:52
lifelessvigneshvar: ok so 0.1.10 is very very old06:52
lifelessvigneshvar: I think it predates sofware config in the heat templates06:52
lifelessvigneshvar: so you'll need a newer os-collect-config06:53
*** dtantsur|afk is now known as dtantsur06:53
lifelessyeah, 16158684a68f8e5c8d1b896f8bc12bbb0e9fc66d is what you need whichisn't in 0.1.1006:53
lifelessand 831ab0be03f2b5635534321611cd0fb1b5a4412706:53
lifeless0.1.13 seems to be the minimum release you need06:54
lifelessvigneshvar: but I recommend 0.1.2006:56
lifelesssince we've been fixing bugs with interactions with heat etc06:56
vigneshvarlifeless, sure i will rebuild a new diskimage and update you06:57
vigneshvari will rebuild with specified version06:58
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-image-elements: Do not use cat in sysctl-set-value
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-image-elements: Add a sysctl-append-value script
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-image-elements: Reserve the keystone high port out of ephemeral
*** xuhaiwei has quit IRC07:03
*** andreaf has quit IRC07:03
jprovazncan someone approve (enough +2 and tests passed)07:03
greghaynesjprovazn: tests passed a while ago :/07:04
lifelessjprovazn: CI is wedged07:05
*** CaptTofu_ has joined #tripleo07:05
*** lifeless changes topic to "CI DOWN, - can't ping instances | Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt"07:05
lifelessjprovazn: we're all trying to reproduce it atm07:05
lifelessso we can fixenate it07:06
*** e0ne has quit IRC07:06
jprovazngreghaynes: well, there is no policy about tests freshness07:07
greghayneshehe, true07:07
mariosjprovazn: why don't you approve it?07:08
*** shardy_afk is now known as shardy07:08
* marios looks at comments07:08
lifelessjprovazn: actually there is one for openstack as a whole07:08
marioslifeless: i have just seen one, don't know if it the same one you're talkinga bout. am trying to investigate a bit more ( /#/c/95396/ Check for relevant environment variables )07:08
lifelessmarios: jprovazn: please don't land stuff other than fixing CI - last thing we need is more destabilisation07:08
marioslifeless: ah thanks07:09
marioslifeless: you might want to send a firedrill type thing to list for better coverage, if it's like that07:09
lifelessmarios: I might tag you to do that, since I'm out for the day - I've got a virus or some such07:09
marioslifeless: basically there the overcloud can't ping the demo box it launches07:09
lifelessmarios: yes07:09
lifelessthere's a bug for it07:09
marioslifeless: awesome thanks, no?07:09
marioslifeless: (number)07:10
lifelessI pushed up I0f27e95c63ee4de023a7c8248054f668fa7274cf as one attempt07:10
*** CaptTofu_ has quit IRC07:10
lifelessand I7ed40078f487459dee9055ef41f10a9b60a0e674 as another07:10
*** jcoufal has joined #tripleo07:10
marioslifeless: great thx07:10
lifelessanyhow, me -> out. to sleep perchance feel less ass07:10
marioslifeless: can you set chan topic "please don't land stuff other than fixing CI - last thing we need is more  destabilisation07:12
marioslifeless: ah... :) irssi truncates it for me07:13
marioslifeless: have a good rest07:13
*** lifeless changes topic to "CI DOWN, - can't ping instances; only land CI fixes | Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt"07:14
lifelessmarios: tada; cat food changed, now sleep.07:14
*** ajc_ has joined #tripleo07:17
*** athomas has joined #tripleo07:21
*** marios_ has joined #tripleo07:24
*** hewbrocca has joined #tripleo07:26
*** jeremydei has quit IRC07:27
*** marios has quit IRC07:27
*** jeremydei has joined #tripleo07:28
*** pcrews has quit IRC07:37
*** yolanda has joined #tripleo07:44
yolandahi, having a problem here:
d0ugallsmola__: Unless you can provide a complete example of a full template model I'm not sure I know how to fix it based on those comments07:44
yolandawith undercloud-precise and overcloud-f28 jobs07:44
d0ugallsmola__: I didn't fully understand it which was why I didn't write that bit ;)07:44
yolandaGheRivero, lifeless, can you help me ?07:45
d0ugallsmola__: So, I'll try making a guess but I expect the spec will need to be updated when somebody can correct me.07:45
lsmola__d0ugal: so the template definition should start at line 3407:45
lsmola__d0ugal: dependencies there should be just some string indexes (like filename or type)07:46
lsmola__d0ugal: template should not point to other template, you are passing that relation to heat stack create07:46
lsmola__d0ugal: saying this string 'compute.yaml' stands for this json_dumped_object, files={'compute.yaml': json_dump}07:47
d0ugallsmola__: Where does it point to another template?07:48
lsmola__d0ugal: the spec?07:48
d0ugallsmola__: Are you referring to how the dependency has been done?07:48
lsmola__d0ugal: I am reffering to how tzumainn was defining it in the etherpad07:49
d0ugallsmola__: I can't see the template pointing to another template?07:49
d0ugalI might be reading it wrong, and I've not fully read your additions yet07:49
lsmola__d0ugal: well is is poiting to some index, so it you have in your template get_file: compute.yaml07:50
d0ugalI think we should wait and have a chat with tzumainn later07:50
lsmola__d0ugal: it's not pointing to a template compute.yaml, it's just index, and you will define the content in files dictionary passed to stack-create07:50
lsmola__d0ugal: so you can then send files={'compute:yaml': "dumbed_compute_version2"} or files={'compute:yaml': "some other template"}07:52
lsmola__d0ugal: I am just making sure you are not planning to relate Template object to other Template objects or Provider resource objects on this level07:53
lsmola__d0ugal: I've added step to line 166, which should be handling all of these relations07:56
d0ugallsmola__: Thanks, I'll try and take this into account07:57
lsmola__d0ugal: ok07:57
d0ugallsmola__: I'm a bit confused still TBH, but when I get back to this bit I'll ping you07:57
d0ugallsmola__: It might be easier with tzumainn around too, as he knows exactly what he intended here :)07:57
lsmola__d0ugal: ok07:57
*** xuhaiwei has joined #tripleo07:59
*** gfidente has joined #tripleo08:00
*** nati_ueno has quit IRC08:01
*** e0ne has joined #tripleo08:05
*** markmc has joined #tripleo08:05
yolandaGheRivero, lifeless, now only one of the jobs fail:
*** e0ne has quit IRC08:13
*** e0ne has joined #tripleo08:23
*** jp_at_hp has joined #tripleo08:32
openstackgerritOm Kumar proposed a change to openstack/diskimage-builder: Remove unwanted kernel/initrd.
*** pelix has joined #tripleo08:35
*** andreaf has joined #tripleo08:37
*** yolanda has left #tripleo08:40
*** yolanda has joined #tripleo08:42
*** nosnos has quit IRC08:44
*** lucasagomes has joined #tripleo08:45
*** nosnos has joined #tripleo08:46
*** rcarrill` has quit IRC08:50
*** CaptTofu_ has joined #tripleo08:53
*** rakesh_hs2 has quit IRC08:54
*** rakesh_hs2 has joined #tripleo08:58
*** CaptTofu_ has quit IRC08:58
*** untriaged-bot has joined #tripleo09:01
untriaged-botUntriaged bugs so far:09:01
*** untriaged-bot has quit IRC09:01
uvirtbotLaunchpad bug 1331931 in tripleo "NeutronControlPlaneID should not be set if building image only" [Undecided,In progress]09:01
uvirtbotLaunchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New]09:01
*** rcarrillocruz has joined #tripleo09:01
rushiagrHi All!09:05
rushiagrCan I create ubuntu 'trusty' images using DIB?09:05
rushiagrI have a precise machine.09:05
*** e0ne_ has joined #tripleo09:11
*** e0ne has quit IRC09:13
*** openstackgerrit has quit IRC09:14
*** martyntaylor has joined #tripleo09:17
*** jtomasek has joined #tripleo09:20
gfidenteguys, which version of ubuntu do we use for tripleo-ci ?09:28
jprovazngfidente: "precise" I suppose - check jobs are names like this: "check-tripleo-undercloud-precise"09:35
gfidentejprovazn, ah yeah :( sorry about the stupid question09:35
jprovazngfidente: np09:36
*** e0ne_ has quit IRC09:38
*** e0ne has joined #tripleo09:39
*** nosnos has quit IRC09:40
*** jprovazn has quit IRC09:42
*** e0ne has quit IRC09:43
*** matsuhashi has quit IRC09:44
*** matsuhashi has joined #tripleo09:44
*** matsuhashi has quit IRC09:46
*** matsuhashi has joined #tripleo09:47
*** IvanBerezovskiy1 has joined #tripleo09:49
*** IvanBerezovskiy1 has left #tripleo09:50
*** IvanBerezovskiy1 has joined #tripleo09:50
*** akuznetsov has quit IRC09:52
*** xuhaiwei has quit IRC10:03
*** jprovazn has joined #tripleo10:14
*** yamahata has quit IRC10:16
*** akrivoka has joined #tripleo10:17
*** CaptTofu_ has joined #tripleo10:36
shadowerStevenK: ping10:40
shadowerStevenK: I've a few questions about os-cloud-config, looks like you may be the right person to ask10:41
*** matsuhashi has quit IRC11:00
*** matsuhashi has joined #tripleo11:01
*** jtomasek has quit IRC11:01
*** e0ne has joined #tripleo11:05
*** matsuhashi has quit IRC11:05
*** dtantsur is now known as dtantsur|lunch11:11
*** jtomasek has joined #tripleo11:14
*** yamahata has joined #tripleo11:15
*** matsuhashi has joined #tripleo11:16
*** CaptTofu_ has quit IRC11:22
*** CaptTofu_ has joined #tripleo11:23
*** CaptTofu_ has quit IRC11:27
*** weshay has joined #tripleo11:28
*** julim has joined #tripleo11:36
*** akuznetsov has joined #tripleo11:39
*** ajc_ has quit IRC11:50
*** ajc_ has joined #tripleo11:50
StevenKshadower: Sorry, I was off having dinner and such, what can I help you understand?11:50
shadowerno worries11:52
shadowerStevenK: so if IIUIC, os-cloud-config exists because we can't express some init calls within Heat templates, right?11:53
shadowerthe keystone role and user config API calls specificaly11:54
*** bmahalakshmi has quit IRC11:54
*** ajc_ has quit IRC11:55
StevenKshadower: Well, we need to do Keystone init and such like before we can invote heat, yes -- and we need to share code between tripleo-incubator (specifically, devtest) and tuskar11:55
shadowerStevenK: I meant the undercloud Heat. I.e. if I want to deploy the overcloud, why do I need occ -- why can't I put that stuff in say the software config for the controller node in the overcloud heat template11:56
shadowerStevenK: what I'd like to understand is -- is this a limitation of Heat that we should (eventually) try to add there? The stuff that occ does sounds like orchestration to me11:57
StevenKshadower: Right, we will use os-cloud-config for things shared between the seed and the undercloud.11:57
StevenKRegistering nodes, that kind of thing.11:58
shadowerStevenK: oh. So it's not going to be used for the overcloud?11:58
StevenKshadower: I don't believe it will be used for the overcloud, I think that's all Heat.11:58
shadowerright okay11:58
shadowerI do understand why we need to have something like this for the seed11:58
*** rbrady has joined #tripleo12:11
*** matsuhashi has quit IRC12:13
*** matsuhashi has joined #tripleo12:14
StevenKgreghaynes: Thanks for the pointer to the other review -- I've commented and suggested he follow my work.12:16
lsmola__stevebaker: I believe you are talking about functionality that even shouldn't be in the os-cloud-config from the beginning12:16
lsmola__stevebaker: dam, sorry that was for StevenK12:16
StevenKlsmola__: Registering nodes? Initializing Keystone?12:17
lsmola__StevenK: the main part that is shared everywhere is init-keystone, register-endpoints12:17
lsmola__StevenK: possibly setup of neutron12:17
StevenKlsmola__: register-nodes is called for both the seed and the undercloud.12:18
*** matsuhas_ has joined #tripleo12:18
*** matsuhashi has quit IRC12:18
lsmola__StevenK: yea sure, it can be there12:19
lsmola__StevenK: thought for us the main part was
lsmola__StevenK: to line 426 probably, we will see12:20
lsmola__StevenK: though for UI we are currently not using the registering nodes from there, we just call the API's directly, for CLI it will be handy12:21
StevenKRight :-)12:22
lsmola__StevenK: btw. it was named config, because it was doing these post config stuff, maybe we should rename it to something more general12:22
lsmola__StevenK: registering_nodes doesn't fit much to config :-)12:23
StevenKlsmola__: Personally, I'm happy with the name os-cloud-config12:23
lsmola__StevenK: yeah I guess, it's not worth month long naming conversation :-)12:24
shadowerright, so occ will handle the init-keystone/ keystone role-create, setup end points, user-config and setup-neutron stuff, too, right?12:25
shadowerwhich we need for the overcloud as well12:25
*** CaptTofu_ has joined #tripleo12:29
*** CaptTofu_ has quit IRC12:33
lsmola__shadower: yes, all the config should be extracted to occ12:37
lazy_princehi all.. is there any patch out for uboot support in dib..? i am working on a similar stuff and so need to know.12:39
*** vigneshvar has quit IRC12:39
*** tzumainn has joined #tripleo12:40
*** andreaf has quit IRC12:46
*** morazi has joined #tripleo12:46
*** CaptTofu_ has joined #tripleo12:53
*** lucasagomes is now known as lucas-hungry12:55
*** jcoufal has quit IRC12:56
*** vigneshvar has joined #tripleo13:04
*** mestery has joined #tripleo13:09
*** mestery has quit IRC13:09
*** mestery has joined #tripleo13:10
*** dtantsur|lunch is now known as dtantsur13:12
*** dprince has joined #tripleo13:14
*** jcoufal has joined #tripleo13:23
*** andreaf has joined #tripleo13:24
*** lucas-hungry is now known as lucasagomes13:33
*** d0ugal has quit IRC13:39
*** d0ugal_ has joined #tripleo13:39
*** d0ugal_ is now known as d0ugal13:40
*** sballe has joined #tripleo13:47
*** davidlenwell has quit IRC13:51
*** anteaya has quit IRC13:51
*** anteaya has joined #tripleo13:52
*** davidlenwell has joined #tripleo13:53
*** IvanBerezovskiy1 has left #tripleo13:56
*** dguerri has joined #tripleo13:57
*** TravT has joined #tripleo13:59
*** bcrochet has quit IRC14:00
jang1BLOODY eventlet.14:03
*** bcrochet has joined #tripleo14:03
jang1I'm beginning to think that rewriting ironic to be correctly threaded and to drop eventlet would be a faster way to a more robust approach.14:04
*** e0ne has quit IRC14:10
*** bnemec has joined #tripleo14:14
*** e0ne has joined #tripleo14:17
*** matsuhas_ has quit IRC14:17
*** dguerri is now known as _dguerri14:19
*** matsuhashi has joined #tripleo14:23
*** _dguerri is now known as dguerri14:24
*** jdob has joined #tripleo14:33
*** athomas_ has joined #tripleo14:33
*** rakesh_hs2 has quit IRC14:33
*** matsuhashi has quit IRC14:33
*** matsuhashi has joined #tripleo14:33
*** rwsu has joined #tripleo14:34
*** athomas_ has quit IRC14:36
*** cwolferh has joined #tripleo14:38
*** matsuhashi has quit IRC14:38
*** matsuhashi has joined #tripleo14:40
*** dguerri is now known as _dguerri14:41
*** _dguerri is now known as dguerri14:41
*** rcarrill` has joined #tripleo14:59
*** e0ne has quit IRC14:59
*** mrunge has quit IRC14:59
*** e0ne has joined #tripleo15:00
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
uvirtbotLaunchpad bug 1332021 in tripleo "check-tripleo-overcloud-precise job fails when collecting host state" [Undecided,New]15:00
*** untriaged-bot has quit IRC15:00
uvirtbotLaunchpad bug 1331931 in tripleo "NeutronControlPlaneID should not be set if building image only" [Undecided,In progress]15:00
uvirtbotLaunchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New]15:00
*** rcarrillocruz has quit IRC15:02
*** ddieterly has joined #tripleo15:04
*** e0ne has quit IRC15:04
*** jprovazn is now known as jprovazn_afk15:05
*** rcarrill` has quit IRC15:05
*** rcarrillocruz has joined #tripleo15:07
*** matsuhashi has quit IRC15:07
*** casanch1 has joined #tripleo15:07
*** matsuhashi has joined #tripleo15:07
jdoblifeless mentioned in the tripleo meeting "since dib had a regression you might want to see if the regression was released or not; if it wasn't, we should fix it before releasing", but i'm having trouble tracking down what regression he's talking about. anyone know?15:11
*** matsuhashi has quit IRC15:11
bnemecjdob: I _think_ he was referring to
*** blamar has joined #tripleo15:12
*** rcarrill` has joined #tripleo15:13
jdobdo you know if the f20 gate is happy again? I'll kick off a recheck on it and try to get that in before I release15:13
dprincejdob: it isn't happy yet.15:15
dprincejdob: also, it isn't Fedora specific15:15
*** rcarrillocruz has quit IRC15:15
jdobit meaning what, the reason it's not happy or that bug?15:16
dprincejdob: we ran check experimental and the ubuntu overcloud job failed in the same way15:16
dprincejdob: the issue is the ping to the overcloud instance (the use image) fails15:16
jdobah, ok. so i'm new to the release process and looking for advice. should I wait and hope this patch lands before late tomorrow or just release what we have and it'll get into next week's?15:18
*** athomas_ has joined #tripleo15:18
*** athomas_ has quit IRC15:18
*** bcrochet has quit IRC15:18
*** athomas has quit IRC15:19
dprincejdob: not sure, I've reviewed most of our commits and none of them look suspicious. "our" meaning the TripleO specific stuff15:19
dprincejdob: this could either be an environmental thing... or some add neutron/nova change is causing the issue15:20
*** jcoufal has quit IRC15:21
gfidentedprince, I was looking into today15:23
uvirtbotLaunchpad bug 1226310 in tripleo "Nova bm operations fail when LIBVIRT_DEFAULT_URI not set" [Medium,Triaged]15:23
*** bcrochet has joined #tripleo15:23
gfidentecan you help me figure where is that we set the default uri for the CI jobs? it definitely have to be set somewhere15:23
gfidente*libvirt default uri15:23
dprincegfidente: does it15:25
*** bcrochet has quit IRC15:25
gfidentedprince, yeah but that only applies to the scripts launched by devtest15:25
*** jtomasek has quit IRC15:26
dprincegfidente: we call directly in (in the tripleo-ci project)15:26
gfidentethe problem we're seeing is that the virtual power driver, when connects to the 'hosting' machine to poweron the baremetal node, doesn't connect to qemu:///system15:26
*** TravT has quit IRC15:26
dprincegfidente: SO I think this is outside of the scope of TripleO. I mean we don't own the seed host entirely...15:27
*** bcrochet has joined #tripleo15:27
dprincegfidente: you'd need to set that in the .bashrc of the host machine, or source it everytime...15:27
dprincegfidente: or perhaps just configure it correctly in the libvirt.conf file (in /etc)15:28
dprincegfidente: lots of ways to do it. perhaps one of those could got into tripleo-incubator as part of the seed host setup15:28
gfidentedprince, indeed15:28
*** athomas has joined #tripleo15:28
gfidenteI was trying to figure how is that accomplished for the CI jobs15:29
*** TravT has joined #tripleo15:35
*** ramishra has joined #tripleo15:38
*** jml has quit IRC15:39
*** jml has joined #tripleo15:40
*** ramishra_ has joined #tripleo15:41
dprincegfidente: ah, I see. So I'm not sure exactly why it is working there then. For CI we do setup our host machines a bit differently15:42
*** ramishra has quit IRC15:43
gfidentedprince, heh I'm still learning about it ... but I'd expect the undercloud would at some point poweron the overcloud VMs and that should take the same route15:44
*** vigneshvar has quit IRC15:44
gfidentevirtual power driver in the undercloud is going to ssh to the system hosting the test environment and powering up a VM15:44
dprincegfidente: will see if I can trace this and figure it out in a bit. Right now I'm debugging something else15:44
gfidentesure, understand that15:45
gfidentesorry :)15:45
*** eghobo has joined #tripleo15:52
*** eghobo has quit IRC15:52
*** eghobo has joined #tripleo15:53
*** eghobo has quit IRC15:53
*** eghobo has joined #tripleo15:54
*** e0ne has joined #tripleo15:57
*** lazy_prince has quit IRC16:00
*** e0ne has quit IRC16:01
*** e0ne has joined #tripleo16:03
*** jcoufal has joined #tripleo16:06
*** blamar has quit IRC16:18
*** martyntaylor has left #tripleo16:23
*** nati_ueno has joined #tripleo16:23
*** regebro|afk is now known as regebro16:30
*** akrivoka has quit IRC16:35
*** jcoufal has quit IRC16:41
*** jcoufal has joined #tripleo16:42
*** e0ne has quit IRC16:43
*** e0ne has joined #tripleo16:43
andreafhi - any core reviewer with a sec for a mini-patch with a +2 already?16:44
*** e0ne has quit IRC16:48
*** dkehn_ has joined #tripleo16:52
*** dkehn_ is now known as dkehnx216:53
*** dkehnx2 is now known as dkehnc16:53
*** dkehn has quit IRC16:54
*** dkehnc has quit IRC16:54
*** dkehnc has joined #tripleo16:55
*** dkehnx has quit IRC16:55
*** dkehnc is now known as dkehnx16:56
*** dkehn has joined #tripleo16:56
*** nati_ueno has quit IRC16:59
*** vinsh_zzz is now known as vinsh17:00
*** vinsh has quit IRC17:00
*** vinsh has joined #tripleo17:00
*** nati_ueno has joined #tripleo17:02
*** pcrews has joined #tripleo17:05
*** jang1 has quit IRC17:07
*** dkehn_ has joined #tripleo17:08
*** lucasagomes has quit IRC17:09
andrearosaSpamapS: if you are around and you have time, please can you see my reply to your comment on this bug
uvirtbotLaunchpad bug 1324598 in tripleo "Passthrough-enable for nova.conf mustache template is not correct" [High,In progress]17:10
*** dkehnx has quit IRC17:11
*** dkehn has quit IRC17:11
*** dkehn has joined #tripleo17:13
*** blamar has joined #tripleo17:15
*** dguerri has quit IRC17:19
SpamapSandrearosa: here now. looking.17:21
andrearosacool, thanks17:22
*** athomas has quit IRC17:22
SpamapSandrearosa: responded, and +2'd17:23
andrearosaSpamapS: thanks for your time.17:24
*** proffalken has quit IRC17:24
*** hewbrocca has quit IRC17:26
mordredSpamapS: hey - are we expecting f20 jobs to be successful in tripleo ci?17:27
mordred(I know we have issues with f20 nodepool, but I haven't really been fully following)17:28
mordredSpamapS: yolanda has been trying to get moving, but I think we're not sure if the errors it's seeing are from her, or are known things or what17:28
*** e0ne has joined #tripleo17:29
yolandamodred, that kind of errors shouldn't come from my change17:29
vinsh/tmp/in_target.d/pre-install.d/05-mysql: line 40: add-apt-repository: command not found17:31
vinshwhere is that found usually?17:31
bnemecmordred: No, our overcloud jobs are broken right now. :-(17:32
bnemecNetworking issues to the user instance in the overcloud.17:32
SpamapSmordred: yes we are expecting them to work17:33
SpamapSmordred: but I think dprince was working on an issue w/ pinging the end-user booted instance17:33
*** ramishra_ has quit IRC17:34
SpamapSoh add-apt-repository is running on fedora?!17:34
vinshthat^ where is that pulled in from? that tool?17:34
* SpamapS looks17:34
SpamapSoh wait no17:34
SpamapSthat is separate issue17:34
SpamapSvinsh: that is in python-software-properties17:34
vinshhm. not sure where that comes from in this stack.17:35
SpamapSvinsh: it is included by default on Ubuntu cloud images.17:35
SpamapSso probably being assumed for all dpkg based systems17:35
vinshgrr.. wonder if it is included in hlinux.17:35
* vinsh goes to see.17:35
dprincemordred: the ping failures we are currently seeing for the overcloud jobs seem to affect all distros. We ran a 'check experimental' yesterday and saw the same failure w/ Ubuntu as well17:35
*** yamahata has quit IRC17:36
*** eghobo has quit IRC17:37
*** eguz has joined #tripleo17:37
mordreddprince: awesome. well, good to know17:37
*** Shrews has quit IRC17:37
*** jcoufal has quit IRC17:38
*** pelix has quit IRC17:41
SpamapSdprince: need any help debugging/testing on that?17:46
*** morazi has quit IRC17:46
dprinceSpamapS: maybe. I'm currently using a vnc console to actually attach to the testenv17:47
SpamapSdprince: nasty17:47
SpamapSdprince: is it not reproducible on a local devtest run?17:47
dprinceSpamapS: yeah, well noone can reproduce locally17:47
* bnemec did17:47
bnemecBut I don't know enough about multi-node Neutron to figure out what's wrong.17:47
dprincebnemec: oh, so you hit it locally?17:47
bnemecdprince: Yeah, I randomly hit it on a devtest run yesterday.17:48
dprincebnemec: seriously, tell somebody!17:48
bnemecI'm pretty sure I mentioned it multiple times right here. ;-)17:48
dprincebnemec: yeah, I've seen a few CI passes in the past 24 hours as well (2)17:48
dprincebnemec: I missed it.17:48
bnemecYeah, lot of discussion going on in the channel lately.17:49
*** eguz has quit IRC17:49
dprincebnemec: can you add details here
uvirtbotLaunchpad bug 1292105 in tripleo "CI failed pinging overcloud instance" [Critical,In progress]17:50
*** eghobo has joined #tripleo17:50
bnemecI'm working on setting up a parallel devtest system to compare with (assuming that one doesn't hit the bug as well).17:50
*** Shrews has joined #tripleo17:50
*** proffalken has joined #tripleo17:51
*** e0ne has quit IRC17:52
dprincebnemec: so have a look at the iptables rules on the controller17:52
dprincebnemec: you'll have to use ip netns exec <namespace> iptables-save17:53
*** e0ne has joined #tripleo17:53
dprincebnemec: to get at them17:53
bnemecdprince: qrouter or qdhcp?17:53
dprincebnemec: qrouter17:53
*** blamar has quit IRC17:57
*** e0ne has quit IRC17:57
*** dkehn_ is now known as dkehnx17:58
*** andreaf_ has joined #tripleo18:05
*** bcrochet has quit IRC18:08
*** sballe has quit IRC18:08
*** tzumainn has quit IRC18:08
*** jp_at_hp has quit IRC18:08
*** victor_lowther has quit IRC18:08
*** noslzzp has quit IRC18:08
*** regebro has quit IRC18:08
*** yolanda has quit IRC18:08
*** dshulyak has quit IRC18:08
*** ccrouch1 has quit IRC18:08
*** fabbione has quit IRC18:08
*** ohadlevy has quit IRC18:08
*** andreaf has quit IRC18:08
*** dprince has quit IRC18:08
*** julim has quit IRC18:08
*** mkerrin has quit IRC18:08
*** jcooley has quit IRC18:08
*** cody-somerville has quit IRC18:08
*** bkero has quit IRC18:08
*** rushiagr has quit IRC18:08
*** russell_h has quit IRC18:08
*** eghobo has quit IRC18:08
*** vinsh has quit IRC18:08
*** cwolferh has quit IRC18:08
*** wendar has quit IRC18:08
*** agordeev has quit IRC18:08
*** yjiang5 has quit IRC18:08
*** adam_g has quit IRC18:08
*** rainya has quit IRC18:08
*** stevebaker has quit IRC18:08
*** edunham has quit IRC18:08
*** zigo has quit IRC18:08
*** chmouel has quit IRC18:08
*** antonym has quit IRC18:08
*** clarkb has quit IRC18:08
*** uvirtbot has quit IRC18:08
*** eghobo has joined #tripleo18:10
*** vinsh has joined #tripleo18:10
*** bcrochet has joined #tripleo18:10
*** cwolferh has joined #tripleo18:10
*** sballe has joined #tripleo18:10
*** dprince has joined #tripleo18:10
*** tzumainn has joined #tripleo18:10
*** julim has joined #tripleo18:10
*** yolanda has joined #tripleo18:10
*** jp_at_hp has joined #tripleo18:10
*** victor_lowther has joined #tripleo18:10
*** noslzzp has joined #tripleo18:10
*** edunham has joined #tripleo18:10
*** regebro has joined #tripleo18:10
*** dshulyak has joined #tripleo18:10
*** mkerrin has joined #tripleo18:10
*** ccrouch1 has joined #tripleo18:10
*** fabbione has joined #tripleo18:10
*** wendar has joined #tripleo18:10
*** ohadlevy has joined #tripleo18:10
*** jcooley has joined #tripleo18:10
*** agordeev has joined #tripleo18:10
*** cody-somerville has joined #tripleo18:10
*** yjiang5 has joined #tripleo18:10
*** bkero has joined #tripleo18:10
*** rushiagr has joined #tripleo18:10
*** adam_g has joined #tripleo18:10
*** rainya has joined #tripleo18:10
*** russell_h has joined #tripleo18:10
*** uvirtbot has joined #tripleo18:10
*** clarkb has joined #tripleo18:10
*** antonym has joined #tripleo18:10
*** chmouel has joined #tripleo18:10
*** zigo has joined #tripleo18:10
*** stevebaker has joined #tripleo18:10
*** dguerri has joined #tripleo18:10
*** jp_at_hp has quit IRC18:11
*** jp_at_hp has joined #tripleo18:11
*** thegodfather has joined #tripleo18:14
*** bcrochet has quit IRC18:14
*** sballe has quit IRC18:14
*** tzumainn has quit IRC18:14
*** jp_at_hp has quit IRC18:14
*** victor_lowther has quit IRC18:14
*** noslzzp has quit IRC18:14
*** regebro has quit IRC18:14
*** yolanda has quit IRC18:14
*** dshulyak has quit IRC18:14
*** ccrouch1 has quit IRC18:14
*** fabbione has quit IRC18:14
*** ohadlevy has quit IRC18:14
*** dprince has quit IRC18:14
*** julim has quit IRC18:14
*** mkerrin has quit IRC18:14
*** jcooley has quit IRC18:14
*** cody-somerville has quit IRC18:14
*** bkero has quit IRC18:14
*** rushiagr has quit IRC18:14
*** russell_h has quit IRC18:14
*** eghobo has quit IRC18:14
*** vinsh has quit IRC18:14
*** cwolferh has quit IRC18:14
*** wendar has quit IRC18:14
*** agordeev has quit IRC18:14
*** yjiang5 has quit IRC18:14
*** adam_g has quit IRC18:14
*** rainya has quit IRC18:14
*** stevebaker has quit IRC18:14
*** edunham has quit IRC18:14
*** zigo has quit IRC18:14
*** chmouel has quit IRC18:14
*** antonym has quit IRC18:14
*** clarkb has quit IRC18:14
*** uvirtbot has quit IRC18:14
*** eghobo has joined #tripleo18:16
*** vinsh has joined #tripleo18:16
*** bcrochet has joined #tripleo18:16
*** cwolferh has joined #tripleo18:16
*** sballe has joined #tripleo18:16
*** dprince has joined #tripleo18:16
*** tzumainn has joined #tripleo18:16
*** julim has joined #tripleo18:16
*** yolanda has joined #tripleo18:16
*** jp_at_hp has joined #tripleo18:16
*** victor_lowther has joined #tripleo18:16
*** noslzzp has joined #tripleo18:16
*** edunham has joined #tripleo18:16
*** regebro has joined #tripleo18:16
*** dshulyak has joined #tripleo18:16
*** mkerrin has joined #tripleo18:16
*** ccrouch1 has joined #tripleo18:16
*** fabbione has joined #tripleo18:16
*** wendar has joined #tripleo18:16
*** ohadlevy has joined #tripleo18:16
*** jcooley has joined #tripleo18:16
*** agordeev has joined #tripleo18:16
*** cody-somerville has joined #tripleo18:16
*** yjiang5 has joined #tripleo18:16
*** bkero has joined #tripleo18:16
*** rushiagr has joined #tripleo18:16
*** adam_g has joined #tripleo18:16
*** rainya has joined #tripleo18:16
*** russell_h has joined #tripleo18:16
*** uvirtbot has joined #tripleo18:16
*** clarkb has joined #tripleo18:16
*** antonym has joined #tripleo18:16
*** chmouel has joined #tripleo18:16
*** zigo has joined #tripleo18:16
*** stevebaker has joined #tripleo18:16
dprincebnemec: at least that is what I'm seeing18:16
dprincebnemec: On your nova compute node go have a look in /mnt/state/var/lib/nova/instances/<uuid>18:16
dprincebnemec: you'll have to go to the compute where the instance is running (virsh list...)18:16
dprincebnemec: on mine the instance just plain doesn't get its dhcp address18:17
*** fabbione has quit IRC18:17
bnemecdprince: Yeah, that's what I'm seeing too.  I left a note on the bug.18:17
bnemecdprince: (also, I think I missed part of that due to the netsplit.  The first line I saw was "at least that is what I'm seeing")18:18
*** bmahalakshmi has joined #tripleo18:18
dprincebnemec: thanks18:18
dprincebnemec: well at least we are seeing the same thing18:18
bnemecI just kicked off my second devtest run.  I'm going to go grab lunch and hopefully I'll have something to look at when I'm done.18:22
*** dkehn_ has joined #tripleo18:25
*** dkehn_ has quit IRC18:27
*** dkehn has quit IRC18:28
*** dkehnx has quit IRC18:28
*** dkehn_ has joined #tripleo18:28
*** dkehn has joined #tripleo18:29
*** yolanda has quit IRC18:33
*** tzumainn has quit IRC18:33
*** regebro has quit IRC18:33
*** yolanda has joined #tripleo18:33
*** sballe has quit IRC18:33
*** tzumainn has joined #tripleo18:33
*** regebro has joined #tripleo18:33
*** jang has joined #tripleo18:33
SpamapSdprince: FYI, 'nova show' will tell admins which compute host has an instance.18:39
dprinceSpamapS: if you don't have creds then virsh works just fine too ;)18:41
*** dtantsur is now known as dtantsur|afk18:48
d0ugaltzumainn: Would you object if I removed the Provider Resource model?18:54
tzumainnd0ugal, nope, go for it18:55
d0ugaltzumainn: ok, we can maybe add it later, but I'm trying to justify it in the spec, but I can't :)18:56
tzumainnyep, fair enough!18:56
d0ugaltzumainn: cool, thanks18:58
SpamapSdprince: yeah thats fine for when you're testing 1 or 2 compute nodes :)19:01
*** dshulyak_ has joined #tripleo19:08
*** nati_uen_ has joined #tripleo19:10
*** nati_ueno has quit IRC19:14
NobodyCamI have a heat question. Will heat read tempate valuse from enviroment vars? I.E. export UNDERCLOUD_NeutronPublicInterface=blah19:16
NobodyCamvalues even19:16
*** dkehn_ has quit IRC19:24
SpamapSNobodyCam: Heat?19:24
SpamapSNobodyCam: heat only knows -P and --environment. It ignores the "environment variables" in your shell.19:25
NobodyCamthats what I thought19:25
NobodyCamSpamapS: could you jump in hipchat - ironic channel19:26
*** dkehn has quit IRC19:28
*** e0ne has joined #tripleo19:28
*** casanch1_ has joined #tripleo19:31
*** casanch1 has quit IRC19:34
*** dkehn has joined #tripleo19:35
*** dkehn_ has joined #tripleo19:35
*** casanch1_ has quit IRC19:35
*** bmahalakshmi has quit IRC19:35
*** e0ne has quit IRC19:35
*** e0ne has joined #tripleo19:36
*** dkehn_ has quit IRC19:36
*** dkehn_ has joined #tripleo19:37
*** dkehn_ is now known as dkehnx19:37
*** e0ne has quit IRC19:38
*** e0ne has joined #tripleo19:41
*** e0ne has quit IRC19:49
*** e0ne has joined #tripleo19:49
*** akuznetsov has quit IRC19:49
*** e0ne has quit IRC19:52
*** jang1 has joined #tripleo19:53
*** akuznetsov has joined #tripleo19:55
vinshhmmmr "/opt/stack/os-config-refresh/configure.d/51-mysql-init: line 13: /usr/local/mysql/scripts/mysql_install_db: No such file or directory"19:55
vinshShould be.. if 10-mysql actually ran during DIB.19:56
* vinsh curls up in a ball and screams.19:56
lifelessSpamapS: what do you think of the idea of heat being allowed to ignore excess parameters rather than failing ?19:57
lifelessvinsh: chmod ?19:57
*** e0ne has joined #tripleo19:57
vinshelements/mysql/install.d/10-mysql:    ln -sf /usr/local/bin/mysql_install_db /usr/bin/mysql_install_db19:58
vinshelements/mysql/install.d/10-mysql:    sed -i -e 's,^basedir="",basedir="/usr/local/mysql",' scripts/mysql_install_db19:58
vinshelements/mysql/install.d/10-mysql:    install -o root -g root -m 0755 scripts/mysql_install_db /usr/local/bin19:58
greghaynesvinsh: are you using the switch to package based install thing?19:58
*** dkehn_ has joined #tripleo19:58
vinshfor some reason.. the install.d job isn't doing its thing correctly19:58
lifelessjdob: bug 132906819:58
uvirtbotLaunchpad bug 1329068 in trove-integration "The trove guest-image is unable to publish the guest-config via cloud-init" [Critical,In progress]
vinshgreghaynes, I dont' know19:58
greghaynesvinsh: well, its a patch youd have to manually pull in, but if you did that stuff is possibly in the wrong place19:59
lifelessjdob: if we release with a regression, its an API break and thats bad19:59
*** akuznetsov has quit IRC19:59
vinshgreghaynes, I just see that during DIB.. 10-mysql says it ran.. yet on the resulting image.. the link it should have made when it ran.. isn't there.20:00
*** dkehn_ has quit IRC20:00
*** dkehn has quit IRC20:00
*** dkehn_ has joined #tripleo20:00
*** dkehnx has quit IRC20:00
jdobok, ya, this is the review that ben linked me earlier. lifeless, what's the right way to do this, since CI is failing. hold off on the release until CI passes for that review?20:00
lifelessjdob: hold off until the bug is fix committed20:01
jdoblifeless: also, when you get a second, are you the one who can add me to tripleo-ptl so I can do the release?20:01
lifelessjdob: when that is is a separate discussion :).20:01
lifelessjdob: sure, whats your gerrit username ?20:01
lifelessjdob: you've read the docs on the wiki ?20:02
jdobthe release management page? yes20:02
jang1Hrm. I think it's all over...20:02
lifelessjdob: and spoken with e.g. rpodolyaka to get familiar with process?20:02
*** dkehn has joined #tripleo20:02
jdobnot yet20:02
*** e0ne has quit IRC20:02
lifelessjdob: ok, so infra really don't like mistakes in tagging :)20:02
lifelessjdob: so I'll hold off until you've got a green light from one of the existing releasers :)20:03
*** dprince has quit IRC20:03
lifeless(thats why there is a separate team to the tripleo-core)20:03
jdobgreen light in the sense that I've spoken with someone to make sure I'm clear on the process?20:03
vinshgreghaynes, anything that might cause the install.d stuff to not really do what it should during DIB like this?20:03
lifelessjdob: yup20:04
jdobcool, that makes sense to me20:04
lifelessjdob: we don't have a formal test, though that would be cool20:04
lifelessvinsh: it should be set  -e, so will error if it tried and failed20:04
lifelessvinsh: did it run it? Is it in the log ?20:05
vinsh10-os-config-applier                          3.30020:05
vinsh10-os-collect-config                          9.96320:05
vinsh10-mysql                                     59.28620:05
vinsh10-locale                                     1.68120:05
vinshappears to be?20:05
jdoblifeless: what's the timeframe on the release? if the bug doesn't land tomorrow is there a problem with the release sliding?20:05
lifelesscertainly looks that way higher up you should see its invocation20:05
lifelessjdob: there's no problem20:05
lifelessjdob: we release to keep inventory low20:05
jdobgotcha, so its an informal thing20:06
lifelessjdob: right now with CI wedged, inventory isn't building up in the fix-committed part :)20:06
vinshlifeless, I don't see the invocation of any of them in the devtest_overcloud ouput.. just that list showing times taken20:06
lifelessvinsh: look for something like dib-run-parts Fri Jun 20 08:07:41 NZST 2014 Running /tmp/image.l4xdukW4/hooks/extra-data.d/75-inject-pip-manifests20:08
*** casanch1 has joined #tripleo20:08
vinshYeah, nadda20:10
vinshonly appears in that list with the time take listed next to it :-/20:10
vinshunless its in another log besides devtest_overcloud output20:10
lifelessthats very very odd20:11
vinshMaybe just deleting the image and everything from triple-root isn't enough20:11
lifelessdo you have those 'Running ....' outputs at all ?20:11
vinshcache=1 might still use old data?20:11
vinshI do have those.. but are for cleanup.d things only20:12
lifelessI have them for install.d:20:12
lifelessdib-run-parts Thu Jun 19 20:11:29 UTC 2014 100-horizon completed20:12
*** CaptTofu_ has quit IRC20:13
vinshI am guessing that usecache=1 got me. even with deleting the image from root20:14
vinshwild guess maybe20:14
greghaynesit shouldnt :/20:14
greghaynesif its not there20:15
lifelessvinsh: very much doubt that20:15
greghaynesProbably gamma rays20:15
vinshwell, I have the entire console log.. and can see items like 99-up-to-date invoked in output20:16
vinshbut not 10-mysql.. though it shows as completed in the end list20:16
lifelessyet its there with timing data20:16
vinshdoesn't show up as an image like you posted though..20:17
lifelessits also set -x so you should see its actions20:17
lifelessits missing -u which is bad form20:17
vinshI see it now!20:18
vinshAFTer i set20:18
vinshexport DIB_DEBIAN_USE_CACHED_IMAGE=020:18
vinshand USE_CACHE=020:18
vinshnot present in out put otherwise.20:19
vinshback to you non vinsh program. will see what I get in these images. ;)20:19
*** CaptTofu_ has joined #tripleo20:21
*** nati_uen_ has quit IRC20:25
*** nati_ueno has joined #tripleo20:25
*** nati_ueno has quit IRC20:27
*** nati_ueno has joined #tripleo20:28
*** blamar has joined #tripleo20:28
*** rcarrillocruz has joined #tripleo20:29
*** rcarrill` has quit IRC20:30
*** akuznetsov has joined #tripleo20:34
*** casanch1 has quit IRC20:34
*** casanch1 has joined #tripleo20:34
*** mestery has quit IRC20:42
*** mestery has joined #tripleo20:43
tchaypoJdob: pong20:44
jdobtchaypo: er, sorry, unping20:45
*** mestery has quit IRC20:47
*** akuznetsov has quit IRC20:51
*** jomara has quit IRC20:51
*** jomara has joined #tripleo20:57
*** rcarrill` has joined #tripleo20:57
*** jp_at_hp has quit IRC20:57
*** jp_at_hp has joined #tripleo20:58
*** jdob has quit IRC20:58
*** rcarrillocruz has quit IRC21:00
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
uvirtbotLaunchpad bug 1331931 in tripleo "NeutronControlPlaneID should not be set if building image only" [Undecided,In progress]21:00
uvirtbotLaunchpad bug 1332321 in tripleo "Undercloud horizon resource tab returns an error due to missing ceilometer components " [Undecided,New]21:00
uvirtbotLaunchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New]21:00
*** untriaged-bot has quit IRC21:00
lifelessany further progress on CI ?21:03
SpamapSlifeless: I believe dprince was finding that DHCP was failing "sometimes" by attaching to the VNC console of the instance21:10
SpamapSoh but you probably know that21:10
lifelessI'm trying to repro myself, currently on an abnormally long overcloud stack-ready wait21:12
*** casanch1 has quit IRC21:13
lifelessNg: around ?21:18
lifelesstchaypo: when you get up, lets talk about getting CI regions up21:18
lifelessgreghaynes: I think I'm going to ask for more volunteers to context switch to this, as capacity matters.21:18
lifelessgreghaynes: if you / adam_g are interested (or know of other interested folk) that would rock21:19
uvirtbotLaunchpad bug 1201656 in tripleo "nova baremetal ties nova-compute to one architecture" [High,Fix released]21:20
jogowith nova barmetal being deprecated is that bug still valid?21:20
lifelessjogo: depends on nova's position on bugs in deprecated but not deleted code.21:21
lifelessho we did actually land the three controller patch21:22
lifelessI wonder21:22
*** dshulyak_ has quit IRC21:22
jogolifeless: well bugs in deprecated code aren't high priority21:22
jogolet me ask a different question, will tripleo being fixing it?21:22
lifelessno, never planned to. Its there because it is a bug, and folk will a) run into it and b) we don't want a hundred and one near-dups filed.21:23
bnemecFrom icmp_seq=1 Destination Host Unreachable21:24
bnemecWhy am I the only one who can reproduce this locally?21:24
*** morganfainberg is now known as morganfainberg_Z21:24
lifelessbnemec: have you tried a single node ctl plane ?21:24
bnemeclifeless: That's how I'm running.  Single control, single compute.21:26
Nglifeless: yep. ssl certs for hp2 are ordered, awaiting verification21:26
devanandalifeless: have you seen and
devanandai'd like to be sure that tripleo-incubator and related tooling starts migrating to use instance_info soon, ideally by J2, so we can finish deprecating the use of driver_info for non-driver-specific things21:28
*** jprovazn_afk has quit IRC21:32
*** dkehn_ is now known as dkehnx21:32
*** openstackgerrit has joined #tripleo21:34
*** jang1 has quit IRC21:36
lifelessdevananda: I don't believe we do any direc tmanipulation of instance info21:45
lifelessdevananda: the only things we do with ironic's api is register nodes and turn them off21:46
tchaypolifeless: Have to go to my sister in law's birthday breakfast, but I should be in the office with Steven about 1021:47
lifelesstchaypo: ok cool.21:47
*** dkehn_ has joined #tripleo21:48
*** dkehn_ has quit IRC21:49
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Split into multiple files
*** dkehn has quit IRC21:49
*** dkehn_ has joined #tripleo21:49
devanandalifeless: this patch means you will no longer set the pxe_deploy_* options on a flavor,a nd instead pass them in when enrolling a node in ironic21:50
lifelessbnemec: Ah,I had manually set controlscale to 321:50
*** dkehnx has quit IRC21:50
devanandalifeless: so i think it will require changes in tripleo-incubator21:50
lifelessdevananda: I am confuse, instance_info will be wiped every time21:50
lifelessdevananda: so how can that tie into what you just said21:51
devanandadriver info21:51
devanandajugglign too many meetings :p21:51
*** dkehn has joined #tripleo21:51
lifelessdevananda: ok yes we need to a) switch to the occ register-nodes and b) have that fix in there21:52
devanandatldr; you will need to put *deploy* image UUIDs on driver_info instead of on nova flavor21:52
devanandaright now, ironic supports both21:52
lifelessdevananda: well, backwards compat. We need to set it on both.21:52
devanandaand there's a migration script proposed to mvoe live nodes over21:52
*** noslzzp has quit IRC21:52
devanandalifeless: right21:52
lifelessdevananda: is there a bug filed on os-cloud-config and tripleo for this?21:53
devanandanot yet, no21:53
*** ddieterly has quit IRC21:53
lifelesscould you please file one? Since you have the details. I'll point folk at it in next weeks meeting21:53
devanandaNobodyCam: we need the tripleo-undercloud test going soon ...21:53
devanandalifeless: ack21:53
*** morganfainberg_Z is now known as morganfainberg21:55
openstackgerritBen Nemec proposed a change to openstack-infra/tripleo-ci: Try reverting Neutron ovs-agent change
bnemecRestarting l3-agent and ovs-agent on my controller node seems to have fixed the problem in one of my environments.22:02
bnemeclifeless: Figured I would give ^ a shot.22:02
bnemecI think that merged around the time we starting having issues and it looked like about the only thing that could be related.22:03
NobodyCamdevananda: its got two +2's22:06
NobodyCamI'll poke infra22:06
*** CaptTofu_ has quit IRC22:06
devanandaNobodyCam: awesome, ty22:06
*** CaptTofu_ has joined #tripleo22:07
*** nati_ueno has quit IRC22:07
*** nati_ueno has joined #tripleo22:07
lifelessbnemec: yeah, longshot, but worth checking22:10
lifelessbnemec: if that isn't it I think we need to write some code to bisect the openstack global timeline22:10
*** CaptTofu_ has quit IRC22:11
*** nati_ueno has quit IRC22:11
*** nati_ueno has joined #tripleo22:12
*** davidlenwell is now known as davidlenwell_22:14
*** davidlenwell_ is now known as davidlenwell22:14
*** ChanServ sets mode: +v davidlenwell22:14
bnemeclifeless: That should be fun. :-)  Probably be useful again in the future though.22:16
bnemeclifeless: Unfortunately I'm not going to be around much tomorrow again.22:16
*** CaptTofu_ has joined #tripleo22:20
*** jp_at_hp has quit IRC22:24
lifeless sudo tail -F /var/log/upstart/os-collect-config.log22:24
lifelesssudo: error in /etc/sudo.conf, line 0 while loading plugin `sudoers_policy'22:24
lifelesssudo: unable to load /usr/lib/sudo/ (null)22:24
lifelesssudo: fatal error, unable to load plugins22:24
*** ccrouch1 has quit IRC22:25
*** jp_at_hp has joined #tripleo22:26
*** andreaf_ has quit IRC22:26
lifelessnow + user-config22:29
lifelessERROR: HTTPConnectionPool(host='', port=8774): Max retries exceeded with url: /v2/73a06626a3264a31a82d7075ac2878cf/os-keypairs (Caused by <class 'httplib.BadStatusLine'>: '')22:29
lifelesssomething verra odd there22:29
*** jp_at_hp has quit IRC22:32
*** blamar has quit IRC22:34
*** openstackgerrit has quit IRC22:34
*** 20WAAHXAJ has joined #tripleo22:36
*** ccrouch has joined #tripleo22:39
lifelesshmm, keystone is whinging22:44
lifeless2014-06-19 22:23:29.001 6918 WARNING keystoneclient.middleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead.22:44
lifelessmorganfainberg: whats ^ about ?22:45
*** openstack has joined #tripleo22:51
*** CaptTofu_ has quit IRC22:55
*** CaptTofu_ has joined #tripleo22:55
SpamapSlifeless: yeah there's a bug there22:56
SpamapSlifeless: its the middleware22:56
SpamapSlifeless: they want us to just feed a URL in, instead of the component pieces22:56
lifelessSpamapS: shouldn't it look it up in keystone?22:56
SpamapSlifeless: this is _to find keystone_22:57
morganfainberglifeless, uhm22:57
morganfainberglifeless, uhhhhhhhh22:57
morganfainberglifeless, wrong config option used? identity_url should be the option instead?22:57
SpamapSmorganfainberg: is that different than auth_uri ?22:58
morganfainbergSpamapS, yeah, but i can't remember off the top of my head the difference, i think this is about the unversioned endpoing discovery22:59
morganfainberglet me check w/ jamielennox22:59
*** CaptTofu_ has quit IRC23:00
SpamapSmorganfainberg: for reference,
morganfainbergno i know what it is23:01
morganfainberginstead of using auth_host, auth_port, auth_protocol,23:01
lifelessok, second time + user-config23:01
lifelessERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-8a04c55a-38d3-44ee-bb89-c2dc29dd5d77)23:01
lifeless has failed23:01
morganfainbergyou can use identity_uri=http://{{}}:35357 etc23:02
SlickNikSpamapS: Commented on
morganfainberglifeless, the construction from the bits and peices is deprecated, but will still work23:03
morganfainbergSpamapS, ^23:03
SpamapSSlickNik: ty23:03
SlickNikSpamapS: You were mentioning that we might want to look into changing the defaults for 12.04, so I looked into what was getting set in a 12.04 image (without the element).23:05
SlickNikSpamapS: In that case, there is a 90_dpkg.cfg file which has:23:05
SlickNik"# to update this file, run dpkg-reconfigure cloud-init23:05
SlickNikdatasource_list: [ NoCloud, ConfigDrive, OVF, MAAS, Ec2 ]"23:05
*** CaptTofu_ has joined #tripleo23:05
SpamapSmorganfainberg: .. different fields .. so confusing. :(23:06
morganfainbergSpamapS, *blink*23:07
morganfainbergyeah :(23:07
morganfainbergSpamapS, this is part of what we'll be fixing with the extracted middleware23:07
morganfainbergSpamapS, /me is working on extracting middleware into it's own package23:07
lifeless/var/log/upstart/nova-api.log:2014-06-19 23:01:02.560 6933 ERROR nova.api.openstack [req-8a04c55a-38d3-44ee-bb89-c2dc29dd5d77 None] Caught error: [Errno 12] Cannot allocate memory23:07
morganfainbergwe're going to aim to fix some of this wonkyness with the 1.0 release23:08
SpamapSmorganfainberg: cool23:08
SpamapSlifeless: nova's getting old23:08
morganfainberg(first release will be 1.0.0)23:08
morganfainbergSpamapS, but splitting and merging in git repositories from all over is ... uhhhh23:08
lifelessSpamapS: this is trunk23:08
morganfainbergi think i'm up to 4 split/merges23:08
morganfainbergSpamapS, it's safe (yeah i know deprecation warning) to use the old style constructed uri23:09
lifelesswtf             total       used       free     shared    buffers     cached23:09
lifelessMem:          2.0G       1.9G       100M       940K        10M        73M23:09
lifeless-/+ buffers/cache:       1.8G       184M23:09
lifelessopenstack not fitting in a 2G VM. Fark.23:09
morganfainbergwith the spec to split the middleware we wont be changing the middleware except for security-fixes in the current location(keystoneclient) so don't worry about the options disappearing23:09
SpamapSmorganfainberg: just promise you'll run 'check experimental' before you land it :)23:13
morganfainbergSpamapS, before we land the change to convert people over to it, i will absolutely do so23:14
SpamapSmorganfainberg: man.. I'm craving shabu shabu again.. :)23:18
morganfainbergSpamapS, that stuff was awesome!23:19
bnemecWTH?  The VM pinged fine for about 30 seconds then it died again.23:19
SpamapSyeah.. want23:19
* bnemec gives up23:19
morganfainbergi'm def. up to go again.23:19
SpamapSbnemec: :-/23:19
SpamapSbnemec: overlay networks are the pits23:19
*** yamahata has joined #tripleo23:19
SpamapSmorganfainberg: yeah let's try to go next week some time.23:20
bnemecIf you can believe it, I actually have less pleasant things than debugging CI failures that I need to go do. :-/23:20
morganfainbergSpamapS, sounds like a plan!23:20
*** dkehn_ is now known as dkehnx23:20
*** noslzzp has joined #tripleo23:21
20WAAHXAJTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Split into multiple files
lifelessinteresting nick there gerritbot23:27
lifelessok so23:29
*** 20WAAHXAJ has quit IRC23:29
lifelessopenstack has got fat23:29
*** openstackgerrit has joined #tripleo23:31
lifelessbnemec: I finally got an end to end run ok23:31
openstackgerritA change was merged to openstack/diskimage-builder: Do not use DatasourceNone for precise cloud-init
lifelessbnemec: by increasing my node size to 3G23:32
lifelessbnemec: I'm just looking to see what arch we use in CI23:32
ccrouchon fedora it should be amd6423:32
bnemeclifeless: I was already running 3G or my undercloud OOMs every time.23:32
lifelessbnemec: CI is running default node size23:33
lifelesserm, no its not we pull it in from gearman-worker.mem-per-env23:33
bnemeclifeless: CI never uses the undercloud though.23:33
bnemecI think it boots fine, but when you try to load all the overcloud stuff it dies.23:33
lifelessbnemec: that was overcloud control config that fails for me23:33
lifelessbnemec: with 2G nodes23:33
lifelesswe make amd64 arch nodes23:34
SpamapSon purpose?23:35
lifelessSpamapS: I think so, fidelity.23:35
SpamapSJust spun up a devstack today..23:35
SpamapS             total       used       free     shared    buffers     cached23:35
SpamapSMem:          7985       3987       3997          0         94       196323:35
SpamapS-/+ buffers/cache:       1929       605523:35
lifelessso I'm wondering23:35
lifeless4G lol23:35
SpamapSubuntu   12171  0.4  1.1 289744 96164 pts/8    S+   23:26   0:02 /usr/bin/python /usr/local/bin/nova-api23:36
uvirtbotLaunchpad bug 12171 in gaim "/usr/share/doc/gaim is empty" [Unknown,Fix released]
SpamapSubuntu   12172  0.4  1.1 289712 96188 pts/8    S+   23:26   0:02 /usr/bin/python /usr/local/bin/nova-api23:36
uvirtbotLaunchpad bug 12172 in ubuntu "Keyboard language menu in the installer is not sorted (dup-of: 9520)" [Low,Invalid]
uvirtbotLaunchpad bug 9520 in kbd-chooser "Ubuntu should have a better method of selecting the keyboard layout" [Medium,Fix released]
SpamapStop two processes23:36
SpamapS(rss anyway)23:36
lifelessoh uvirtbot no you didn't23:36
SpamapSubuntu 66623:37
uvirtbotLaunchpad bug 666 in launchpad "can't file a bug on Ubuntu" [Medium,Invalid]
* SpamapS needs to take a break before diving back into heat dev23:37
lifelessok so we grab 'free' output23:38
lifelesshmm, I'd prefer free -h23:38
openstackgerritlifeless proposed a change to openstack-infra/tripleo-ci: Use free -h for us mere mortals.
*** xuhaiwei has joined #tripleo23:44
*** mestery has joined #tripleo23:46
lifelessok so the CI fail isn't memory23:48
lifeless             total       used       free     shared    buffers     cached23:48
lifelessMem:       4048704    3148732     899972        444      71808     35239623:48
lifeless-/+ buffers/cache:    2724528    132417623:48
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Warn about 2GB w/amd64.
*** dkehn_ has joined #tripleo23:50
*** dkehn_ has quit IRC23:51
*** dkehn has quit IRC23:51
lifelessbnemec: two runs now, works both times.23:51
*** dkehn_ has joined #tripleo23:51
*** dkehnx has quit IRC23:51
lifelessbnemec: AIUI its fedora and ubuntu right? I don't need to build a fedora set of images?23:51
bnemeclifeless: Yeah, the check experimental on Ubuntu failed too.23:52
lifelessbnemec: I'm looping on overcloud only now.23:52
lifelessbnemec: cached image, again and again23:52
*** dkehn has joined #tripleo23:52
*** nati_ueno has quit IRC23:54
bnemeclifeless: Yeah, I seem to be the only one hitting this in local devtest.  I've gotten it every time I've run it in the past two days though.23:54
*** nati_ueno has joined #tripleo23:55
*** nati_ueno has quit IRC23:55
*** nati_ueno has joined #tripleo23:56
lifelessmaybe we're missing something23:58
lifelesslike firewall rules23:58
lifelesson the jenkins slave23:58
lifelessor seed23:58
lifelessip forwarding23:58
* lifeless throws ideas out23:58

Generated by 2.14.0 by Marius Gedminas - find it at!